Information processing device, information processing system, information processing method, and program

Abstract

To more efficiently remove a question sentence that is to be removed from question sentences included in interaction history information.SOLUTION: An information processing device includes: extraction means that extracts, based on a predetermined condition, classification candidate question sentences from interaction history information of a service provision system that returns an answer sentence to a question sentence from a user terminal; determination means that determines question sentences to be removed from among the classification candidate question sentences; screen generation means that displays the classification candidate question sentences from which the question sentences to be removed have been removed, and generates data of a screen for receiving selection of a question sentence to be removed from a user; and removal means that removes the question sentence selected by the user.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, an information processing method, and a program.

Background Art

[0002] There has conventionally been known an automatic response service in which an information processing apparatus such as a computer responds (answers) to an inquiry from a user (for example, an input such as a question regarding a product, service, or system), so-called a chatbot (sometimes simply referred to as a bot).

[0003] For example, Patent Document 1 discloses a technique in which, in a chatbot system, after a response by a bot, the chat history by the bot is saved and analyzed by statistical processing or the like with respect to the history.

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, when a chatbot system is used for customer inquiry response work, the chat history (dialog history information) by the bot includes question texts from customers that can be utilized for marketing.

[0005] However, the question texts from customers included in the dialog history information may include inappropriate question texts such as pranks. In order to efficiently grasp the voices of customers from such question texts from customers, there is a problem that it is necessary to remove inappropriate question texts from the question texts from customers included in the dialog history information. Note that since the chatbot system has different variations of question texts from customers depending on the web page where it is installed and the question-and-answer data in which the question texts and answer texts used for chatbot processing are associated, it has been difficult to remove inappropriate question texts. Note that the conventional technology such as Patent Document 1 does not solve such problems.

[0006] One embodiment of the present invention has been made in view of the above-mentioned problems, and aims to more efficiently remove question sentences that should be removed from question sentences included in dialogue history information. [Means for solving the problem]

[0007] One embodiment of the present invention provides a service that responds to questions from a user terminal, and based on the dialogue history information of the service provider system, it provides answers based on predetermined conditions. Inappropriate question suggestions Questions for classification candidates as The system includes an extraction means for extracting questions, a determination means for determining which questions should be excluded from the candidate questions for classification, a screen generation means for generating screen data to display the candidate questions from which the questions to be excluded have been removed and to accept the user's selection of questions to be removed, and a removal means for removing the questions selected by the user. Furthermore, the extraction means is characterized by extracting candidate question sentences for classification from the results of classifying a plurality of question sentences recorded in the dialogue history information based on similarity. It is an information processing device. [Effects of the Invention]

[0008] This allows for more efficient removal of questions that should be removed from the dialogue history information. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating an example of an information processing system according to this embodiment. [Figure 2] This is a hardware configuration diagram of an example of a computer according to this embodiment. [Figure 3] This is a functional configuration diagram of an example of an information processing system according to this embodiment. [Figure 4] This is a flowchart of an example of the processing procedure of the service provision system according to this embodiment. [Figure 5] This is an explanatory diagram illustrating an example of dialogue history information. [Figure 6] This flowchart illustrates an example of a processing procedure for extracting candidate question sentences for classification. [Figure 7] This is an explanatory diagram illustrating an example of the process for extracting question sentences that are candidates for classification. [Figure 8] It is an explanatory diagram of an example of a process of determining and excluding question sentences to be excluded from candidate question sentences of classification candidates that are candidate inappropriate question sentences. [Figure 9] It is an explanatory diagram of an example of a process of determining and excluding question sentences to be excluded from candidate question sentences of classification candidates that are candidate inappropriate question sentences. [Figure 10] It is an image diagram of an example of an administrative screen displayed on an administrator terminal. [Figure 11] It is a functional configuration diagram of an example of an information processing system according to the present embodiment. [Figure 12] It is a flowchart of an example of a processing procedure of a service providing system according to the present embodiment. [Figure 13] It is an explanatory diagram of an example of dialogue history information. [Figure 14] It is a flowchart showing an example of a processing procedure for extracting candidate classification question sentences. [Figure 15] It is an explanatory diagram of an example of a candidate classification question sentence. [Figure 16] It is an explanatory diagram of an example of a question sentence for which an answer has been found. [Figure 17] It is an explanatory diagram of an example of question-and-answer data. [Figure 18] It is an explanatory diagram of an example of the confidence level of an answer to a question sentence. [Figure 19] It is an explanatory diagram of an example of a candidate classification question sentence. [Figure 20] It is an explanatory diagram of an example of a feature amount of a candidate classification question sentence. [Figure 21] It is an explanatory diagram of an example of a feature amount of a sentence on a web page. [Figure 22] It is an explanatory diagram of an example of a comparison result. [Figure 23] It is an explanatory diagram of an example of a question sentence to be removed.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. [First Embodiment] <System Configuration> Figure 1 is a configuration diagram of an example of an information processing system according to this embodiment. In this embodiment, the information processing system 1 has a configuration in which a service provision system 10, a user terminal 14, and an administrator terminal 15 are connected to each other via a network 18 such as the Internet or a LAN (Local Area Network).

[0011] The service provision system 10 provides a chatbot service. The service provision system 10 in Figure 1 includes an information processing device 11 that functions as a web server, an information processing device 12 that functions as a chatbot server, and an information processing device 13 that functions as an aggregation server.

[0012] The information processing device 11, which functions as a web server, provides data from a web page, such as a homepage where a chatbot is installed, to the user terminal 14. The information processing device 12, which functions as a chatbot server, performs chatbot processing, for example, by answering questions from the user terminal 14. For example, the information processing device 12, which functions as a chatbot server, provides a chatbot service that responds with answers to questions entered by the user.

[0013] Furthermore, the information processing device 12, which functions as a chatbot server, stores information such as the chat history of questions and answers exchanged by the chatbot as dialogue history information. Dialogue history information is an example of a name.

[0014] The information processing device 13, which functions as an aggregation server, assists in removing inappropriate questions from the question texts included in the dialogue history information. Inappropriate questions include, for example, questions that do not make sense in Japanese, or questions that are unrelated to the web page on which the chatbot is installed. By assisting users such as administrators in removing inappropriate questions, the information processing device 13, which functions as an aggregation server, makes it easier for users to understand customer feedback from the question texts included in the dialogue history information.

[0015] The user terminal 14 receives operations from the user, such as displaying a web page and entering questions into a chatbot installed on the web page. The user terminal 14 also displays answers to questions entered by the user into the chatbot, in accordance with the chatbot processing of the information processing device 12, which functions as a chatbot server.

[0016] The user terminal 14 may be a notebook PC (Personal Computer), desktop PC, smartphone, tablet device, mobile phone, or PDA (Personal Digital Assistant). Alternatively, the user terminal 14 may be a printer, scanner, fax machine, multifunction peripheral (MFP), projector, display device with electronic whiteboard functionality, digital signage output device, HUD (Head Up Display) device, industrial machinery, imaging device, sound collection device, medical equipment, networked home appliance, automobile (Connected Car), game console, etc.

[0017] The administrator terminal 15 is operated by an administrator or other user. The administrator terminal 15 accepts operations from the user, such as displaying candidates for inappropriate question sentences identified by the information processing device 13, which functions as an aggregation server, and specifying question sentences to be removed from the list of candidates for inappropriate question sentences. The administrator terminal 15 may be a notebook PC, desktop PC, smartphone, tablet device, mobile phone, or PDA.

[0018] Note that the configuration of the information processing system 1 shown in Figure 1 is just one example. The service provision system 10 in Figure 1 may be implemented using a single computer or multiple computers, or it may be implemented using cloud services.

[0019] <Hardware Configuration> "computer" The information processing devices 11, 12, 13, user terminal 14, and administrator terminal 15 in Figure 1 are implemented by a computer 500 with the hardware configuration shown in Figure 2, for example. Figure 2 is a hardware configuration diagram of an example of a computer according to this embodiment.

[0020] Computer 500 is equipped with a CPU (Central Processing Unit) 501, ROM (Read Only Memory) 502, RAM (Random Access Memory) 503, HD 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, data bus 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, and media I / F 516.

[0021] Of these components, the CPU 501 controls the operation of the entire computer 500 according to the program. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501.

[0022] The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection interface 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory. The network interface 509 is an interface for data communication using the network 18. The data bus 510 is an address bus, data bus, etc., for electrically connecting various components such as the CPU 501.

[0023] The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 514 controls the reading or writing of various data to the DVD-RW 513, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0024] Note that the hardware configuration shown in Figure 2 is just one example, and it is not necessary to include all of the components shown in Figure 2, or to include components other than those shown in Figure 2.

[0025] <Functional Configuration> Figure 3 is a functional configuration diagram of an example of an information processing system according to this embodiment. Note that components unnecessary for the explanation of this embodiment have been appropriately omitted from the functional configuration diagram in Figure 3.

[0026] The service provision system 10 realizes the functional configuration shown in Figure 3 by executing an OS (Operating System) and programs on, for example, the information processing devices 11, 12, and 13 shown in Figure 1.

[0027] The service provision system 10 in Figure 3 includes a web page provision unit 30, a web page data storage unit 32, a chatbot processing unit 34, a question answer data storage unit 36, a dialogue history information storage unit 38, an extraction unit 40, a first feature calculation unit 42, a second feature calculation unit 44, a comparison unit 46, a discrimination unit 48, a screen generation unit 50, a removal unit 52, and an aggregation unit 54.

[0028] Furthermore, the user terminal 14 implements the functional configuration shown in Figure 3 by executing the OS and programs. The user terminal 14 has an input unit 60 and an output unit 62. In addition, the administrator terminal 15 implements the functional configuration shown in Figure 3 by executing the OS and programs. The administrator terminal 15 has an input unit 80 and an output unit 82.

[0029] The web page provision unit 30 provides the data of the web page on which the chatbot is installed to the user terminal 14. The web page data storage unit 32 stores the data of the web page on which the chatbot is installed. For example, content such as a homepage (an example of a web page) is one such example.

[0030] The chatbot processing unit 34 performs chatbot processing to provide the user terminal 14 with an answer to a question from the user terminal 14. The question and answer data storage unit 36 ​​stores question and answer data, which is used for chatbot processing and associates question and answer sentences. For example, the question and answer data can be in the form of Q&A (Question and Answer) data, where question and answer sentences are paired. Alternatively, the question and answer data can be in the form of FAQ (Frequently Asked Questions) data, which is a collection of frequently asked questions and their answers.

[0031] The dialogue history information storage unit 38 stores information such as the chat history of the chatbot, including the exchange of questions and answers, as dialogue history information. The extraction unit 40 extracts candidate questions for classification from the dialogue history information, such as scribbles, as candidate questions, as described below.

[0032] The first feature calculation unit 42 calculates the features of the question sentences of the classification candidates, which are candidates for inappropriate questions. For example, the first feature calculation unit 42 calculates a sentence vector as a feature to calculate the features of the classification candidate question sentences. The second feature calculation unit 44 calculates the features of the question sentences included in the question answer data stored in the question answer data storage unit 36. For example, the second feature calculation unit 44 calculates a sentence vector of the question sentences included in the question answer data as a feature.

[0033] The comparison unit 46 compares the feature quantities of the candidate question sentences with the feature quantities of the question sentences contained in the question answer data stored in the question answer data storage unit 36. For example, the comparison unit 46 compares the sentence vector of the candidate question sentences with the sentence vector of the question sentences contained in the question answer data. As a result of the comparison, the comparison unit 46 calculates the similarity between the feature quantities of the candidate question sentences and the feature quantities of the question sentences contained in the question answer data.

[0034] The discrimination unit 48 determines which question sentences should be excluded from the candidate question sentences for classification based on the comparison results of the comparison unit 46. For example, the discrimination unit 48 determines which candidate question sentences should be excluded from the candidate question sentences for classification if there are question sentences in the question answer data with a similarity of a threshold or higher, based on the similarity between the feature quantities of the candidate question sentences for classification and the feature quantities of the question sentences included in the question answer data. The discrimination unit 48 also determines which candidate question sentences should be excluded from the candidate question sentences for classification if there are no question sentences in the question answer data with a similarity of a threshold or higher. In this implementation, the case where the similarity is a threshold or higher is explained as an example, but it is also possible to include cases such as "higher than the threshold," which can similarly determine which question sentences should be excluded based on the threshold. In this way, the discrimination unit 48 can determine which question sentences should be excluded from the candidate question sentences for classification based on the relationship between similarity and the threshold.

[0035] The screen generation unit 50 displays the question sentences of the classification candidates identified by the discrimination unit 48, generates screen data such as an administration screen for accepting the user's selection of question sentences to be removed, and provides the screen data to the administrator terminal 15.

[0036] The administrator terminal 15, which is provided with the screen data, displays the question sentences of the classification candidates determined by the discrimination unit 48. The administrator terminal 15, which is provided with the screen data, also accepts an operation from the user to select a question sentence to be removed from the classification candidate question sentences.

[0037] The removal unit 52 removes the question selected by the user of the administrator terminal 15 from the list of candidate question sentences for classification. The aggregation unit 54 can aggregate multiple question sentences included in the dialogue history information after removing inappropriate questions, thereby improving the aggregation accuracy.

[0038] The input unit 60 of the user terminal 14 receives operations from the user, such as displaying a web page and entering a question into the chatbot installed on the web page. The output unit 62 displays the web page and the answer to the question entered by the user into the chatbot.

[0039] The input unit 80 of the administrator terminal 15 accepts operations such as displaying the question sentences of the classification candidates that have been identified, and allowing the user to select question sentences to be removed from among the classification candidate question sentences. The output unit 82 displays the question sentences of the classification candidates that have been identified, and displays a screen for accepting the user's selection of question sentences to be removed.

[0040] <Processing> The following describes the process after the service provision system 10 performs chatbot processing to provide the user terminal 14 with answers to questions from the user terminal 14, and the dialogue history information is stored in the dialogue history information storage unit 38. The service provision system 10 assists in removing inappropriate questions, such as scribbles, from the questions included in the dialogue history information, for example, by the processing procedure shown in Figure 4.

[0041] Figure 4 is a flowchart of an example of the processing procedure of the service provision system according to this embodiment. In step S10, the extraction unit 40 of the service provision system 10 reads, for example, the dialogue history information shown in Figure 5 from the dialogue history information storage unit 38.

[0042] Figure 5 is an explanatory diagram of an example of dialogue history information. As shown in Figure 5, the dialogue history information includes information about the question text from the user terminal 14. The question text "I am a cat" in Figure 5 is an example of an inappropriate question text, as it is unrelated to the location where the chatbot is installed, such as the web page. The question text "aaaa" in Figure 5 is an example of an inappropriate question text, as it does not make sense in Japanese.

[0043] In step S12, the extraction unit 40 extracts candidate inappropriate question sentences from the dialogue history information as candidate question sentences for classification. Figure 6 is a flowchart showing an example of the processing procedure for extracting candidate question sentences for classification. Figure 7 is an explanatory diagram of an example of the process for extracting candidate question sentences for classification.

[0044] In step S30, the extraction unit 40 classifies the multiple question sentences from the user terminal 14 recorded in the dialogue history information in Figure 5 into multiple clusters based on similarity, for example, as shown in Figure 7. The processing in step S30 applies a clustering algorithm that handles noise (outliers), such as DBSCAN (Density-based spatial clustering of applications with noise).

[0045] In clustering methods that handle noise, such as DBSCAN, similar question sentences are classified into the same cluster, such as "Cluster 1" and "Cluster 2," while question sentences for which no other similar question sentences were found are classified as noise. For example, in Figure 7, the question sentences "Regarding next month's seminar," "The price is too high," "Failed to start," "I am a cat," and "aaaa" are classified as noise.

[0046] In step S32, the extraction unit 40 extracts the question sentences classified as noise from the results of the classification in step S30 as candidate question sentences for classification, which are candidates for inappropriate question sentences. In Figure 7, the question sentences classified as noise, "Regarding next month's seminar," "The price is too high," "Failed to start," "I am a cat," and "aaaa," are extracted as candidate question sentences for classification, which are candidates for inappropriate question sentences.

[0047] Returning to step S14 in Figure 4, the discrimination unit 48 identifies and excludes the question sentences that should be excluded from the classification candidate question sentences, which are candidates for inappropriate question sentences extracted in step S12. The process in step S14 is performed, for example, as shown in Figures 8 and 9. Figures 8 and 9 are explanatory diagrams of an example of the process of identifying and excluding question sentences that should be excluded from the classification candidate question sentences, which are candidates for inappropriate question sentences.

[0048] The first feature calculation unit 42 calculates a sentence vector as a feature for each question sentence of the classification candidate, which is a candidate for an inappropriate question. The second feature calculation unit 44 calculates a sentence vector as a feature for each question sentence of the question answer data stored in the question answer data storage unit 36.

[0049] The comparison unit 46 compares each of the candidate question sentences in the classification group, which are candidates for inappropriate questions, with each of the question sentences in the question-answer data, and calculates the similarity. For example, the comparison unit 46 calculates the cosine similarity between the sentence vector of each candidate question sentence in the classification group, which are candidates for inappropriate questions, and the sentence vector of each question sentence in the question-answer data, using a brute-force method.

[0050] Cosine similarity is a measure of the similarity between two vectors, and is the cosine value of the angle between the two vectors in a vector space. The cosine similarity between the sentence vectors of each question sentence in the classification candidate data (which are candidates for inappropriate questions) and the sentence vectors of each question sentence in the question answer data approaches "1.0" when the meanings of the question sentences are similar, and approaches "-1.0" when the meanings of the sentences are different.

[0051] For example, in the example in Figure 8, the candidate inappropriate question "Regarding next month's seminar" has a similarity score calculated for both the question "Regarding the seminar" and "Regarding the materials" in the question-answer data. Similarly, the candidate inappropriate questions "The price is too high," "Failed to start," "I am a cat," and "aaaa" also have similarity scores calculated for both the question "Regarding the seminar" and "Regarding the materials" in the question-answer data.

[0052] The discrimination unit 48 determines which question sentences should be excluded from the list of inappropriate question sentence candidates based on the comparison results of the comparison unit 46. For example, based on the comparison results of the comparison unit 46, the discrimination unit 48 determines that if there is a candidate for an inappropriate question sentence among the question sentences included in the question answer data that has a similarity score of 1 or higher than a threshold, that question sentence should be excluded from the list of inappropriate question sentence candidates. Conversely, the discrimination unit 48 determines that if there is a candidate for an inappropriate question sentence among the question answer data that does not have a similarity score of 1 or higher than a threshold, that question sentence should not be excluded from the list of inappropriate question sentence candidates.

[0053] For example, in Figure 8, the similarity between the candidate inappropriate question "Regarding next month's seminar" and the question "Regarding the seminar" included in the question-answer data is above the threshold. Also, in Figure 8, for example, the similarity between the candidate inappropriate questions "The price is too high," "Failed to start," "I am a cat," and "aaaa" and the question "Regarding the seminar" and "Regarding the materials" in the question-answer data is below the threshold.

[0054] As shown in Figure 9, the discrimination unit 48 excludes the candidate for inappropriate question sentence "Regarding next month's seminar" from the list of inappropriate question sentence candidates if there is a question sentence in the question answer data that is similar to or above the threshold. Also, as shown in Figure 9, the discrimination unit 48 leaves the candidates for inappropriate question sentences "The price is too high," "Failed to start," "I am a cat," and "aaaa" as inappropriate question sentence candidates if there is no question sentence in the question answer data that is similar to or above the threshold.

[0055] Returning to step S16 in Figure 4, the screen generation unit 50 displays candidate inappropriate question sentences identified by the discrimination unit 48, generates screen data such as an administration screen for accepting selection of question sentences to be removed, and provides the screen data to the administrator terminal 15, thereby displaying a UI (User Interface) such as the one shown in Figure 10 on the administrator terminal 15.

[0056] Figure 10 is an illustrative image of an example of an administration screen displayed on an administrator terminal. In the administration screen 1000a of Figure 10, candidates for inappropriate questions have been removed in step S14. Users such as administrators can operate the administration screen 1000a to select an inappropriate question from the list of candidates. Figure 10 shows the administration screen 1000a with "I am a cat" and "aaaa" selected as inappropriate questions.

[0057] When the "Next" button on the administration screen 1000a is pressed, the administrator terminal 15 displays the administration screen 1000b shown in Figure 10. The administration screen 1000b accepts the user's selection of a removal pattern for the inappropriate question selected on the administration screen 1000a. The administration screen 1000b shows, as examples of removal patterns, "Pattern 1: Remove the selected question," "Pattern 2: Remove all questions asked by the questioner of the selected question," and "Pattern 3: Remove all questions asked by other questioners who asked the same question as the questioner of the selected question."

[0058] Removal pattern "Pattern 1: Remove selected question text" is an example of designating the selected question text in the administration screen 1000a as an inappropriate question text. Removal pattern "Pattern 2: Remove all questions asked by the questioner of the selected question text" is an example of designating all questions asked by the questioner of the selected question text in the administration screen 1000a as inappropriate questions. Removal pattern "Pattern 3: Remove all questions from other questioners who asked the same question as the questioner of the selected question text" is an example of designating all questions asked by other questioners who asked the same question as the questioner of the selected question text in the administration screen 1000a as inappropriate questions.

[0059] Returning to step S18 in Figure 4, the administrator terminal 15 receives the specification of inappropriate question texts to be removed from the user operating the management screen 1000a, and also receives the specification of removal patterns from the user operating the management screen 1000b.

[0060] In step S20, the removal unit 52 removes inappropriate question sentences according to the specification of inappropriate question sentences to be removed and the specification of removal patterns received by the administrator terminal 15 from a user such as an administrator.

[0061] For example, when "Pattern 1: Remove selected question sentences" is pressed on the administration screen 1000b, the administrator terminal 15 displays the administration screen 1000c shown in Figure 10. The administration screen 1000c displays the question sentences "I am a cat" and "aaaa" as a list of questions to be removed. Users such as administrators can remove the question sentences "I am a cat" and "aaaa" displayed as a list of questions on the administration screen 1000c by pressing the "Remove" button. The administration screen 1000c may also allow users to individually select the questions to be removed.

[0062] According to the first embodiment, inappropriate questions, such as scribbles (including those that appear to be incorrect inputs), can be more efficiently removed from the question texts included in the dialogue history information, based on the texts included in the relevant web page (or its content), and identified as question texts that should be removed. [Second Embodiment] In the first embodiment, an example was described in which candidate question sentences for classification, which are candidates for inappropriate question sentences, are extracted from dialogue history information by applying a clustering algorithm. The extraction of candidate question sentences for classification, which are candidates for inappropriate question sentences, may be performed, for example, as follows. The second embodiment overlaps with the description of the first embodiment, so some explanations will be omitted as appropriate.

[0063] The system configuration and hardware configuration are the same as in the first embodiment. Figure 11 is a functional configuration diagram of an example of the information processing system according to this embodiment.

[0064] <Functional Configuration> Figure 11 is a functional configuration diagram of an example of an information processing system according to this embodiment. Note that components unnecessary for the explanation of this embodiment have been appropriately omitted from the functional configuration diagram in Figure 11. The information processing system 1 in Figure 11 is a configuration in which a confidence calculation unit 56 has been added to the functional configuration diagram in Figure 3.

[0065] The confidence calculation unit 56 calculates a confidence score that represents the likelihood of the answer sentence that the chatbot was able to provide in response to the question. For example, the confidence calculation unit 56 calculates the confidence score of the answer sentence based on the similarity between the sentence vector of the question sentence stored in the dialogue history information and the sentence vector of the question sentence in the question answer data stored in the question answer data storage unit 36.

[0066] The first feature calculation unit 42 calculates the features of the question sentences of the classification candidates, which are candidates for inappropriate questions. For example, the first feature calculation unit 42 calculates the sentence vector of the classification candidate question sentence as a feature. The second feature calculation unit 44 calculates the features of the sentences contained in the web page on which the chatbot is installed. For example, the second feature calculation unit 44 calculates the sentence vector of the sentences contained in the web page on which the chatbot is installed as a feature.

[0067] The comparison unit 46 compares the feature quantities of the candidate question sentence with the feature quantities of the sentences contained on the web page where the chatbot is installed. For example, the comparison unit 46 compares the sentence vector of the candidate question sentence with the sentence vector of the sentences contained on the web page where the chatbot is installed. As a result of the comparison, the comparison unit 46 calculates the similarity between the feature quantities of the candidate question sentence and the feature quantities of the sentences contained on the web page where the chatbot is installed.

[0068] The discrimination unit 48 determines which question sentences should be excluded from the list of candidate question sentences based on the comparison results of the comparison unit 46. For example, the discrimination unit 48 determines which question sentences should be excluded from the list of candidate question sentences based on the similarity between the feature quantities of the candidate question sentences and the feature quantities of the sentences contained in the web page on which the chatbot is installed, if there are sentences in the web page with a similarity of a threshold or higher. The discrimination unit 48 also determines which question sentences should be excluded from the list of candidate question sentences if there are no sentences in the web page on which the chatbot is installed with a similarity of a threshold or higher. The configuration of the other aspects of the second embodiment is the same as that of the first embodiment, so a description will be omitted.

[0069] <Processing> The following describes the process after the service provision system 10 performs chatbot processing to provide the user terminal 14 with answers to questions from the user terminal 14, and the dialogue history information is stored in the dialogue history information storage unit 38. The service provision system 10 assists in removing inappropriate questions, such as scribbles, from the questions included in the dialogue history information, for example, by the processing procedure shown in Figure 12.

[0070] Figure 12 is a flowchart of an example of the processing procedure of the service provision system according to this embodiment. In step S50, the extraction unit 40 of the service provision system 10 reads the dialogue history information shown in Figure 13, for example, from the dialogue history information storage unit 38.

[0071] Figure 13 is an explanatory diagram of an example of dialogue history information. As shown in Figure 13, the dialogue history information includes information on the chatbot's response status (whether an answer was found or not) to the question from the user terminal 14. By using the dialogue history information shown in Figure 13, the extraction unit 40 can select from the dialogue history information either a question for which an answer was found or a question for which no answer was found. The question in Figure 13, "Tell me how to make peperoncino," is an example of a question that is unrelated to the location where the chatbot is installed, such as a webpage, and is an example of an inappropriate question.

[0072] In step S52, the extraction unit 40 extracts candidate inappropriate question sentences from the dialogue history information as candidate question sentences for classification, for example, using the procedure shown in Figure 14. Figure 14 is a flowchart showing an example of the processing procedure for extracting candidate question sentences for classification.

[0073] In step S70, the extraction unit 40 refers to the chatbot's response status in the dialogue history information shown in Figure 13 and selects, for example in Figure 15, the questions from the user terminal 14 for which the chatbot could not find an answer. Figure 15 is an explanatory diagram of an example of a question for classification. In Figure 15, two questions whose chatbot response status is "No answer found" are selected from the dialogue history information in Figure 13 as questions for removal.

[0074] In step S72, the extraction unit 40 refers to the chatbot's response status in the dialogue history information shown in Figure 13 and selects, for example, the questions from the user terminal 14 for which the chatbot has found an answer, as shown in Figure 16. Figure 16 is an explanatory diagram of an example of questions for which an answer has been found. In Figure 16, eight questions whose chatbot response status is "Answer found" have been selected from the dialogue history information in Figure 13.

[0075] In step S74, the extraction unit 40 reads question-answer data from the question-answer data storage unit 36, for example, the question-answer data shown in Figure 17. Figure 17 is an explanatory diagram of an example of question-answer data. As shown in Figure 17, the question-answer data is information that associates question sentences and answer sentences used for chatbot processing, and can use data in the format of Q&A or FAQ, for example.

[0076] In step S76, the confidence calculation unit 56 calculates the confidence level representing the likelihood of the answer the chatbot was able to give to the question in Figure 17, as follows: The confidence calculation unit 56 calculates the sentence vector (vector data of several hundred to several thousand dimensions) for each question in Figure 16 and the sentence vector for each question in Figure 17.

[0077] The confidence calculation unit 56 calculates the cosine similarity between the sentence vector of the question in Figure 16 and the sentence vector of the question in Figure 17 by brute force. The cosine similarity between the sentence vector of the question in Figure 16 and the sentence vector of the question in Figure 17 approaches "1.0" when the meanings of the sentences are similar, and approaches "-1.0" when the meanings of the sentences are different.

[0078] The confidence calculation unit 56 selects the largest cosine similarity among the calculated cosine similarities as the "confidence level of the answer" for each question in Figure 16. Figure 18 is an explanatory diagram of an example of the confidence level of the answer for a question. In Figure 18, for each question in Figure 16, the confidence level of the answer and the question in the question answer data that yielded the largest cosine similarity are shown.

[0079] In step S78, the extraction unit 40 selects question sentences from among those for which a chatbot response was found that have a confidence level lower than the threshold as candidate question sentences for classification. For example, if the threshold is set to "0.8", the extraction unit 40 selects three question sentences from Figure 18 with a threshold lower than "0.8" as candidate question sentences for classification, as shown in Figure 19. Figure 19 is an explanatory diagram of an example of a question sentence to be removed.

[0080] Through the process shown in Figure 14, the extraction unit 40 can extract the candidate question sentences for classification shown in Figures 15 and 19. Returning to Figure 12, in step S54, the first feature calculation unit 42 of the service provision system 10 calculates a sentence vector as a feature for each candidate question sentence, which is a candidate for an inappropriate question sentence as shown in Figures 15 and 19, for example as shown in Figure 20. Figure 20 is an explanatory diagram of an example of the feature of a candidate question sentence for classification.

[0081] In step S56, the second feature calculation unit 44 obtains data from the web page data storage unit 32 for the web page on which the chatbot is installed. The second feature calculation unit 44 obtains text data from the acquired web page data and divides it into sentences. The second feature calculation unit 44 calculates a sentence vector as a feature for each sentence contained in the web page on which the chatbot is installed, for example as shown in Figure 21. Figure 21 is an explanatory diagram of an example of the feature of sentences on a web page.

[0082] In step S58, the comparison unit 46 compares the feature quantities of the candidate question sentences shown in Figure 20 with the feature quantities of the text on the web page where the chatbot is installed, shown in Figure 21, and calculates the similarity as follows: The comparison unit 46 calculates the cosine similarity between the text vector of the candidate question sentences shown in Figure 20 and the text vector of the text on the web page where the chatbot is installed, shown in Figure 21, by exhaustive calculation.

[0083] Furthermore, the cosine similarity between the sentence vectors of the candidate questions shown in Figure 20 and the sentence vectors of the text on the web page where the chatbot is installed, shown in Figure 21, approaches "1.0" if the meanings of the sentences are similar, and approaches "-1.0" if the meanings of the sentences are different.

[0084] The comparison unit 46 selects the largest cosine similarity among the calculated cosine similarities as the similarity to the text on the web page where the chatbot shown in Figure 22 is installed, for each of the candidate question sentences shown in Figure 20, as shown in Figure 22. Figure 22 is an explanatory diagram of an example of the comparison results.

[0085] In step S60, the discrimination unit 48 identifies and excludes questions that should be excluded from the candidate questions for classification based on the comparison results from step S58. For example, the discrimination unit 48 identifies and excludes questions that should be excluded from the candidate questions for classification shown in Figure 22, which have a similarity threshold of 0.8 or higher: "About the service environment," "How much is the monthly fee?", "Are there any discount sales?", and "Please tell me the sales procedure."

[0086] Furthermore, the discrimination unit 48 can identify the question sentence "How do I make peperoncino?" whose similarity is less than the threshold "0.8" as a question sentence that should be kept as a candidate for classification (a question sentence to be removed), as shown in Figure 23. Figure 23 is an explanatory diagram of an example of a question sentence to be removed.

[0087] In step S62 of Figure 4, the screen generation unit 50 displays candidate inappropriate question sentences identified by the discrimination unit 48, generates screen data such as an administration screen for accepting selection of question sentences to be removed, and provides the screen data to the administrator terminal 15, thereby displaying the UI on the administrator terminal 15.

[0088] The UI displays a list of inappropriate question candidates, excluding those that should be excluded in step S60. The processing in steps S64 to S66 is the same as in the first embodiment. In step S64, a user such as an administrator can operate the UI to select and remove an inappropriate question from the list of inappropriate question candidates.

[0089] According to the second embodiment, by comparing the feature quantities of sentences contained in the relevant web page (content), inappropriate questions, such as scribbles (including those that appear to be incorrect inputs), can be more efficiently removed from the question sentences included in the dialogue history information as question sentences that should be removed.

[0090] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0091] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. The present invention is not limited by these embodiments, and the components in these embodiments include those readily conceivable to those skilled in the art, those substantially identical, and those within the scope of so-called equivalents. Furthermore, various omissions, substitutions, modifications, and combinations of components can be made without departing from the spirit of these embodiments.

[0092] Examples of the present invention are as follows: <1> An extraction means for extracting candidate question sentences for classification based on predetermined conditions from the dialogue history information of a service provision system that provides answers to questions from user terminals, A determination means for determining which question sentences should be excluded from the aforementioned candidate question sentences for classification, A screen generation means generates screen data for displaying the candidate question sentences from which the question sentences to be excluded have been excluded, and for receiving the user's selection of the question sentences to be removed. A removal means for removing the question selected by the user, An information processing device having <2> The extraction means extracts candidate question sentences for classification from the results of classifying a plurality of question sentences recorded in the dialogue history information based on similarity. The aforementioned <1> The information processing device described. <3> The determination means determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity of the candidate question sentences and the question sentences included in the question answer data used by the service provision system for providing the answers, and a threshold. The aforementioned <1> or <2> The information processing device described. <4> The system further includes a confidence calculation means for calculating the confidence level of the answer to the question that the service provision system was able to provide to the question, The extraction means extracts candidate questions from the question sentences that the service provider system was able to answer based on the confidence level of the answer sentence, and also extracts candidate questions from the question sentences that the service provider system was unable to answer as candidate questions. The aforementioned <1> The information processing device described. <5> The discrimination means determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity calculated by comparing the feature quantities of the candidate question sentences with the feature quantities of the sentences contained in the content of the web page displayed on the user terminal when the user terminal received the question sentences from the user terminal, and a threshold. The aforementioned <1> or <4> The information processing device described. <6> The screen generation means accepts the selection of the question sentence to be removed according to the removal pattern specified by the user from among a plurality of removal patterns. The aforementioned <1> ~ <5> An information processing device as described in any one of the following items. <7> An information processing system in which an information processing device, a user terminal, and an administrator terminal are connected in a manner that enables communication, The aforementioned information processing device is An extraction means for extracting candidate question sentences for classification based on predetermined conditions from the dialogue history information of a service provision system that provides answers to questions from user terminals, A determination means for determining which question sentences should be excluded from the aforementioned candidate question sentences for classification, A screen generation means generates screen data for displaying the candidate question sentences from which the question sentences to be excluded have been excluded, and for receiving the user's selection of the question sentences to be removed. A removal means for removing the question selected by the user, It has, The aforementioned administrator terminal is An output means that displays on the screen the question sentences of the classification candidates from which the question sentences to be excluded have been excluded, according to the data, An input means for receiving the user's selection of the question sentences to be removed, An information processing system having <8> The extraction means extracts candidate question sentences for classification from the results of classifying a plurality of question sentences recorded in the dialogue history information based on similarity. The aforementioned <7> The information processing system described. <9> The determination means determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity of the candidate question sentences and the question sentences included in the question answer data used by the service provision system for providing the answers, and a threshold. The aforementioned <7> or <8> The information processing system described. <10> The system further includes a confidence calculation means for calculating the confidence level of the answer to the question that the service provision system was able to provide to the question, The extraction means extracts candidate questions from the question sentences that the service provider system was able to answer based on the confidence level of the answer sentence, and also extracts candidate questions from the question sentences that the service provider system was unable to answer as candidate questions. The aforementioned <7> The information processing system described. <11> The discrimination means determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity calculated by comparing the feature quantities of the candidate question sentences with the feature quantities of the sentences contained in the content of the web page displayed on the user terminal when the user terminal received the question sentences from the user terminal, and a threshold. The aforementioned <7> or <10> The information processing system described. <12> The screen generation means accepts the selection of the question sentence to be removed according to the removal pattern specified by the user from among a plurality of removal patterns. The aforementioned <7> ~ <11> An information processing system as described in any one of the following items. <13> An information processing method performed by an information processing device, An extraction procedure for extracting candidate question sentences for classification based on predetermined conditions from the dialogue history information of a service provision system that provides answers to questions from user terminals, A determination procedure for identifying question sentences that should be excluded from the aforementioned candidate question sentences for classification, A screen generation procedure that generates screen data for displaying the candidate question sentences from which the question sentences to be excluded have been excluded, and for receiving the user's selection of the question sentences to be removed, A removal procedure for removing the question selected by the user, An information processing method having <14> The extraction procedure involves extracting candidate question sentences for classification from the results of classifying multiple question sentences recorded in the dialogue history information based on similarity. The aforementioned <13> The information processing method described. <15> The aforementioned determination procedure determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity of the candidate question sentences and the question sentences included in the question answer data used by the service provision system for providing the answers, and a threshold. The aforementioned <13> or <14> The information processing method described. <16> The system further includes a confidence calculation procedure for calculating the confidence level of the answer to the question that the service provision system was able to provide in response to the question. The extraction procedure involves extracting candidate questions from the question sentences that the service provider system was able to answer based on the confidence level of the answer sentence, and also extracting candidate questions from the question sentences that the service provider system was unable to answer as candidate questions for classification. The aforementioned <13> The information processing method described. <17> The aforementioned discrimination procedure determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity calculated by comparing the feature quantities of the candidate question sentences with the feature quantities of the sentences contained in the content of the web page displayed on the user terminal when the user terminal received the question sentences from the user terminal, and a threshold. The aforementioned <13> or <16> The information processing method described. <18> The screen generation procedure accepts the selection of the question sentence to be removed according to the removal pattern specified by the user from among multiple removal patterns. The aforementioned <13> ~ <17> The information processing method described in any one of the following items. <19> In an information processing device, An extraction procedure for extracting candidate question sentences for classification based on predetermined conditions from dialogue history information of a service provision system that provides answers to questions from user terminals. A procedure for determining which question sentences should be excluded from the aforementioned candidate question sentences for classification. A screen generation procedure for generating screen data to display the question sentences from the classification candidates that have been excluded from the question sentences to be excluded, and to allow the user to select which question sentences to remove. A removal procedure to remove the question sentence selected by the user. A program to execute. [Explanation of symbols]

[0093] 1. Information Processing System 10. Service Delivery System 11-13 Information Processing Equipment 14. User terminals 15 Administrator terminal 18 Network 30 Webpage Provision Department 32 Web page data storage unit 34 Chatbot Processing Unit 36 Question Answer Data Storage Unit 38 Dialogue History Information Storage Unit 40 Extraction part 42 First Feature Calculation Unit 44. Second Feature Calculation Unit 46 Comparison Section 48 Discrimination part 50 Screen generation section 52 Removal part 54. Aggregation Department 56 Confidence calculation part 60, 80 Input section 62, 82 Output section [Prior art documents] [Patent Documents]

[0094] [Patent Document 1] Japanese Patent Publication No. 2021-93133

Claims

1. An extraction means for extracting candidate questions for classification based on predetermined conditions from the dialogue history information of a service provision system that provides answers to questions from user terminals, A determination means for determining which question sentences should be excluded from the aforementioned candidate question sentences for classification, A screen generation means generates screen data for displaying the candidate question sentences from which the question sentences to be excluded have been excluded, and for receiving the user's selection of the question sentences to be removed. A removal means for removing the question selected by the user, It has, The extraction means extracts candidate question sentences for classification from the results of classifying a plurality of question sentences recorded in the dialogue history information based on similarity. An information processing device characterized by the following.

2. The determination means determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity of the candidate question sentences and the question sentences included in the question answer data used by the service provision system for providing the answers, and a threshold. The information processing apparatus according to claim 1, characterized in that

3. An extraction means for extracting candidate question sentences for classification based on predetermined conditions from dialogue history information of a service provision system that provides answers to question sentences from a user terminal, A determination means for determining which question sentences should be excluded from the aforementioned candidate question sentences for classification, A screen generation means generates screen data for displaying the candidate question sentences from which the question sentences to be excluded have been excluded, and for receiving the user's selection of the question sentences to be removed. A removal means for removing the question selected by the user, The system includes a confidence calculation means for calculating the confidence level of the answer to the question that the service provision system was able to provide to the question, The extraction means extracts candidate questions from the question sentences that the service provider system was able to answer based on the confidence level of the answer sentence, and also extracts candidate questions from the question sentences that the service provider system was unable to answer as candidate questions. An information processing device characterized by the following.

4. The discrimination means determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity calculated by comparing the feature quantities of the candidate question sentences with the feature quantities of the sentences contained in the content of the web page displayed on the user terminal when the user terminal received the question sentences from the user terminal, and a threshold. An information processing apparatus according to claim 1 or 3, characterized by the above.

5. The screen generation means accepts the selection of the question sentence to be removed according to the removal pattern specified by the user from among a plurality of removal patterns. An information processing apparatus according to claim 1 or 3, characterized by the above.

6. An information processing system in which an information processing device, a user terminal, and an administrator terminal are connected in a manner that enables communication, The aforementioned information processing device is An extraction means for extracting candidate questions for classification based on predetermined conditions from the dialogue history information of a service provision system that provides answers to questions from user terminals, A determination means for determining which question sentences should be excluded from the aforementioned candidate question sentences for classification, A screen generation means generates screen data for displaying the candidate question sentences from which the question sentences to be excluded have been excluded, and for receiving the user's selection of the question sentences to be removed. A removal means for removing the question selected by the user, It has, The aforementioned administrator terminal is An output means that displays on the screen the question sentences of the classification candidates from which the question sentences to be excluded have been excluded, according to the data, An input means for receiving the user's selection of the question sentences to be removed, It has, The extraction means extracts candidate question sentences for classification from the results of classifying a plurality of question sentences recorded in the dialogue history information based on similarity. An information processing system characterized by the following.

7. The determination means determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity of the candidate question sentences and the question sentences included in the question answer data used by the service provision system for providing the answers, and a threshold. The information processing system according to claim 6, characterized by the following:

8. An information processing system in which an information processing device, a user terminal, and an administrator terminal are connected in a communicative manner, The aforementioned information processing device is An extraction means for extracting candidate questions for classification based on predetermined conditions from the dialogue history information of a service provision system that provides answers to questions from user terminals, A determination means for determining which question sentences should be excluded from the aforementioned candidate question sentences for classification, A screen generation means generates screen data for displaying the candidate question sentences from which the question sentences to be excluded have been excluded, and for receiving the user's selection of the question sentences to be removed. A removal means for removing the question selected by the user, A confidence calculation means for calculating the confidence level of the answer to the question that the service provision system was able to answer to the question, It has, The aforementioned administrator terminal is An output means that displays on the screen the question sentences of the classification candidates from which the question sentences to be excluded have been excluded, according to the data, An input means for receiving the user's selection of the question sentences to be removed, It has, The extraction means extracts candidate questions from the question sentences that the service provider system was able to answer based on the confidence level of the answer sentence, and also extracts candidate questions from the question sentences that the service provider system was unable to answer as candidate questions. An information processing system characterized by the following.

9. The discrimination means determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity calculated by comparing the feature quantities of the candidate question sentences with the feature quantities of the sentences contained in the content of the web page displayed on the user terminal when the user terminal received the question sentences from the user terminal, and a threshold. The information processing system according to claim 6 or 8, characterized by the above.

10. The screen generation means accepts the selection of the question sentence to be removed according to the removal pattern specified by the user from among a plurality of removal patterns. The information processing system according to claim 6 or 8, characterized by the above.

11. An information processing method performed by an information processing device, An extraction procedure for extracting candidate questions for classification based on predetermined conditions from the dialogue history information of a service provision system that provides answers to questions from user terminals, and A determination procedure for identifying question sentences that should be excluded from the aforementioned candidate question sentences for classification, A screen generation procedure that generates screen data for displaying the candidate question sentences from which the question sentences to be excluded have been excluded, and for receiving the user's selection of the question sentences to be removed, A removal procedure for removing the question selected by the user, It has, The extraction procedure involves extracting candidate question sentences for classification from the results of classifying multiple question sentences recorded in the dialogue history information based on similarity. An information processing method characterized by the following.

12. The aforementioned determination procedure determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity of the candidate question sentences and the question sentences included in the question answer data used by the service provision system for providing the answers, and a threshold. The information processing method according to claim 11, characterized by the above.

13. An information processing method performed by an information processing device, An extraction procedure for extracting candidate questions for classification based on predetermined conditions from the dialogue history information of a service provision system that provides answers to questions from user terminals, and A determination procedure for identifying question sentences that should be excluded from the aforementioned candidate question sentences for classification, A screen generation procedure that generates screen data for displaying the candidate question sentences from which the question sentences to be excluded have been excluded, and for receiving the user's selection of the question sentences to be removed, A removal procedure for removing the question selected by the user, The system has a confidence calculation procedure for calculating the confidence level of the answer to the question that the service provider has been able to answer to the question, The extraction procedure involves extracting candidate questions from the question sentences that the service provider system was able to answer based on the confidence level of the answer sentence, and also extracting candidate questions from the question sentences that the service provider system was unable to answer as candidate questions for classification. An information processing method characterized by the following.

14. The aforementioned discrimination procedure determines which question sentences to be excluded from the candidate question sentences based on the relationship between the similarity calculated by comparing the feature quantities of the candidate question sentences with the feature quantities of the sentences contained in the content of the web page displayed on the user terminal when the user terminal received the question sentences from the user terminal, and a threshold. The information processing method according to claim 11 or 13, characterized by the above.

15. The screen generation procedure accepts the selection of the question sentence to be removed according to the removal pattern specified by the user from among multiple removal patterns. The information processing method according to claim 11 or 13, characterized by the above.

16. In an information processing device, An extraction procedure for extracting candidate inappropriate questions as classification candidates from the dialogue history information of a service provision system that provides answers to questions from user terminals, based on predetermined conditions. A procedure for determining which question sentences should be excluded from the aforementioned candidate question sentences for classification. A screen generation procedure for generating screen data to display the question sentences from the classification candidates that have been excluded from the question sentences to be excluded, and to allow the user to select which question sentences to remove. A removal procedure to remove the question sentence selected by the user. Make it run, The extraction procedure involves extracting candidate question sentences for classification from the results of classifying multiple question sentences recorded in the dialogue history information based on similarity. A program characterized by the following.

17. Information processing device, An extraction procedure for extracting candidate inappropriate questions as classification candidates from the dialogue history information of a service provision system that provides answers to questions from user terminals, based on predetermined conditions. A procedure for determining which question sentences should be excluded from the aforementioned candidate question sentences for classification. A screen generation procedure for generating screen data to display the question sentences from the classification candidates that have been excluded from the question sentences to be excluded, and to allow the user to select which question sentences to remove. A removal procedure to remove the question sentence selected by the user. A confidence calculation procedure for calculating the confidence level of the answer to the aforementioned question that the service provision system was able to provide in response to the aforementioned question, Make it run, The extraction procedure involves extracting candidate questions from the question sentences that the service provider system was able to answer based on the confidence level of the answer sentence, and also extracting candidate questions from the question sentences that the service provider system was unable to answer as candidate questions for classification. A program characterized by the following.

Images