Information processing device, information processing system, information processing method, and program

Abstract

To determine a question sentence that is to be removed from question sentences included in interaction history information.SOLUTION: An information processing device includes: extraction means that extracts removal candidate question sentences from interaction history information of a service provision system that replies to a question sentence from a user terminal; first feature amount calculation means that calculates a feature amount of each removal candidate question sentence; second feature amount calculation means that calculates a feature amount of a sentence included in a content of a web page in which an input of the question sentence has been received; comparison means that compares between the feature amount of the removal candidate question sentence and the feature amount of a sentence included in the content of the web page; and determination means that determines a question sentence to be removed, from among the removal candidate question sentences, based on a comparison result.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, an information processing method, and a program.

Background Art

[0002] An automatic response service in which an information processing apparatus such as a computer responds (answers) to an inquiry from a user (for example, an input such as a question about a product, service, or system), so-called a chatbot (sometimes simply called a bot), has been conventionally known.

[0003] For example, Patent Document 1 discloses a technique in which, in a chatbot system, after a response by a bot, the chat history by the bot is stored and analyzed by statistical processing or the like with respect to the history.

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, when a chatbot system is used for customer inquiry response work, the chat history (dialogue history information) by the bot includes question texts from customers that can be utilized for marketing by aggregation.

[0005] However, the question texts from customers included in the dialogue history information may include question texts (questions to be removed) that disrupt the aggregation accuracy, such as pranks. There has been a problem that, in order to improve the aggregation accuracy, it is necessary to discriminate and remove the question texts to be removed from the question texts from customers included in the dialogue history information. Note that the conventional technology such as Patent Document 1 does not solve such problems.

[0006] <固定ID= One embodiment of the present invention has been made in view of the above problems, and an object thereof is to discriminate a question text to be removed from the question texts included in the dialogue history information.

Means for Solving the Problems

[0007] One embodiment of the present invention is an information processing device comprising: an extraction means for extracting candidate question sentences from dialogue history information of a service provision system that answers question sentences from a user terminal; a first feature calculation means for calculating the feature quantities of the candidate question sentences; a second feature calculation means for calculating the feature quantities of sentences contained in the content of a web page that received the input of the question sentences; a comparison means for comparing the feature quantities of the candidate question sentences with the feature quantities of sentences contained in the content of the web page; and a discrimination means for determining which question sentences to be removed from the candidate question sentences based on the results of the comparison. [Effects of the Invention]

[0008] It is possible to identify which questions should be removed from the dialogue history information. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating an example of an information processing system according to this embodiment. [Figure 2] This is a hardware configuration diagram of an example of a computer according to this embodiment. [Figure 3] This is a functional configuration diagram of an example of an information processing system according to this embodiment. [Figure 4] This is a flowchart of an example of the processing procedure of the service provision system according to this embodiment. [Figure 5] This is an explanatory diagram illustrating an example of dialogue history information. [Figure 6] This flowchart shows an example of a processing procedure for extracting question sentences that are candidates for removal. [Figure 7] This is an explanatory diagram of an example of a question that could be removed. [Figure 8] This is an explanatory diagram of an example of a question for which an answer was found. [Figure 9] This is an explanatory diagram of an example of question and answer data. [Figure 10] This is an illustrative diagram illustrating an example of the level of confidence in an answer to a question. [Figure 11]This is an explanatory diagram of an example of a question that could be removed. [Figure 12] This is an explanatory diagram illustrating an example of the feature quantities of a question text that could be removed. [Figure 13] This is an explanatory diagram illustrating an example of the text features of a web page. [Figure 14] This is an explanatory diagram illustrating an example of the comparison results. [Figure 15] This is an explanatory diagram of an example of a question that should be removed. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the attached drawings.

[0011] <System Configuration> Figure 1 is a configuration diagram of an example of an information processing system according to this embodiment. In this embodiment, the information processing system 1 has a configuration in which a service provision system 10, a user terminal 14, and an administrator terminal 15 are connected to each other via a network 18 such as the Internet or a LAN (Local Area Network).

[0012] The service provision system 10 provides a chatbot service. The service provision system 10 in Figure 1 includes an information processing device 11 that functions as a web server, an information processing device 12 that functions as a chatbot server, and an information processing device 13 that functions as an aggregation server.

[0013] The information processing device 11, which functions as a web server, provides data from a web page, such as a homepage where a chatbot is installed, to the user terminal 14. The information processing device 12, which functions as a chatbot server, performs chatbot processing, for example, by answering questions from the user terminal 14. For example, the information processing device 12, which functions as a chatbot server, provides a chatbot service that responds with answers to questions entered by the user.

[0014] In addition, the information processing device 12 that functions as a chatbot server stores information such as the exchange of question texts and answer texts, such as the chat history by the chatbot, as dialogue history information. The dialogue history information is an example of a name.

[0015] The information processing device 13 that functions as an aggregation server discriminates question texts to be removed from the question texts included in the dialogue history information. The question texts to be removed are, for example, question texts that disrupt the aggregation accuracy, such as pranks. The information processing device 13 that functions as an aggregation server can improve the aggregation accuracy by aggregating the question texts after removing the question texts determined to be removed.

[0016] The user terminal 14 accepts operations from the user, such as operations to display a web page, and operations to input a question text or the like to a chatbot installed on the web page. In addition, the user terminal 14 displays an answer text for the question text input by the user according to the chatbot process of the information processing device 12 that functions as a chatbot server.

[0017] The user terminal 14 is, for example, a notebook PC (Personal Computer), a desktop PC, a smartphone, a tablet terminal, a mobile phone, or a PDA (Personal Digital Assistant). In addition, the user terminal 14 may be an output device such as a printer, a scanner, a facsimile machine, a multifunction peripheral (MFP), a projector, a display device having an electronic blackboard function, a digital signage, a HUD (Head Up Display) device, an industrial machine, an imaging device, a sound collection device, a medical device, a network home appliance, a connected car, or a game machine.

[0018] The administrator terminal 15 is operated by an administrator or the like. The administrator terminal 15 accepts operations from the administrator, such as operations to display the result of the aggregation performed by the information processing device 13 that functions as an aggregation server. <Note that the configuration of the information processing system 1 shown in Figure 1 is just one example. The service provision system 10 in Figure 1 may be implemented using a single computer or multiple computers, or it may be implemented using cloud services.

[0020] <Hardware Configuration> "computer" The information processing devices 11, 12, 13, user terminal 14, and administrator terminal 15 in Figure 1 are implemented by a computer 500 with the hardware configuration shown in Figure 2, for example. Figure 2 is a hardware configuration diagram of an example of a computer according to this embodiment.

[0021] Computer 500 is equipped with a CPU (Central Processing Unit) 501, ROM (Read Only Memory) 502, RAM (Random Access Memory) 503, HD 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, data bus 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, and media I / F 516.

[0022] Of these components, the CPU 501 controls the operation of the entire computer 500 according to the program. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501.

[0023] The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection interface 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory. The network interface 509 is an interface for data communication using the network 18. The data bus 510 is an address bus, data bus, etc., for electrically connecting various components such as the CPU 501.

[0024] The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 514 controls the reading or writing of various data to the DVD-RW 513, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0025] Note that the hardware configuration shown in Figure 2 is just one example, and it is not necessary to include all of the components shown in Figure 2, or to include components other than those shown in Figure 2.

[0026] <Functional Configuration> Figure 3 is a functional configuration diagram of an example of an information processing system according to this embodiment. Note that components unnecessary for the explanation of this embodiment have been appropriately omitted from the functional configuration diagram in Figure 3.

[0027] The service provision system 10 realizes the functional configuration shown in Figure 3 by executing an OS (Operating System) and programs on, for example, the information processing devices 11, 12, and 13 shown in Figure 1.

[0028] The service provision system 10 in Figure 3 includes a web page provision unit 30, a web page data storage unit 32, a chatbot processing unit 34, a question answer data storage unit 36, a dialogue history information storage unit 38, an extraction unit 40, a first feature calculation unit 42, a second feature calculation unit 44, a confidence level calculation unit 46, a comparison unit 48, a discrimination unit 50, a removal unit 52, and an aggregation unit 54.

[0029] Furthermore, the user terminal 14 implements the functional configuration shown in Figure 3 by executing the OS and programs. The user terminal 14 has an input unit 60 and an output unit 62. In addition, the administrator terminal 15 implements the functional configuration shown in Figure 3 by executing the OS and programs. The administrator terminal 15 has an input unit 80 and an output unit 82.

[0030] The web page provision unit 30 provides the data of the web page on which the chatbot is installed to the user terminal 14. The web page data storage unit 32 stores the data of the web page on which the chatbot is installed. For example, content such as a homepage (an example of a web page) is one such example.

[0031] The chatbot processing unit 34 performs chatbot processing to provide the user terminal 14 with an answer to a question from the user terminal 14. The question and answer data storage unit 36 ​​stores question and answer data, which is used for chatbot processing and associates question and answer sentences. For example, the question and answer data can be in the form of Q&A (Question and Answer) data, where question and answer sentences are paired. Alternatively, the question and answer data can be in the form of FAQ (Frequently Asked Questions) data, which is a collection of frequently asked questions and their answers.

[0032] The dialogue history information storage unit 38 stores information such as the chat history of the chatbot, including question and answer exchanges, as dialogue history information. The extraction unit 40 extracts question sentences that are candidates for removal, such as scribbles, from the dialogue history information as described below.

[0033] The first feature calculation unit 42 calculates the features of the candidate question sentences for removal. For example, the first feature calculation unit 42 calculates a sentence vector as a feature to calculate the features of the candidate question sentences for removal. The second feature calculation unit 44 calculates the features of the sentences contained in the web page on which the chatbot is installed. For example, the second feature calculation unit 44 calculates a sentence vector as a feature of the sentences contained in the web page on which the chatbot is installed.

[0034] The confidence calculation unit 46 calculates a confidence score that represents the likelihood of the answer sentence that the chatbot was able to provide in response to the question. For example, the confidence calculation unit 46 calculates the confidence score of the answer sentence based on the similarity between the sentence vector of the question sentence stored in the dialogue history information and the sentence vector of the question sentence in the question answer data stored in the question answer data storage unit 36.

[0035] The comparison unit 48 compares the feature quantities of the candidate question text with the feature quantities of the text contained on the web page where the chatbot is installed. For example, the comparison unit 48 compares the sentence vector of the candidate question text with the sentence vector of the text contained on the web page where the chatbot is installed. As a result of the comparison, the comparison unit 48 calculates the similarity between the feature quantities of the candidate question text and the feature quantities of the text contained on the web page where the chatbot is installed.

[0036] The discrimination unit 50 identifies the question sentences to be removed from the list of candidate question sentences based on the comparison results of the comparison unit 48. For example, the discrimination unit 50 identifies question sentences to be removed based on the similarity between the feature quantities of the candidate question sentences to be removed and the feature quantities of the sentences contained in the web page where the chatbot is installed, and identifies question sentences to be removed whose similarity is lower than a threshold. In this embodiment, the case where the similarity is lower than a threshold is described as an example of a predetermined condition, but other conditions such as "less than" and "less than or equal to" that can similarly identify candidate question sentences to be removed relative to a threshold are also included. In this way, the discrimination unit 50 can identify question sentences to be removed from the list of candidate question sentences based on the relationship between similarity and a threshold.

[0037] The removal unit 52 removes the question sentences that the discrimination unit 50 has identified as needing to be removed from the multiple question sentences that the aggregation unit 54 aggregates. Therefore, the aggregation unit 54 can aggregate the question sentences after the question sentences that need to be removed have been removed, thus improving the aggregation accuracy.

[0038] The input unit 60 of the user terminal 14 receives operations from the user, such as displaying a web page and entering a question into the chatbot installed on the web page. The output unit 62 displays the web page and the answer to the question entered by the user into the chatbot.

[0039] The input unit 80 of the administrator terminal 15 receives requests from the administrator to display the results of the aggregation performed by the information processing device 13, which functions as an aggregation server, and to display the questions that should be removed. The output unit 82 displays the aggregation results, the questions that should be removed, etc.

[0040] <Processing> The following describes the process after the service provision system 10 performs chatbot processing to provide the user terminal 14 with answers to questions from the user terminal 14, and the dialogue history information is stored in the dialogue history information storage unit 38. The service provision system 10 identifies questions that should be removed from the dialogue history information, such as those containing scribbles, which disrupt the accuracy of the aggregation, for example, using the processing procedure shown in Figure 4.

[0041] Figure 4 is a flowchart of an example of the processing procedure of the service provision system according to this embodiment. In step S10, the extraction unit 40 of the service provision system 10 reads the dialogue history information shown in Figure 5, for example, from the dialogue history information storage unit 38.

[0042] Figure 5 is an explanatory diagram of an example of dialogue history information. As shown in Figure 5, the dialogue history information includes information on the chatbot's response status (whether an answer was found or not) to the question from the user terminal 14. By using the dialogue history information shown in Figure 5, the extraction unit 40 can select from the dialogue history information either a question for which an answer was found or a question for which no answer was found.

[0043] In step S12, the extraction unit 40 extracts candidate question sentences for removal from the dialogue history information, for example, using the procedure shown in Figure 6. Figure 6 is a flowchart showing an example of the processing procedure for extracting candidate question sentences for removal.

[0044] In step S30, the extraction unit 40 refers to the chatbot's response status in the dialogue history information shown in Figure 5 and selects, for example as shown in Figure 7, the questions from the user terminal 14 for which the chatbot could not find an answer. Figure 7 is an explanatory diagram of an example of a question for removal. In Figure 7, two questions whose chatbot response status is "No answer found" are selected from the dialogue history information in Figure 5 as questions for removal.

[0045] In step S32, the extraction unit 40 refers to the chatbot's response status in the dialogue history information shown in Figure 5 and selects, for example, the questions from the user terminal 14 for which the chatbot has found an answer, as shown in Figure 8. Figure 8 is an explanatory diagram of an example of a question for which an answer has been found. In Figure 7, eight questions with a chatbot response status of "Answer found" have been selected from the dialogue history information in Figure 5.

[0046] In step S34, the extraction unit 40 reads, for example, the question-answer data shown in Figure 9 from the question-answer data storage unit 36. Figure 9 is an explanatory diagram of an example of question-answer data. As shown in Figure 9, the question-answer data is information that associates question sentences and answer sentences used for chatbot processing, and can use data in the format of, for example, Q&A or FAQ.

[0047] In step S36, the confidence calculation unit 46 calculates the confidence level representing the likelihood of the answer sentence that the chatbot was able to provide to the question sentence in Figure 8, as follows: The confidence calculation unit 46 calculates the sentence vector (vector data of several hundred to several thousand dimensions) for each question sentence in Figure 8 and the sentence vector for each question sentence in Figure 9.

[0048] The confidence calculation unit 46 calculates the cosine similarity between the sentence vector of the question in Figure 8 and the sentence vector of the question in Figure 9 by brute force. Cosine similarity is a measure of the similarity between two vectors, and is the cosine value of the angle between the two vectors in the vector space. The cosine similarity between the sentence vector of the question in Figure 8 and the sentence vector of the question in Figure 9 approaches "1.0" when the meanings of the sentences are similar, and approaches "-1.0" when the meanings of the sentences are different.

[0049] The confidence calculation unit 46 selects the highest cosine similarity among the calculated cosine similarities as the "confidence level of the answer" for each question in Figure 8. Figure 10 is an explanatory diagram of an example of the confidence level of the answer for a question. In Figure 10, for each question in Figure 8, the confidence level of the answer and the question in the question-answer data that yielded the highest cosine similarity are shown.

[0050] In step S38, the extraction unit 40 selects questions from which the chatbot has provided an answer that have a confidence level lower than a threshold, and these questions are candidates for removal. For example, if the threshold is set to "0.8", the extraction unit 40 selects three questions from Figure 10 with a threshold lower than "0.8" as candidates for removal, as shown in Figure 11. Figure 11 is an explanatory diagram of an example of a candidate question for removal.

[0051] The extraction unit 40 can extract the candidate question sentences for removal shown in Figure 7 and Figure 11 through the process shown in Figure 6. Returning to Figure 4, in step S14, the first feature calculation unit 42 of the service provision system 10 calculates a sentence vector as a feature for each candidate question sentence for removal shown in Figure 7 and Figure 11, for example as shown in Figure 12. Figure 12 is an explanatory diagram of an example of the feature of a candidate question sentence for removal.

[0052] Returning to the explanation of Figure 4, in step S16, the second feature calculation unit 44 obtains data from the web page data storage unit 32 for the web page on which the chatbot is installed. The second feature calculation unit 44 obtains text data from the acquired web page data and divides it into sentence units. The second feature calculation unit 44 calculates a sentence vector as a feature for each sentence contained in the web page on which the chatbot is installed, for example as shown in Figure 13. Figure 13 is an explanatory diagram of an example of the feature of a web page sentence. In this embodiment, the unit for calculating the feature is a sentence, but it may be a document unit, a sentence unit, or a word unit.

[0053] In step S18, the comparison unit 48 compares the feature quantities of the candidate question sentences shown in Figure 12 with the feature quantities of the text on the web page where the chatbot is installed, shown in Figure 13, and calculates the similarity as follows: The comparison unit 48 calculates the cosine similarity between the text vector of the candidate question sentences shown in Figure 12 and the text vector of the text on the web page where the chatbot is installed, shown in Figure 13, by exhaustive calculation.

[0054] Furthermore, the cosine similarity between the sentence vectors of the candidate questions for removal shown in Figure 12 and the sentence vectors of the text on the web page where the chatbot is installed shown in Figure 13 approaches "1.0" if the meanings of the sentences are similar, and approaches "-1.0" if the meanings of the sentences are different.

[0055] The comparison unit 48 selects the largest cosine similarity among the calculated cosine similarities as the similarity to the text on the web page where the chatbot shown in Figure 13 is installed, for each of the candidate question sentences for removal shown in Figure 12, as shown in Figure 14. Figure 14 is an explanatory diagram of an example of the comparison results.

[0056] In step S20, the discrimination unit 50 determines the question sentences to be removed based on the comparison results from step S18. For example, from the candidate question sentences to be removed shown in Figure 14, the discrimination unit 50 can determine that the question sentence "How do you make peperoncino?" which has a similarity score lower than the threshold "0.8" should be removed, as shown in Figure 15. Figure 15 is an explanatory diagram of an example of a question sentence to be removed.

[0057] The processing performed by the discrimination unit 50 in step S20 is to determine that a question entered from a chatbot installed on a web page is an inappropriate question that disrupts the accuracy of the aggregation, such as a scribble, if it is not similar to any of the sentences contained on that web page.

[0058] As shown in Figure 4, the service provision system 10 can identify inappropriate questions, such as scribbles, from among the user questions included in the dialogue history information, which would disrupt the accuracy of the aggregation performed by the aggregation unit 54.

[0059] The removal unit 52 of the service provision system 10 removes the question sentences to be removed, as determined by the process shown in Figure 4, from the multiple question sentences that the aggregation unit 54 aggregates. Therefore, the aggregation unit 54 can aggregate the question sentences after the question sentences to be removed have been removed, thereby improving the aggregation accuracy.

[0060] Furthermore, according to this embodiment, even when the chatbot system is used for handling customer inquiries, it is possible to remove questions containing scribbles or other nonsense, and then accurately aggregate the questions entered into the chatbot to recognize the voices of users.

[0061] In this embodiment, questions that are not similar to the content of the web page on which the chatbot is installed, and questions that are not similar to the content of the question-answer data used by the chatbot, are identified as questions that should be removed.

[0062] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0063] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. The present invention is not limited by these embodiments, and the components in these embodiments include those readily conceivable to those skilled in the art, those substantially identical, and those within the scope of so-called equivalents. Furthermore, various omissions, substitutions, modifications, and combinations of components can be made without departing from the spirit of these embodiments.

[0064] Examples of the present invention are as follows: <1> An extraction means for extracting candidate question sentences from the dialogue history information of a service provision system that answers questions from user terminals, A first feature calculation means for calculating the feature quantities of the question texts that are candidates for removal, A second feature calculation means for calculating the feature quantities of sentences contained in the content of a web page that received the input of the aforementioned question, A comparison means for comparing the feature quantities of the question sentences of the candidates for removal with the feature quantities of the sentences contained in the content of the web page, Based on the results of the above comparison, a determination means for determining which question sentences to be removed from the candidate question sentences to be removed, An information processing device having <2> The system further includes a confidence calculation means for calculating the confidence level of the answer to the question that the service provision system was able to provide to the question, The extraction means extracts candidate question sentences for removal from the dialogue history information based on the confidence level of the answer sentence. The aforementioned <1> The information processing device described. <3> The extraction means extracts the question sentences that the service provision system could not answer as candidate question sentences for removal. The aforementioned <1> or <2> The information processing device described. <4> The comparison means calculates the similarity by comparing the feature quantities of the question text of the candidate to be removed with the feature quantities of the sentences contained in the content of the web page. The aforementioned <1> ~ <3> An information processing device as described in any one of the following items. <5> The determination means determines which question sentences should be removed from the candidate question sentences for removal based on the relationship between the similarity and the threshold. The aforementioned <4> The information processing device described. <6> The content of the aforementioned webpage is the data of the webpage that was displayed on the user terminal when the question was received from the user terminal. The aforementioned <1> ~ <5> An information processing device as described in any one of the following items. <7> An information processing system in which an information processing device and a user terminal are connected in a manner that enables communication, The aforementioned information processing device is An extraction means for extracting candidate question sentences from the dialogue history information of a service provision system that responds to question sentences from the user terminal, A first feature calculation means for calculating the feature quantities of the question texts that are candidates for removal, A second feature calculation means for calculating the feature quantities of sentences contained in the content of a web page that received the input of the aforementioned question, A comparison means for comparing the feature quantities of the question sentences of the candidates for removal with the feature quantities of the sentences contained in the content of the web page, Based on the results of the above comparison, a determination means for determining which question sentences to be removed from the candidate question sentences to be removed, It has, The user terminal has an input means for receiving the question text from the user. An information processing system having <8> The aforementioned information processing device is The system further includes a confidence calculation means for calculating the confidence level of the answer to the question that the service provision system was able to provide to the question, The extraction means extracts candidate question sentences for removal from the dialogue history information based on the confidence level of the answer sentence. The aforementioned <7> The information processing system described. <9> The extraction means extracts the question sentences that the service provision system could not answer as candidate question sentences for removal. The aforementioned <7> or <8> The information processing system described. <10> The comparison means calculates the similarity by comparing the feature quantities of the question text of the candidate to be removed with the feature quantities of the sentences contained in the content of the web page. The aforementioned <7> ~ <9> An information processing system as described in any one of the following items. <11> The determination means determines which question sentences should be removed from the candidate question sentences for removal based on the relationship between the similarity and the threshold. The aforementioned <10> The information processing system described. <12> The content of the aforementioned webpage is the data of the webpage that was displayed on the user terminal when the question was received from the user terminal. The aforementioned <7> ~ <11> An information processing system as described in any one of the following items. <13> An information processing method performed by an information processing device, An extraction procedure for extracting candidate questions for removal from the dialogue history information of a service provision system that answers questions from user terminals, A first feature calculation procedure for calculating the feature quantities of the question text of the candidate for removal, A second feature calculation procedure for calculating the feature quantities of sentences contained in the content of the web page that received the input of the aforementioned question, A comparison procedure for comparing the feature quantities of the question text of the candidate for removal with the feature quantities of the sentences contained in the content of the web page, A determination procedure for identifying which question sentences to remove from the candidate question sentences to remove based on the results of the above comparison, An information processing method having <14> The system further includes a confidence calculation procedure for calculating the confidence level of the answer to the question that the service provision system was able to provide in response to the question. The extraction procedure involves extracting candidate question sentences for removal from the dialogue history information based on the confidence level of the answer sentence. The aforementioned <13> The information processing method described. <15> The extraction procedure involves extracting the question sentences that the service provision system was unable to answer as candidate question sentences for removal. The aforementioned <13> or <14> The information processing method described. <16> The comparison procedure involves comparing the feature quantities of the question text of the candidate for removal with the feature quantities of the sentences contained in the content of the web page to calculate the similarity. The aforementioned <13> ~ <15> The information processing method described in any one of the following items. <17> The aforementioned determination procedure involves determining which question sentences should be removed from the list of candidate question sentences based on the relationship between the similarity and the threshold. The aforementioned <16> The information processing method described. <18> The content of the aforementioned webpage is the data of the webpage that was displayed on the user terminal when the question was received from the user terminal. The aforementioned <13> ~ <17> The information processing method described in any one of the following items. <19> In an information processing device, An extraction procedure for extracting candidate questions for removal from the dialogue history information of a service provision system that answers questions from user terminals. A first feature calculation procedure for calculating the feature quantities of the question text of the candidate for removal, A second feature calculation procedure for calculating the feature quantities of sentences contained in the content of the web page that received the input of the aforementioned question, A comparison procedure for comparing the feature quantities of the question text of the candidate for removal with the feature quantities of the sentences contained in the content of the web page. A determination procedure for identifying which question sentences to remove from the candidate question sentences to remove, based on the results of the above comparison, A program to execute. [Explanation of Symbols]

[0065] 1. Information Processing System 10. Service Delivery System 11-13 Information Processing Equipment 14. User terminals 15 Administrator terminal 18 Network 30 Webpage Provision Department 32 Web page data storage unit 34 Chatbot Processing Unit 36 Question Answer Data Storage Unit 38 Dialogue History Information Storage Unit 40 Extraction part 42 First Feature Calculation Unit 44. Second Feature Calculation Unit 46 Confidence calculation part 48 Comparison Section 50 Discrimination part 52 Removal part 54. Aggregation Department 60, 80 Input section 62, 82 Output section [Prior art documents] [Patent Documents]

[0066] [Patent Document 1] Japanese Patent Publication No. 2021-93133

Claims

1. An extraction means for extracting candidate question sentences from the dialogue history information of a service provision system that answers questions from user terminals, A first feature calculation means for calculating the feature quantities of the question texts that are candidates for removal, A second feature calculation means for calculating the feature quantities of sentences contained in the content of a web page that has received the input of the aforementioned question, A comparison means for comparing the feature quantities of the question sentences of the candidates for removal with the feature quantities of the sentences contained in the content of the web page, Based on the results of the above comparison, a determination means for determining which question sentences to be removed from the candidate question sentences to be removed, An information processing device having

2. The system further includes a confidence calculation means for calculating the confidence level of the answer to the question that the service provision system was able to provide to the question, The extraction means extracts candidate question sentences for removal from the dialogue history information based on the confidence level of the answer sentence. The information processing apparatus according to claim 1, characterized in that

3. The extraction means extracts the question sentences that the service provision system could not answer as candidate question sentences for removal. An information processing apparatus according to claim 1 or 2, characterized by the above.

4. The comparison means calculates the similarity by comparing the feature quantities of the question text of the candidate to be removed with the feature quantities of the sentences contained in the content of the web page. An information processing apparatus according to claim 1 or 2, characterized by the above.

5. The determination means determines which question sentences should be removed from the candidate question sentences for removal based on the relationship between the similarity and the threshold. The information processing apparatus according to claim 4, characterized by the following:

6. The content of the aforementioned webpage is the data of the webpage that was displayed on the user terminal when the question was received from the user terminal. An information processing apparatus according to either claim 1 or 2, characterized by the following:

7. An information processing system in which an information processing device and a user terminal are connected in a manner that enables communication, The aforementioned information processing device is An extraction means for extracting candidate question sentences from the dialogue history information of a service provision system that responds to question sentences from the user terminal, A first feature calculation means for calculating the feature quantities of the question texts that are candidates for removal, A second feature calculation means for calculating the feature quantities of sentences contained in the content of a web page that has received the input of the aforementioned question, A comparison means for comparing the feature quantities of the question sentences of the candidates for removal with the feature quantities of the sentences contained in the content of the web page, Based on the results of the above comparison, a determination means for determining which question sentences to be removed from the candidate question sentences to be removed, It has, The user terminal has an input means for receiving the question text from the user. An information processing system having [a certain feature].

8. The aforementioned information processing device is The system further includes a confidence calculation means for calculating the confidence level of the answer to the question that the service provision system was able to provide to the question, The extraction means extracts candidate question sentences for removal from the dialogue history information based on the confidence level of the answer sentence. The information processing system according to claim 7, characterized by the following:

9. The extraction means extracts the question sentences that the service provision system could not answer as candidate question sentences for removal. The information processing system according to claim 7 or 8, characterized by the above.

10. The comparison means calculates the similarity by comparing the feature quantities of the question text of the candidate to be removed with the feature quantities of the sentences contained in the content of the web page. The information processing system according to claim 7 or 8, characterized by the above.

11. The determination means determines which question sentences should be removed from the candidate question sentences for removal based on the relationship between the similarity and the threshold. The information processing system according to claim 10, characterized by the following:

12. The content of the aforementioned webpage is the data of the webpage that was displayed on the user terminal when the question was received from the user terminal. The information processing system according to claim 7 or 8.

13. An information processing method performed by an information processing device, An extraction procedure for extracting candidate questions for removal from the dialogue history information of a service provision system that answers questions from user terminals, A first feature calculation procedure for calculating the feature quantities of the question text of the candidate for removal, A second feature calculation procedure for calculating the feature quantities of sentences contained in the content of the web page that received the input of the aforementioned question, A comparison procedure for comparing the feature quantities of the question text of the candidate for removal with the feature quantities of the sentences contained in the content of the web page, A determination procedure for identifying which question sentences to remove from the candidate question sentences to remove based on the results of the above comparison, An information processing method having

14. The system further includes a confidence calculation procedure for calculating the confidence level of the answer to the question that the service provision system was able to provide in response to the question. The extraction procedure involves extracting candidate question sentences for removal from the dialogue history information based on the confidence level of the answer sentence. The information processing method according to claim 13, characterized by the above.

15. The extraction procedure involves extracting the question sentences that the service provision system was unable to answer as candidate question sentences for removal. The information processing method according to claim 13 or 14, characterized by the above.

16. The comparison procedure involves comparing the features of the candidate question text with the features of the text contained in the web page content to calculate the similarity. The information processing method according to claim 13 or 14, characterized by the above.

17. The aforementioned determination procedure involves determining which question sentences should be removed from the list of candidate question sentences based on the relationship between the similarity and the threshold. The information processing method according to claim 16, characterized by the above.

18. The content of the aforementioned webpage is the data of the webpage that was displayed on the user terminal when the question was received from the user terminal. The information processing method according to claim 13 or 14.

19. In an information processing device, An extraction procedure for extracting candidate questions for removal from the dialogue history information of a service provision system that answers questions from user terminals. A first feature calculation procedure for calculating the feature quantities of the question text of the candidate for removal, A second feature calculation procedure for calculating the feature quantities of sentences contained in the content of the web page that received the input of the aforementioned question, A comparison procedure for comparing the feature quantities of the question text of the candidate for removal with the feature quantities of the sentences contained in the content of the web page. A determination procedure for identifying which question sentences to remove from the candidate question sentences to remove, based on the results of the above comparison, A program to execute.

Images