QA data evaluation device

Abstract

This QA data evaluation device is provided with: an acquisition means that acquires QA data including the content of a user's questions to a chatbot and the content of the chatbot's answers to the questions, and log information about the user's use of the chatbot; an extraction means that extracts, from the log information, a feature quantity relating to the temporal behavior of the user's use of the chatbot; and a generation means that generates QA data evaluation information indicating the quality of the QA data on the basis of the feature quantity.

Description

[Technical field]

[0001] The present invention relates to a QA data evaluation device, a QA data evaluation method, and a recording medium. [Background technology]

[0002] An information processing system that presents an appropriate response text to a chat user in response to a question text sent from the chat user has been proposed or put into practical use as a chatbot system. The chatbot system refers to a QA data DB (database) that stores QA data in which expected question texts and response texts to the question texts are associated with each other, acquires a response text corresponding to the question text sent from the chat user, and presents the response text to the chat user. Therefore, it is no exaggeration to say that the reliability of a chatbot system is determined by the quality of the QA data. Therefore, in order to improve the quality of the QA data, an administrator of the chatbot system creates learning data that indicates the quality of the QA data based on the results of actual operation, and performs maintenance such as correction, deletion, and addition of the QA data based on the learning data. The quality of the QA data can be evaluated by the chat user inputting evaluation information that indicates whether the response included in the QA data is appropriate for the question. In this way, evaluation information that is actively input by the chat user is referred to as "active evaluation information" below. On the other hand, evaluation information that is not actively input by the chat user is referred to as "non-active evaluation information" below.

[0003] For example, Patent Document 1 discloses a method of acquiring the intonation and pitch of a chat user's voice after presenting a response as non-active evaluation information, and creating learning data representing the quality of the QA data based on the acquired information.

[0004] Patent document 2 also discloses that the following types of non-active evaluation information are obtained: text information obtained by converting the speech made by the chat user in response to the chatbot's response, audio data obtained by digitizing the audio of the above speech, image data obtained by digitizing an image of the chat user's appearance when hearing the response, and biometric information of the chat user (pulse, heart rate, blood pressure, brain waves, respiratory rate, etc.) around the time the response was heard.

[0005] Additionally, as a technology related to chatbots, Patent Document 3 discloses a technology for searching for a chatbot service that can be reliably used from among a large number of chatbot services based on the number of users per unit time, the average usage time of users, and chat content information. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-91513 A [Patent Document 2] JP 2019-45978 A [Patent Document 3] JP 2019-185614 A [Patent Document 4] Patent No. 5817531 Summary of the Invention [Problem to be solved by the invention]

[0007] However, it can be difficult to obtain passive evaluation information from chat users.

[0008] A primary object of the present invention is to provide an information processing device that makes it possible to easily obtain passive evaluation information. [Means for solving the problem]

[0009] A QA data evaluation device according to one embodiment of the present invention includes: An acquisition means for acquiring QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user; An extraction means for extracting features related to a temporal behavior of the user's use of the chatbot from the log information; A generating means for generating QA data evaluation information indicating the quality of the QA data based on the feature amount; The device is configured to include:

[0010] Further, a QA data evaluation method according to an embodiment of the present invention includes: Acquire QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user; Extracting features related to the temporal behavior of the user's use of the chatbot from the log information; The apparatus is configured to generate QA data evaluation information indicating whether the QA data is good or bad based on the feature amount.

[0011] Further, a computer-readable recording medium according to an embodiment of the present invention comprises: On the computer, A process of acquiring QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user; A process of extracting features related to a temporal behavior of the user's use of the chatbot from the log information; A process of generating QA data evaluation information indicating the quality of the QA data based on the feature amount; The recording medium is configured to record a program for causing the recording medium to perform the above steps. Effect of the Invention

[0012] According to the present invention, by having the above-mentioned configuration, non-active evaluation information can be easily obtained. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram of an information processing apparatus according to a first embodiment of the present invention. [Diagram 2] 2 is a diagram showing an example of the configuration of a QA data DB in the information processing device according to the first embodiment of the present invention. FIG. [Diagram 3] 2 is a diagram showing an example of a configuration of a chat log DB in the information processing device according to the first embodiment of the present invention; FIG. [Figure 4] 2 is a diagram illustrating an example of the configuration of a cluster DB in the information processing device according to the first embodiment of the present invention. FIG. [Figure 5A] 2 is a diagram showing an example of the configuration of a rule DB in the information processing device according to the first embodiment of the present invention. FIG. [Figure 5B] FIG. 2 is a diagram showing an example of rules in the information processing device according to the first embodiment of the present invention. [Figure 5C] FIG. 11 is a diagram showing another example of the rules in the information processing device according to the first embodiment of the present invention. [Figure 6] 2 is a diagram showing an example of a configuration of a learning data DB in the information processing device according to the first embodiment of the present invention; FIG. [Figure 7] 5 is a flowchart showing an example of chatbot processing and chat log collection processing in the information processing device according to the first embodiment of the present invention. [Figure 8] 5 is a flowchart showing an example of a learning data generation process in the information processing device according to the first embodiment of the present invention. [Figure 9] 3 is a diagram showing an example of chat log information in the information processing device according to the first embodiment of the present invention; FIG. [Figure 10] 3 is a diagram showing an example of a document generated by collecting question texts and response texts in log information in the information processing device according to the first embodiment of the present invention. FIG. [Figure 11]10 is a flowchart showing an example of processing executed in step S25 by a learning data generating unit in the information processing device according to the first embodiment of the present invention. [Figure 12] FIG. 2 is a diagram showing an example of a chatbot management screen in the information processing device according to the first embodiment of the present invention. [Figure 13] FIG. 11 is a block diagram of a QA data evaluation device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Next, an embodiment of the present invention will be described in detail with reference to the drawings. [First embodiment] Fig. 1 is a block diagram of an information processing device 100 according to a first embodiment of the present invention. Referring to Fig. 1, the information processing device 100 has a chatbot function of outputting an appropriate response text to a terminal device operated by a chat user in response to a question text received from the terminal device operated by the chat user, and a function of evaluating QA data used by the chatbot. The information processing device 100 includes, as main components, a communication I / F (interface) unit 110, an operation input unit 120, a screen display unit 130, a storage unit 140, and an arithmetic processing unit 150.

[0015] The communication I / F unit 110 is composed of a data communication circuit, and is configured to perform data communication with one or more user terminals 160 wirelessly or by wire. The user terminal 160 is an information processing device used by a user (chat user) who chats with the chatbot. The user terminal 160 is, for example, a personal computer, a smartphone, a tablet terminal, or the like equipped with a communication function. Any external device (not shown) other than the user terminal 160 may be connected to the communication I / F unit 110. The operation input unit 120 is composed of devices such as a keyboard and a mouse, and is configured to detect an operator's operation and output it to the arithmetic processing unit 150. The screen display unit 130 is composed of a device such as an LCD (Liquid Crystal Display), and is configured to display various information on the screen in response to an instruction from the arithmetic processing unit 150.

[0016] The storage unit 140 is composed of one or more storage devices such as a hard disk or a memory, and is configured to store processing information and a program 141 required for various processes in the arithmetic processing unit 150. The program 141 is a program that is read into the arithmetic processing unit 150 and executed to realize various processing units, and is read in advance from an external device or recording medium (not shown) via a data input / output function such as the communication I / F unit 110 and stored in the storage unit 140. Main processing information stored in the storage unit 140 includes a QA data DB 142, a chat log DB 143, a cluster DB 144, a rule DB 145, and a learning data DB 146.

[0017] The QA data DB 142 is a database that stores QA data in which question text and response text are associated with each other. FIG. 2 shows an example of the configuration of the QA data DB 142. The QA data DB 142 in this example is composed of a plurality of entries, each of which stores one QA data 1420. The QA data 1420 stored in each entry is composed of a QA data ID 1421, a question text 1422, and a response text 1423. In the QA data ID 1421 field, an ID such as a number for uniquely identifying the QA data 1420 is set. In the question text 1422 field, text information related to a question anticipated to be asked by a chat user is set. In the response data 1424 field, text information related to a response to an inquiry based on the question text 1422 is set.

[0018] The chat log DB 143 is a database that stores log information of chats between a chatbot and a chat user. FIG. 3 shows an example of the configuration of the chat log DB 143. The chat log DB 143 in this example is composed of a plurality of entries, each of which stores log information 1430 of one chat. The chat log information 1430 stored in each entry is composed of a chat user ID 1431, a chat ID 1432, and a plurality of event data 1433. In the chat user ID 1431 field, an ID for uniquely identifying a chat user is set. In the chat ID 1432 field, an ID such as a number for uniquely identifying each chat conducted with the chat user identified by the chat user ID 1431 is set. In the event data 1433 field, data related to the event of the chat is set.

[0019] The event data 1433 is composed of a date and time 14331, a type 14332, a text 14333, and a QA data ID 14334. The type of event data is set in the type 14332 field. There are four types of event data: session establishment, session release, question, and response. Session establishment means that a chat session is established (connected) between the chatbot and the chat user. Session release means that a session established between the chatbot and the chat user is released (disconnected). Question means that the chatbot has received a question text from the chat user. Response means that the chatbot has sent a response text to the chat user. The date and time 14331 field is set to the date and time when the event of the relevant type occurred, for example, in the format of "year, month, day, hour, minute, second, comma second". When the type is a question, text information of the question is set in the text 14333 field, and when the type is a response, text information of the response is set. When the type is session establishment or session release, for example, a NULL value is set in the text 14333 field. When the type is question, if QA data containing question text matching the text 14333 related to the question exists, the ID of the QA data is set in the QA data ID 14333 field, and when no QA data exists, information to the effect that no matching question text is registered is set. When the type is response, the QA data ID 14333 field is set with the same information as that set in the QA data ID 14333 field in the event data 1433 of the question that is the premise of the response. When the type is session establishment or session release, for example, a NULL value is set in the QA data ID 14333 field.

[0020] The cluster DB 144 is a database that stores information about one or more clusters generated by clustering the log information 1430 of a plurality of chats stored in the chat log DB 143 into the same cluster, with semantically similar log information being clustered together. FIG. 4 shows an example of the configuration of the cluster DB 144. The cluster DB 144 in this example is composed of a plurality of entries, each of which stores one cluster 1440. The cluster 1440 stored in each entry is composed of a cluster ID 1441, a question label 1442, the number of chat logs 1443, and a list 1434 of chat log IDs. In the item of the cluster ID 1441, an ID such as a number for uniquely identifying the cluster 1440 is set. In the item of the question label 1442, a question text commonly included in the log information of the chats belonging to the cluster 1440 is set as a question label. In the item of the list 1434 of the chat log IDs, a list of chat log IDs for identifying the log information 1430 of the chats belonging to the cluster 1440 is set. The chat log ID may be configured, for example, by a combination of a chat user ID 1431 and a chat ID 1432 shown in FIG.

[0021] The rule DB 145 is a database that stores rules for creating learning data that indicates the quality of QA data from log information in a cluster stored in the cluster DB 144. FIG. 5A shows an example of the configuration of the rule DB 145. The rule DB 145 in this example is composed of multiple entries, each of which stores one rule 1450. The rule 1450 stored in each entry is composed of a rule ID 1451, a feature type 1452, learning target QA data 1453, and an evaluation value calculation criterion 1454. In the rule ID 1451 item, an ID such as a number for uniquely identifying the rule 1450 is set. In the feature type 1452 item, a type of feature of the chat user's temporal behavior during a chat is set, which is calculated from the log information in the cluster 1440 stored in the cluster DB 144. The temporal behavior includes the elapsed time from receiving a response to asking a question, the elapsed time from receiving a response to the end of the chat, the number of questions per unit time, and the elapsed time from the start to the end of the chat. In the item of learning target QA data 1453, data is set that specifies QA data for which learning data is to be created based on the feature set in the item of feature type 1452. In the item of evaluation value calculation criterion 1454, a criterion for calculating an evaluation value that indicates the quality of the QA data set in the item of learning target QA data 1453 is set.

[0022] 5B is a diagram showing an example of rules stored in the rule DB 145. In the rule 1450-1 of this example, the feature type 1452 is set to "time T1 from when the chat user receives the response text to the last question until the chat ends," the learning target QA data 1453 is set to "QA data related to the last question," and the evaluation value calculation criterion 1454 is set to "the higher the ratio of chats in which the time T1 is less than the predetermined time TH1, the lower the evaluation value." This rule 1450-1 utilizes the tendency of chat users that, when an accurate response (answer) is returned to a question, the chat user takes a reasonable amount of time to try to understand the content of the response, but when an unexpected or irrelevant response is returned, the chat user may give up on solving the problem with the chatbot at the moment of glancing at the response and close the chat screen immediately.

[0023] 5C is a diagram showing another example of a rule stored in the rule DB 145. In this example, the rule 1450-2 has the following set in the feature type 1452 field: "frequency N1 of asking the next question before a predetermined time has elapsed since the previous question," the learning target QA data 1453 field: "QA data related to the question content commonly included in the log information in the cluster," and the evaluation value calculation criterion 1454 field: "the higher the proportion of chats whose frequency N1 is equal to or greater than a predetermined frequency TH2, the lower the evaluation value." This rule 1450-2 utilizes the tendency of chat users to rephrase their question and sometimes ask the question multiple times if they do not receive an appropriate response (answer) to their question.

[0024] The rules 1450 stored in the rule DB 145 are not limited to the above-mentioned rules 1450-1 and 1450-2, and may be rules with other content, or there may be three or more rules. For example, a rule may be used in which the evaluation value calculation criterion 1454 of the rule 1450-1 is replaced with "the higher the percentage of chats whose time T1 is equal to or longer than a predetermined time TH1, the higher the evaluation value." Also, a rule may be used in which the item of the evaluation value calculation criterion 1454 of the rule 1450-2 is replaced with "the higher the percentage of chats whose frequency N1 is less than a predetermined frequency TH2, the higher the evaluation value."

[0025] Referring again to FIG. 1, the learning data DB 146 is a database that stores learning data that indicates the quality of QA data. FIG. 6 shows an example of the configuration of the learning data DB 146. The learning data DB 146 in this example is composed of a plurality of entries, each of which stores one learning data. The learning data 1460 stored in each entry is composed of a learning data ID 1461, a question text 1462, a response text 1463, a QA data ID 1464, an evaluation value 1465, a cluster ID 1466, a rule ID 1467, a confirmation flag 1468, and an administrator name 1469. In the item of the learning data ID 1461, an ID such as a number for uniquely identifying the learning data is set. In the items of the question text 1462 and the response text 1463, the QA data to be evaluated, i.e., the question text and the response text exchanged between the chat user and the chatbot, are set. In the QA data ID 1464 field, when QA data including a question text matching the question text set in the question text 1462 exists, the ID of the existing QA data is set, and when the existing QA data does not exist, information that the matching question text has not been registered is set. In the evaluation value 1465 field, a value indicating the quality of the QA data to be evaluated is set. The evaluation value 1465 may be, for example, a binary value of a value indicating that the QA data is good (e.g., 1) and a value indicating that the QA data is bad (e.g., 0). Alternatively, the evaluation value 1465 may be a multi-value that allows the quality of the QA data to be set in three or more stages (e.g., 10 stages). Alternatively, the evaluation value 1465 may further include a value indicating that the evaluation value is not determined (e.g., a NULL value). In the cluster ID 1466 field, the cluster ID 1441 of the cluster 1440 used to generate the learning data is set. In the item of rule ID 1467, a rule ID 1451 of the rule 1450 used to generate the learning data is set. In the item of confirmation flag 1468, a state indicating whether the learning data 1460 has been confirmed or not is set, for example, a value of 1 when confirmed and a value of 0 when not confirmed. In the item of administrator name 1469, the name of the administrator of the chatbot who confirmed the learning data 1460 for maintenance of QA data, or the like is set.

[0026] The arithmetic processing unit 150 has one or more processors such as MPUs and their peripheral circuits, and is configured to read and execute the program 141 from the storage unit 140, thereby implementing various processing units by having the above hardware and the program 141 work together. The main processing units implemented by the arithmetic processing unit 150 are a chatbot 151, a chat log collection unit 152, a learning data generation unit 153, and a QA data management unit 154. Here, the chat log collection unit 152, the learning data generation unit 153, and the QA data management unit 154 configure a QA data evaluation device.

[0027] The chatbot 151 is configured to chat with a chat user. The chatbot 151 establishes a chat session with the chat user in response to a request from the chat user. In addition, when a question text is transmitted from the chat user through the established session, the chatbot 151 receives the question text, searches the QA data DB 142 for QA data including a question text that semantically matches the received question text, and acquires a response text included in the searched QA data. In addition, when the QA data including a question text that semantically matches the received question text does not exist in the QA data DB 142, the chatbot 151 generates a response text such as a predefined fixed phrase, for example, "I could not recognize your question. Please ask again in a different way." Then, the chatbot 151 transmits the acquired or generated response text to the user terminal 160 of the chat user who made the inquiry, and displays it on the terminal screen of the user terminal 160. In addition, the chatbot 151 releases the chat session established with the chat user in response to a request from the chat user.

[0028] The chat log collection unit 152 is configured to collect log information of chat between the chatbot 151 and the chat user and store it in the chat log DB 143. For example, when the chatbot 151 establishes a new chat session with the chat user, the chat log collection unit 152 secures a new entry in the chat log DB 143 and sets the chat user ID 1431, the chat ID 1432, and event data 1433 related to the establishment of the session (date and time 1431 when the session is established, type 14332 indicating the establishment of the session, text 14333 of a NULL value, and QA data ID 14334) in the entry. In addition, when the chatbot 151 receives a question text from the chat user through the session, the chat log collection unit 152 sets the event data 1433 related to the question (date and time 1431 when the question is received, type 14332 indicating the question, text 14333 indicating the question text information, and QA data ID 14334) in the secured entry in the chat log DB 143. Furthermore, when the chatbot 151 transmits a response text to the chat user through the session, the chat log collection unit 152 sets event data 1433 relating to the response (date and time 1431 when the response was transmitted, type 14332 indicating the response, text 14333 indicating the response text information, and QA data ID 14334) in the secured entry of the chat log DB 143. Furthermore, when the chatbot 151 releases the session, the chat log collection unit 152 sets event data 1433 relating to session release (date and time 1431 when the session was released, type 14332 indicating session release, text 14333 of a NULL value, and QA data ID 14334) in the secured entry of the chat log DB 143.

[0029] The learning data generating unit 153 is configured to generate learning data representing the quality of QA data by using the chat log information stored in the chat log DB 143 and the rules stored in the rule DB 145, and store the generated learning data in the learning data DB 146. The learning data generating unit 153 starts the generation process of the learning data, for example, when a certain amount of log information is accumulated in the chat log DB 143, when a certain time has passed since the previous generation of learning data, periodically, or when instructed by an operator. For example, the learning data generating unit 153 clusters the log information of a plurality of chats stored in the chat log DB 143 into the same cluster if the log information is semantically similar, and stores the generated cluster in the cluster DB 144. In addition, the learning data generating unit 153 applies the rules stored in the rule DB 145 to each cluster stored in the cluster DB 144 to calculate feature amounts from the chat log information in the cluster, perform statistical processing of the calculated feature amounts, and calculate an evaluation value based on the result of the statistical processing, thereby generating learning data, and stores the generated learning data in the learning data DB 146. Statistical processing includes creating frequency distributions, histograms, means, medians, modes, etc.

[0030] The QA data management unit 154 is configured to support the administrator of the chatbot in performing maintenance such as modifying, deleting, and adding the QA data stored in the QA data DB 142 based on the learning data stored in the learning data DB 146. For example, the QA data management unit 154 displays a list of the learning data stored in the learning data DB 146 on the screen display unit 130, thereby enabling the administrator to refer to the contents of the learning data. In addition, the QA data management unit 154 displays a list of the QA data stored in the QA data DB 142 on the screen display unit 130, allowing the administrator to interactively modify, delete, and add the QA data.

[0031] Next, the operation of the information processing device 100 will be described in detail.

[0032] The operation of the information processing device 100 is roughly divided into chatbot processing performed when an inquiry (question) from a chat user is accepted, and QA data evaluation processing. The QA data evaluation processing is roughly divided into chat log collection processing, learning data generation processing for generating learning data, and maintenance processing for maintaining QA data.

[0033] <Chatbot processing and chat log collection processing> First, chatbot processing and chat log collection processing will be described with reference to the flowchart in Fig. 7. Chatbot processing and chat log collection processing are performed by chatbot 151 and chat log collection unit 152 for each chat user and for each chat.

[0034] When the chatbot 151 of the information processing device 100 accepts an operation for starting a chat on the user terminal 160 by the chat user, the chatbot 151 performs a chat start process (step S1). In the chat start process of step S1, the chatbot 151 performs a process for establishing a session for chatting between the chatbot 151 and the user terminal 160 used by the chat user. In addition, in the chat start process of step S1, the chatbot 151 may further display a standard phrase at the start of a chat (for example, text such as "Please enter the details of your inquiry") on the screen of the user terminal 160 used by the chat user through the established session.

[0035] When a chat session is established between the chat user and the chatbot 1510, the chat log collection unit 152 performs a chat log collection process (step S2). In the chat log collection process of step S2, the chat log collection unit 152 secures one new entry in the chat log DB 143, and sets the chat user ID 1431, the chat ID 1432, and event data 1433 related to the establishment of the session (date and time 1431 when the session was established, type 14332 indicating the establishment of the session, text 14333 of a NULL value, and QA data ID 14334) in the secured entry (hereinafter, referred to as an entry of interest).

[0036] Next, the chatbot 151 checks whether there is a new question from the chat user (step S3). A new question is a new chat input by the chat user. When there is no new chat input, the chatbot 151 proceeds to the process of step S9. When there is a new chat input, the chatbot 151 acquires the input chat content (question text) (step S4). When the chatbot 151 acquires a new question from a chat user, the chat log collection unit 152 additionally sets event data 1433 consisting of date and time 1431 when the question was received, type 14332 representing the question, text 14333 representing the question text information, and QA data ID 14334 (set to a NULL value at this point) to the entry of interest in the chat log DB 143 (step S5).

[0037] Next, the chatbot 151 searches the QA data DB 142 for QA data including question text that semantically matches the question text acquired from the chat user, and generates a response text included in the searched QA data as a response to the chat user (step S6). If the QA data including question text that semantically matches the question text acquired from the chat user does not exist in the QA data DB 142 in step S6, the chatbot 151 generates a preset template as a response to the chat user. If the QA data including question text that semantically matches the question text acquired from the chat user exists in the QA data DB 142, the chat log collection unit 152 sets the ID of the existing QA data to the QA data ID 14333 of the event data 1433 added in step S5, and if the QA data does not exist, sets a message to that effect in the QA data ID 14333.

[0038] Next, the chatbot 151 transmits the generated response to the user terminal 160 used by the chat user, and displays it on the screen of the user terminal 160 (step S7). When the chatbot 151 transmits the response to the user terminal 160 of the chat user, the chat log collection unit 152 additionally sets event data 1433 consisting of the date and time 1431 when the response was transmitted, a type 14332 representing the response, text 14333 representing the response text information, and a QA data ID 14334, to the entry of interest in the chat log DB 143 (step S8). Then, the chatbot 151 proceeds to the process of step S9.

[0039] In step S9, the chatbot 151 judges whether or not the end of the chat has been detected. For example, the chatbot 151 may judge that the end of the chat has been detected when it is detected that the chat user has expressed an intention to end the chat on the user terminal 160. If the chatbot 151 judges that the end of the chat has not been detected, it returns to the process of step S3 and repeats the same process as the above-mentioned process. If the chatbot 151 detects the end of the chat, it performs a chat end process (step S10). In the chat end process of step S10, the chatbot 151 performs a process of releasing (disconnecting) the session established with the chat user. In the chat end process of step S10, the chatbot 151 may further display a standard phrase at the time of chat end (for example, text such as "Thank you for using our service") on the screen of the user terminal 160 used by the chat user through the session before the release.

[0040] When the chatbot 151 releases a chat session, the chat log collection unit 152 sets event data 1433 related to the session release (date and time 1431 when the session was released, type 14332 indicating the session release, NULL value text 14333, and QA data ID 14334) in the entry of interest in the chat log DB 143 (step S11).

[0041] <Learning data generation process> Next, the learning data generation process will be described with reference to the flowchart in Fig. 8. The learning data generation process is performed by the learning data generation unit 153.

[0042] When the learning data generating unit 153 of the information processing device 100 starts the process of generating learning data, it first reads chat log information to be used for generating learning data from the chat log DB 143 (step S21). For example, the learning data generating unit 153 may read all log information stored in the chat log DB 143 as log information to be used for generating learning data. Alternatively, the learning data generating unit 153 may refer to the date and time set in the date and time 14331 and read, for example, all log information after a predetermined date and time specified by an administrator or the like, all log information before the predetermined date and time, or all log information after the predetermined start date and time and before the predetermined end date and time from the chat log DB 143 as log information to be used for generating learning data.

[0043] Next, the learning data generating unit 153 clusters the read log information into the same cluster if the log information is semantically similar (step S22). Being semantically similar means that the contents of the question text and the response text exchanged between the chat log information are overall and semantically similar. For example, "I want to cancel my vacation request" and "I want to withdraw my vacation" are examples of chat log information that are semantically similar to each other. In addition, "expensive" and "expensive" and "looks great" and "looks great" are other examples of chat log information that are semantically similar to each other. The method of clustering semantically similar chat log information into the same cluster is arbitrary. For example, the question text and the response text in each chat log information are collected into one document, and the above clustering may be performed by applying a known document clustering method to this document group, which classifies similar documents into the same cluster.

[0044] A known document clustering method is, for example, the document clustering method described in Patent Document 4, but is not limited thereto. In the document clustering method described in Patent Document 4 (hereinafter, referred to as the document clustering method related to the present invention), first, for any combination of two terms, that is, a term appearing in one document and a term appearing in the other document, among terms appearing in two documents included in a document group, a concept tree structure is obtained that represents a hierarchical relationship between the concepts of the two terms. Next, for the any combination, a concept similarity is obtained that is an index indicating the closeness of the concepts of the two terms, which is maximum when the occurrence frequency in the document group of a common hypernym of the two terms or a term in a subordinate series of the hypernym in the obtained concept tree structure matches the occurrence frequency in the document group of each of the two terms, and is minimum when there is no common hypernym of the two terms in the concept tree structure. Next, an inter-document similarity, which is the degree of semantic similarity of the two documents included in the document group, is obtained based on the concept similarity. Next, the documents in the document group are clustered based on the inter-document similarity.

[0045] For example, consider that the learning data generating unit 153 clusters a log information group including two chat log information LU11 and LU21 shown in Fig. 9 using a document clustering method related to the present invention. In Fig. 9, the chat log information LU11 on the left side indicates the log information of the chat between the chat user U01 and the chatbot 151, and the log information LU21 on the right side indicates the log information of the chat between the chat user U02 and the chatbot 151. In Fig. 9, the two-way arrow indicates an event of establishing or releasing a chat session, and the speech bubble indicates an event of a response comment sent from the chatbot 151 to a chat user, or a question comment received by the chatbot 151 from a chat user. In addition, the date and time written under each event indicates the date and time of the event occurrence. In order to identify each event, each event is conveniently assigned with the symbols LU111 to LU117 and LU211 to LU217. In the case of such chat log information, the learning data generation unit 153 collects the question texts and response texts in the log information LU11 shown in Fig. 9 to generate one document LU11B as shown in Fig. 10. Note that in the example of Fig. 10, fixed phrases common to all chats, such as "Please enter your inquiry" and "Thank you for using our service", which are presented to the chat user by the chatbot 151 at the start and end of a chat, are excluded. In addition, the learning data generation unit 153 collects the question texts and response texts in the log information LU21 to generate one document LU21B as shown in Fig. 10. Then, the learning data generation unit 153 performs clustering on a document group including documents LU11B and LU21B by applying a document clustering method related to the present invention. As a result, in the case of the two pieces of log information LU11 and LU21 shown in Figure 9, even though the question text of event LU113 "I would like to cancel my vacation request" and the question text of event LU213 "I would like to withdraw my vacation" which are semantically the same but differ at the phrase level, exist in different log information, the two pieces of log information LU11 and LU21 will be clustered into the same cluster.

[0046] In step S22, the learning data generation unit 153 generates a cluster 1440 consisting of a cluster ID 1441, a question label 1442, the number of chat logs 1443, and a list 1434 of chat log IDs for each of the clusters generated by the clustering, and stores the cluster 1440 in the cluster DB 144. For example, the learning data generation unit 153 sets the question text "How to cancel vacation", which appears in common in the log information of multiple chats, to the question label 1442 of the cluster to which the two pieces of log information shown in Fig. 9 belong.

[0047] Next, the learning data generation unit 153 focuses on one cluster 1440 among the one or more clusters stored in the cluster DB 144 (step S23). Next, the learning data generation unit 153 focuses on one rule 1450 among the one or more rules stored in the rule DB 145 (step S24). Next, the learning data generation unit 153 creates learning data 1460 based on the focused cluster 1440 and the focused rule 1450, and stores the data in the learning data DB 146 (step S25).

[0048] FIG. 11 is a flowchart showing an example of the process executed by the learning data generating unit 153 in step S25. Referring to FIG. 11, the learning data generating unit 153 first calculates the feature quantity of the type set in the feature quantity type 1452 item of the rule 1450 under focus from each piece of chat log information 1430 of the cluster 1440 under focus (step S31). For example, in the case of the rule 1450-1, the learning data generating unit 153 calculates "the time T1 from when the chat user is presented with a response text to the last question to when the chat ends" from each piece of chat log information. For example, in the case of the log information LU11 shown in FIG. 9, since the event LU116 is a response to the chat user's last question, the time from the date and time of the event LU116 to when the chat of the event LU117 ends is calculated as the time T1. Also, for example, in the case of the rule 1450-2, the learning data generating unit 153 calculates "the frequency N1 of asking the next question before a predetermined time has elapsed since the previous question" from each piece of chat log information. 9, there are two questions, for events LU113 and LU115, so if the time that has elapsed from event LU113 to event LU115 is less than a predetermined time, the frequency N1 will be 1, and if it is equal to or greater than the predetermined time, the frequency N1 will be 0. Incidentally, in the case of chat log information with a total of M questions, the maximum value of frequency N1 will be M-1.

[0049] Next, the learning data generating unit 153 performs statistical processing of the feature amount calculated from each of the chat log information based on the evaluation value calculation standard 1454 of the rule 1450 in focus (step S32). For example, in the case of the rule 1450-1, the learning data generating unit 153 first calculates the total number S1 of chat log information whose time T1 is less than a predetermined time TH1. Next, the learning data generating unit 153 calculates the ratio R1 of the total number S1 to the total number S0 of chat logs in the cluster in focus. In addition, in the case of the rule 1450-2, the learning data generating unit 153 first calculates the total number S1 of chat log information whose frequency N1 is equal to or greater than a predetermined frequency TH2. Next, the learning data generating unit 153 calculates the ratio R1 of the total number S1 to the total number S0 of chat logs in the cluster in focus.

[0050] Next, the learning data generating unit 153 calculates an evaluation value from the result of the statistical processing (step S33). For example, in the case of the rules 1450-1 and 1450-2, the higher the ratio R1, the lower the evaluation value is set by the learning data generating unit 153. For example, if the ratio R1 is 80% or more, the evaluation value is 0; if it is 60% or more and less than 80%, the evaluation value is 2; if it is 40% or more and less than 60%, the evaluation value is 5; if it is 20% or more and less than 40%, the evaluation value is 8; and if it is less than 20%, the evaluation value is 10. Here, the evaluation value indicates a higher evaluation as the numerical value increases.

[0051] Next, the learning data generation unit 153 creates learning data 1460 by setting necessary information in the items of learning data ID 1461, question text 1466, response text 1463, QA data ID 1464, evaluation value 1464, cluster ID 1466, and rule ID 1467, setting a value indicating an unconfirmed state in confirmation flag 1468, and setting a NULL value in administrator name 1469, and stores the created learning data in the learning data DB 146. The learning data generation unit 153 sets cluster ID 1441 of the cluster 1440 in focus and rule ID 1451 of the rule 1450 in focus in the items of cluster ID 1466 and rule ID 1467. The learning data generation unit 153 also sets the evaluation value calculated in step S33 in the item of evaluation value 1465. In addition, the learning data generation unit 153 sets the question text, response text, and QA data ID 1421 of the QA data containing them specified by the learning target QA data 1453 of rule 1450 identified by rule ID 1467 in the fields of question text 1462, response text 1463, and QA data ID 1464.

[0052] Referring again to FIG. 8, when the learning data generation unit 153 finishes the process of step S25, it moves its attention to one of the rules stored in the rule DB 145 that have not yet been applied to the target cluster (step S28), returns to step S25 via step S27, and repeats the same process as described above using another rule for the target cluster. Further, when the learning data generation unit 153 finishes applying all the rules to the target cluster (YES in step S27), it moves its attention to one of the clusters stored in the cluster DB 144 that have not yet been processed (step S28), returns to step S24 via step S29, and repeats the same process as described above for another cluster. Further, when the learning data generation unit 153 finishes paying attention to all the clusters (YES in step S29), it ends the process of FIG. 8.

[0053] <Maintenance Process of QA Data> Next, the data maintenance process will be described. The data maintenance process is performed by the QA data management unit 154.

[0054] FIG. 12 shows an example of a chatbot management screen 170 displayed on the screen display unit 130 when the QA data management unit 154 is activated by the administrator of the information processing apparatus 100. The chatbot management screen 170 in this example has a learning data list display area 171, a QA data editing area 172, a cluster display area 173, a rule display area 174, and a chat log display area 175.

[0055] The learning data list display area 171 is an area for displaying a list of one or more learning data 1460 stored in the learning data DB 146. The QA data management unit 154 may read out all the learning data 1460 stored in the learning data DB 146 and display them in the learning data list display area 171. Alternatively, the QA data management unit 154 may selectively read out some of the learning data 1460 from all the learning data stored in the learning data DB 146 and display them in the learning data list display area 171. As the part of the learning data, the confirmation flag 1468 may be learning data indicating an unconfirmed state. Alternatively, the part of the learning data may be learning data whose evaluation value 1465 is higher or lower than the evaluation value designated by the administrator. The QA data management unit 154 sets one of the learning data displayed in the learning data list display area 171 as the current learning data. The QA data management unit 154 clearly indicates the current learning data to the administrator by highlighting the current learning data, for example. Furthermore, the QA data management unit 154 sets "confirmed" in the confirmation flag 1468 field of the current learning data, and sets the name of the administrator logged in to the management screen in the administrator name 1469 field. When a change is instructed by the administrator's cursor operation, the QA data management unit 154 switches the current learning data to another learning data as instructed.

[0056] The QA data editing area 172 is an area for performing editing such as updating, deleting, and adding QA data. The QA data editing area 172 has a QA data ID field 1721, a question text field 1722, a response text field 1723, an update button 1724, a delete button 1725, and an add button 1726. The QA data management unit 154 displays the QA data ID 1464, question text 1462, and response text 1463 of the current learning data in the QA data ID field 1721, the question text field 1722, and the response text field 1723. The QA data management unit 154 also edits the contents of the question text field 1722 and the response text field 1723 in response to an editing operation of the operation input unit 120 by the administrator. Furthermore, when the administrator presses an Update button 1724, the QA data management unit 154 updates (overwrites) the QA data in the QA data DB 142 identified by the QA data ID set in the QA data ID field 1721 with the contents of the question text and response text set in the edited question text field 1722 and response text field 1723. Furthermore, when the administrator presses a Delete button 1725, the QA data management unit 154 deletes the QA data in the QA data DB 142 identified by the QA data ID set in the QA data ID field 1721. Furthermore, when the administrator presses an Add button 1726, the QA data management unit 154 creates QA data having a new QA data ID and the contents of the question text and response text set in the edited question text field 1722 and response text field 1723, and adds the QA data to the QA data DB 142 as new QA data.

[0057] The cluster display area 173 displays the contents of the cluster 1440, that is, the cluster ID 1441, the question label 1442, the number of chat logs 1443, and the list 1434 of chat log IDs. The QA data management unit 154 reads out the contents of the cluster 1440 having the cluster ID 1441 that matches the cluster ID 1466 of the current learning data from the cluster DB 144, and displays it in the cluster display area 173. The QA data management unit 154 sets one of the chat log IDs in the list 1434 of chat log IDs displayed in the cluster display area 173 as the current chat log ID. The QA data management unit 154 clearly displays the current chat log ID to the administrator by highlighting it, for example. The QA data management unit 154 switches the current chat log ID to a specified chat log ID in the list 1434 of chat log IDs in response to an instruction to change the current chat log ID by the administrator's cursor operation.

[0058] The chat log display area 175 is an area for displaying chat log information. The QA data management unit 154 reads out chat log information having a chat log ID that matches the current chat log ID from the chat log DB 143 and displays it in the chat log display area 175.

[0059] The rule display area 174 is an area for displaying the contents of the rule 1450, that is, the rule ID 1451, the feature type 1452, the learning target QA data 1453, and the evaluation value calculation criterion 1454. The QA data management unit 154 reads out the rule 1450 having the rule ID 1451 that matches the rule ID 1467 of the current learning data from the rule DB 145, and displays it in the rule display area 174.

[0060] Since the QA data management unit 154 performs the above-mentioned process using the chatbot management screen 170 shown in FIG. 12, the administrator of the information processing device 100 can interactively modify, delete, and add the QA data that is the learning target while individually referring to the learning data 1460 stored in the learning data DB 146. In addition, the QA data management unit 154 displays the contents of the cluster 1440 used to create the learning data 1460 in the cluster display area 173 and displays details of the chat log information constituting the cluster 1440 in the chat log display area 175, so that the administrator can modify, delete, and add the QA data while checking what kind of cluster 1440 and what collection of chat log information the learning data 1460 was generated from. In addition, the QA data management unit 154 displays the contents of the rule 1450 used to create the learning data 1460 in the rule display area 174, so that the administrator can modify, delete, and add the QA data while checking what kind of rule 1450 the learning data 1460 was generated using.

[0061] As described above, the information processing device 100 according to this embodiment can easily acquire passive evaluation information. The reason is that the information processing device 100 can collect chat log information, calculate feature amounts from the collected chat log information, and calculate evaluation values ​​based on the calculated feature amounts, all on the information processing device 100 side, and it is not necessary for the chat user side to be equipped with special devices such as a microphone, a camera, or a biosensor.

[0062] Furthermore, according to the information processing device 100 of this embodiment, multiple pieces of log information that are semantically similar are clustered into the same cluster, predetermined features are extracted from each of the multiple pieces of log information that belong to the same cluster, and learning data is created that indicates the quality of Q&A data related to question texts commonly included in the log information in the cluster based on the results of statistical processing of the multiple extracted features. This makes it possible to reduce the variation in evaluation due to the behavior of a specific chat user.

[0063] Furthermore, according to the information processing device 100 of this embodiment, the features used are "the time from the presentation of a response to the last question to the end of the chat" (rule 1450-1) and "the frequency with which the next question was asked before a predetermined time had elapsed since the previous question" (rule 1450-2), making it possible to create learning data that reflects the opinions of the silent majority.

[0064] [Second embodiment] Next, a QA data evaluation device according to a second embodiment of the present invention will be described with reference to the drawings. Fig. 13 is a block diagram of a QA data evaluation device 200 according to this embodiment.

[0065] With reference to FIG. 13, the QA data evaluation device 200 includes an acquisition unit 201 , an extraction unit 202 , and a generation unit 203 .

[0066] The acquiring means 201 is configured to acquire QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user.

[0067] The extraction means 202 is configured to extract, from the log information, features related to the temporal behavior of the user's use of the chatbot.

[0068] The generating means 203 is configured to generate QA data evaluation information indicating the quality of the QA data based on the feature amount.

[0069] The QA data evaluation device 200 configured in this manner operates as follows. That is, first, the acquisition means 201 acquires QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user. Next, the extraction means 202 extracts, from the log information, features related to the temporal behavior of the use of the chatbot by the user. Next, the generation means 203 generates QA data evaluation information indicating the quality of the QA data based on the features.

[0070] According to the QA data evaluation device 200 configured and operating as described above, it is possible to easily obtain non-active evaluation information. The reason is that the QA data evaluation device 200 can collect chat log information, calculate features from the collected chat log information, and calculate evaluation values ​​based on the calculated features, all of which are performed on the QA data evaluation device 200 side, and it is not necessary for the chat user side to have special devices such as a microphone, a camera, or a living body detection sensor.

[0071] Although the present invention has been described with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. For example, the following modifications are also included in the present invention.

[0072] In the above-described embodiment, the feature amount of the chat user's behavior over time during a chat is calculated from the chat log information, and learning data representing the quality of the QA data is created based on the calculated feature amount. However, in addition to the feature amount of the chat user's behavior over time during a chat calculated from the chat log information, learning data may be created taking other information into consideration. Examples of other information include active evaluation information, the chat user's voice, image, and biological information (pulse, heart rate, blood pressure, brain waves, respiratory rate, etc.), URL selection, usage date and time, and user terminal information (PC, smartphone, etc.).

[0073] The active evaluation information is created based on information on the reaction of the chat user who received the response during the operation of the chatbot. The active evaluation information is information that the chat user actively takes the trouble to input in order to evaluate the presented response. Examples of active evaluation information include utterances, text, emoticons, stamps, etc., such as "Like," "Amazing," and "Clever," which indicate a good evaluation, and "No Good," "No Good," which indicate a bad evaluation. In addition, the active evaluation information is input, for example, by a social button indicating "Like" or "Bad." However, since active evaluation information is not always obtained, the active evaluation information required for generating learning data may be insufficient. It is said that active evaluation information is obtained for about 10% of all questions. Therefore, it is important to create learning data by measuring the satisfaction level and evaluation of the chat user for the presented response using information other than active evaluation information, i.e., non-active evaluation information. According to the present invention, such non-active evaluation information can be easily created. [Industrial Applicability]

[0074] The present invention can be applied to the operation and management of chatbots, and can be used, for example, for maintaining QA data.

[0075] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. [Appendix 1] An acquisition means for acquiring QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user; An extraction means for extracting features related to a temporal behavior of the user's use of the chatbot from the log information; A generating means for generating QA data evaluation information indicating the quality of the QA data based on the feature amount; A QA data evaluation device comprising: [Appendix 2] The method further comprises clustering means for clustering a plurality of pieces of log information into a plurality of groups according to semantic similarity of the pieces of log information, The extraction means extracts the feature amount from each of a plurality of pieces of log information belonging to each of the plurality of clusters; the generating means generates the QA data evaluation information based on a result of statistical processing of a plurality of feature amounts extracted from each of the plurality of pieces of log information. 2. The QA data evaluation device according to claim 1. [Appendix 3] the feature amount is a feature amount related to a time from outputting a response to a last question from the user to an end of the chat; 3. The QA data evaluation device according to claim 1 or 2. [Appendix 4] the feature quantity is a feature quantity related to a frequency of inputting the content of another question within a predetermined time period from the time point when the content of a question is inputted from the user to the chatbot, 3. The QA data evaluation device according to claim 1 or 2. [Appendix 5] The system further includes a QA data management unit for displaying the generated QA data evaluation information. 5. A QA data evaluation device according to any one of appendices 1 to 4. [Appendix 6] The QA data management means updates, deletes, or adds the QA data in response to an operation input by an administrator of the chatbot for the QA data. 6. The QA data evaluation device according to claim 5. [Appendix 7] the QA data management means displays the log information used in generating the QA data evaluation information. 7. The QA data evaluation device according to claim 5 or 6. [Appendix 8] the QA data management means displays a rule including the type of the feature used to create the QA data evaluation information from the log information and a calculation criterion for an evaluation value representing the quality of the QA data. 8. A QA data evaluation device according to any one of appendices 5 to 7. [Appendix 9] Acquire QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user; Extracting features related to the temporal behavior of the user's use of the chatbot from the log information; generating QA data evaluation information indicating the quality of the QA data based on the feature amount; QA data evaluation methods. [Appendix 10] On the computer, A process of acquiring QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user; A process of extracting features related to a temporal behavior of the user's use of the chatbot from the log information; A process of generating QA data evaluation information indicating the quality of the QA data based on the feature amount; A computer-readable recording medium having a program recorded thereon for causing a computer to carry out the above. [Explanation of symbols]

[0076] 100 Information processing device 110 Communication I / F section 120 Operation input section 130 Screen display section 140 Storage section 141 Programs 142 QA Data DB 143 Chat log DB 144 Cluster DB 145 Rules DB 146 Learning Data DB 150 Processing section 151 Chatbots 152 Chat Log Collection Department 153 Learning Data Generation Unit 154 QA Data Management Department

Claims

1. An acquisition means for acquiring QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user; An extraction means for extracting features related to a temporal behavior of the user's use of the chatbot from the log information; A generating means for generating QA data evaluation information indicating the quality of the QA data based on the feature amount; A QA data evaluation device comprising:

2. The method further comprises clustering means for clustering a plurality of pieces of log information into a plurality of groups according to semantic similarity of the pieces of log information, The extraction means extracts the feature amount from each of a plurality of pieces of log information belonging to each of the plurality of groups; The generating means generates the QA data evaluation information based on a result of statistical processing of a plurality of feature amounts extracted from each of the plurality of pieces of log information. The QA data evaluation device according to claim 1 .

3. the feature amount is a feature amount related to a time from outputting a response to a last question from the user to an end of the chat; 3. The QA data evaluation device according to claim 1 or 2.

4. the feature quantity is a feature quantity related to a frequency of inputting the content of another question within a predetermined time period from the time point when the content of a question is inputted from the user to the chatbot, The QA data evaluation device according to claim 1 or 2.

5. The system further includes a QA data management unit for displaying the generated QA data evaluation information.

5. The QA data evaluation device according to claim 1.

6. The QA data management means updates, deletes, or adds the QA data in response to an operation input by an administrator of the chatbot for the QA data. The QA data evaluation device according to claim 5.

7. The QA data management means displays the log information used in generating the QA data evaluation information.

7. The QA data evaluation device according to claim 5 or 6.

8. the QA data management means displays a rule including the type of the feature used to create the QA data evaluation information from the log information and a calculation criterion for an evaluation value representing the quality of the QA data.

8. The QA data evaluation device according to claim 5.

9. A method for evaluating QA data by a computer, comprising: The computer acquires QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user; The computer extracts features related to the temporal behavior of the user's use of the chatbot from the log information; The computer generates QA data evaluation information indicating the quality of the QA data based on the feature amount. QA data evaluation method.

10. On the computer, A process of acquiring QA data including the content of a question from a user to a chatbot and the content of a response from the chatbot to the question, and log information regarding the use of the chatbot by the user; A process of extracting features related to a temporal behavior of the user's use of the chatbot from the log information; A process of generating QA data evaluation information indicating the quality of the QA data based on the feature amount; A program to carry out the above.

Images