Information processing system, information processing method, and information processing apparatus

Abstract

The present application has an object to make it easy to prepare a set of learning data for ensuring the performance of a mechanical learning model. The present application relates to an information processing system, an information processing method, and an information processing apparatus. The information processing system acquires a set of learning data including input data and a label as teaching data for the input data, causes a mechanical learning model to learn through the set of learning data, inputs test data to the mechanical learning model that has learned through the set of learning data, and evaluates whether the performance of the mechanical learning model satisfies a specified condition based on the output of the mechanical learning model to which the test data has been input. In a case where it is evaluated that the performance of the mechanical learning model does not satisfy the specified condition, the set of learning data is updated, and the mechanical learning model is re-learned through the updated set of learning data. The information processing system repeats the update of the data set, the re-learning, and the evaluation in accordance with the evaluation.

Description

Technical Field

[0001] This invention relates to an information processing system, an information processing method, and an information processing device. Background Technology

[0002] In recent years, people have been developing systems that automate the response to queries, such as chatbots. These systems, upon receiving a query, determine which of a set of pre-defined tags the query corresponds to, and then output the answer corresponding to the determined tag. In recent years, machine learning models have been widely used in this type of NLU (Natural Language Understanding) processing, which determines the corresponding tag based on the query text.

[0003] Japanese Patent Application Publication No. 2004-5648 discloses a method that supports annotating training data used to train a natural language understanding system. Summary of the Invention

[0004] [The problem the invention aims to solve]

[0005] It is known that the performance of a machine learning model after learning varies depending on the composition of the set of learning data used by the model. Therefore, when creating the set of learning data, the administrator must investigate and rectify any problems with the set. For the administrator, parsing the set of learning data becomes a significant burden.

[0006] The present invention was made in view of the aforementioned problems, and its object is to provide a technique to facilitate the preparation of a set of learning data used to ensure the performance of a machine learning model.

[0007] [Technical means to solve the problem]

[0008] To address the aforementioned problem, the information processing system of the present invention includes: a learning server that enables a machine learning model to learn using a set of learning data comprising input data and labels serving as teaching data for the input data; and a response server that inputs user-inputted data into the learned machine learning model and outputs response data based on labels output by the machine learning model; and further includes: an initial data acquisition mechanism that acquires the set of learning data; a learning mechanism that enables the machine learning model to learn using the set of learning data; and an evaluation mechanism that inputs test data into the machine learning model learned using the set of learning data and, based on labels output by the machine learning model, outputs response data. The system comprises: an output mechanism for a machine learning model that has been fed with the test data; an unfolding mechanism that, if the performance of the machine learning model is deemed to meet the specified conditions, unfolds the learned machine learning model onto the response server; a data updating mechanism that, if the performance of the machine learning model is deemed not to meet the specified conditions, updates the set of learning data; a relearning mechanism that, using the updated set of learning data, enables the machine learning model to relearn; and repeating the processes of the data updating mechanism, the relearning mechanism, and the evaluation mechanism based on the evaluation of the evaluation mechanism.

[0009] Furthermore, the information processing method of the present invention includes the following steps: obtaining a set of learning data containing input data and tags, wherein the tags are teaching data for the input data used to generate response data; enabling a machine learning model to learn using the set of learning data; inputting test data into the machine learning model after learning using the set of learning data, and evaluating whether the performance of the machine learning model meets specified conditions based on the output of the machine learning model with the input test data; if the performance of the machine learning model meets the specified conditions, expanding the machine learning model with evaluated performance to a response server, the response server inputting user-inputted input data into the learned machine learning model, and outputting response data based on the tags output by the machine learning model; if the performance of the machine learning model does not meet the specified conditions, updating the set of learning data; and enabling the machine learning model to relearn using the updated set of learning data; and repeating the steps of updating the set of learning data, enabling the machine learning model to relearn, and evaluating the performance of the machine learning model according to the evaluation.

[0010] Furthermore, the information processing apparatus of the present invention includes: an initial data acquisition mechanism for acquiring a set of learning data including input data and tags, wherein the tags are teaching data for the input data used to generate response data; a learning mechanism for enabling a machine learning model to learn using the set of learning data; an evaluation mechanism for inputting test data into the machine learning model learned using the set of learning data and evaluating whether the performance of the machine learning model meets specified conditions based on the output of the machine learning model with the input test data; an expansion mechanism for expanding the machine learning model with evaluated performance to a response server when the performance of the machine learning model meets the specified conditions, wherein the response server inputs user-inputted input data into the learned machine learning model and outputs response data based on the tags output by the machine learning model; a data update mechanism for updating the set of learning data when the performance of the machine learning model does not meet the specified conditions; and a relearning mechanism for enabling the machine learning model to relearn using the updated set of learning data; and repeating the processing of the data update mechanism, the relearning mechanism, and the evaluation mechanism according to the evaluation of the evaluation mechanism.

[0011] In one embodiment of the invention, the information processing system may further include a detection mechanism that, when the performance of the machine learning model is deemed not to meet the specified conditions, determines whether the set of learning data meets the detection conditions; if the detection conditions are met, the data updating mechanism updates the set of learning data.

[0012] In one embodiment of the invention, the detection mechanism may determine whether the number of input data for each label in the set of learning data meets the detection conditions, and if the detection conditions are met, the data update mechanism updates the set of learning data.

[0013] In one embodiment of the invention, if the performance of the machine learning model is assessed as not meeting the specified conditions, the data update mechanism updates the set of learning data based on improvement parameters, and the information processing system further includes a parameter update mechanism that updates the improvement parameters according to the update of the set of learning data.

[0014] In one embodiment of the invention, the information processing system may further include a logging mechanism that stores input data in a log storage device when input data is input by the user; and the detection mechanism determines whether there are labels with insufficient input data in the set of learning data. If it is determined that there are labels with insufficient input data, the data update mechanism extracts the input data corresponding to the label from the input data stored in the log storage device and adds learning data containing the extracted input data and the label to the set of learning data.

[0015] In one embodiment of the invention, if it is determined that there are insufficient labels for the input data, the data update mechanism extracts the input data corresponding to the label from the input data stored in the log storage based on the input data corresponding to the label in the set of learning data and the input data stored in the log storage.

[0016] In one embodiment of the invention, the information processing system may further include: a log mechanism that stores input data in a log storage device when input data is input by the user; and a test data appending mechanism that extracts input data corresponding to any tag from the input data stored in the log storage device and appends the extracted input data and the tag set to the learning data.

[0017] [The effects of the invention]

[0018] This invention makes it easy to prepare the set of learning data used to ensure the performance of machine learning models. Attached Figure Description

[0019] Figure 1 This is a diagram illustrating an example of an information processing system according to an embodiment of the present invention.

[0020] Figure 2 It is a block diagram representing the functions implemented by an information processing system.

[0021] Figure 3 This is a graph representing an example of a learning dataset.

[0022] Figure 4 This is a block diagram representing the functional structure of the question and answer section.

[0023] Figure 5 It is a flowchart that roughly represents the processes related to learning a machine learning model in an information processing system.

[0024] Figure 6 This diagram illustrates the processing procedures of the Problem Detection Department and the Data Change Department.

[0025] Figure 7 This is a flowchart illustrating an example of processing related to data parsing.

[0026] Figure 8 This is a graph illustrating the adjustment of the number of samples in the learning dataset.

[0027] Figure 9 This is a flowchart illustrating an example of the inquiry and response process.

[0028] Figure 10 This is a diagram representing an example of a query log.

[0029] Figure 11 This is a flowchart illustrating an example of data appending processing.

[0030] Figure 12 This diagram illustrates the relationship between similarity and the extraction of user queries. Detailed Implementation

[0031] Hereinafter, embodiments of the present invention will be described based on the accompanying drawings. For components marked with the same symbols, repeated descriptions will be omitted. In this embodiment, an information processing system will be described that handles user queries like a chatbot, determines which of a plurality of pre-set tags the query corresponds to, and then outputs a response corresponding to the determined tag.

[0032] The following explanation addresses the case where queries are input in text format, but queries can also be input in voice format. Furthermore, this information processing system utilizes a machine learning model for Natural Language Understanding (NLU). The system trains the machine learning model using a set of training data, and the trained model is then used to analyze user queries.

[0033] Figure 1 This diagram illustrates an example of an information processing system according to an embodiment of the present invention. The information processing system includes a learning management server 1 and a query response server 2. The learning management server 1 and the query response server 2 are connected via a network, and the query response server 2 is connected via the network to multiple user terminals 3. The user terminals 3 are users of the services provided by the information processing system, operated by users who wish to make inquiries. User terminals 3 may be, for example, smartphones or personal computers.

[0034] Learning management server 1 includes a processor 11, a storage unit 12, a communication unit 13, and an input / output unit 14. Furthermore, learning management server 1 is a server computer. Although not shown here, query response server 2 is also a server computer, and like learning management server 1, it includes a processor 11, a storage unit 12, a communication unit 13, and an input / output unit 14. The functions of learning management server 1 and query response server 2, which will be described below, can also be implemented using multiple server computers.

[0035] The processor 11 operates according to the program stored in the storage unit 12. Additionally, the processor 11 controls the communication unit 13 and the input / output unit 14. Furthermore, the program may be provided via the Internet or the like, or it may be provided in a storage medium that can be read by a computer, such as a flash memory or a DVD (Digital Versatile Disc)-ROM (Read Only Memory).

[0036] Storage unit 12 includes memory components such as RAM (Random Access Memory) and flash memory, as well as external storage devices such as hard disk drives. Storage unit 12 stores the program. In addition, storage unit 12 stores information input from processor 11, communication unit 13, and input / output unit 14, as well as calculation results.

[0037] The communication unit 13 is a functional unit that enables communication with other devices, such as including integrated circuits that implement wireless LAN (Local Area Network) and wired LAN. Based on the control of the processor 11, the communication unit 13 inputs information received from other devices to the processor 11 or the storage unit 12, and sends information to other devices.

[0038] The input / output unit 14 includes a video controller for controlling the display output element and a controller for acquiring data from the input element. Input elements include a keyboard, mouse, touch panel, etc. Based on the control of the processor 11, the input / output unit 14 outputs display data to the display output element and acquires data input by the user through operation of the input element. The display output element is, for example, an external display device.

[0039] Secondly, the functions provided by the information processing system will be explained. Figure 2This is a block diagram illustrating the functions implemented by the information processing system. Functionally, the information processing system includes an initial data determination unit 51, a learning unit 52, a performance evaluation unit 53, a problem detection unit 54, a data modification unit 55, a model expansion unit 56, a learning control unit 57, and a query response unit 58. The functions of the initial data determination unit 51, learning unit 52, performance evaluation unit 53, problem detection unit 54, data modification unit 55, model expansion unit 56, and learning control unit 57 are implemented by the processor 11 included in the learning management server 1 executing the program stored in the storage unit 12 and controlling the communication unit 13, etc. The query response unit 58 is implemented by querying the processor 11 included in the response server 2 to execute the program stored in the storage unit 12 and controlling the communication unit 13, etc.

[0040] In addition, the information processing system also includes a learning dataset 61 and a query log 62 as data. These can be primarily stored in the storage unit 12, or they can be stored in a database or storage device implemented by other servers.

[0041] The initial data determination unit 51 acquires an initial learning dataset 61. The learning dataset 61 contains multiple learning datasets. Each learning dataset contains query data and labels that serve as teaching data for that query data.

[0042] Figure 3 This is a graph representing an example of the learning dataset 61. The query data is, for example, as follows: Figure 3 The query text shown represents data indicating a query input in natural language. The query data can also be the result of parsing the query text using methods such as lexical analysis. During learning, the learning unit 52 can also use lexical analysis to convert the query data consisting of the query text into query data consisting of word groups, and then input the converted query data into the machine learning model. The labels stored in the learning data are any one of a number of pre-defined labels. A label is information representing the intent of the user who inputs the query; hereinafter, it is also referred to as "intent". There are multiple sets of learning data containing a certain label; for ease of explanation, query data (query text) containing learning data with a certain label is referred to as query data (query text) belonging to that label.

[0043] The learning unit 52 enables the machine learning model to learn using the learning dataset 61. Furthermore, after updating the learning data, the learning unit 52 enables the machine learning model to learn again using the updated learning dataset 61.

[0044] The machine learning model is configured to output one of multiple labels after receiving input query data. In this embodiment, the machine learning model can be, for example, a machine learning model constructed using so-called deep learning such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), or BERT (Bidirectional Encoder Representations from Transformers), to construct a machine learning model that takes input words segmented by lexical analysis. Alternatively, a machine learning model can be constructed that takes input vectors composed of feature words extracted from the lexical analysis words and is equipped with machine learning methods such as random forests or support vector machines (SVMs). Furthermore, the machine learning model can also be provided by an external system, where the processing details are unknown.

[0045] The performance evaluation unit 53 inputs test data into the machine learning model learned through the learning dataset 61, and evaluates whether the performance of the machine learning model meets specified conditions based on the output of the machine learning model after receiving the test data. The test data contains multiple records, each containing query data and a label as the answer to the query data. For example, the performance evaluation unit 53 can calculate the accuracy rate for each label individually and evaluate whether the performance meets the specified conditions based on whether there are labels with an accuracy rate lower than a specified threshold.

[0046] If the performance is deemed not to meet the specified conditions, the problem detection unit 54 determines whether the learning dataset 61 meets the pre-set problem detection conditions.

[0047] If the detection criteria for the problem are met, the data modification department 55 updates the learning dataset 61. The specific methods for problem detection and updating will be described below.

[0048] If the performance is determined to meet the specified conditions, the model unfolding unit 56 unfolds the learned machine learning model onto the query response server 2, which answers actual queries from users. The unfolding of the machine learning model can be performed by copying the parameters of the learned machine learning model to the query response server 2, or by copying a virtual environment containing the learned machine learning model to the query response server 2. Alternatively, it can be performed by switching the input address of the query data to input the query data from the actual user query into the learned machine learning model built in the cloud.

[0049] The learning control unit 57 controls the initial data determination unit 51, the learning unit 52, the performance evaluation unit 53, the problem detection unit 54, and the data modification unit 55, and controls the preparation of the learning dataset 61 and the learning of the machine learning model. In addition, if the performance is determined by the performance evaluation unit 53 to meet the specified conditions, the model expansion unit 56 is instructed to expand the learned machine learning model.

[0050] The query response unit 58 receives the query input by the user from the user terminal 3 and then outputs the answer to the query. Additionally, it stores information representing the input query in the query log 62.

[0051] Figure 4 This is a block diagram illustrating the functional configuration of the query-response unit 58. Functionally, the query-response unit 58 includes a natural language processing unit 71, a dialogue management unit 72, and a response generation unit 73. These functions are achieved by querying the processor 11 included in the response server 2 to execute programs stored in the storage unit 12, and controlling the communication unit 13, etc.

[0052] The Natural Language Processing (NLP) unit 71 performs so-called Natural Language Understanding (NLU) functions. The NLP unit 71 performs lexical analysis and contains a machine learning model that generates query data from the query text through lexical analysis as input, and outputs labels. The NLP unit 71 can also send query text or query data to a natural language understanding function implemented through another server and obtain its results. Furthermore, the query response unit 58 may also include an ASR (Automatic Speech Recognition) / STT (Speech to Text) function that parses query speech input by the user's voice, and its output is input to the NLP unit 71.

[0053] Based on the tags output from the natural language processing unit 71, the dialogue management unit 72 obtains the text of the answer to the query from the answer generation unit 73 and sends the answer text to the user terminal 3. In addition, the query answer unit 58 may also include a TTS (Text to Speech) function to convert the answer text into speech, so that the query answer unit 58 outputs the converted speech to the user terminal 3 instead of the answer text.

[0054] Here, a user's query and its response are defined as one turn. The query response server 2 can be configured to output the user's final desired answer through multiple turns. More specifically, the dialogue management unit 72 can manage state transitions based on the tags output for a particular query text or query data, causing the response generation unit 73 to generate an answer corresponding to the state transition. For example, if the tag "forget_password" is output for the user's query text "Forgot password" in the first turn, the dialogue management unit 72 causes the response generation unit 73 to generate "Do you know your email address? (Yes / No)" as the answer. If the tag "yes" is output for the next user's query text "Yes, I know," the dialogue management unit 72 causes the dialogue management unit 73 to generate an answer corresponding to the state transition from the tag "forget_password" to "yes": "Please reset your password through the following link."

[0055] In addition, the dialogue management unit 72 stores the query text or query data, information indicating whether a tag has been determined, the determined tag, and the user's feedback on the answer to the query log 62.

[0056] The response generation unit 73, under the control of the dialogue management unit 72, generates response text corresponding to the determined tags. Furthermore, details regarding the processing of the natural language processing unit 71, the dialogue management unit 72, and the response generation unit 73 will be described below.

[0057] The following will further explain the learning of the machine learning model and the preparation of the learning dataset 61 implemented by the initial data determination unit 51, the learning unit 52, the performance evaluation unit 53, the problem detection unit 54, the data modification unit 55, the model expansion unit 56, and the learning control unit 57. Figure 5 It is a flowchart that roughly represents the processes related to learning a machine learning model in an information processing system.

[0058] First, based on the instructions of the learning control unit 57, the initial data determination unit 51 obtains an initial learning dataset 61 and a set of test data (hereinafter referred to as the test dataset) (step S101). The test dataset contains multiple test data, which includes query data and labels to be output for the query data.

[0059] Next, the learning control unit 57 causes the learning unit 52 to begin processing. The learning unit 52 uses the learning dataset 61 to enable the machine learning model to learn (step S102). The learning unit 52 enables the machine learning model to learn using the learning dataset 61 obtained by the initial data determination unit 51 when step S102 is initially executed.

[0060] After the machine learning model learns, the learning control unit 57 initiates processing by the performance evaluation unit 53, which determines whether the learned machine learning model meets the performance conditions (step S104). More specifically, the performance evaluation unit 53 inputs query data contained in the test data into the learned machine learning model for multiple test data sets, and determines whether the output is the same as the label contained in the test data (whether it is correct). Then, the performance evaluation unit 53 calculates the correctness ratio for each label in the learned data and determines whether there are any labels whose calculated ratio is lower than the determination threshold. If there are no labels whose calculated ratio is lower than the determination threshold, the performance evaluation unit 53 determines that the performance conditions are met; otherwise, it determines that the performance conditions are not met. Alternatively, the performance evaluation unit 53 may calculate the correctness ratio for each label output by the machine learning model and determine whether the performance conditions are met based on the ratio.

[0061] If the learned machine learning model is determined to be unsatisfactory (step S104 is "No"), the learning control unit 57 adjusts the improvement strategy (step S105). Then, the learning control unit 57 initiates processing by the problem detection unit 54, which detects problem points in the learning dataset 61 (step S106). The learning control unit 57 may submit improvement parameters to the problem detection unit 54 based on the improvement strategy, thereby enabling the problem detection unit 54 to detect problem points in the learning dataset 61 based on these improvement parameters. Details regarding the improvement strategy and improvement parameters will be described below.

[0062] After the problem detection unit 54 processes the data, the learning control unit 57 initiates the data modification unit 55 to begin processing. The data modification unit 55 updates the learning dataset 61 based on the detected problem points (step S107). Then, returning to step S102, the learning control unit 57 initiates the learning unit 52 to begin processing. The learning unit 52 uses the updated learning dataset 61 to enable the machine learning model to relearn. The processing from step S103 onwards is the same as the first time, so its description is omitted.

[0063] On the other hand, when it is determined in step S104 that the learned machine learning model meets the performance conditions (step S104 is "yes"), the learning control unit 57 causes the model expansion unit 56 to start processing, and the model expansion unit 56 expands the learned machine learning model to the query response server 2 (step S108).

[0064] The number of detection methods for problem points in Problem Detection Department 54 and the number of update methods for data in Data Change Department 55 are both multiple. Figure 6This diagram illustrates the processing of the problem detection unit 54 and the data modification unit 55. The detection processing column indicates the types of methods used by the problem detection unit 54 to detect problems in the learning dataset 61, the problem column indicates the types of problems detected, and the data modification processing indicates the types of update methods used to improve the detected problems in the learning dataset 61. (About...) Figure 6 The method shown and the details of the processing performed in the method will be described below.

[0065] First, let’s describe “Data statistics”. Figure 7 This is a flowchart illustrating an example of processing related to data parsing. Figure 7 The processing shown is derived from the data parsing-related processes extracted from the details of steps S105, S106, and S107.

[0066] exist Figure 7 In the process shown, firstly, the learning control unit 57 determines the upper and lower limits of the number of data items as improvement parameters included in the improvement strategy (step S201). The upper and lower limits of the number of data items can be determined to be different from the upper and lower limits used previously. Furthermore, if the improvement parameters for other problem detection methods are changed, the previous upper and lower limits can be used directly without changing them. The upper and lower limits can be determined by selecting one from a set of pre-set candidate values.

[0067] The learning control unit 57 uses the upper and lower limits as arguments to retrieve the data parsing API (Application Programming Interface) from the problem detection unit 54. The problem detection unit 54 counts the number of learning data items for each label in the learning dataset 61 (step S202).

[0068] The upper and lower limits may not be parameters used when calling the data parsing API. For example, the data parsing API may count the number of learning data items for each tag, and the learning control unit 57 may determine the upper and lower limits based on the number of learning data items for each tag. Alternatively, the learning control unit 57 may obtain the maximum and minimum values ​​of the number of learning data items for each tag, set the value smaller than the maximum value by a specified value as the upper limit, and set the value larger than the minimum value by a specified value as the lower limit.

[0069] After counting the number of items, the learning control unit 57 determines whether there are any tags whose count exceeds the upper limit (step S203).

[0070] If there are tags whose number of samples exceeds the upper limit (step S203 is "Yes"), the learning control unit 57 considers it to have detected a "Too many samples" problem and uses the tags exceeding the upper limit and the upper limit as arguments to retrieve the "Sample Reduction" processing API from the data modification unit 55. Then, the data modification unit 55 reduces the number of learning data samples for tags whose number of samples exceeds the upper limit (step S204). The data modification unit 55 can also calculate the sum of similarity between each of the multiple learning data samples containing tags exceeding the upper limit and the query data containing the same tags, and determine the learning data to be deleted based on the ranking determined by the sum. For example, learning data with a pre-set ranking can be deleted. Alternatively, the learning data to be deleted can be randomly determined. If there are no tags whose number of samples exceeds the upper limit, the processing in step S204 is skipped.

[0071] On the other hand, if there are tags with a number of items below the lower limit (step S205 is "Yes"), the learning control unit 57 considers it to have detected a "Lack of samples" problem and uses the tags below the lower limit and the lower limit as arguments to retrieve the API for "New sample collection" (learning data acquisition) processing from the data modification unit 55. In the learning data acquisition process, the data modification unit 55 extracts the query data corresponding to the tag from the query log 62 and appends learning data containing the extracted query data and the tag to the learning dataset 61. Furthermore, the query data to be appended is extracted from the query log 62 based on the query data corresponding to the tag in the learning dataset 61 and the query data stored in the query log 62. Details of these processes will be described below.

[0072] Figure 8 This is a graph illustrating the adjustment of the number of items in the learning dataset 61. Figure 8 The data includes three graphs, with the vertical axis of each graph representing the number of training data items, and each label indicating the number of training data items. Figure 8 In this context, Nmax represents the upper limit of the number of items, and Nmin represents the lower limit of the number of items.

[0073] Figure 8 The graph above represents the number of training data items for each label obtained during data parsing. In this example, the number of training data items for two labels exceeded the upper limit. Therefore, "Sample Reduction" was performed on these two labels to change the training dataset 61 (see reference 61) so that the number of training data items was below the upper limit. Figure 8(The two labels on the left side of the middle curve). Additionally, in this example, the number of training data items for four labels was below the lower limit. Therefore, a "New sample collection" (data appending) was performed on these four labels to change the training dataset 61 so that the number of training data items was above the lower limit (see reference 61). Figure 8 (The four labels on the right side of the graph below).

[0074] When the number of training data items varies significantly across different labels, machine learning models tend to output labels with an unnecessarily large number of items. Adjusting the number of training data items in this way ensures the accuracy of the machine learning model.

[0075] Here, the processing of queries by the query response unit 58 of server 2 after the machine learning model is expanded, and the query log 62 are explained. Figure 9 This is a flowchart illustrating an example of the processing of the query response unit 58. First, the natural language processing unit 71 of the query response unit 58 acquires query data based on input from the user (step S501). The natural language processing unit 71 can acquire the query data by either directly obtaining the query text input by the user or by performing lexical analysis on the query text.

[0076] The natural language processing unit 71 inputs the acquired query data into the learned machine learning model (step S502). Then, if the machine learning model fails to determine the label (step S503 is "No"), the dialogue management unit 72 sends a message to the user indicating that it cannot answer the query, and stores the information indicating the query data along with the message indicating that the corresponding label was not detected in the query log 62 (step S504).

[0077] If the machine learning model has determined the label (step S503 is "Yes"), the dialogue management unit 72 submits the determined label to the response generation unit 73, which generates a response to the determined label (step S505). The response generation unit 73 may generate a response simply by retrieving the text of the response stored in association with the label, or it may dynamically generate a response using information recorded in association with the user or the organization being queried.

[0078] The dialogue management unit 72 outputs the generated answer to the user terminal 3 (step S506). Here, the user terminal 3 outputs the answer, a screen for inputting whether the answer is appropriate for the query, and sends the input from the user to the query response server 2. The dialogue management unit 72 obtains feedback information indicating whether the answer is appropriate from the user terminal 3 (step S507). If information indicating that the answer is inappropriate is obtained (step S507 is "No"), the dialogue management unit 72 stores the information indicating the query data, the determined label, and the message indicating that the answer is inappropriate in the query log 62 (step S509). If information indicating that the answer is appropriate is obtained (step S507 is "Yes"), the dialogue management unit 72 stores the information indicating the query data, the determined label, and the message indicating that the answer is appropriate in the query log 62 (step S510).

[0079] Figure 10 This is a diagram representing an example of query log 62. "User Query" represents the query data, which is the text of the query entered by the user. "Corresponding Label Detection" indicates whether the machine learning model has determined the label; for this value, "Yes" indicates a label has been determined, and "No" indicates no label has been determined. "Inferred Label" is the label determined by the machine learning model. "Response Appropriateness" is feedback from the user indicating whether the response is appropriate; for this value, "Yes" indicates an appropriate response, and "No" indicates an inappropriate response.

[0080] The information stored in query log 62 is used as part of the processing of the problem detection unit 54 and the data change unit 55, which will be described below. Query log 62 is generated after the machine learning model is deployed. For example, in cases where the machine learning model needs to be relearned due to changes in the situation, the processing using query log 62 can be performed.

[0081] The following section provides a more detailed explanation of the "New sample collection" (data appending) process. Figure 11 This is a flowchart illustrating an example of data appending processing. Figure 11 The processing is performed separately for each label in the learning dataset 61 that is considered to have a small number of items.

[0082] The data modification unit 55 selects learning data containing labels as processing objects from the learning dataset 61 (step S301). Then, the data modification unit 55 calculates the similarity between each of the multiple user queries contained in the query log 62 and the query data contained in the selected learning data for each of the selected learning data (step S302). In addition, the similarity can be calculated for all combinations of one of the selected learning data and one of the multiple user queries.

[0083] The data modification unit 55 can generate article vectors using keywords extracted from the user's query text through lexical analysis or other means, or keywords contained in the query data, and calculate the similarity of the generated article vectors, thereby calculating the similarity between the user's query and the query data. Alternatively, a machine learning model that directly converts text into article vectors can be pre-built using so-called deep learning. The data modification unit 55 inputs the user's query text and the query data text into this machine learning model and calculates the similarity of the output article vectors.

[0084] After calculating the similarity, the data modification unit 55 extracts from multiple user queries those training data whose similarity is calculated to be greater than the first similarity threshold (e.g., 0.9) and the number of such data is N (e.g., 3) or more (step S303: first method). This is the process of extracting queries that are similar to a large number of training data.

[0085] Next, the data modification unit 55 extracts from multiple user queries those in which the number of learning data with a similarity calculated to be greater than the first similarity threshold is M (e.g., 1) or less (step S304: second method). This is the process of extracting queries that are similar to fewer learning data.

[0086] Next, the data modification unit 55 extracts from multiple user queries the number of learning data points that have a similarity calculated to be greater than the second similarity threshold (e.g., 0.6) and less than the first similarity threshold, and the number of such points is one or more (step S305: third method). This is the process of extracting queries used to expand the scope of queries corresponding to the labels in the learning data.

[0087] The data modification unit 55 adds learning data to the learning dataset 61. The learning data includes query data of user queries extracted through the processing in steps S303 to S305, and labels as processing objects (step S306).

[0088] Furthermore, if the number of user queries extracted in step S306 exceeds the lower limit minus the number of learning data items for the tags being processed (an additional predetermined number), the data modification unit 55 can select an additional predetermined number of user queries from the extracted user queries and add learning data to the selected user queries. In this process, the data modification unit 55 can randomly select user queries, or it can pre-set a reference ratio for each of the first to third methods, and reduce the number of user queries extracted by dividing the number of user queries extracted in each of the first to third methods by the total number of user queries extracted, thereby selecting an additional predetermined number of user queries.

[0089] Figure 12This diagram illustrates the relationship between similarity and the extraction of user queries. Figure 12 In the example, a solid line connecting the records in the learning data to the records in query log 62 indicates that the similarity calculated based on these records is above the first similarity threshold, and a dashed line indicates that the similarity calculated based on these records is above the second similarity threshold but below the first similarity threshold. No line indicates that no similarity above the second similarity threshold was calculated.

[0090] If N is 3 and M is 1, then Figure 12 Of the four user queries in query log 62 shown, the first user query has a similarity score of at least the first similarity threshold with three (N or more) query texts, and is therefore extracted using method 1. The second user query has a similarity score of at least the first similarity threshold with one (M or less) query text, and is therefore extracted using method 2. The third user query has a similarity score of at least the second similarity threshold with one query text, and is therefore extracted using method 3. This approach does not only extract user queries with high similarity to query texts in the larger training dataset, but also extracts other user queries, thus preventing a bias towards queries corresponding to labels that could lead to a decrease in accuracy.

[0091] The “New sample collection” process can also be used to prepare test datasets. For example, a label with a smaller number of data items can be identified from the test data contained in the test dataset, and the “New sample collection” process can be performed on that label. In this case, the test dataset can be used instead of the training dataset in all processes.

[0092] User queries can also be extracted using other methods. For example, a learning dataset 61 can be used to learn an evaluation machine learning model that calculates a score indicating whether a user query corresponds to a label of a specified object. The data change unit 55 extracts user queries based on whether the score output after inputting user queries from query log 62 into the evaluation machine learning model exceeds a threshold.

[0093] Alternatively, a machine learning model can be constructed using so-called deep learning to extract article vectors from input query text. The data modification unit 55 inputs query text, which is stored as learning data along with labels used for processing, contained in the learning dataset 61, into this machine learning model and obtains the average value of the output article vectors. Furthermore, the data modification unit 55 can also input user queries from the query log 62 into this machine learning model and extract user queries based on whether the similarity between the output article vectors and the average value exceeds a threshold.

[0094] Next, the processing of "Overlap detection" and "Overlap resolution" to solve the problem of "Overlapped samples" will be explained. After the problem detection unit 54 calls the "Overlap detection" API, it detects learning data that are similar to each other but have been assigned different labels. More specifically, for query texts (denoted as object query texts) belonging to labels (object labels) that have been judged by the performance evaluation unit 53 to have an accuracy rate below the threshold, the problem detection unit 54 performs the following two processes respectively. The first process is to calculate a first index representing the similarity between other query texts belonging to the object label and the object query text. The second process is to calculate a second index representing the similarity between each query text belonging to other labels and the object query text.

[0095] Then, regarding the object query text, if the second metric indicates similarity to any query text belonging to other labels, and the first metric indicates that the similarity to other query texts belonging to the object label is lower than the baseline state, the data modification unit 55, after calling the "Overlap resolution" API, will delete the learning data containing the object query text from the learning dataset 61.

[0096] exist Figure 3 In the training dataset 61 shown, the fourth and sixth training data points have very similar query meanings but are assigned different labels. In this case, the machine learning model cannot learn successfully, and the accuracy of the output labels decreases. In this situation, removing inappropriate training data can improve the accuracy of the machine learning model.

[0097] This section explains the handling of "Out-of-scope" and "Create intent" methods used to address the "Lack of intents" problem. The problem detection unit 54, after calling the "Out-of-scope" API, determines whether the number of user queries in query log 62 without a defined tag exceeds a threshold. If the threshold is exceeded, there may be insufficient tags corresponding to the queries.

[0098] Then, the data modification unit 55, after invoking the "Create intent" API, aggregates the text of multiple user queries for which tags have not been determined. If a cluster exists where the number of user queries exceeds a specified value, the user queries belonging to that cluster are output as tag candidates to the administrator of the learning management server 1. The administrator inputs tags to be added based on the output candidates, and user queries from the output user queries that correspond to the added tags. The data modification unit 55 then adds learning data containing the input user queries and tags.

[0099] At once Figure 10 For the fourth data point in query log 62 shown, the machine learning model failed to determine the corresponding label. In cases with a large number of such instances, it is easier to append learning data using the "Out-of-scope" and "Create intent" methods.

[0100] The handling of "Prediction Failure," used to address the "Misunderstanding" problem, is explained. The problem detection unit 54, after calling the "Prediction Failure" API, counts the number of user queries deemed inappropriately answered and the total number of user queries in the query log 62, tag by tag. Then, it determines, tag by tag, whether the metric value calculated by dividing the number of inappropriately answered user queries by the total number exceeds a specified threshold. If the threshold is exceeded, it indicates a high number of misjudgments for that tag.

[0101] If there are labels exceeding the threshold, the "New sample collection" API is invoked. The data modification unit 44 extracts user queries from the query log 62 and appends learning data containing user queries to the learning dataset 61 so that the number of learning data items for that label is greater than the current number of items.

[0102] At once Figure 10 For the fifth data point in query log 62 shown, the label output by the machine learning model is inappropriate. Given a large number of such cases, adjusting the learning dataset 61 through handling "Prediction failure" and "New sample collection" can easily improve the accuracy of the machine learning model.

[0103] As explained above, by conducting machine learning model evaluation, problem detection of the learning dataset 61 based on the evaluation, and changes to the learning dataset 61 within a unified and controlled environment, administrators can more easily prepare the learning dataset 61 to ensure the performance of the machine learning model. Furthermore, the reduced learning time of the machine learning model makes it easier to improve query responses using the machine learning model in response to changes in the environment.

Claims

1. An information processing system that receives inquiries from users, determines which of a plurality of pre-defined tags the inquiries corresponds to, and outputs an answer corresponding to the determined tag, comprising: The learning server enables the machine learning model to learn through a set of learning data containing input data and labels that serve as teaching data for that input data. The system and its response server input user-generated input data into the learned machine learning model and output response data based on the labels generated by the machine learning model. This information processing system includes: The initial data acquisition organization acquires the set of learning data; The learning institution uses the set of learning data to enable the machine learning model to learn; An evaluation agency inputs test data into the machine learning model, which has been learned through the set of learning data, and evaluates whether the performance of the machine learning model meets specified conditions based on the output of the machine learning model that has been input with the test data. The unfolding mechanism, when the performance of the machine learning model is evaluated as meeting the specified conditions, unfolds the learned machine learning model onto the response server; The data updating mechanism updates the set of learning data when the performance of the machine learning model is deemed not to meet the specified conditions. and The relearning mechanism enables the machine learning model to relearn using the updated set of learning data; and Based on the evaluation of the evaluation agency, the processes of the data update agency, the relearning agency, and the evaluation agency are repeated; If the performance of the machine learning model is deemed not to meet the specified conditions, the data update mechanism updates the set of learning data based on improvement parameters for the upper and lower thresholds of the number of data items in the set containing the learning data. The information processing system also includes a parameter update mechanism that, when the performance of the machine learning model is deemed not to meet the specified conditions, changes the upper and lower threshold values ​​of the number of data items.

2. The information processing system according to claim 1, wherein... It also includes a testing organization that, when the performance of the machine learning model is deemed not to meet the specified conditions, determines whether the set of learning data meets the testing conditions. If the detection conditions are met, the data update mechanism updates the set of learning data.

3. The information processing system according to claim 2, wherein... The detection agency determines whether the number of input data for each label in the set of learning data meets the detection conditions. If the detection conditions are met, the data update mechanism updates the set of learning data.

4. The information processing system according to claim 2, wherein... It also includes a logging mechanism that, upon receiving input data from the user, stores the input data in a log storage device; and The detection agency determines whether there are labels in the set of learning data that have insufficient input data. If it is determined that there are insufficient labels for the input data, the data update mechanism extracts the input data corresponding to the label from the input data stored in the log storage, and adds the extracted input data and the label to the set of learning data.

5. The information processing system according to claim 4, wherein... If it is determined that there are labels with insufficient input data, the data update mechanism extracts the input data corresponding to the label from the input data stored in the log storage, based on the input data corresponding to the label in the set of learning data and the input data stored in the log storage.

6. The information processing system according to any one of claims 1 to 3, wherein Also includes: The logging mechanism, upon receiving input data from the user, stores the input data in a log storage device; and The test data appending mechanism extracts input data corresponding to any label from the input data stored in the log storage and appends the extracted input data and the set of the label to the learning data.

7. The information processing system according to claim 2, wherein... The testing agency calculates a first indicator and a second indicator. The first indicator represents the similarity between the object input data, which is the input data of the label with an accuracy rate below a threshold, and other input data belonging to that label. The second indicator represents the similarity between the object input data and the input data of the label that belongs to a different label. The data update mechanism deletes the object input data from the set of learning data when the second indicator indicates similarity to any one of the object input data and input data belonging to a label different from the label to which the object input data belongs, and the first indicator indicates that the similarity to other input data belonging to the label to which the object input data belongs is lower than the baseline state.

8. The information processing system according to claim 2, wherein The detection agency determines whether the number of tags in the input data input by the user into the machine learning model exceeds the threshold. The data update organization categorizes the input data that has not been labeled into clusters, and proposes label candidates to the administrator based on the categorized clusters; The data update mechanism will add the set of candidate tags and input data to the learning data based on the tags provided by the administrator.

9. The information processing system according to claim 2, wherein It also includes a logging mechanism that stores input data to a log storage device when the user inputs input data. The detection agency determines the labels to be added to the learning data based on the number of input data items that the user answered as inappropriate by the labels determined by the machine learning model, which are input data input by the user and entered into the machine learning model. The data update mechanism extracts input data corresponding to the label of the object from the input data stored in the log storage, and appends learning data containing the extracted input data and the label to the set of learning data.

10. An information processing method, comprising receiving an inquiry from a user, determining which of a plurality of pre-defined tags the inquiry corresponds to, and outputting an answer corresponding to the determined tag, and including the following steps: Obtain a collection of learning data containing input data and labels, where the labels are instructional data for the input data, used to generate response data; The machine learning model learns by using the set of learning data; Input test data into the machine learning model that has been learned through the set of learning data, and evaluate whether the performance of the machine learning model meets the specified conditions based on the output of the machine learning model that has been input with the test data. If the performance of the machine learning model is evaluated to meet the specified conditions, the machine learning model with evaluated performance is expanded to a response server. The response server inputs the user-inputted input data into the learned machine learning model and outputs response data based on the labels output by the machine learning model. If the performance of the machine learning model is deemed not to meet the specified conditions, the set of learning data is updated. and The machine learning model is relearned using the updated set of learning data; and In the step of updating the set of learning data, if the performance of the machine learning model is deemed not to meet the specified conditions, the set of learning data is updated based on improvement parameters for the upper and lower thresholds of the number of data items in the set containing the learning data. Based on the evaluation, the steps of repeatedly updating the set of learning data, relearning the machine learning model, and evaluating the performance of the machine learning model are performed. It also includes a parameter update step, which changes the upper and lower threshold values ​​of the number of data items if the performance of the machine learning model is deemed not to meet the specified conditions.

11. An information processing apparatus, comprising: receiving an inquiry from a user, determining which of a plurality of pre-set tags the inquiry corresponds to, and outputting an answer corresponding to the determined tag, and comprising: The initial data acquisition organization acquires a collection of learning data containing input data and labels. The labels are teaching data for the input data and are used to create response data. The learning institution uses the set of learning data to enable the machine learning model to learn; An evaluation agency inputs test data into the machine learning model, which has been learned through the set of learning data, and evaluates whether the performance of the machine learning model meets specified conditions based on the output of the machine learning model that has been input with the test data. The unfolding mechanism, when the performance of the machine learning model is evaluated to meet the specified conditions, unfolds the machine learning model with evaluated performance to the response server, which inputs the user-inputted input data into the learned machine learning model and outputs response data based on the labels output by the machine learning model; The data updating mechanism updates the set of learning data when the performance of the machine learning model is deemed not to meet the specified conditions. and The relearning mechanism enables the machine learning model to relearn using the updated set of learning data; and Based on the evaluation of the evaluation agency, the processes of the data update agency, the relearning agency, and the evaluation agency are repeated; If the performance of the machine learning model is deemed not to meet the specified conditions, the data update mechanism updates the set of learning data based on improvement parameters for the upper and lower thresholds of the number of data items in the set containing the learning data. The information processing device also includes a parameter update mechanism that, when the performance of the machine learning model is assessed as not meeting the specified conditions, changes the upper and lower threshold values ​​of the number of data items.

Images