Program, and Automatic Response System

Abstract

To provide an automated answering system capable of smoothly giving an appropriate answer even when a corresponding conversation pattern is not prepared.SOLUTION: A program causes a processor of a computer to execute the steps of: receiving input of a question text from a user; causing a learned model to output an appropriate answer text as an answer to the input question text out of a plurality of answer candidates registered in advance in a database and presenting the output answer text to the user; receiving input of an evaluation of the answer text from the user; when a negative evaluation is input from the user, presenting the question text to a person in charge of management; receiving input of a re-answer to the question text from the person in charge of management; and registering a content of the re-answer in the database as a new answer candidate.SELECTED DRAWING: Figure 20

Description

[Technical field]

[0001] The present invention relates to a program, a method, and a system. [Background technology]

[0002] In recent years, chatbots, which are automated systems that respond to user inquiries, have become widely used. For example, the system described in Patent Document 1 performs a process of generating an appropriate answer to an input question through machine learning using conversation patterns. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-020995 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, because such automated response systems answer questions through machine learning using conversation patterns, they may not be able to properly learn and provide an answer to specific inquiries for which no conversation patterns have been prepared.

[0005] The present invention provides an automatic response system that can smoothly provide an appropriate response even if a corresponding conversation pattern is not prepared. [Means for solving the problem]

[0006] One aspect of the present invention is a program used by a computer that answers questions from a user through an automatic response, which causes a processor of the computer to execute the following steps: accepting input of a question sentence from a user; causing a trained model to output an appropriate answer sentence as an answer to the input question sentence from among a plurality of answer candidates pre-registered in a database, and presenting the output answer sentence to the user; accepting input of an evaluation of the answer sentence from the user; if the user inputs a negative evaluation of the answer sentence, presenting the question sentence to an administrative personnel; accepting input of a re-answer to the question sentence from the administrative personnel; and registering the content of the re-answer in the database as a new answer candidate. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram showing the overall configuration of an automatic response system according to an embodiment of the present invention; [Diagram 2] 2 is a block diagram showing a configuration of an employee terminal according to the embodiment; FIG. [Diagram 3] FIG. 2 is a block diagram showing a configuration of an administrator terminal according to the present embodiment. [Figure 4] FIG. 2 is a block diagram showing a configuration of a server according to the embodiment. [Diagram 5] FIG. 2 is a diagram showing the first half of the procedure for using the automatic response system according to the embodiment. [Figure 6] FIG. 11 is a diagram showing the second half of the usage procedure in the automatic response system according to the embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a data structure of an employee database according to the embodiment. [Figure 8] FIG. 1 is a first diagram showing an overview of the data structure of a conversation pattern database according to the present embodiment. [Figure 9] FIG. 2 is a second diagram showing an overview of the data structure of the conversation pattern database according to the present embodiment. [Figure 10] FIG. 2 is a diagram showing a first editing screen for the conversation pattern database according to the present embodiment. [Figure 11] FIG. 13 is a diagram showing a second editing screen for the conversation pattern database according to the present embodiment. [Figure 12] FIG. 6 is a diagram for explaining the details of the processing from steps P01 to P07 in FIG. 5 among the processing by the automatic response system. [Figure 13] FIG. 6 is a diagram for explaining the details of the processing from steps P08 to P10 in FIG. 5 among the processing by the automatic response system. [Figure 14] FIG. 7 is a diagram for explaining the details of the processing from steps P11 to P19 in FIG. 6 among the processing by the automatic response system. [Figure 15] FIG. 7 is a diagram for explaining the details of the processing from steps P20 to P21 in FIG. 6 among the processing by the automatic response system. [Figure 16] 13 is a diagram showing a first example screen P10 shown in FIG. 12, among examples of screens in the automatic answering system. FIG. [Figure 17] FIG. 14 is a diagram showing a second example screen P11 shown in FIG. 13, among examples of screens in the automatic response system. [Figure 18] FIG. 16 is a diagram showing a third example screen P12 shown in FIG. 15, among examples of screens in the automatic answering system. [Figure 19] FIG. 17 is a diagram showing a fourth example screen P13 shown in FIG. 15, among examples of screens in the automatic answering system. [Figure 20] FIG. 17 is a diagram showing a fifth example screen P14 shown in FIG. 15, among examples of screens in the automatic answering system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and the repeated description will be omitted.

[0009] (1) Configuration of automatic response system 1 The configuration of an automatic response system 1 (hereinafter simply referred to as system 1) will be described. FIG. 1 is a block diagram showing the overall configuration of system 1 according to this embodiment. System 1 constitutes a part of an in-house system used within a company, for example. System 1 is a system used when an employee (user) makes an inquiry about in-house rules to a management department, such as the human resources department or general affairs department. Automatic response system 1 is a chatbot that automatically outputs an appropriate answer to a question entered by text input.

[0010] As shown in FIG. 1, the system 1 includes a plurality of employee terminals 10, a plurality of manager terminals 20, a server 30, and a conversation pattern DB (database) 40. The employee terminal 10, the manager terminal 20, the server 30, and the conversation pattern DB 40 are connected via a network 50 (for example, the Internet or an intranet).

[0011] The employee terminal 10 and the manager terminal 20 are, for example, a smartphone, a tablet terminal, or a personal computer. The employee terminal 10 is a terminal used by an employee of the company in which the system 1 is operated. In this description, an employee refers to a person who is engaged in the business of the company in which the system 1 is operated, regardless of the employment status. Note that the employee terminal 10 in this invention may also include a terminal used by an executive who is not included in the category of an employee.

[0012] The administrator terminal 20 is a terminal used by an administrator of the system 1. The administrator refers to an employee belonging to a supervising department that is responsible for the maintenance and operation of the system 1 in the company where the system 1 is operated. An example of a supervising department is a management department that receives many inquiries from each department, such as general affairs, human resources, accounting, and finance. Note that the supervising department is not limited to a management department, and for example, a sales support department that receives frequent inquiries from the sales department, or a manufacturing department that receives frequent inquiries from the sales support department, may be the supervising department of the system 1.

[0013] Here, some of the employee terminals 10 correspond to the manager terminal 20, and the terminal is distinguished as either an employee terminal 10 or a manager terminal 20 depending on the purpose of the operation of the terminal. That is, when an employee belonging to a supervisory department checks the company rules using the system 1, the terminal used by the employee corresponds to the employee terminal 10 in this description. On the other hand, when the employee performs operations related to the maintenance of the system 1, which will be described later, as part of his / her job, the terminal used by the employee corresponds to the employee terminal 10 in this description.

[0014] The server 30 executes calculation processing based on information acquired from the multiple employee terminals 10 and information input from the multiple manager terminals 20, and outputs various information described below. The server 30 may be configured by one or more physical computers. The server 30 provides the employee terminal 10 with a response in response to the request sent from the employee terminal 10 . The server 30 provides the administrator terminal 20 with a response in response to the request sent from the administrator terminal 20 .

[0015] (1-1) Configuration of employee terminal 10 The following describes the configuration of the employee terminal 10. Fig. 2 is a block diagram showing the configuration of the employee terminal 10 of this embodiment. As shown in FIG. 2, the employee terminal 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14.

[0016] The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a Read Only Memory (ROM), a Random Access Memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0017] The programs include, for example, the following programs: ·OS (Operating System) programs Application programs that perform information processing

[0018] The data includes, for example, the following data: Databases referenced in information processing Data obtained by performing information processing (i.e., the results of performing information processing)

[0019] The processor 12 is a computer that realizes the functions of the employee terminal 10 by starting a program stored in the storage device 11. The processor 12 is, for example, at least one of the following: ·CPU(Central Processing Unit) ·GPU(Graphic Processing Unit) ·ASIC(Application Specific Integrated Circuit) ·FPGA(Field Programmable Array)

[0020] The input / output interface 13 is configured to obtain information (eg, user instructions) from an input device connected to the employee terminal 10, and to cause an output device connected to the employee terminal 10 to output information. The input device is, for example, a physical button, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 15, a speaker, a lamp, or a combination thereof.

[0021] The communication interface 14 is configured to control communication between the employee terminal 10 and an external device (eg, the manager terminal 20, the server 30, the conversation pattern DB 40, or a combination thereof). The display 15 is configured to display an image (still image or moving image). The display 15 is, for example, a liquid crystal display or an organic EL display.

[0022] (1-2) Configuration of administrator terminal 20 The following describes the configuration of the administrator terminal 20. Fig. 4 is a block diagram showing the configuration of the administrator terminal 20 of this embodiment.

[0023] As shown in FIG. 4, the administrator terminal 20 includes a storage device 21, a processor 22, an input / output interface 23, a communication interface 24, and a display 25.

[0024] The storage device 21 is configured to store programs and data, and is, for example, a combination of a ROM, a RAM, and a storage.

[0025] The programs include, for example, the following programs: -OS programs Applications that process information (e.g. web browsers)

[0026] The data includes, for example, the following data: Databases referenced in information processing Results of information processing

[0027] The processor 22 is a computer that realizes the functions of the administrator terminal 20 by starting a program stored in the storage device 21. The processor 22 is, for example, at least one of the following. ·CPU GPU ·ASIC FPGA

[0028] The input / output interface 23 is configured to obtain information (for example, a user's instruction) from an input device connected to the administrator terminal 20, and to cause an output device connected to the administrator terminal 20 to output the information. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 25, a speaker, or a combination thereof.

[0029] The communication interface 24 is configured to control communication between the administrator terminal 20 and an external device (eg, the server 30, the conversation pattern DB 40, or a combination thereof). The display 25 is configured to display an image (still image or moving image). The display 25 is, for example, a liquid crystal display or an organic EL display.

[0030] (1-3) Configuration of Server 30 The following describes the configuration of the server 30. Fig. 3 is a block diagram showing the configuration of the server 30 of this embodiment.

[0031] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0032] The storage device 31 is configured to store programs and data, and is, for example, a combination of a ROM, a RAM, and a storage device.

[0033] The programs include, for example, the following programs: -OS programs · Application programs that perform information processing (e.g., automatic response models)

[0034] The data includes, for example, the following data: Databases referenced in the processing of information (e.g. employee databases) Results of information processing

[0035] The processor 32 is a computer that realizes the functions of the server 30 by starting a program stored in the storage device 31. The processor 32 is, for example, at least one of the following. ·CPU GPU ·ASIC FPGA

[0036] The input / output interface 33 is configured to obtain information (eg, a user's instruction) from an input device connected to the server 30, and to cause an output device connected to the server 30 to output information. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0037] The communication interface 34 is configured to control communication between the server 30 and an external device (eg, the employee terminal 10, the manager terminal 20, the conversation pattern DB 40, or a combination thereof).

[0038] Next, the automatic response model stored in the storage device 31 of the server 30 will be described. The automatic response model is a trained model for realizing an automatic response function as a chatbot. For example, the automatic response model is stored in advance in the storage device 31 when the server 30 provides an automatic response service. The automatic response model is re-trained as needed based on the processing described below. For example, the automatic response model trains conversation patterns stored in the conversation pattern DB 40 as learning data.

[0039] The automatic response model is obtained by making the machine learning model perform machine learning according to a model learning program based on the learning data. In this embodiment, the automatic response model is trained to output a predetermined answer sentence to an input question sentence together with the question category (QA item) to which the answer sentence belongs. At this time, the learning data is, for example, a plurality of question patterns according to the type of conversation as input data, and the answer pattern to this question pattern and its category are correct answer output data. More specifically, the auto-answer model performs text mining processing on the input question text and selects a question pattern that has features that are highly similar to the features of the question text. It then selects an answer pattern that corresponds to the question category (QA item) to which the question pattern is linked. At this time, the auto-answer model assigns a recommendation score to the selected question category (QA item) according to the similarity, and recommends multiple question categories (QA items) and answer patterns with high recommendation scores.

[0040] The machine learning model according to the present embodiment is, for example, a parameterized composite function in which a plurality of functions are combined. The parameterized composite function is defined by a combination of a plurality of adjustable functions and parameters. The machine learning model according to the present embodiment may be any parameterized composite function that satisfies the above requirements, but is assumed to be a multi-layered network model (hereinafter referred to as a multi-layered network). An automatic response model using a multi-layered network has an input layer that inputs a question, an output layer that outputs an answer to the question, and at least one intermediate layer or hidden layer provided between the input layer and the output layer. The automatic response model is expected to be used as a program module that is a part of artificial intelligence software.

[0041] As the multi-layered network according to the present embodiment, for example, a deep neural network (DNN), which is a multi-layered neural network that is the subject of deep learning, may be used. As the DNN, for example, a recurrent neural network (RNN) that targets time-series information or the like may be used. In addition, a network with a self-attention mechanism that targets natural language may be used.

[0042] The storage device 31 stores a dialogue program for answering questions from a user using an automatic response model. The dialogue program may be stored in advance in the storage device 31. Alternatively, the dialogue program may be stored in a non-transient storage medium and distributed, and may be read out from the non-transient storage medium and installed in the storage device 31.

[0043] (1-4) Structure of conversation pattern DB40 1 is a database that stores conversation patterns that serve as learning data for an automatic response model. The conversation patterns include question patterns that are candidates for question sentences, and answer patterns that are candidates for answer sentences. The conversation pattern DB 40 stores answer patterns in association with QA items, which are question categories classified according to the content of the question. The conversation pattern DB 40 also stores multiple answer patterns in association with one QA item (i.e., one answer pattern). The structure of the conversation pattern DB 40 will be described in detail later.

[0044] (2) Overview of the embodiment The outline of this embodiment will be described with reference to Figs. 5 and 6. Fig. 5 is a diagram showing the first half of the procedure for using the system 1. Fig. 6 is a diagram showing the second half of the procedure for using the system 1. Unlike a typical chatbot, the system 1 has a function for notifying the administrator of information urging maintenance when an appropriate response cannot be output. This point will be described in detail.

[0045] As shown in Figure 5, the first half of the steps in System 1 are listed below: Step P01: The user inputs a question. Step P02: Output the answer to the user. Step P03: The user inputs an evaluation for the answer output from System 1. Step P04: If the user's evaluation is positive, the process ends. · Step P05: If the user's evaluation is negative, check the reason.

[0046] Here, there are mainly two reasons why a user may give a negative evaluation: The system 1 notifies the administrator of the content of the negative evaluation. Reason 1: If the answer contains insufficient or excessive information (step P07) Reason 2: The QA item in the answer text is inappropriate (step P06).

[0047] Reason 1 is when the system 1 recommends an appropriate question category (QA item) for a question from a user, but the answer content is not updated or contains unnecessary information, and it is considered to be incomplete. In this case, the administrator needs to update and correct the information about the answer pattern linked to the corresponding QA item.

[0048] Therefore, the process to resolve the deficiency due to reason 1 is carried out as follows: - Step P08: Recommend to the administrator to update the answer pattern information. - Step P09: The administrator updates the answer pattern information. · Step P10: Based on the updated information, a new answer is provided to the user who asked the question. This makes it possible to provide the user with appropriate information in response to a problem in the reply due to reason 1.

[0049] On the other hand, reason 2 in step P06 is a case where the system 1 is unable to recommend an appropriate question category (QA item) for a question from a user, and answers the question with information that is less relevant to the question, resulting in a defect. In this case, there is a concern that the question pattern linked to the answer pattern needs to be revised, or that a QA item corresponding to the question has not been set in the first place.

[0050] Therefore, the process for resolving the defect due to reason 2 is carried out as follows, as shown in FIG. · Step P11: Output other answer candidates. At this time, the previously output QA item is excluded, and other multiple answer candidates are output, for example, multiple options with the names of the QA items are listed, and the user is asked to select the answer candidate that he / she thinks is appropriate. Also, in the case where there is no option that seems appropriate, an option that indicates that fact ("not applicable") is also output.

[0051] First, a case where the user selects any one of the QA items will be described. Step P12: Display answer patterns for the Q&A item selected by the user. - Step P13: Enter an evaluation for the new answer.

[0052] If a positive evaluation is input (step P14), the following process is carried out. · Step P15: Add the question as a new question pattern. That is, it is determined that the question pattern linked to the answer pattern was insufficient, and the question is linked to the answer pattern as a new question pattern and registered in the conversation pattern DB 40. The automatic response model then re-learns, updating it so that when a similar question is input in the future, the new answer can be output.

[0053] On the other hand, if a negative evaluation is input in step P13 (step P16), the reason for this is confirmed. In this case too, the reason for the negative evaluation from the user is one of the following. Reason 1: If the answer contains insufficient or excessive information (step P18) Reason 2: The Q&A item in the answer is inappropriate (step P17).

[0054] Here, in the case of reason 1, it is assumed that an appropriate Q&A item has been recommended, and it is determined that the answer needs to be updated, and the process from steps P08 to P10 in Fig. 5 is carried out. This makes it possible to provide the user with appropriate information when the answer is incomplete due to reason 1.

[0055] On the other hand, in the case of reason 2, it is determined that appropriate QA items cannot be recommended as in the previous case, and the process of step P11 is repeated again. In other words, the QA items output in the past two times are excluded, and options for the other multiple answer candidates are output to the user again.

[0056] Next, a case where there is no suitable option for another answer candidate newly output in step P10 (step P19) will be described. - Step P20: Perform escalation processing. Step P21: The administrator's response is notified to the user. The escalation process is a process in which it is determined that there is no appropriate Q&A item set to answer a user's question, and the administrator responds by entering text. In addition, the administrator can also register the answer as a new conversation pattern at his or her discretion.

[0057] For example, if there is a possibility that the question will be asked frequently from other employees in the future, the answer entered as text can be treated as an answer pattern, and the question can be treated as a question pattern, and these can be linked to the newly set QA item and stored in the conversation pattern DB 40. On the other hand, if the question is about the user's personal circumstances, it may be difficult to arrive at a generalized answer, and therefore it may be determined that there is no need to register a new conversation pattern.

[0058] By notifying the user of the answer entered as text in this manner, the user can provide an appropriate answer even for a question for which an appropriate Q&A item has not been set. In addition, the automatic response model will then be re-trained, so that if a separate question similar to the newly registered question pattern is asked, the response sentence set as the new response pattern will be output.

[0059] (3) Database The database used by the system 1 according to this embodiment will be described.

[0060] (3-1) Employee DB The employee DB (database) according to this embodiment will be described. Fig. 7 is a diagram showing an example of the data structure of the employee DB according to this embodiment. The employee DB is stored in the storage device 31 of the server 30.

[0061] The employee DB stores employee information. The employee information is information about employees who belong to a company. The employee DB may also include information about executives who are the employers, and may include information about all users who make inquiries through the system 1.

[0062] 7, the employee DB includes an "employee ID" field, a "name" field, a "department" field, a "job title" field, a "contact information" field, and a "personal information" field. Each field is associated with the others.

[0063] The "Employee ID" field stores the employee ID. The employee ID is a code for identifying an employee and is set for each employee.

[0064] The "Name" field stores the name of the employee corresponding to the employee ID.

[0065] The "Department" field stores information about the current department of the employee corresponding to the employee ID. For example, the "Department" field stores the names of departments such as Sales Department, Development Department, Legal Department, General Affairs Department, and Manufacturing Department.

[0066] The "position" field stores information about the current position of the employee corresponding to the employee ID. The "position" field is not limited to the position in the department such as team leader, section manager, department manager, etc., but may store information about the role in the business case such as project manager. Also, instead of or in addition to the position, information about the employee's job position may be stored.

[0067] The "Personal Information" field stores the personal information of the employee corresponding to the employee ID. The employee's personal information includes, for example, the following information: Information about your birth, such as your age, sex, and date of birth Geographical information closely related to employees, such as nationality, place of origin, place of residence, etc. Information about your career history, such as your educational background and work history (including past positions and duties in your current job) Family information, such as whether you have a spouse and children Each of these pieces of information may have a column in the employee DB for each item.

[0068] Furthermore, the employee DB is not limited to these attributes and can store any information related to employee attributes.

[0069] (3-2) Conversation Pattern Database The conversation pattern DB 40 will now be described. Fig. 8 is a first diagram showing an outline of the data structure of the conversation pattern DB 40 according to this embodiment. Fig. 9 is a second diagram showing an outline of the data structure of the conversation pattern DB 40 according to this embodiment.

[0070] As shown in FIG. 8, the conversation pattern DB40 has areas for storing conversation patterns divided into categories of questions called QA items, which are classified according to the content of the question. A conversation pattern is composed of a question pattern and an answer pattern. A question pattern is text data indicating a question sentence set in anticipation of a question from a user. An answer pattern is answer information including text data indicating an answer sentence that is an answer candidate set as an answer to a question corresponding to the question pattern. An answer pattern may include document data such as a PDF containing the answer content, address information of a site managed by an internal system or a government office, and image data.

[0071] Usually, multiple question patterns are set for one answer pattern. In other words, even if the same answer is required, it is assumed that each user will ask a question in a different way. For this reason, multiple question patterns are set for one answer pattern so that QA items can be appropriately recommended for a wide range of questioning ways.

[0072] As shown in Fig. 9, the conversation pattern DB 40 has a hierarchical structure. Q&A items that are highly related to each other are classified into medium divisions called small categories. Furthermore, multiple closely related small categories are grouped together to form larger categories. By managing Q&A items in this way, conversation patterns can be edited smoothly.

[0073] In the illustrated example, the following major divisions are set as major categories: Attendance management: information on correcting attendance records for daily work, applying for leave, etc. Labor procedures: Changes to employee addresses, names, family structures, etc. Expenses-related: Confirmation of settlement methods for various expenses, etc. Personnel-related matters: Internal job postings, transfer requests, etc. General affairs-related matters: Contents related to how to use office facilities or requests for facility repairs, etc. Other: Anything that does not fall under any of the above It should be noted that this major division is merely an example and can be changed or added to as desired.

[0074] In the illustrated example, at least the following small categories are set as medium divisions in the major category of attendance management. · Travel expenses related: matters related to settlement of expenses incurred during business trips - Employee benefits related content: for example, information about employee benefits including daily transportation expenses Note that this intermediate classification is merely an example, and changes and additions can be made as desired.

[0075] In the illustrated example, at least the following Q&A items are set as question categories in the travel expenses-related subcategory. - Regarding travel expense claims: Information regarding the settlement method for travel expenses associated with business trips Regarding how to reserve transportation: Information on how to reserve transportation tickets for business trips Regarding accommodation cost conditions: Information regarding accommodation conditions (costs) at the business trip destination Note that these QA items are merely examples and can be changed or added as desired.

[0076] Each of these hierarchically structured sections can be edited as desired. Figure 10 is a diagram showing a first edit screen for conversation pattern DB 40. Figure 11 is a diagram showing a second edit screen for conversation pattern DB 40. As shown in Fig. 10, when editing a QA item, the administrator selects a major division that includes the QA item to be edited from the major categories displayed on the left side of the QA item setting screen displayed on the administrator terminal 20. Furthermore, on the right side of the screen, the administrator selects a minor category that includes the QA item to be edited, and selects the QA item to be edited ("Regarding travel expense application" in the illustrated example) (symbol G). This causes the screen display to transition to that shown in Fig. 11.

[0077] Next, the administrator edits the answer patterns for the displayed Q&A items in Fig. 11. Specifically, as shown in the upper part of Fig. 11, the administrator inputs text information as the answer sentences and various file data including the answer contents. The administrator can also add question patterns on this screen. The administrator can expand the questioning methods and set them from the perspective of what questions the user is likely to ask about the relevant Q&A item. Finally, the inputted content is saved by clicking the Save button H. This sets the new Q&A item. In this way, answer patterns for questions of various contents are registered in the conversation pattern DB.

[0078] (4) Information processing Next, the information processing of this embodiment will be described. FIG. 12 is a diagram for explaining the details of the processes in steps P01 to P07 in FIG. 5 among the processes performed by the system 1. In FIG.

[0079] (4-1) Steps P01 to P07 As shown in FIG. 12, first, the employee terminal 10 accepts input of a question from the user (step S101). Specifically, when an employee user has an unclear point (problem) in the course of daily work that requires confirmation from the management department, the employee operates the employee terminal 10 to start the chat function of the system 1 and inputs a question in a conversational format in the chat field. The processor 12 of the employee terminal 10 outputs the text information input via the communication interface 14 to the server 30.

[0080] After step S101, the server 30 acquires a question input by the user (step S301). Specifically, the processor 32 of the server 30 acquires, via the communication interface 34, text information that is the question output from the employee terminal 10.

[0081] After step S301, the server 30 selects an optimal answer sentence for the question sentence (step S302). Specifically, the processor 32 of the server 30 inputs a question into the automatic response model, and selects the QA item and answer pattern with the highest recommendation score that are the answer candidates output from the automatic response model from among multiple answer candidates pre-registered in a database.

[0082] After step S302, the server 30 outputs the selected answer sentence (step S303). Specifically, the processor 32 outputs the answer pattern selected in step S301 to the employee terminal 10 via the communication interface .

[0083] After step S303, the employee terminal 10 displays the reply text output from the server 30 (step S102). Specifically, the processor 12 of the employee terminal 10 acquires the answer sentence output from the server 30 via the communication interface 14 and displays it on the display 15. In this way, the answer to the question is presented to the user who asked the question.

[0084] After step S102, the employee terminal 10 accepts an input of an evaluation on the answer sentence from the user (step S103). Specifically, after displaying the answer, the processor 12 of the employee terminal 10 displays a questionnaire to the user asking whether the answer is appropriate. At this time, the questionnaire asks the user to confirm at least the following contents. ·Whether the answer is appropriate If the answer is inappropriate, the reason why

[0085] In addition, if the reply is inappropriate, the following options are presented to the user to select the reason. · The answer contained in the answer is insufficient or excessive The question category (QA item) of the answer contained in the answer is inappropriate In addition, when the answer sentence is inappropriate, the reason may be explained by a survey in which the user inputs text without presenting options. The processor 12 outputs the evaluation result input via the communication interface 14 to the server 30. The first screen example P10 displayed on the display 15 of the employee terminal 10 at this time will be described later.

[0086] After step S103, the server 30 acquires the evaluation result input by the user (step S304). Specifically, the processor 32 of the server 30 acquires the content of the answer entered by the user via the communication interface 14.

[0087] In step S304, if a positive evaluation is input by the user (Yes in step S305), the server 30 ends the process. This is because it can be determined that the answer to the user's question is appropriate and has resolved the user's problem.

[0088] On the other hand, if a negative evaluation is input by the user in step S304 (No in step S305), the server 30 branches the process depending on the reason for the negative evaluation.

[0089] Specifically, when the user inputs that the question category (QA item) of the answer content included in the answer sentence is inappropriate (X in step S306), the server 30 performs the processes from connector B onwards. On the other hand, if the user inputs that the answer content included in the answer sentence is insufficient or excessive (Y in step S306), the server 30 performs the processing from connector C onwards. Here, the processing from connector C onwards will be explained first. Fig. 13 is a diagram for explaining the details of the processing from steps P08 to P10 in Fig. 5, among the processing by the system 1 as the processing from connector C onwards.

[0090] (4-2) Steps P08 to P10 After step S306, the server 30 notifies the administrator terminal 20 of the recommendation to update the information (step S311). Specifically, the processor 32 of the server 30 outputs a notification to the administrator terminal 20 via the communication interface 24, which notifies the administrator that information updating is recommended for the answer pattern linked to the recommended QA item. This notification includes, for example, the following information: -Title of the QA item that needs updating Number of inquiries related to the QA item - Information about the user who pointed out the deficiency in the answer In other words, since the user has pointed out that the answer content contained in the answer text is insufficient or excessive, it is considered necessary to update the information of the answer text in order to eliminate the deficiency.

[0091] After step S311, the administrator terminal 20 displays a notification of the recommendation to update information on the display 25 (step S211). Specifically, the processor 22 of the administrator terminal 20 receives the notification of the recommendation to update information output from the server 30 via the communication interface 24, and presents the contents of the notification to the administrator by displaying them on the display 25. This allows the administrator to understand the contents of the QA items that need to be updated. A second screen example P11 displayed on the display 25 of the administrator terminal 20 at this time will be described later.

[0092] After step S211, the administrator updates the information on the answer content (step S212). Specifically, the administrator operates the administrator terminal 20 to edit and update the contents of the answer sentences for the QA items that require information updating. At this time, the QA items can be edited on the QA item editing screen described above (see Figs. 10 and 11). The processor 22 of the administrator terminal 20 outputs a notice of the update and its contents to the server 30 via the communication interface 24.

[0093] After step S212, the server 30 obtains the updated information (step S312). Specifically, the processor 32 of the server 30 receives the information output from the administrator terminal 20 via the communication interface .

[0094] After step S312, the server 30 outputs the updated information (step S313). Specifically, the processor 32 outputs information about the updated answer text to the user who pointed out the deficiency in the answer text via the communication interface 34. At this time, when notifying the user of the new answer text, the administrator notifies the user that the information included in the answer text has been updated.

[0095] After step S313, the employee terminal 10 displays the output information (step S111). Specifically, the processor 12 of the employee terminal 10 acquires the information output from the server 30 via the communication interface 14, and displays the content of the answer on the display 15. By checking the content of the answer displayed on the display 15, the user can confirm the appropriate answer sentence with updated information for the original question. With the above, the process from connector C in Fig. 12 (steps P08 to P10 in Fig. 5) is completed. Next, the process from connector B in Fig. 12 will be described.

[0096] (4-3) Steps P11 to P19 FIG. 14 is a diagram for explaining the details of the processes from steps P11 to P19 in FIG. 6 among the processes performed by the system 1. In FIG. As shown in Fig. 14, after step S306, the server 30 presents other answer candidates to the user (step S321). That is, since the user has evaluated the QA item of the answer content included in the answer sentence as inappropriate in the previous process, it can be considered that the QA item recommended from the question sentence was not appropriate in the first place. Therefore, the server 30 newly outputs the other multiple answer sentences output by the automatic response model as candidates to be answered next to the answer sentence already presented to the user.

[0097] Specifically, the processor 32 selects multiple QA items having recommendation scores of 2 or higher from among the multiple QA items recommended by the automatic response model in step S302 of Fig. 13, and outputs the selected QA items to the employee terminal 10. At this time, the processor 32 outputs information on the selected QA items and answer patterns linked to the QA items.

[0098] After step S321, the employee terminal 10 displays other answer candidates output from the server 30 (step S121). Specifically, the processor 12 of the employee terminal 10 displays a plurality of answer candidates (for example, the names of three QA items with recommendation scores 2 to 4) output from the server 30 on the display 15. At this time, the processor 12 accepts an input from the user to select a QA item that is likely to correspond to the answer sentence to be confirmed. The processor 12 also displays an option of "No option" in case there is no option for an appropriate QA item.

[0099] After step S121, the employee terminal 10 accepts the user's selection from other answer candidates (step S122). Specifically, the processor 12 of the employee terminal 10 accepts an input of an operation in which the user selects one of the options for the multiple Q&A items displayed on the display 15 and the option indicating “no option.” The processor 12 outputs the selected information to the server 30.

[0100] In step S122, if there is no option for the appropriate QA item, that is, if the option "No option" is selected by the user (No in step S123), the process from connector D onwards is executed as an escalation process that entrusts the subsequent response to the administrator. This escalation process will be described later with reference to FIG.

[0101] In step S122, if any of the options for the Q&A item is selected by the user (Yes in step S123), the employee terminal 10 displays the content of the selected answer candidate (step S124). Specifically, the processor 12 of the employee terminal 10 displays, as an answer sentence, an answer pattern that is associated with the Q&A item selected by the user and stored on the display 15. In this way, a new answer sentence is presented to the user.

[0102] After step S124, the employee terminal 10 accepts an input of an evaluation for the answer text (step S125). Specifically, the processor 12 of the employee terminal 10 displays a questionnaire to the user asking whether the newly presented answer sentence is appropriate. This questionnaire is the same as the questionnaire displayed in step S103 of Fig. 12. Specifically, this questionnaire asks the user to confirm at least the following contents: ·Whether the answer is appropriate If the answer is inappropriate, the reason why

[0103] Also, just like the previous time, if the reply is inappropriate, the following options are presented to the user to select the reason. · The answer contained in the answer is insufficient or excessive The question category (QA item) of the answer contained in the answer is inappropriate

[0104] After step S125, the server 30 acquires the evaluation result input by the user (step S322). Specifically, the processor 32 of the server 30 acquires the content of the answer entered by the user via the communication interface 14.

[0105] In step S322, if a positive evaluation is input by the user (Yes in step S323), the server 30 registers the question sentence from the user in the conversation pattern DB 40 as a new question pattern corresponding to the re-answer that was an appropriate answer. In this case, it is considered that an appropriate question pattern was not linked to the answer pattern of the re-answer that should have originally answered the question from the user. Therefore, the question sentence is linked to the answer pattern related to the re-answer as a new question pattern and stored in the conversation pattern DB 40. Thereafter, the automatic response model re-learns the new conversation pattern, so that it can provide appropriate answers to similar questions in the future. Thereafter, the server 30 ends the process.

[0106] On the other hand, if a negative evaluation is input by the user in step S322 (No in step S323), the server 30 branches the process depending on the reason for the negative evaluation.

[0107] Specifically, if the user inputs that the question category (QA item) of the answer content included in the answer sentence is inappropriate (X in step S325), the server 30 executes the process of step S321 again as the process after connector B. That is, the server 30 outputs other QA items that have not yet been presented as answer candidates according to the recommended order. The subsequent processes are executed in the same manner.

[0108] On the other hand, if the user inputs that the answer content included in the answer sentence is insufficient or excessive (Y in step S325), the server 30 executes the process from step S311 onwards shown in Fig. 13 again as the process from connector C onwards. That is, the server 30 outputs a notification to the administrator terminal 20 that it is recommended to update the information on the answer pattern linked to the QA item. With the above, the processing from steps P11 to P19 shown in FIG. 6 is completed.

[0109] (4-4) Steps P20 to P21 Next, as the process after connector D, the process from steps P20 to P21 in FIG. 6 including the escalation process will be described. FIG. 15 is a diagram for explaining the details of the processing from steps P20 to P21 in FIG. 6 among the processing by the system 1.

[0110] As shown in FIG. 15, the server 30 outputs a question to the manager terminal 20 (step S331). Specifically, when the user determines that an appropriate Q&A item is not recommended even when multiple new answer candidates are presented, the processor 32 of the server 30 outputs the question to the administrator terminal 20. This is to prompt the administrator to perform escalation processing. Escalation processing refers to an individual response to a question from a user by the administrator directly checking the question and inputting the answer as text.

[0111] After step S331, the administrator terminal 20 displays the question (step S231). Specifically, the administrator terminal 20 displays the question text output from the server 30 on the display 25. This presents the question text to the administrator. A third screen example P12 displayed on the display 25 of the administrator terminal 20 at this time will be described later.

[0112] After step S231, the administrator terminal 20 accepts an answer input for the re-answer (step S232). Specifically, the administrator checks the presented question and inputs the answer by operating the input / output interface 33 of the administrator terminal 20. A fourth screen example P13 displayed on the display 25 of the administrator terminal 20 at this time will be described later.

[0113] After step S232, the administrator terminal 20 accepts a selection as to whether or not to add a new conversation pattern to the conversation pattern DB 40 (step S233). Specifically, the processor 12 of the administrator terminal 20 displays on the display 25 an option to select whether or not to register the content entered by the administrator as a new conversation pattern. When registering as a new conversation pattern, the administrator inputs the category of the new QA item to which the answer sentence will be linked along with an instruction to register. Specifically, the administrator inputs the name of the category of the QA item to which the re-answer answer sentence will be newly registered. The administrator can also make any edits required for registering the question sentence as a new question pattern. The fifth screen example P14 displayed on the display 25 of the manager terminal 20 at this time will be described later.

[0114] After step S233, the administrator terminal 20 outputs the input information to the server 30 (step S234). Specifically, the processor 22 of the administrator terminal 20 outputs, for example, the following information to the server 30 as the information input by the administrator: - Information on new responses entered by the administrator Information entered by the administrator as to whether or not the answer sentence is to be newly registered in the conversation pattern DB40 Information about the name of a new Q&A item to which the conversation sentence is to be linked, when the answer sentence is newly registered in the conversation pattern DB 40, input by the administrator -Text data showing new question patterns created by editing the question text

[0115] After step S234, the server 30 acquires the input information (step S332). Specifically, the processor 32 of the server 30 acquires information about the new answer output from the administrator terminal 20.

[0116] After step S332, the server 30 registers a second response (step S333). Specifically, the processor 32 of the server 30 associates the re-answered answer sentence as an answer pattern, and the question sentence from the user (or text data edited from the question sentence) as a question pattern, with a new QA item having the name input by the user. Then, the re-answered answer sentence and the question sentence from the user are registered in the conversation pattern DB 40. As a result, a new answer pattern and question pattern are registered for the new QA item. After that, the automatic response model re-learns the new conversation pattern, so that the automatic response model recommends the new QA item for a similar question, and the content of the re-answered answer from the manager is automatically output as an answer sentence to the employee terminal 10 of the questioner.

[0117] After step S333, the server 30 outputs the re-answer information to the employee terminal 10 (step S334). Specifically, the processor 32 of the server 30 outputs the re-answer information to the employee terminal 10 in order to present it to the user. At this time, the information that the administrator has sent a re-answer may be sent to the user's contact information.

[0118] After step S334, the employee terminal 10 displays the re-answer (step S131). Specifically, the processor 12 of the employee terminal 10 presents the content of the re-answer to the user by displaying the re-answer on the display 15. This allows the user to check the re-answer and obtain an appropriate answer to the question he or she asked, thereby resolving the problem. With the above, the process from steps P20 to P21 in FIG. 6, which includes the escalation process, is completed.

[0119] (5) Screen examples Next, examples of screens of the system 1 will be described with reference to FIGS.

[0120] (5-1) First screen example P10 FIG. 16 is a diagram showing the first example screen P10 shown in FIG. As shown in Fig. 16, this screen displays a questionnaire regarding the content of the answer to the question that the user inquired about after the answer has been presented to the user and the user has confirmed the answer. First, the following options are displayed in questionnaire A1 shown at the top. -Selection indicating the problem has been resolved (positive evaluation) · Option indicating the problem was not resolved (negative evaluation) Here, if the problem is solved by the answer, this can be regarded as a positive evaluation that the answer was appropriate (Yes in step S305 in FIG. 12). On the other hand, if the problem is not solved by the answer, it can be regarded as a negative evaluation that the answer was inappropriate (No in step S305 in FIG. 12).

[0121] Next, the following options are displayed in survey A2 shown below. The answer recommendation is incorrect The answer is inappropriate Here, when the user selects "the recommended answer is wrong", this corresponds to a case where the QA item of the answer content included in the answer sentence is inappropriate (X in step S306 in FIG. 12). On the other hand, when the user selects "the content of the answer is inappropriate", this corresponds to a case where the content of the answer contained in the answer sentence is either excessive or insufficient (Y in step S306 in FIG. 12). In this way, by displaying the questionnaire on the employee terminal 10, it is possible to confirm the evaluation from the user.

[0122] (5-2) First screen example P11 FIG. 17 is a diagram showing the second example screen P11 shown in FIG. 17, the second screen example P11 displays a list L1 of QA items for which updating of information included in the answer text is recommended, which includes multiple categories and QA items. In the example shown, the titles of the major categories and QA items are displayed. List L1 displays the number of inquiries. The number of inquiries is a number indicating the number of questions about the Q&A item in a given period. List L1 may be displayed so that, for example, Q&A items with a large number of inquiries are at the top. When the administrator clicks on the edit button B1 in the list L1, an edit screen for the Q&A item shown in FIG. 11 is displayed, and the administrator can update the answer by editing it.

[0123] (5-3) First screen example P12 FIG. 18 is a diagram showing the third example screen P12 shown in FIG. 18, the third example screen P13 displays a list L2 of questions to be answered by the administrator as escalation processing, which includes multiple categories and Q&A items. In the example shown, the questions that are judged to require escalation processing are displayed. When the administrator clicks on the reply button B2 in the list L1, the fourth example screen P13 shown in FIG. 19 is displayed, and the administrator can input the reply.

[0124] (5-4) First screen example P13 FIG. 19 is a diagram showing the fourth example screen P13 shown in FIG. As shown in Fig. 19, the fourth screen example P13 displays a screen for confirming the contents of the administrator's reply. The administrator confirms the question that is the subject of the escalation process, and enters an appropriate reply in the reply field. At this time, image data, video data, document data, etc. may be attached. After inputting an appropriate response, the administrator clicks the "Reply to Inquiry" button B3 displayed at the bottom, which causes a fifth example screen P14 shown in FIG.

[0125] (5-5) First screen example P14 FIG. 20 is a diagram showing the fifth example screen P14 shown in FIG. As shown in FIG. 20, the fifth example screen P14 displays a check box B4 for allowing the administrator to select whether or not to register the content of the re-answer as a new answer candidate. The fifth example screen P14 also displays an input field B5 for specifying the question category to which the re-answer belongs. The input field B5 displays a selection field B6 for selecting a large category or a medium category as the category to which the re-answer belongs, and an input field B7 for inputting the name of the new Q&A item. After the user has completed the necessary input, he or she clicks the send button B8, and the input information is output to the server 30. Although not shown in the figure, the question text input by the user may be modified to an appropriate expression as a question pattern. This modification of the question text is optional and can be performed as necessary.

[0126] (6) Summary As described above, in the system 1 according to the present embodiment, after a user gives a negative evaluation to a response text by an automatic response, a question text is output to the administrator as an escalation process, and a second response to the question text is received from the administrator. The system 1 then registers the content of the second response as a new response candidate in the conversation pattern DB 40. This allows a smooth and appropriate response to be made even if a corresponding conversation pattern is not prepared. In addition, for new inquiries for which a response text has not yet been prepared, the response text from the administrator can be made available for subsequent answering by an automatic response.

[0127] Incidentally, in conventional chatbots for general consumers, the ratio of specific individual inquiries is low, and there is little motivation to train responses to specific inquiries because of the need for versatility that can be applied to a variety of situations. System 1 is particularly specialized in labor, so there is considerable merit in training it to train responses to specific inquiries.

[0128] Furthermore, in system 1, prior to the step of registering the content of the re-answer, an input of a choice as to whether or not to register the content of the re-answer as a new answer candidate is received from the administrator. This allows the administrator to choose not to newly register an answer that should not be generalized as a conversation pattern, such as when the answer contains personal information, thereby improving the convenience of maintenance by the administrator.

[0129] The system 1 also receives an input from the administrator specifying the question category to which the content of the re-answer belongs, thereby making it possible to register a new conversation pattern in an appropriate category according to its content.

[0130] Furthermore, in the step of registering the content of the re-answer, system 1 registers the content of the re-answer for the question category selected by the administrator. This makes it possible to smoothly manage the conversation pattern DB classified by categories with a hierarchical structure.

[0131] Furthermore, in the step of registering the content of the re-answer, the system 1 registers the question sentence for the selected question category, thereby making it possible to manage the answer sentence and the question sentence in association with each other for the question category.

[0132] Furthermore, the system 1 presents the content of the negative evaluation entered by the user to the administrator, thereby enabling the administrator to be accurately informed of the results of the action taken against the user. By being able to grasp the problematic aspects in a negative evaluation and by allowing the manager to understand the content of the negative evaluation, appropriate measures can be taken thereafter.

[0133] In addition, in response to a negative evaluation input indicating that the question category (QA item) of the answer content is inappropriate, the system 1 presents to the user a plurality of new answer sentences that the automatic response model has output as candidates to be answered next to the answer sentence. This allows the automatic response model to present a plurality of next candidate answer sentences to the user, thereby providing an appropriate answer.

[0134] In addition, in response to a user's input of an evaluation indicating that none of the newly presented multiple answer sentences includes an appropriate answer, the system 1 outputs the question sentence to the administrator, thereby urging the administrator to appropriately perform escalation processing.

[0135] (7) Variations The storage device 31 may exist outside the server 30 and be connected to the server 30 via a network NW. The conversation pattern information stored in the conversation pattern DB 40 may be stored in the storage device 31 of the server 30.

[0136] Each step of the above information processing can be executed by any of the employee terminal 10, the server 30, and the manager terminal 20. In the above description, an example was given in which each step in each process was executed in a specific order, but the execution order of each step is not limited to the example described above as long as there is no dependency between the steps. In other words, the order can be changed as desired as long as no contradiction occurs in the process.

[0137] The content of the categories in which the above-mentioned conversation patterns are stored can be changed as desired. That is, the administrator can set the major categories, minor categories, and Q&A items as desired.

[0138] The system 1 may have other functions. For example, the system 1 may automatically select a person in charge of re-answering a question that is the target of escalation processing. In this function, the processor 32 of the server 30 sets a highly relevant department according to the contents of the QA item recommended by the automatic response model. Then, by referring to the employee DB, the server 30 selects a person in charge of answering set as a point of contact for inquiries in the department. The server 30 sends a notification requesting a re-answer to the contact information of the selected person in charge of answering as a person in charge of re-answering in the escalation processing. The person in charge of answering transmits the contents of the re-answered answer to the administrator. The administrator checks the contents of the answer from the person in charge of answering and performs the processes from step S233 in FIG. 15 onward, in which the administrator checks the contents of the answer from the person in charge of answering and selects whether to add it as a new QA item. With this function, even if the inquiry contains highly specialized content, an appropriate department staff member can be selected as the person to respond, making it possible to smoothly respond to the user's inquiry.

[0139] In addition, when a second response by an administrator is registered in a new QA item, the major and minor categories to which the QA item belongs are not limited to existing categories. In other words, when registering a new QA item, a new major or medium category may be created and the new QA item may be registered in that category. For example, a major or medium category dedicated to second responses, to which only the QA items to which the content of the second response is linked belong, may be provided separately from the existing major and medium categories.

[0140] Although the embodiment of the present invention has been described in detail above, the scope of the present invention is not limited to the above embodiment. Furthermore, the above embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above embodiment and the modified examples can be combined. In addition, the above-described embodiment and modified examples may be partially omitted. For example, when a negative evaluation of an answer is received from a user who is a questioner, escalation processing may be performed without checking the reason.

[0141] (7) Additional Notes The contents of the present invention are described below.

[0142] (Appendix 1) A program for use in a computer that answers questions from a user through an automatic response, comprising: The computer processor A step of accepting input of a question sentence from a user (step S101); A step of causing the trained model to output an appropriate answer sentence as an answer to the input question sentence from among a plurality of answer candidates registered in advance in a database, and presenting the output answer sentence to the user (step S102); A step of accepting an input of an evaluation of the answer sentence from a user (step S103); a step of presenting the question sentence to the administrator when a negative evaluation is input from the user to the answer sentence (step S231); A step of accepting input of a second answer to the question from the administrator (step S232); and a step of registering the content of the re-answer in the database as a new answer candidate (step S333).

[0143] (Appendix 2) The processor further comprises: Prior to the step of registering the content of the re-answer in the database (step S333), The program according to claim 1, which executes a step (step S233) of accepting, from the administrator, a selection input of whether or not to register the content of the re-answer as a new answer candidate.

[0144] (Appendix 3) In the step of accepting the selection input (step S233), 3. The program according to claim 1 or 2, which accepts an input specifying a question category to which the content of the re-answer belongs.

[0145] (Appendix 4) In the step of registering the content of the re-answer in the database (step S333), A program according to any one of appendices 1 to 3, which registers the content of the re-answer in association with the selected question category.

[0146] (Appendix 5) In the step of registering the content of the re-answer in the database (step S333), A program according to any one of appendices 1 to 4, which links the question text with the selected question category, edits the question text as necessary, and then registers the question text.

[0147] (Appendix 6) The evaluation of the answer text is as follows: Negative evaluations point out that the answer contains insufficient or excessive information. and a negative evaluation indicating that the question category of the answer content included in the answer text is inappropriate.

[0148] (Appendix 7) In the step of presenting a question to an administrator (step S231), 7. The program of claim 6, which presents to an administrator the contents of the negative evaluation entered by the user.

[0149] (Appendix 8) A program as described in Appendix 7, in which, in a step of accepting input of an evaluation of the answer sentence (step S103), in response to input of a negative evaluation indicating that the question category of the answer content is inappropriate, the program presents to the user a number of new answer sentences that the trained model has output as candidates to be answered next to the answer sentence.

[0150] (Appendix 9) The program described in Appendix 8, which executes a step (step S231) of presenting a question sentence to an administrator in response to a user inputting an evaluation indicating that none of the newly presented multiple answer sentences contains an appropriate answer.

[0151] (Appendix 10) The program described in Appendix 8, which executes a step (step S324) of registering the question sentence in the database as a new question pattern corresponding to an appropriate answer in response to a user inputting an evaluation that the newly presented multiple answer sentences contain an appropriate answer.

[0152] (Appendix 11) 1. A method for use in a computer to answer user queries through an automated response, comprising: The computer processor A step of accepting input of a question sentence from a user (step S101); A step of causing the trained model to output an appropriate answer sentence as an answer to the input question sentence from among a plurality of answer candidates registered in advance in a database, and presenting the output answer sentence to the user (step S102); A step of accepting an input of an evaluation of the answer sentence from a user (step S103); a step of presenting the question sentence to the administrator when a negative evaluation is input from the user to the answer sentence (step S231); A step of accepting input of a second answer to the question from the administrator (step S232); A step of registering the content of the re-answer in the database as a new answer candidate (step S333).

[0153] (Appendix 12) An automatic response system having a computer that answers questions from a user by automatic response, The computer processor is A means for accepting input of a question sentence from a user; A means for causing the trained model to output an appropriate answer sentence as an answer to the input question sentence from among a plurality of answer candidates registered in advance in a database, and presenting the output answer sentence to a user; A means for receiving an input of an evaluation of the answer sentence from a user; a means for presenting the question text to an administrator when a negative evaluation is input by the user in response to the answer text; A means for accepting re-entry of answers to the questions from the administrator; A system having a means for registering the content of the re-answer in a database as a new answer candidate. [Explanation of symbols]

[0154] 1: Information processing system 10: Terminal 11:Storage device 12: Processor 13: Input / Output Interface 14: Communication interface 20: Administrator terminal 21:Storage device 22: Processor 23: Input / Output Interface 24: Communication interface 30: Server 31:Storage device 32 : Processor 33: Input / Output Interface 34: Communication interface

Claims

1. A program used for a computer that answers questions from users through automatic responses, wherein the processor of the computer executes steps of receiving an input of a question text from a user, causing an answer text appropriate as an answer to the input question text to be output to a trained model including a self-attention mechanism network for natural language, and presenting the output answer text to the user, and receiving an input of an evaluation of the answer text from the user.

2. The user includes employees of a company, and the question from the user is an inquiry from the user to the management department of the company. The program according to Claim 1.

3. In the step of receiving an input of an evaluation of the answer text, when a negative evaluation is input from the user for the answer text, a new answer text output by the trained model as a candidate for the next answer to the answer text is presented to the user. The program according to Claim 1.

4. The evaluation of the answer text includes a negative evaluation indicating that the answer content included in the answer text is inappropriate. The program according to Claim 1.

5. An automatic response system having a computer that answers questions from an employee terminal, wherein the employee terminal includes a communication unit that transmits a question text to the computer and an input unit that receives an input of an evaluation of an answer text to the question corresponding to the question text, and the computer includes an output unit that causes an answer text appropriate as an answer to the input question text to be output to a trained model including a self-attention mechanism network for natural language, and a presentation unit that presents the output answer text to the employee terminal. An automatic response system.

6. When a negative evaluation is input from an employee using the employee terminal for the answer text, the presentation unit causes the employee terminal to newly present an answer text output by the trained model as a candidate for the next answer to the answer text. The automatic response system according to Claim 5.

7. The evaluation of the answer text includes a negative evaluation indicating that the answer content included in the answer text is inappropriate. The automatic response system according to Claim 5.

8. The automatic response system according to claim 5, wherein the question from the employee terminal includes content related to the internal rules of the company to which the employee using the employee terminal belongs.

Images