Dialog processing apparatus and dialog processing method

Abstract

To improve the efficiency of office work by reducing a flow from manual consultation to reexamination.SOLUTION: An interactive processor 10 for executing interactive processing by using a model obtained by learning questions and answers acquires a primary answer to an inputted question by using the model by a primary answer generation and primary determination part 19, executes an audit from a plurality of different audit viewpoints to the acquired primary answer by an audit and evaluation part 22, and determines the necessity of re-observation by consultation by a re-observation determination part 25.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to, for example, an interaction processing device and an interaction processing method. [Background technology]

[0002] Financial institutions are now adopting a system whereby responses to customer inquiries are audited from multiple perspectives, i.e., through a panel of experts. If an appropriate response is obtained, the bank will proceed to a re-examination process.

[0003] Nowadays, non-face-to-face dialogue systems such as chatbots are becoming popular, and a system that answers questions using a learning model has been proposed (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-86817 Summary of the Invention [Problem to be solved by the invention]

[0005] In dialogue systems for financial institutions, high accuracy of responses is particularly required, and it is desirable to be able to reduce the process from consultation to re-examination even if only a little.The realization of such a system is anticipated.

[0006] The present invention has been made in consideration of the above background, and aims to provide a dialogue processing device and a dialogue processing method that can reduce the manual process from consultation to re-consideration, thereby improving the efficiency of administrative work. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the above-mentioned objectives, one embodiment of the present invention is a dialogue processing device that performs dialogue processing using a model that has learned questions and answers, and is characterized by comprising: a primary response unit that obtains a primary response to an input question using the model; an audit unit that performs audits from multiple different audit perspectives on the primary response obtained by the primary response unit; and a re-supervision determination unit that holds a meeting based on the audit results of the audit unit to determine whether re-supervision is necessary.

[0008] Another embodiment of the present invention is an interaction processing method for a device that performs interaction processing using a model that has learned questions and answers, and is characterized by including a primary response step in which a primary response is obtained using the model for an input question, an audit step in which an audit is performed from a plurality of different audit perspectives on the primary response obtained in the primary response step, and a re-supervision determination step in which a discussion is held based on the audit results of the audit step to determine whether re-supervision is necessary. [Effects of the Invention]

[0009] According to the present invention, it is possible to reduce the manual process from consultation to re-examination, thereby improving the efficiency of administrative work. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing the functions of an interaction processing device according to an embodiment of the present invention. [Figure 2] 1 is a configuration diagram showing a hardware configuration of an interaction processing device according to an embodiment of the present invention; [Figure 3] FIG. 1 is an explanatory diagram illustrating a QATBL according to one embodiment of the present invention. [Figure 4] FIG. 10 is an explanatory diagram illustrating a method for calculating a collegial score according to an embodiment of the present invention. [Figure 5] FIG. 10 is an explanatory diagram illustrating a method for improving the accuracy of a collegial score according to an embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating a process for improving the accuracy of a collegial score according to an embodiment of the present invention. [Figure 7] FIG. 1 is an explanatory diagram illustrating an example of a flow from a question to obtaining an answer according to an embodiment of the present invention. [Figure 8] FIG. 10 is an explanatory diagram illustrating score calculation according to an embodiment of the present invention. [Figure 9] FIG. 10 is an explanatory diagram illustrating the flow up to re-supervision of the dialogue processing according to one embodiment of the present invention. [Figure 10] FIG. 2 is a diagram illustrating different audits according to one embodiment of the present invention. [Figure 11] FIG. 1 is an explanatory diagram illustrating an audit procedure according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the following description and drawings are merely examples for explaining the present invention, and some omissions and simplifications have been made as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Furthermore, unless otherwise specified, each component may be singular or plural.

[0012] In the following description, identical or similar components may be designated by the same reference numerals, and redundant description may be omitted. Furthermore, in the following description, various types of information may be described using expressions such as "information" and "table," but the various types of information may also be expressed using other data structures. Furthermore, identification information may be expressed using expressions such as "identification information," "identifier," "name," "ID," and "number," but these are interchangeable. Furthermore, in the following description, "database" will be referred to as "DB," and "table" as "TBL."

[0013] First, the dialogue processing device will be described with reference to Figures 1 and 2. Figure 1 is a block diagram showing the functions of the dialogue processing device according to this embodiment, Figure 2 is a configuration diagram showing the hardware configuration of the dialogue processing device according to this embodiment, and Figure 3 is an explanatory diagram explaining the QATBL according to this embodiment.

[0014] In this embodiment, an example of an interaction processing device 10 that is a server of a financial institution such as a bank realizes interaction processing with a customer terminal 50 accessed by a customer via a network 200 will be given.

[0015] Of course, the dialogue processing device 10 can also be applied when installed in a branch of a financial institution, in which case it will also be equipped with a keyboard (software or hardware) for customers to operate and input, and a display for displaying information on the screen.

[0016] The functions of the dialogue processing device 10 are, for example, as shown in Figure 1, composed of a communication control unit 11, a user authentication unit 12, a memory unit 13, a primary answer generation and primary judgment unit 19 which is a primary answer unit, an answer generation unit 20, an answer proposal creation unit 21, an audit and evaluation unit 22, an answer generation LLM 23, an evaluation LLM 24, a reassessment unit 25, an improvement unit 26, etc.

[0017] The communication control unit 11 communicates with external devices such as a customer terminal 50 or an operator terminal 70 operated by a user (a customer or an operator of a financial institution) via a network 200. The user authentication unit 12 authenticates a user who accesses using the customer terminal 50 or the operator terminal 70.

[0018] The storage unit 13 stores a user management TBL 14, a primary determination rule 15, a knowledge DB 16, a QATBL 17, an audit DB 18, etc. Although not shown, the user management TBL 14 is configured to refer to the user management TBL 14 and store information identifying a customer, transaction information with the customer, a history of past interactions with the customer, etc., in association with a customer ID.

[0019] The primary determination rules 15, which will be described later, are information for primary determination of primary answers generated in response to questions from customers. The knowledge DB 16 is configured to register knowledge data for answers to questions in advance through an operator's operation, via an administrative manual such as a PDF created in text format.

[0020] QATBL 17 is configured to store customer questions and answers in association with each other, as shown in Figure 3. In the example of Figure 3, the answer to the question "Can I open a second account?" is "I already have a regular account...no." The consensus score for this answer is 5.0. Audit DB 18 is configured so that review perspectives and weighting coefficients for each review can be input and set in advance by an operator for answers to questions.

[0021] The primary answer generation and primary judgment unit 19 generates a primary answer and makes a primary judgment by referring to the primary judgment rules 15. The answer generation unit 20 generates an answer by referring to the knowledge DB 16 and using the answer generation LLM 23 based on the judgment result of the primary answer.

[0022] The answer proposal creation unit 21 refers to the QATBL 17 and creates an answer proposal based on the answer generated by the answer generation unit 20. The audit and evaluation unit 22 refers to the audit DB 18 and evaluation LLM 24 of audit standards prepared in advance for a plurality of different departments, performs a plurality of different audits, and calculates a score as an evaluation.

[0023] The answer generation LLM 23 is an answer generation model for generating answers that is improved through learning and fine tuning. The evaluation LLM 24 is a model that supports evaluation to calculate a score in the audit and evaluation unit 22.

[0024] The re-evaluation unit 25 uses the evaluation results of the audit and evaluation unit 22 to instruct the improvement unit 26 to fine-tune if the audit results determined by the panel of auditors through a full audit meet the predetermined criteria, and returns a response to the customer terminal 50 via the communication control unit 11. Furthermore, the re-evaluation unit 25 instructs the response generation unit 20 to generate a response again if the above criteria are not met. The improvement unit 26 fine-tunes the regeneratively generated LLM 23 in accordance with the instructions of the re-evaluation unit 25.

[0025] As shown in FIG. 2, the dialogue processing device 10 of this embodiment is composed of a CPU 101 that controls the entire device, a memory 103 that stores programs 102 for processing according to this embodiment to be executed by the CPU 101 (such as the operations of FIGS. 6, 9, and 11, which will be described later) and stores various data during execution, an operation device 104 that has a keyboard and a display device, an external storage device 105 that registers (stores) the memory unit 13 in the form of a DB, TBL, etc., and stores and manages the answer generation LLM 23 and the evaluation LLM 24 in a model format that is operated by a program, a communication IF 106 that is connected to a network 200 as a communication control unit 11 and controls communication with external devices, and a bus 107 that is connected to each unit within the device and controls communication of data, signals, etc. within the device.

[0026] It goes without saying that the answer generation LLM 23 and the evaluation LLM 24 may be configured as a single LLM, or may be configured independently from the external storage device 105 as different functions.

[0027] Next, the collegial score will be explained using Figures 4, 5, and 6. Figure 4 is an explanatory diagram illustrating a method for calculating the collegial score according to this embodiment, Figure 5 is an explanatory diagram illustrating a method for improving the accuracy of the collegial score according to this embodiment, and Figure 6 is a flowchart illustrating the process for improving the accuracy of the collegial score according to this embodiment.

[0028] In the dialogue processing device 10, in calculating the collegial score, the procedure is carried out in the order of a question phase, an answer generation phase, an answer proposal creation phase, a check phase, an evaluation phase, and a collegial phase, as shown in FIG.

[0029] When a question is posed from the customer terminal 50 in the question phase, the knowledge DB 16 is referenced in the answer generation phase, and an answer is generated by the answer generation LLM 23. In the answer proposal creation phase, the QATBL 17 is referenced, and an answer proposal is created based on the generated answer.

[0030] In the following check phase, the audit DB 18 that stores answer-specific view checklists is referenced, and checks are carried out using checklists appropriate for each audit department. For example, if there are three audit departments, A, B, and C, checks are carried out using checklists A, B, and C appropriate for each department, as shown in Figure 4.

[0031] Then, in the evaluation phase, the evaluation LLM24 is referenced, an evaluation corresponding to the results of each check is performed, and each evaluation result is normalized and output in the form of a score. The evaluation result of checklist A might be a score of "7.0," the evaluation result of checklist B might be a score of "2.0," and the evaluation result of checklist C might be a score of "4.0." In the deliberation phase, a deliberation is held based on the normalized evaluation results (scores), and the deliberation result might be a score of "5.0," for example.

[0032] Here, the answer generation LLM23 learns answers that are in line with the contents of the checklist, improving accuracy, as shown in Figure 5. At that time, answers with high scores in the discussion are referenced in QATBL17, and a periodic execution batch for fine-tuning is processed.

[0033] Now, to be more specific about the processing, as shown in FIG. 6, the primary answer generation and primary judgment unit 19, the answer generation unit 20, and the answer proposal creation unit 21 generate a primary answer using RAG (Retrieval-Augmented Generation) by referring to the knowledge DB 16 (step S61), and then the audit and evaluation unit 22 calculates an audit score corresponding to each audit department (step S62).

[0034] Then, the set of question, answer, and audit score is registered (stored) in the QATBL 17 (step S63), and combinations of questions and answers having audit scores (good accuracy) equal to or greater than a predetermined threshold are extracted from the QATBL 17 (step S64). Using the data of the combinations extracted in this way, fine tuning is performed on the answer generation LLM 23 to improve it (step S65). It is preferable to perform this fine tuning periodically once a certain amount of data has been accumulated, as this will increase accuracy.

[0035] Next, the operation of the dialogue processing will be described with reference to Fig. 7 to Fig. 11. Fig. 7 is an explanatory diagram illustrating an example of the flow from asking a question to obtaining an answer according to this embodiment, Fig. 8 is an explanatory diagram illustrating score calculation according to this embodiment, Fig. 9 is an explanatory diagram illustrating the flow of the dialogue processing up to re-monitoring according to this embodiment, Fig. 10 is an explanatory diagram illustrating a different audit according to this embodiment, and Fig. 11 is an explanatory diagram illustrating the audit procedure according to this embodiment.

[0036] Here, the operation at the time of initial registration will be explained using Figure 7. In this figure, "Q", "A", and "S" shown in square frames represent a question from a customer, an answer from the answer generation LLM, and an evaluation LLM, respectively.

[0037] On the display of the customer terminal 50, for example, a message such as "Please ask a bank employee" is displayed in the explanation field, and an operable inquiry button is displayed, allowing the customer to enter the question text and ask a question ((1) in the same figure).

[0038] In the case of voice input, the dialogue processing device 10 converts voice to text, and the question is input in text form ((2) in the figure). In other words, in this case, it is assumed that a device for converting voice to text is provided, although not shown.

[0039] The knowledge DB 16, which is a vector DB, is configured by vectorizing business manuals in file formats such as PDF. The dialogue processing device 10 also plays the role of a query server, referring to the knowledge DB 16 in response to a question (Q) ((3) in the figure), generating a response using the answer generation LLM 23, and returning it to the query server ((4) in the figure).

[0040] The inquiry server requests an audit from the evaluation LLM 24, which performs audits and evaluations by referring to the audit DB 18 ((5) in the figure). The evaluation LLM 24 obtains the audit score and collegial score, which are the evaluation results corresponding to each audit department, and returns them to the inquiry server ((6) in the figure).

[0041] The inquiry server registers the audit score, the collaborating score, and the question and answer combination in QATBL17 ((7) in the figure), and then proceeds to re-supervision, which is a manual process, for question and answer combinations whose collaborating score is lower than a pre-defined threshold. These re-supervision cases are registered in QATBL17 ((8) in the figure).

[0042] If the consensus score is equal to or greater than the threshold, the inquiry server returns the answer as is to the customer terminal 50 ((10) in the figure). The inquiry server also executes a similarity search ((9) in the figure).

[0043] Here, (5) Audit Request and (6) Audit Score in FIG. 7 will be described with reference to FIG. 8. First, a system prompt is entered in advance. In this embodiment, the system prompt may set a framework for specializing in finance, define ethical standards, or set links to specific knowledge, data sets, reference materials, etc., in order to improve the accuracy of answers to questions.

[0044] Then, in (5) Audit Request shown in Figure 7, the customer answer proposal is input from the answer generation LLM 23, and a normalized score is calculated. Then, in (6) Audit Score, the normalized audit score is returned.

[0045] To specify the processing for the audit request in (5) above, the review result response is set to "0" if there is a problem, and "1" if there is no problem, as shown in Figure 8. Then, the audit result for each review perspective is calculated, and further, the reliability for each review perspective is calculated.

[0046] Next, an audit score for each review perspective is calculated. This audit score is obtained by multiplying the review result, confidence level, and weighting coefficient. Since an audit score is obtained for each review perspective, the total is divided by the maximum audit score to output a normalized audit score, and (6) the audit score is returned.

[0047] Next, re-supervision will be explained using Fig. 9. When the dialogue processing device 10 receives a question from a customer through the customer terminal 50, the dialogue processing device 10 derives an answer through the orchestrator processing, primary search processing, audit processing, processing of the answer generation LLM 23 and evaluation LLM 24, and QA processing of the QATBL 17, and responds with the answer. In addition, processing related to re-supervision is executed by operating the operator terminal 70.

[0048] When a customer posts a question via the customer terminal 50, the knowledge DB 16 is referenced, knowledge data related to the question is searched, and the search results are returned to the orchestrator. The orchestrator then passes the question and the search results together to the answer generation LLM 23, and obtains the answer generated by the answer generation LLM 23 in text format.

[0049] Then, in the collaborative process (step S91), the orchestrator first requests the audit and evaluation unit 22 to audit each audit department, and the audit and evaluation unit 22 then carries out the audit in accordance with the audit content registered in the audit DB 18. Regarding the audit content, for example, in the example of Fig. 10, five types of audit functions are prepared from multiple different perspectives. The five types of audit functions are quality control, training and feedback, policy and process audit, customer satisfaction, and ensuring compliance.

[0050] In the case of quality control, the review content is "Monitor the quality of the services provided by the support team and register review points for continuous improvement." The review point is "Is the language too colloquial? Is the content too specific?" This quality control is assigned an audit score of A, with a score of 0.6.

[0051] In the case of training and feedback, the review content is "Register the regular training content and feedback content for staff as review points." The review point is "Is the amount of text appropriate? Does it follow the template?" This training and feedback is assigned an audit score of B, 0.9.

[0052] In the case of a policy and process audit, the review content is "Register customer support policy as the review perspective." The review perspective is "Are responses being provided from the customer's perspective?" This policy and process audit is assigned an audit score of C, with a score of 0.3.

[0053] In the case of customer satisfaction, the review content is "Register the review viewpoint based on customer feedback." The review viewpoint is "Are proper nouns used in the text easy to understand?" This customer satisfaction audit is assigned an audit score of D, with a score of 0.5.

[0054] In the case of ensuring compliance, the review content is "Register the contents of laws and regulations and internal company rules as review viewpoints." The review viewpoint is also to register the contents of laws and regulations and internal company rules. This compliance audit is associated with audit score E, which is assigned a score of 0.2.

[0055] In this way, when the orchestrator obtains the audit score, it outputs the previously obtained answer and the audit score to be obtained later to processing using QATBL 17. If the audit score falls below a predetermined threshold (stored in the memory unit 13), an artificial re-supervision is performed, in which case the answer as is is not permitted, i.e., a non-permitted answer permission flag (NG) is returned to the orchestrator. Also, if the answer is above the threshold, an answer permission flag (OK) is returned to the orchestrator to permit the answer as is.

[0056] Specifically, a vector approximation search is performed by QATBL17, which has already registered the answer as a query, and if the approximation value is less than or equal to a threshold, a new answer (text) is registered.On the other hand, if an approximation value greater than or equal to the threshold is found and the audit score is greater than or equal to the threshold, an answer permission flag (OK) is returned.

[0057] Then, in the re-supervision process (step S94), the QATBL 17 is referenced, and newly registered answers with audit scores lower than the threshold are returned to the operator terminal 60 as question-answer combinations (step S95), and the operator manually performs re-supervision on the answers with the lower audit scores. Then, when the re-supervision is completed, the results of the re-supervision are registered for the corresponding data in the QATBL 17.

[0058] 11 shows the specific processing of the audit in FIG. 9 (step S92). For example, assume that the audit requires audits by audit departments A, B, and C. When the audit and evaluation unit 22 receives an audit request from the orchestrator, it performs each audit by audit departments A, B, and C and finally returns an audit score.

[0059] The audit and evaluation unit 22 first executes a search in the audit DB 18 in response to the prompt to perform an audit for audit department A. As a result, a review perspective is acquired as a response from the audit department A table (for example, Key-Value type) of quality control review contents and perspectives shown in Fig. 10.

[0060] The audit and evaluation unit 22 requests the answer generation LLM 23 to perform a review using the obtained answers, prompts, and review perspectives, and obtains a score for each review perspective and a reliability for each review perspective from the answer generation LLM 23 as a review result.

[0061] The audit and evaluation unit 22 then calculates and normalizes the audit score for each review perspective by multiplying the review result, the reliability, and the weighting coefficient.

[0062] Similarly, the process up to normalization is also carried out for audit department B, but in this example, the sequence for audit department C is omitted. Review results (the sum of the score for each review perspective and the reliability for each review perspective) are obtained for each department that requires consultation.

[0063] In this way, once the audit and evaluation unit 22 has completed the normalization of the necessary audit departments, it calculates an overall score from the scores and weightings of each audit department and performs normalization (multiplying the audit results by the weightings). The final normalized score is also a consensus score and is returned to the orchestrator as the audit score.

[0064] As described above, according to this embodiment, it is possible to reduce the manual process from consultation to re-viewing, thereby improving the efficiency of administrative work.

[0065] Since re-supervision generally increases the reliability and transparency of companies and financial institutions, the mitigation measures against re-supervision in this embodiment will also be effective for insurance companies in addition to banks.

[0066] Furthermore, the above-described configurations, functional units, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), an IC card, an SD card, a DVD, or other recording media.

[0067] Furthermore, the above-described layout of the various functional units, processing units, and databases of the dialogue processing device 10 is merely an example. The layout of the various functional units, processing units, and databases can be changed to an optimal layout in terms of the performance, processing efficiency, communication efficiency, etc. of the hardware and software that these devices are equipped with.

[0068] Furthermore, the configuration (schema, etc.) of the database that stores the various types of data described above can be flexibly changed from the perspective of efficient use of resources, improved processing efficiency, improved access efficiency, improved search efficiency, and the like. [Explanation of symbols]

[0069] 10 Dialogue processing device 11 Communication control section 12 User authentication section 13 Storage section 14 User Management TBL 15 Primary Judgment Rules 16 Knowledge DB 17 QATBL 18 Audit DB 19 Primary answer generation and primary judgment section 20 Answer generation part 21 Answer Draft Creation Department 22 Audit and Evaluation Department 23 Answer Generation LLM 24 Rating LLM 25 Reexamination Department 26 Improvement Department 101 CPU that controls the entire device 102 Programs 103 memory 104 Operating Devices 105 External storage device 106 Communication Interface 107 Bus

Claims

1. A dialogue processing device that executes dialogue processing using a model that has learned questions and answers, a primary answering unit that obtains a primary answer to an input question using the model; an auditing unit that executes audits from a plurality of different auditing perspectives on the primary response obtained by the primary response unit; A re-inspection determination unit that determines whether re-inspection is necessary by conducting a consultation based on the inspection results of the inspection unit; 1. A dialogue processing device comprising:

2. 2. The dialogue processing apparatus according to claim 1, wherein the auditing unit calculates a score based on the audit from each of the audit viewpoints.

3. 3. The dialogue processing apparatus according to claim 2, wherein the auditing unit calculates a score for the collegial body based on the calculated score.

4. 4. The dialogue processing device according to claim 3, wherein the re-supervision determination unit determines whether re-supervision is necessary based on the score of the collegial body.

5. 4. The dialogue processing apparatus according to claim 3, further comprising an improvement unit that improves the model based on the scores corresponding to the respective audit viewpoints calculated by the audit unit.

6. 6. The dialogue processing device according to claim 5, wherein the improvement unit prepares a threshold value in advance and extracts combinations of questions and answers that result in a score equal to or higher than the threshold value to improve the model.

7. 6. The dialogue processing apparatus according to claim 5, wherein said improving section is a fine tuning section.

8. 8. The dialogue processing apparatus according to claim 7, wherein said improvement section further performs fine tuning periodically.

9. 2. The dialogue processing device according to claim 1, further comprising: when the re-supervision determination unit obtains a determination result indicating that re-supervision is required, the re-supervision determination unit outputs a message indicating that re-supervision is required.

10. A dialogue processing method for a device that executes dialogue processing using a model that has learned questions and answers, comprising: a primary answer step of obtaining a primary answer to an input question using the model; an audit step of executing an audit from a plurality of different audit viewpoints on the primary response obtained in the primary response step; A re-supervision determination step in which a consultation is carried out based on the audit results of the audit step to determine whether re-supervision is necessary; 10. An interactive processing method comprising:

Images