Apparatus, method and program for improving automatic response data

Abstract

The present invention provides a highly convenient automatic response data improving device, method, and program that can effectively present to a user information that is the basis for determining learning data suitable for input. [Solution] In order to improve the answer accuracy of an automatic answering device 2, the device analyzes and presents information that serves as a clue to learning data that is recommended to be input, and is composed of a user evaluation means 11, an answer evaluation means 12, an evaluation data analysis means 13, a topic estimation means 14, an answer evaluation display means 15, a simulation means 16, and a topic correction means 17. The user evaluation means 11 accepts user evaluations. The answer evaluation means 12 generates answer evaluation data that evaluates the quality of automatic answers. The topic estimation means 14 performs natural language processing and classifies into categories using topic modeling. The evaluation data analysis means 13 analyzes the quality of automatic answers for each category.

Description

[Technical field]

[0001] The present invention relates to a technology for improving the accuracy of responses from a device that automatically responds to questions from users by using pre-input learning data and generation AI. [Background technology]

[0002] In recent years, systems that automatically respond to inputs such as questions from users using a program have become widespread (see, for example, Patent Document 1). While some applications use such systems to convey information about the company to external parties such as customers, they are also used by companies and the like to share information about their internal agreements and the products and services they provide within their organizations. In such automatic response systems, there is a demand for improved response accuracy.

[0003] In addition, services using generative AI are rapidly increasing, and large-scale language models (LLMs) are rapidly advancing as AI models for text. Large-scale language models are natural language processing models built using huge amounts of text data and deep learning technology. Large-scale language models are also used in automatic response systems, and companies and other organizations can operate automatic response systems by inputting large amounts of text data they own into large-scale language models.

[0004] However, the data input into large-scale language models is entered manually, and it is not clear in advance what data and how much should be input. This means that problems exist such as not being able to obtain sufficiently accurate answers because the necessary data is not input. Therefore, currently, the provider of the automatic response system consults the user about the learning data suitable for input. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2016-99968 A Summary of the Invention [Problem to be solved by the invention]

[0006] In view of this situation, the present invention aims to provide a highly convenient automatic response data improvement device, method, and program that can effectively present to the user information that serves as the basis for determining learning data suitable for input. [Means for solving the problem]

[0007] In order to solve the above problems, the automatic response data improvement device of the present invention is a device that improves a device that automatically responds to questions from users by using pre-input learning data and a generation AI, and is equipped with a user evaluation means that accepts user evaluations regarding answer data and stores them in a database as user evaluation data, an answer evaluation means that generates answer evaluation data that evaluates the quality of an automatic answer using the question data, the answer data, and the user evaluation data, a topic estimation means that performs natural language processing on the question data and answer data, classifies them into categories using topic modeling, and stores them in the database, and an evaluation data analysis means that uses the answer evaluation data and the categories to analyze the quality of the automatic answer at least for each category. By evaluating the quality of the automatic response based on the user's evaluation and classifying and analyzing the results into categories, it is possible to effectively analyze information that is useful for improving the automatic response data. The analyzed data may be displayed visually on the display of the terminal used by the user, or may be output as audio or the like. The term "automatic response data" used here includes both question data and answer data to the question data. In other words, the purpose is not only to improve the response performance of the automatic response device, but also to improve the quality of question data input by users. The user evaluation received by the user evaluation means may be an evaluation such as "Good" or "Bad" or a graded evaluation such as a five-level evaluation. Also, a configuration may be adopted in which free text input is received and the text is analyzed using natural language processing technology. The topic modeling used in the topic estimation means is not limited to one, and multiple topic modeling may be used to automatically extract optimal topics from the results of the algorithms of each topic modeling.

[0008] The automatic response device to which the automatic response data improvement device of the present invention is applied broadly includes devices that automatically respond to questions from users using pre-input learning data and a generation AI, and is preferably an automatic response device using RAG (Retrieval-Augmented Generation) technology, but may be configured to perform fine tuning, for example. That is, in the case of an automatic response device using RAG technology, the pre-input learning data means learning data pre-input to the automatic response device. In addition, in the case of an automatic response device that performs fine tuning, the pre-input learning data means learning data that has been re-learned by the generation AI. A wide range of data such as text, images, and voice can be input as learning data. As for generative AI, a wide range of generative AIs can be used that utilize well-known large-scale language models (LLMs) such as "BERT" (Bidirectional Encoder Representations from Transformers) and "GPT" (Generative Pre-trained Transformer).

[0009] In the automatic response data improvement device of the present invention, the evaluation data analysis means may have a learning data analysis means for analyzing learning data that is recommended to be input. By having the learning data analysis means, it becomes possible to specifically present data that is preferable to be input as learning data to the user.

[0010] In the automatic response data improvement device of the present invention, the evaluation data analysis means may have a question data analysis means for analyzing question data that is recommended to be input. By having the question data analysis means, it becomes possible to specifically present data that is preferable to be input as question data.

[0011] The automatic response data improvement device of the present invention preferably further comprises an answer evaluation display means for visually displaying on the display at least one of the answer success rate for each category during a predetermined period and a list of questions for which the answer success rate is below a predetermined threshold. By visually displaying the answer success rate for each category during a predetermined period, the user can check at a glance which category of learning data should be input, improving convenience. In addition, a graph of the change in the answer success rate compared to the past and the answer success rate for each number of questions may be displayed. By visually displaying a list of questions whose answer success rate is below a predetermined threshold, users can easily check which specific questions are likely to be answered incorrectly, thereby improving convenience.

[0012] The automatic response data improvement device of the present invention preferably further comprises a simulation means having a simulation data input means for receiving input of learning data for simulation, a question content selection means for selecting from a list a question content for which the answer success rate is below a predetermined threshold, and a preview means for displaying in parallel the answer content related to the answer data before and after the input of the simulation data. By providing the simulation means, it becomes easier to realize the effect of improving answering performance by inputting appropriate learning data for questions with a low answer success rate, in other words, questions that are easy to fail to answer. This can increase the motivation of users to improve the automatic response data.

[0013] It is preferable that the automatic response data improvement device of the present invention further comprises a topic correction means for accepting a correction of the category classification by the topic estimation means, and stores the corrected category classification in the database in association with the question data and the answer data. By providing the topic correction means, a user can manually correct the category classification estimated by the topic estimation means. Furthermore, by storing data on the corrected category classification in the database, it can be used to improve the accuracy of the topic estimation means.

[0014] In the automatic response data improvement device of the present invention, the answer evaluation means may include a relevance determination means for extracting a question immediately before or immediately after the question based on the user information and question date and time contained in the question data, and determining the relevance between the question and the extracted immediately before or immediately after question based on at least one of the question content contained in the question data and the answer content contained in the answer data, and may use the user evaluation data of the immediately before or immediately after question that is confirmed to be relevant to the question for the answer evaluation of the question. Generally, questions asked in close time by the same user are often related questions. Therefore, by using the user evaluation data on the question immediately before or immediately after the question for the answer evaluation of the question, it is possible to perform a more accurate evaluation. In addition, for example, the immediately before or immediately after question may be extracted for questions asked by users who are not the same user but are involved in the same or similar departments or work. The relevance determination based on the question data and the answer data is performed by referring to the similarity of keywords extracted using natural language processing, for example.

[0015] In the automatic response data improvement device of the present invention, the answer evaluation means may include a grouping means for grouping a plurality of questions in a question set based on at least one of the user information, question date and time, question content, and answer content in the answer data, and may evaluate the answers based on at least one of the type, number, and order of user evaluation data in the group. In general, when a user asks a question, the question may be continued multiple times until a sufficient understanding is obtained. Therefore, by regarding multiple questions as belonging to the same group, not limited to those immediately before or after the question, and using the user evaluation data in the group, more accurate evaluation is possible. When grouping, user information is considered, such as the identity of the user, and the identity and similarity of department and work. For question dates and times, the proximity of dates and times is considered, and for question and answer contents, the similarity of keywords extracted using natural language processing is considered. There is no particular limit to the number of questions that make up a group. In addition, the handling of user evaluation data within a group may refer to the evaluation trends of the user.

[0016] The automatic response data improvement method of the present invention is a method for improving a device that automatically responds to questions from users by using previously input learning data and a generation AI, and includes a user evaluation step of accepting user evaluations on answer data and storing them in a database as user evaluation data, an answer evaluation step of generating answer evaluation data that evaluates the quality of an automatic answer using question data, answer data, and user evaluation data, a topic estimation step of performing natural language processing on the question data and answer data, classifying them into categories using topic modeling, and storing them in the database, and an evaluation data analysis step of analyzing the quality of the automatic answer for at least each category using the answer evaluation data and categories. , the computer executes .

[0017] The automatic response data improving program of the present invention causes a computer to execute each step of the automatic response data improving method described above. Effect of the Invention

[0018] The automatic response data improvement device, method, and program of the present invention have the effect of effectively presenting information that is the basis for determining learning data suitable for input to the user. In addition, the device, method, and program have the effect of being highly convenient and increasing the user's motivation to improve answer quality. [Brief description of the drawings]

[0019] [Figure 1] Functional block diagram of the automatic response data improvement device according to the first embodiment [Diagram 2] System configuration diagram of the automatic response system [Diagram 3] 1 is a schematic flow diagram of a method for improving automatic response data according to a first embodiment of the present invention; [Figure 4] User evaluation image [Diagram 5] Explanation of topic inference method [Figure 6] Image of improvement of learning data [Figure 7] Answer evaluation display image [Figure 8] Image of recommendation data input simulation [Figure 9] Functional block diagram of an automatic response data improvement device according to a second embodiment [Figure 10] FIG. 11 is an explanatory diagram of the answer evaluation means of the second embodiment. [Figure 11] Functional block diagram of an automatic response data improvement device according to a third embodiment [Figure 12] FIG. 11 is an explanatory diagram of the answer evaluation means of the third embodiment. [Figure 13] Diagram of the evaluation method when grouped DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. Note that the scope of the present invention is not limited to the following examples and illustrated examples, and many modifications and variations are possible. EXAMPLES

[0021] Fig. 1 shows a functional block diagram of an automatic response data improving device of Example 1. As shown in Fig. 1, an automatic response system 100 includes an automatic response data improving device 1, an automatic response device 2, and a generating AI (LLM) 3. The automatic response device 2 is a device that automatically responds to questions from a user (not shown) using pre-input learning data 4a and a generation AI 3, and includes a database 7. As the generation AI 3, a wide range of generation AIs that utilize known large-scale language models (LLMs) such as "BERT" and "GPT" can be used.

[0022] The automatic answering device 2 uses a search expansion generation technology called RAG. The RAG technology is a technology that improves answer accuracy by combining the search of external information such as in-house data in the generation of answer data using a large-scale language model. For example, when the automatic answering device 2 is used in a certain company, the in-house data is input as learning data 4a and stored in the database 7. The data stored in the database 7 has a vector database structure that saves it as a numerical vector (embedded), and is managed so that it can be searched efficiently. When a user inputs question data 4b into the automatic answering device 2, the automatic answering device 2 automatically searches for data related to the input question data 4b (word, pdf, pptx, excel, etc.), inputs the input question data 4b and the automatically searched related data to the generation AI 3 by API communication, and displays the answer results output by the generation AI 3 on the display equipped with the automatic answering device 2.

[0023] The automatic response data improvement device 1 is a device that analyzes and presents information that serves as a clue to learning data that is recommended to be input in order to improve the response accuracy of the automatic response device 2, and is composed of a user evaluation means 11, an answer evaluation means 12, an evaluation data analysis means 13, a topic estimation means 14, an answer evaluation display means 15, a simulation means 16, and a topic correction means 17. The user evaluation means 11 receives user evaluations regarding the answer data 4c, and stores them in the database 7 as user evaluation data. The answer evaluation means 12 generates answer evaluation data that evaluates the quality of the automatic answer using the question data 4b, the answer data 4c, and the user evaluation data. The topic estimation means 14 performs natural language processing on the question data 4b and the answer data 4c, classifies the data into categories using topic modeling, links the data to the question data 4b and the answer data 4c, and stores the data in the database 7. The topic correction means 17 accepts corrections to the category classification made by the topic estimation means 14, and stores the corrected category classification in the database 7 by linking the data to the question data 4b and the answer data 4c. The evaluation data analysis means 13 uses the answer evaluation data and the categories to analyze the quality of at least the automatic answers for each category, and includes a learning data analysis means 13a and a question data analysis means 13b. The learning data analysis means 13a analyzes the learning data that is recommended for input. Also, the question data analysis means 13b analyzes the question data that is recommended for input.

[0024] The answer evaluation display means 15 visually displays on a display at least one of the answer success rate for each category during a predetermined period and a list of questions whose answer success rate is below a predetermined threshold. The simulation means 16 includes a simulation data input means 161, a question content selection means 162, and a preview means 163. The simulation data input means 161 accepts input of learning data 4a for simulation. The accepted learning data 4a is transmitted to the automatic answering device 2. The question content selection means 162 selects from a list a question content whose answer success rate is below a predetermined threshold. The preview means 163 accepts answer data 4c to the learning data 4a from the automatic answering device 2, and displays the answer content regarding the answer data 4c before and after the input of the simulation data in parallel.

[0025] FIG. 2 shows a system configuration diagram of the automatic response system. As shown in FIG. 2, the automatic response system 100 is a system in which a server (20, 30) transmits and receives data to and from client terminals such as a PC 51, a smartphone 52, and a tablet terminal 53 via a network 6. The server 20 functions as an automatic response device 2 and an automatic response data improvement device 1. The server 30 functions as a generation AI 3. A client terminal such as a PC 51 can be used to input learning data 4a and question data 4b. In addition, the analysis data and recommendation data presented by the automatic response data improvement device 1 are output to a client terminal such as a PC 51 and displayed on a display 51a of the PC 51. The analysis and recommendation data may be presented not only visually but also audibly by voice or the like. In this embodiment, it is assumed that a user uses the automatic response device 2 and the automatic response data improvement device 1 using the PC 51.

[0026] Fig. 3 shows a schematic flow diagram of the automatic response data improvement method of Example 1. As shown in Fig. 3, first, a user evaluation on the answer data 4c is accepted and stored in the database 7 as user evaluation data (step S01: user evaluation step). FIG. 4 shows a user evaluation image diagram. As shown in FIG. 4, an answer 42a by the generation AI 3 is displayed in response to a question 41a by a user. A “Good” evaluation button 11a and a “Bad” evaluation button 11b are displayed below the answer 42a. After examining the content of the answer 42a to the question 41a, if the user feels that the answer is good, the user clicks the “Good” evaluation button 11a, and if the user feels that the answer is bad, the user clicks the “Bad” evaluation button 11b to perform the evaluation. Note that a user evaluation is not essential, and if none of the evaluation buttons are clicked, it is treated as no user evaluation. The input user evaluation data is linked to the question data 4b and the answer data 4c and stored in the database 7.

[0027] Next, as shown in Fig. 3, answer evaluation data is generated by evaluating the quality of the automatic answer using the question data, answer data, and user evaluation data (step S02: answer evaluation step). The quality of the automatic answer is evaluated by the answer evaluation means 12. In this embodiment, for example, when the user evaluation for the answer data is "Good", the automatic answer is evaluated as successful and given a "+1", and when the user evaluation is "Bad", the automatic answer is evaluated as unsuccessful and given a "-1". Also, when there is no user evaluation, the automatic answer is evaluated as neither successful nor unsuccessful and given a "0".

[0028] Furthermore, natural language processing is performed on the question data 4b and the answer data 4c, and the data is classified into categories using topic modeling, and stored in the database 7 (step S03: topic estimation step). Note that the order of steps S02 and S03 does not matter. Fig. 5 is an explanatory diagram of the topic estimation means. As shown in Fig. 5, the database 7 includes a question history database 7a and an answer history database 7b. The question history database 7a stores the question ID, question date and time, question content, and account ID for each question. The answer history database 7b stores the answer ID, answer content, rating, related question ID, and category. Taking a question with a question ID of "q0000001" as an example, the question is stored in the question history database 7a, and "q0000001" is also stored in the answer history database 7b as a related question ID.

[0029] The topic estimation means 14 performs natural language processing on the text of the question "What industry does product X sell well in?", which is the question content of the question ID "q0000001", and the answer "We cannot answer that question because we do not have such information." Specifically, first, each question is divided into words and vectorized, and the frequency of occurrence of each word is recorded (vector conversion). After that, processes such as removing words that have little meaning in the sentence and unifying the prototype of parts of speech are performed (data preprocessing). After vector conversion and data preprocessing, the topic estimation means 14 classifies each question into a similar category group using a topic modeling technique. Note that, since a topic modeling algorithm may be biased if it is judged based on only one algorithm, this embodiment employs a method of automatically extracting the most suitable topic overall by looking at the results of four topic modeling algorithms, namely, "Latent Seman+c Index (LSI)", "Probabilis+c Latent Seman+c Indexing (PLSI)", "Latent Dirichlet Alloca+on (LDA)" and "Cosine Similarity". Data relating to the classified categories is stored in the database 7 in association with the question data 4b and answer data 4c.

[0030] Fig. 6 is an image diagram of improvement of learning data, (1) shows before categorization, (2) shows after categorization, and (3) shows after input of improved data. Of questions (41b to 41e) in question group 410 shown in Fig. 6(1), questions (41b, 41c) were evaluated as "Good" by the user, and the automatic response was evaluated as successful, so they were given a "+1". In contrast, questions (41d, 41e) were evaluated as "Bad" by the user, and the automatic response was evaluated as unsuccessful, so they were given a "-1". As shown in Fig. 6(2), the questions (41b-41e) in the question group 410 shown in Fig. 6(1) are classified into categories 14a and 14b. As a result, the questions (41b, 41c) classified into category 14a are highly evaluated, whereas the questions (41b, 41c) classified into category 14b are highly evaluated, suggesting a high need for improvement. Therefore, by improving the quality of the learning data 4a related to category 14b, the evaluation can be improved, as shown in Fig. 6(3).

[0031] Using the evaluation by the answer evaluation means 12 and the category classification by the topic estimation means 14, the quality of the automatic answer for each category is analyzed (step S04: evaluation data analysis step). In this embodiment, the analysis result is visually displayed on the display 51a using the answer evaluation display means 15. Here, an example of visual display using the answer evaluation display means 15 will be described. FIG. 7 shows an image diagram of the answer evaluation display. As shown in FIG. 7, on the display 51a, answer evaluation results (151a, 151b) are displayed on the left side of the screen, and diagnosis results 152 obtained by analyzing the categories are displayed on the right side of the screen. Note that the answer evaluation results (151a, 151b) can be scrolled up and down to sequentially view a larger number of answer evaluations.

[0032] Buttons (15a to 15c) are arranged above the answer evaluation results (151a, 151b), and by clicking the buttons (15a to 15c), the display method of the answer evaluation results can be switched, such as displaying by "category name", by "answer success rate", or in descending order of "number of questions". In the example shown in Fig. 7, button 15a is clicked and display by "category name" is selected. Of the answer evaluation results (151a, 151b), the answer evaluation result 151a will be used as an example for explanation. In the answer evaluation result 151a, "expense settlement" is displayed as "category 1", and information related to the "expense settlement" category is displayed. The information related to the "expense settlement" category includes the question content in the question data, such as "About guests staying at hotels during remote business trips", the answer evaluation for the question, which is represented by "o" or "x", and the "AI answer success rate", which is represented by a pie chart and a numerical value. The number in parentheses indicates the total number of questions related to the category.

[0033] For "Expenses" in category 1 and "Product details" in category 2, the category estimated by the topic estimation means 14 is initially displayed, but the category name can be edited by clicking button 15d provided to the right of "Expenses" or "Product details." Button 15d activates topic modification means 17. A button 15e provided below the answer evaluation results (151a, 151b) increases the number of questions displayed. If a user wishes to view more questions related to category 1 or category 2, the user clicks on the button 15e. Button 15f located at the bottom right of answer evaluation result 151b is used to change the conditions. By clicking button 15f, the display contents can be narrowed down by conditions such as the period to be analyzed or whether or not to display results belonging to multiple categories.

[0034] The evaluation of the question is displayed as a "Good" or "Bad" evaluation given by the user evaluation means 11, with the answer evaluation means 12 evaluating the automatic response as a success and giving a rating of "+1" being displayed as "◯" or evaluating the automatic response as a failure and giving a rating of "-1" being displayed as "X." Although not shown, if no user evaluation is given and the evaluation by the answer evaluation means 12 is "0," it may be displayed as "△." The question contents and the evaluation of the questions are stored in the database 7 and output and displayed on the display 51a. The "AI answer success rate" indicates the answer rate at which the generated AI 3 correctly answers the user's questions, and the data stored in the database 7 is analyzed by the evaluation data analysis means 13 and displayed on the screen by the answer evaluation display means 15. As shown in FIG. 7, the "AI answer success rate" is visually displayed as a pie chart or a percentage, which promotes intuitive understanding by the user and makes it possible to present information in a highly convenient manner.

[0035] The diagnosis result 152 displayed on the right side of the screen indicates the analysis result of the category, and is explained in easy-to-understand text. The text to be displayed is automatically generated for all sentences using the evaluation data analysis means 13, which allows for a wide variety of recommendations. However, instead of the above configuration, for example, a fixed format sentence may be prepared in advance, and the category name such as "Product details" and numerical values ​​such as "3 months", "500 items", and "50%" may be automatically input based on the analysis results. The example questions 152a are excerpts of questions that were evaluated as being unsuccessful in answering. Reading the example questions 152a together with the explanatory text promotes understanding of the diagnosis results and how to improve them. Clicking the button 15g transitions to a simulation screen.

[0036] FIG. 8 shows an image diagram of a simulation of input of recommendation data. As shown in FIG. 8, on the display 51a, a data addition section 16a, a data list display section 16b, a pull-down menu 16c, and questions (41f-41h) are displayed on the left side of the screen, and a preview display section 16d is displayed on the right side of the screen. It is possible to view more data or questions by scrolling the data list display section 16b or the questions (41f-41h) up and down. The data addition section 16a, the data list display section 16b, the pull-down menu 16c, the questions (41f-41h), and the preview display section 16d function as the simulation means 16. A user operating the PC 51 selects a file in the data adding unit 16a or adds it by dragging and dropping, and uploads the learning data 4a for simulation to the server 20. The uploaded learning data 4a is displayed in the data list display unit 16b. In the data list display unit 16b, the user can check the details, delete data, etc. Next, click on the pull-down menu 16c to select a category. Here, the "Product Details" category is selected. A list of questions in the selected category that were evaluated as being unsuccessful for the generation AI3 to answer is displayed. Here, questions 41f to 41h are displayed. When any question is clicked, the simulation results are displayed in the preview display area 16d. In FIG. 8, question 41f has been clicked.

[0037] The preview display section 16d displays the icon and ID of the user who input the question, the question 41f, the answers (42b, 42c) by the generation AI 3, the suggestion prompt display section 16e, and the message box 16g. The answer 42c when the uploaded learning data 4a is read and answered using the RAG technique (after) and the answer 42b when the answer is answered without the data (before) are displayed side by side, so that it is possible to check in real time how the answer changes before and after the data is input. It can be seen that the answer 42b in the before state was not answered using the RAG technique because there is no data. It is also considered that hallucination has occurred in the second sentence. In contrast, it can be seen that the answer 42c in the after state is an accurate answer to the question. In this way, by providing the simulation means 16, the user can check the improvement results by adding data at a glance and realize the effect of the improvement.

[0038] In addition to improving the quality of the learning data, it is also important to improve the quality of the questions in order to increase the answering accuracy. Therefore, based on the history of successfully answered questions stored in the database 7, a better prompt is automatically generated and displayed on the proposed prompt display unit 16e. The proposed prompt display unit 16e causes the question data analysis means 13b to function. Since it automatically recommends prompts that are likely to be answered successfully, if the suggested prompts are registered in a prompt list that can be used by all users, users who have not been able to get a good answer to the question in the past can also use it, improving convenience. When the user clicks the "Enter into chat" button 16f in the proposed prompt display area 16e, the prompt is automatically entered into the message box, and when the user clicks the send button 16h, the prompt is registered in the prompt list. This has the advantage that the user only needs to fill in the specific information. EXAMPLES

[0039] In general, when questions are asked consecutively, they are often highly related questions. Therefore, in this embodiment, a configuration for evaluating questions that are asked in close time intervals by the same user will be described. Fig. 9 shows a functional block diagram of an automatic response data improvement device of the second embodiment. As shown in Fig. 9, an automatic response system 101 of the second embodiment is composed of an automatic response data improvement device 1a, an automatic response device 2, and a generation AI (LLM) 3. The automatic response data improvement device 1a of the second embodiment is different from the automatic response data improvement device 1 of the first embodiment in that the answer evaluation means 120 includes a relevance determination means 12a. The other configurations are the same as those of the first embodiment. The relevance determination means 12a extracts the question immediately before or immediately after the question in question based on the user information and question date and time contained in the question data 4b, and determines the relevance between the question and the extracted immediately before or immediately after question based on at least one of the question data 4b and the answer data 4c, and the user evaluation data of the immediately before or immediately after question that is confirmed to be related to the question in question is used for evaluating the answer to the question.

[0040] FIG. 10 is an explanatory diagram of the answer evaluation means of the second embodiment, where (1) shows the answer evaluation means 12 of the first embodiment, and (2) shows the answer evaluation means 120 of the second embodiment. In each case, nine questions are asked by the same user from top to bottom, and each user evaluation data (5a to 5i) for the answer is shown as "Good", "Bad" or "-" (no evaluation). As shown in FIG. 10(1), the answer evaluation means 12 of the first embodiment judges the user evaluation data (5a to 5i) individually. Therefore, for example, when the user evaluation data 5e is evaluated using the answer evaluation means 12, evaluation is performed using only the user evaluation data 5e.

[0041] In contrast, as shown in FIG. 10(2), the answer evaluation means 120 of the second embodiment extracts questions immediately before or immediately after each question corresponding to the user evaluation data (5a to 5i) and judges the relevance. Therefore, for example, when the answer evaluation means 120 is used to evaluate the user evaluation data 5e, the immediately preceding user evaluation data 5d and the immediately succeeding user evaluation data 5f of the user evaluation data 5e are extracted based on the identity of the user and the proximity of the question date and time, and then the relevance is judged. The relevance is judged based on the question data and the answer data. Here, the relevance with the user evaluation data 5d is confirmed, and the relevance with the user evaluation data 5f is denied. In contrast, in the case of the user evaluation data 5h, the relevance with both the immediately preceding user evaluation data 5g and the immediately succeeding user evaluation data 5i is confirmed. The user evaluation data 5d whose relevance with the user evaluation data 5e is confirmed is used to evaluate the question data and the answer data corresponding to the user evaluation data 5e. Furthermore, the user evaluation data (5g, 5i) that is confirmed to be related to the user evaluation data 5h is used to evaluate the question data and answer data that correspond to the user evaluation data 5h. In this way, by using the evaluation regarding the immediately preceding or immediately following question, a more accurate evaluation is possible, and useful information can be provided to the user. EXAMPLES

[0042] Fig. 11 shows a functional block diagram of an automatic response data improvement device of the third embodiment. As shown in Fig. 11, an automatic response system 102 of the third embodiment is composed of an automatic response data improvement device 1b, an automatic response device 2, and a generation AI (LLM) 3. The automatic response data improvement device 1b of the third embodiment is different from the automatic response data improvement device 1 of the first embodiment in that the answer evaluation means 121 includes a relevance determination means 12b. The other configurations are the same as those of the first embodiment. The grouping means 12b groups a plurality of questions in a question set based on at least one of the user information, question date and time, and question content contained in the question data, and the answer content contained in the answer data, and evaluates the answers based on at least one of the type, number, and order of user evaluation data within the group.

[0043] Fig. 12 is an explanatory diagram of the answer evaluation means of the third embodiment, where (1) shows the answer evaluation means 12 of the first embodiment, and (2) shows the answer evaluation means 121 of the third embodiment. In each case, nine questions are asked by the same user, starting from the top, and the user evaluation data (5a to 5i) for each answer is shown as "Good", "Bad", or "-" (no evaluation). As shown in Fig. 12(1), the answer evaluation means 12 of the first embodiment judges the user evaluation data (5a to 5i) individually. In contrast, as shown in Fig. 12(2), the answer evaluation means 121 of the third embodiment groups questions corresponding to user evaluation data (5a to 5i). The criteria for grouping are the same user, the proximity of question dates and times, and the relevance of the question contents and answer contents. In the example shown in Fig. 12(2), the user evaluation data (5a to 5c) is determined to be group 8a, the user evaluation data (5d, 5e) to be group 8b, and the user evaluation data (5f to 5i) to be group 8c. For each group (8a to 8c), the answers are evaluated based on the type, number, and order of the user evaluation data. The "type" of user evaluation data refers to "Good", "Bad", or "-" (no evaluation), and the "number" refers to the number of "Good", etc. Also, not only the number but also the percentage may be added to the evaluation. The "order" refers to the order of "Good", "Bad", etc.

[0044] Fig. 13 is an explanatory diagram of the evaluation method when grouping, where (1) shows the case where all the questions and answers ultimately received a "Good" user rating, and (2) shows the case where all the questions and answers ultimately received a "Bad" user rating. In this case, all three questions and answers are determined to be in a group. When all the evaluations in a group are "Good," as in the case of group 8d shown in FIG. 13(1), the answer can be evaluated as successful. Both Group 8e and Group 8f had two "Good" responses and one "Bad" response, but Group 8e rated the first question as "Bad," but the second question improved to "Good." The third question was also rated as "Good." From this, it can be inferred that the first question did not provide an appropriate answer, but the second question provided an appropriate answer, and the third question provided a deeper understanding. In contrast, in group 8f, the first question was rated as "Good", but the second question was rated as "Bad". Then, the third question was rated as "Good". Since the first question was rated as "Good", but the question immediately following it was rated as "Bad", it is possible that the evaluation of the first question was inaccurate. For these reasons, Group 8e can be rated higher than Group 8f.

[0045] Moreover, comparing group 8g and group 8h, group 8g has one "Good" and two "Bad", while group 8h has one "Good" and two "-" (no rating). Here, as in the first embodiment, it is possible to rate "0" when there is no rating, but it is generally said that many users only rate at the end when multiple questions are asked. Then, for example, if it can be determined from the tendency of the user's ratings over a certain period of time that the user only clicks "Good" or "Bad", it is possible to determine that "-" is highly likely to be the opposite rating to the last rating, and to treat the "-" in group 8h as the same as "Bad", and to rate groups 8g and 8h in the same way. In addition, the method of determining whether a user clicks only "Good" or "Bad" based on the user's evaluation tendency in a predetermined period may be, for example, based on the ratio of "Good" or "Bad" to the total number of questions by the user exceeding a predetermined threshold. In addition, the user's evaluation tendency may be determined based on the number of evaluations out of the total number of questions, the number of specific evaluation types, etc.

[0046] Next, when all the evaluations in a group are "Bad," as in group 8i shown in FIG. 13(2), it can be evaluated that the answer was unsuccessful. Both Group 8j and Group 8k had one “Good” and two “Bad” ratings, but Group 8j was rated “Good” on the first question, “Bad” on the second question, and “Bad” on the third question. In contrast, in group 8k, the first question rated the item as “Bad,” but the second question rated it as “Good,” and then the third question rated it as “Bad” again. For example, if the final result includes a series of "Bad" evaluations, the satisfaction with the answers can be evaluated as low. From this, group 8k can be given a higher evaluation than loop 8j.

[0047] Moreover, comparing group 8l and group 8m, group 8l has two "Good"s and one "Bad," while group 8m has one "Bad" and two "-"s (no rating). As with the ratings for groups 8g and 8h described above, if it can be determined from the user's rating tendency over a given period that the user only clicks "Good" or "Bad," it can be determined that there is a high possibility that the "-" is the opposite rating to the last rating, and the "-" for group 8m can be treated the same as "Good," and groups 8l and 8m can be given the same rating. However, in general, if a user continues to receive no ratings from the beginning and then receives a "Bad" rating, this is often the case when the user has not received a satisfactory answer no matter how many times they have asked. For this reason, unless a particular user's rating tendency can be identified, all "-"s in group 8m should be rated as "Bad". For these reasons, Group 8l can be rated higher than Group 8m. It should be noted that the evaluation method shown here is merely one example, and by setting a variety of evaluation criteria, it is possible to achieve a more accurate evaluation. [Industrial Applicability]

[0048] The present invention is useful as a technology for improving response accuracy in an automatic answering device using a generation AI. [Explanation of symbols]

[0049] 1,1a,1b Automatic response data improvement device 2. Auto-answering machine 3 Generation AI 4a Training data 4b Question data 4c Response data 5a~5i User evaluation data 6 Network 7 Database 7a Question History Database 7b Answer history database 8a~8m group 11 User evaluation methods 11a,11b Rating button 12,120,121 Answer evaluation method 12a Relevance determination method 12b Grouping methods 13. Evaluation data analysis methods 13a Training data analysis methods 13b Questionnaire data analysis methods 14 Topic Inference Methods 14a,14b Category 15. Answer evaluation display means 15a~15g,16f buttons 16 Simulation Methods 16a Data Addition Section 16b Data list display section 16c Pulldown 16d Preview display area 16e Proposal prompt display section 16g Message Box 16h Send button 17 Topic Revision Methods 20,30 Server 41a~41h Questions 42a~42c Answer 51 PC 51a Display 52 Smartphone 53 Tablet devices 100~102 Automated Response System 151a, 151b Answer evaluation results 152 Diagnosis 152a Sample Questions 161 Simulation data input means 162 Question content selection method 163 Preview Methods 410 Questions

Claims

1. A device for improving a device that automatically responds to questions from a user using pre-input learning data and generated AI, a user evaluation means for receiving a user evaluation regarding the answer data and storing the user evaluation data in a database; an answer evaluation means for generating answer evaluation data that evaluates the quality of an automatic answer using the question data, the answer data, and the user evaluation data; a topic estimation means for performing natural language processing on the question data and the answer data, classifying the data into categories using topic modeling, and storing the data in a database; an evaluation data analysis means for analyzing the quality of the automatic answers at least for each category using the answer evaluation data and the categories; An automatic response data improving device comprising:

2. 2. The automatic response data improving device according to claim 1, wherein said evaluation data analyzing means includes learning data analyzing means for analyzing learning data recommended for input.

3. 2. The automatic response data improving device according to claim 1, wherein said evaluation data analyzing means comprises question data analyzing means for analyzing question data whose input is recommended.

4. The automatic response data improvement device according to claim 1, further comprising an answer evaluation display means for visually displaying on a display at least one of the answer success rate for each category during a specified period and a list of questions for which the answer success rate is below a specified threshold.

5. a simulation data input means for receiving an input of learning data for a simulation; a question content selection means for selecting from a list a question content for which the answer success rate is below a predetermined threshold; A preview means for displaying the contents of answers regarding the answer data in parallel before and after inputting the simulation data; 5. The automatic response data improving device according to claim 4, further comprising a simulation means having the following:

6. The automatic response data improvement device according to claim 1, further comprising a topic correction means for accepting corrections to category classification made by the topic estimation means, and linking the corrected category classification to the question data and the answer data and storing it in a database.

7. The answer evaluation means includes: extracting questions immediately before or after the question based on the user information and question date and time contained in the question data; a relevance determination means for determining a relevance between the question and the extracted immediately preceding question or immediately following question based on at least one of a question content included in the question data and an answer content included in the answer data, 2. The automatic response data improving device according to claim 1, wherein the user evaluation data of the immediately preceding question or the immediately following question that is confirmed to be related to the question is used for evaluating the answer to the question.

8. The answer evaluation means includes: a grouping means for grouping a plurality of questions in the question set based on at least one of user information, question date and time, question content, and answer content in the answer data, the grouping means comprising:

2. The automatic response data improving device according to claim 1, wherein answers are evaluated based on at least one of the type, number, and order of user evaluation data in the group.

9. A method for improving a device that automatically responds to questions from a user using pre-input learning data and a generating AI, comprising: a user evaluation step of accepting user evaluations regarding the answer data and storing the evaluations in a database as user evaluation data; an answer evaluation step of generating answer evaluation data by evaluating the quality of the automatic answer using the question data, the answer data, and the user evaluation data; a topic estimation step of performing natural language processing on the question data and the answer data, classifying the data into categories using topic modeling, and storing the results in a database; The automatic response data improving method includes an evaluation data analyzing step of analyzing the quality of the automatic response at least for each category by using the response evaluation data and the categories.

10. 10. An automatic response data improving program for causing a computer to execute each step of the automatic response data improving method according to claim 9.

Images