Computer implementation methods, systems, and computer programs

Abstract

Hallucination prevention measures are desired. [Solution] The disclosed system is a computer implementation method performed by one or more computers, which may include generating an answer to a question using a generative artificial intelligence that generates answers to a question input to the generative artificial intelligence, performing an evaluation of the answer generated by the generative artificial intelligence, and displaying the results of the evaluation. Performing the evaluation may include identifying a portion of the answer to be evaluated and performing a partial evaluation of the identified portion to be evaluated.

Description

Technical Field

[0001] The present disclosure relates to computer-implemented methods, systems, and computer programs.

Background Art

[0002] Conventionally, when a user inputs a question, there is a known technique of generating an answer by artificial intelligence (hereinafter referred to as "AI") while referring to sentences corresponding to the user's question using a search model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] Companies have departments such as the personnel department that frequently receive inquiries from employees within the company. In order to reduce the burden of responding to individual inquiries from such departments, it is conceivable to use the above technology.

[0005] However, in the answer by AI, there may occur a phenomenon called hallucination, in which AI generates a plausible lie that is not based on facts, and the answer may not be reliable. Since employees receive the answer from AI as an answer from that department, if hallucination occurs, the reliability of that department may be damaged.

[0006] Therefore, countermeasures against hallucination are desired.

[0007] The disclosed technology is a computer implementation method performed by one or more computers, which may include generating an answer to a question using generative artificial intelligence that generates an answer to a question input to the generative artificial intelligence, performing an evaluation of the answer generated by the generative artificial intelligence, and displaying the results of the evaluation. Performing the evaluation may include identifying a portion of the answer to be evaluated and performing a partial evaluation of the identified portion to be evaluated.

[0008] One aspect of this disclosure is a system. The system of the disclosure may be configured to perform processes including generating answers to questions using generative artificial intelligence that generates answers to questions input to the generative artificial intelligence, performing an evaluation of the answers generated by the generative artificial intelligence, and displaying the results of the evaluation. Performing the evaluation may include identifying portions of the answers to be evaluated and performing a partial evaluation of the identified portions to be evaluated.

[0009] Another aspect of the disclosure is a computer program. The computer program of the disclosure may be a computer program that causes a computer to perform a process which includes generating answers to questions using generative artificial intelligence that generates answers to questions input to the generative artificial intelligence, performing an evaluation of the answers generated by the generative artificial intelligence, and displaying the results of the evaluation. Performing the evaluation may include identifying portions of the answers to be evaluated and performing a partial evaluation of the identified portions to be evaluated.

[0010] Further details will be described in the embodiments below. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a network configuration diagram representing a system to which the computer implementation method according to the embodiment is applied. [Figure 2] Figure 2 is an explanatory diagram showing an example of a user interface display for a correspondence table. [Figure 3] Figure 3 is a diagram illustrating the general process from receiving user-inputted questions to displaying answers and evaluations. [Figure 4] Figure 4 is a sequence diagram showing the details of the process in Figure 3. [Figure 5] Figure 5 shows an example of a question input screen. [Figure 6] Figure 6 is an explanatory diagram showing an example of a contact form generated by the query extension generation AI. [Figure 7] Figure 7 is a flowchart that specifically explains the response and evaluation display process performed by the server. [Figure 8] Figure 8 is an explanatory diagram showing an example of a response display screen on a user terminal. [Figure 9] Figure 9 shows an example of the user terminal display screen when the evaluation is negative, with (A) showing an example of a message indicating that the answer cannot be provided and an evaluation, and (B) showing an example of an inquiry data display. [Modes for carrying out the invention]

[0012] <1. Overview of Computer Implementation Methods, Systems, and Computer Programs> (1) The computer implementation method according to the embodiment is a computer implementation method executed by one or more computers, which may include generating an answer to a question using a generative artificial intelligence that generates answers to questions, performing an evaluation of the answer generated by the generative artificial intelligence, and displaying the result of the evaluation. Performing the evaluation may include identifying the part of the answer to be evaluated and performing a partial evaluation of the identified part to be evaluated. In this case, since the evaluation is only of the part to be evaluated, even if the accuracy of the other parts decreases due to hallucination or the like, the accuracy of the answer of the part to be evaluated can be evaluated more appropriately. Therefore, according to the computer implementation method according to the embodiment, hallucination countermeasures can be taken for the answer generated by the generative AI.

[0013] (2) Identifying the evaluation target part may include obtaining identification information used to identify the evaluation target part and using the identification information to identify the evaluation target part in the response.

[0014] (3) Identifying the evaluation target part may be performed by an artificial intelligence that identifies the evaluation target part in the response using the identification information.

[0015] (4) Generating the response may include generating the response by search expansion generation using search results for the search target data, and performing the evaluation may include evaluating the degree of coincidence between the response generated by the generation artificial intelligence and the search results. In this case, evaluating the degree of coincidence between the response generated by the generation artificial intelligence and the search results is useful as a measure against hallucination particularly when using search expansion generation.

[0016] (5) The search target data may include a plurality of FAQ data, and the FAQ data may be combination data of questions and answers. In this case, it is useful as a measure against hallucination particularly in scenarios where accurate information is required.

[0017] (6) The FAQ data may be combination data of questions and answers regarding the personnel system.

[0018] (7) When the result of the evaluation is negative, it may further include generating inquiry data described in accordance with a preset format based on the content of the question input to the generation artificial intelligence. In this case, even when the result of the evaluation is negative, support for obtaining an answer to the question can be provided.

[0019] (8) The question input to the generation artificial intelligence is generated based on the question input by the user, and further includes displaying the inquiry data together with a display prompting the user to inquire to the inquiry destination using the inquiry data. In this case, even when the generation artificial intelligence cannot solve the problem the user has, support for solving it can be provided, improving user satisfaction.

[0020] (9) The system according to the embodiment may be configured to execute a process including generating an answer to the question input to the generation artificial intelligence by the generation artificial intelligence that generates an answer to the question, performing an evaluation of the answer generated by the generation artificial intelligence, and displaying the result of the evaluation. Performing the evaluation may include identifying an evaluation target part in the answer and performing a partial evaluation on the identified evaluation target part.

[0021] (10) The computer program according to the embodiment may be a computer program for causing a computer to execute a process including generating an answer to the question input to the generation artificial intelligence by the generation artificial intelligence that generates an answer to the question, performing an evaluation of the answer generated by the generation artificial intelligence, and displaying the result of the evaluation. Performing the evaluation may include identifying an evaluation target part in the answer and performing a partial evaluation on the identified evaluation target part.

[0022] The computer program according to the embodiment may be configured to cause the computer to execute the computer implementation method. Also, the computer program according to the embodiment may cause the computer to function as the system. The computer program may be recorded on a computer-readable non-temporary recording medium.

[0023] <2. Examples of Computer Implementation Method, System, and Computer Program> Hereinafter, embodiments will be described in more detail with reference to the drawings.

[0024] Figure 1 is a network configuration diagram representing System 1 to which the computer implementation method according to the embodiment is applied.

[0025] System 1 is an inquiry system that can automatically generate and display answers to inquiries from user 101 via user terminal 100. This system 1 can be used, for example, to reduce the workload on the human resources department 201 (human resources staff) in responding to employee inquiries regarding human resources systems within a company.

[0026] For example, when System 1 receives a question from user 101, who is an employee, it generates an answer based on FAQ data 322 stored in server 300. FAQ data 322 is a combination of frequently asked questions about personnel systems and their answers to the personnel department 201.

[0027] Furthermore, as will be described later, System 1 can not only reduce the load but also improve the accuracy of automatically generated answers by utilizing the generation AI 400 and the search engine 500. System 1 can also generate answers while distinguishing between inquiries that can be answered by System 1 and inquiries that should be answered directly by the Human Resources Department 201, based on the evaluation results. At the same time, it can provide the user 101 with an evaluation of the answer, providing the user 101 with information to determine whether the answer is trustworthy.

[0028] System 1 is composed of, for example, user terminals 100, administrator terminal 200, server 300, generation AI 400, and search engine 500, all connected to each other via a network, working together as needed.

[0029] User terminal 100 is a computer used by user 101, for example, an employee who asks a question. User terminal 100 can send the entered question to server 300 and display the answer or evaluation result obtained from server 300 in response to the question. User terminal 100 is, for example, a smartphone, tablet, or personal computer. User terminal 100 has a processor 110 and memory 120.

[0030] Processor 110 is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or another type of processor.

[0031] The memory 120 may include, for example, a primary storage device and a secondary storage device. The primary storage device is, for example, RAM (Random-Access Memory). The secondary storage device is, for example, an HDD (Hard Disk Drive) or an SSD (Solid-State Drive). The memory 120 has the necessary computer programs that are executed by the processor 110.

[0032] The administrator terminal 200 is a computer used, for example, by the human resources department 201 (human resources department staff) within a company. The administrator terminal 200 has functions for maintaining the correspondence table 323 and FAQ data 322 of the server 300. The administrator terminal 200 is, for example, a smartphone, tablet, or personal computer. The administrator terminal 200 has a processor 210 and memory 220.

[0033] Processor 210 is, for example, a CPU, GPU, or other type of processor.

[0034] The memory 220 may include, for example, a primary storage device and a secondary storage device. The primary storage device is, for example, RAM. The secondary storage device is, for example, an HDD or SSD. The memory 220 has the required computer program executed by the processor 210.

[0035] The user terminal 100 and the administrator terminal 200 are connected to the server 300 via wired or wireless connections, for example, the internet or a company LAN, enabling them to communicate with each other.

[0036] Server 300 is a computer installed, for example, inside or outside a company and operated by the company. Server 300 has a processor 310 and memory 320.

[0037] Processor 310 is, for example, a CPU, GPU, or other type of processor.

[0038] The memory 320 may include, for example, a primary storage device and a secondary storage device. The primary storage device is, for example, RAM. The secondary storage device is, for example, an HDD or SSD. The memory 320 has a required computer program 321 (program 321) that is executed by the processor 310. The program 321 has instructions that are read by the processor 310 and perform processing which will be described in detail later.

[0039] Furthermore, memory 320 contains FAQ data 322 and correspondence table 323. FAQ data 322 and correspondence table 323 can be appropriately referenced as search target data for the generating AI 400 and the search engine 500.

[0040] FAQ data 322 (searchable data) is data created, for example, by the Human Resources Department 201 from the administrator terminal 200. FAQ data 322 is a combination of multiple data sets, each containing a question and an answer. For example, FAQ data 322 could be a combination of the question "What should I do if I need to work on a flexible leave day?" and the answer "If you need to work on a flexible leave day, please submit a cancellation request as described below." Another example of FAQ data 322 could be a combination of the question "What is the deadline for taking flexible leave?" and the answer "The deadline for taking flexible leave is from the grant date of April 1st until March 31st of the following year. Please note that if you do not take the leave by March 31st, it will expire." FAQ data 322 may be periodically maintained by the Human Resources Department 201 from the administrator terminal 200, including modifications and additions.

[0041] FAQ data 322 is made publicly available for user 101 to view via the internet or other networks, and each FAQ data 322 is assigned a URL (Uniform Resource Locator).

[0042] The correspondence table 323 registers keywords included in the FAQ data 322, related terms for those keywords, and explanatory information in an associated manner. The correspondence table 323 can be created in advance, for example, by receiving input from the HR department 201 via the administrator terminal 200. The correspondence table 323 is used as search target data for the search engine 500 to search for keywords (step S302 in Figure 3). The correspondence table 323 is also used as information for the generating AI 400 to perform filtering to extract related keywords (step S303).

[0043] Here, Figure 2 is an explanatory diagram showing an example of the user interface display for correspondence table 323.

[0044] The keywords are terms used as official names within the company, and in FAQ data 322, these official names are used as terms in the questions and answers.

[0045] Related terms are abbreviations or synonyms that User 101 might use in their questions to mean the same thing as the official keyword, either commonly or not commonly used within the company.

[0046] Explanatory information is supplementary information used to help the Generating AI400 understand the meaning of a keyword, especially when the keyword is not a commonly used term that the Generating AI400 can understand, but rather, for example, an internal company term.

[0047] For example, if the keyword is "sick leave," the related term is "sick leave," and the descriptive information is "leave name." The correspondence table 323 can be maintained as needed by the HR department 201 from the administrator terminal 200, including modifications and additions. The HR department 201 can edit the correspondence table 323 by, for example, pressing the edit button 21 or delete button 22 in the "Actions" column of Figure 2.

[0048] The Generative AI 400 (Generative Artificial Intelligence) can generate the necessary information based on instructions (prompts) input from the Server 300 and provide it to the Server 300. The Generative AI 400 is a language model capable of generating text using natural language processing, and is, for example, a Generative AI model that uses Retrieval Augmented Generation (RAG) technology to generate information in cooperation with an internal database. In this embodiment, the Generative AI 400 uses the correspondence table 323 and FAQ data 322 stored in the memory 320 of the Server 300 as search target data, and generates answers etc. using RAG technology that uses the search results for this search target data. The Generative AI 400 is, for example, ChatGPT provided by OpenAI.

[0049] In this system 1, the generation AI 400 functions as a query extension generation AI 401, an answer / evaluation generation AI 402, and an inquiry data generation AI 403. The query extension generation AI 401 performs "preprocessing" to create an inquiry form (a question to be entered into the generation artificial intelligence) based on a question entered by user 101 (user input question) (step S303 in Figure 3). The answer / evaluation generation AI 402 generates an answer to the user input question from the relevant FAQ data 322, etc., and performs an evaluation of that answer (step S305). In addition, if the evaluation result is negative, the inquiry data generation AI 403 generates inquiry data to inquire about the content of the question to the human resources department 201 (step S417 in Figure 4).

[0050] The search engine 500 can retrieve the necessary information based on the search criteria provided by the server 300 and provide it to the server 300. The search engine 500 could be, for example, Azure AI Search provided by Microsoft.

[0051] In this system 1, the search engine 500 functions as a full-text search engine 501 that searches for keywords from the correspondence table 323 (step S302 in Figure 3). The search engine 500 also functions as a hybrid search engine 502 that searches for the corresponding FAQ data 322 from the FAQ data 322 by combining vector search and keyword search based on related keywords and FAQ titles (step S304 in Figure 3).

[0052] Next, we will explain the procedure for the processing performed in System 1. Figure 3 is a diagram illustrating the general flow of the process from receiving user input questions to displaying answers and evaluations. Figure 4 is a sequence diagram showing the details of the process in Figure 3.

[0053] In step S301, System 1 begins processing when the server 300 receives a user input question entered via the user terminal 100. In step S302, the full-text search engine 501 searches the correspondence table 323 for keywords that match the terms in the user input question.

[0054] In step S303, the query extension generation AI401 generates an "inquiry form" based on the searched keywords and user-inputted questions. The inquiry form contains information including "related keywords," "FAQ title," and "questions to be answered." The inquiry form is data used in subsequent processes, and the processing in step S303 can serve as preprocessing for these processes.

[0055] Related keywords are keywords that are highly likely to be related to the user input question, obtained by filtering the keywords acquired in step S302 using the descriptive information in the correspondence table 323. These related keywords will be used in the subsequent process (step S304) to search for FAQ data 322 that matches the user input question. The number of related keywords extracted is, for example, fewer than the number of keywords found by the full-text search engine 501 (e.g., around 3 keywords).

[0056] The FAQ title is the information corresponding to the question among the questions and answers that make up the FAQ data 322. The FAQ title is a question that is expected to contain an answer to the user-input question. This FAQ title becomes the vector query for the vector search in the process (step S304) that is executed later to search for FAQ data 322 that matches the user-input question from the FAQ data 322.

[0057] The questions to be answered are information that specifically expresses the user input questions and are necessary questions (information) to solve the problems that user 101 is facing, as understood from the user input questions. The questions to be answered can be summarized, for example, in a bulleted list. The questions to be answered may include not only questions that user 101 wants answered directly (mandatory questions), but also questions that would be beneficial for user 101 to ask as supplementary information and would be desirable to provide in order to increase user 101's satisfaction (optional questions).

[0058] In step S304, the hybrid search engine 502 uses the relevant keywords extracted in step S303 as keywords for keyword search and the FAQ title as a vector query for vector search to search for matching FAQ data 322. This extracts FAQ data 322 related to the user input question.

[0059] In step S305, the response / evaluation generation AI 402 generates a response to present to user 101 based on the extracted FAQ data 322 (relevant FAQ data) and the inquiry form generated in step S303. The response / evaluation generation AI 402 also obtains identification information to identify the part of the generated response that is subject to evaluation. Using this identification information, the response / evaluation generation AI 402 identifies the part of the response to be evaluated and then performs a partial evaluation of that part. If the evaluation result is negative and a response cannot be generated, the inquiry data generation AI 403 generates "inquiry data" for user 101 to inquire about the content of the question with the human resources department 201 (step S417 in Figure 4).

[0060] In step S306, the answers and evaluations generated in step S305, as well as the inquiry data generated if necessary, are displayed on the user terminal 100.

[0061] Next, we will explain the process in detail using Figure 4.

[0062] In step S401, when the user terminal 100 receives a question input from user 101, in step S402, the user terminal 100 sends the user input question to the server 300. Here, Figure 5 shows an example of the question input screen.

[0063] For example, employee user 101 enters the question, "I have to work on a flexible leave day. What should I do?" into the attendance and leave inquiry form on the website provided by the HR department 201 for employees, and clicks the "Generate Answer" button 51.

[0064] The server 300 (processor 310) receives user input questions from the user terminal 100. In step S403, the server 300 instructs the full-text search engine 501 to refer to the correspondence table 323 and perform a full-text search to extract one or more keywords that match terms included in the user input question, or one or more keywords that correspond to related words that match terms.

[0065] In step S404, the full-text search engine 501 searches the correspondence table 323 for keywords that match terms included in the user input question. In step S405, the full-text search engine 501 sends a predetermined number of keywords (e.g., 20) to the server 300.

[0066] Specifically, the full-text search engine 501 performs a full-text search of the correspondence table 323 using terms such as "flexible," "leave," "day," and "work" contained in the user input question, "I have to work on a day that is supposed to be a flexible leave day. What should I do?" as search queries. From the correspondence table 323, the full-text search engine 501 retrieves about 20 keywords, such as "flexible leave" and "annual paid leave."

[0067] Here, as described above, System 1 generates answers to user-input questions using the Generative AI 400, which employs RAG technology. RAG technology generates output information using information that matches keywords in the database used by the Generative AI 400. Therefore, if the terms used by User 101 in the user-input question are abbreviations or synonyms for keywords, the Generative AI 400 may not be able to perform an appropriate search.

[0068] In response to this, System 1 uses the correspondence table 323 to extract the official names corresponding to the terms included in the user input questions as keywords, and performs a process to replace the terms in the user input questions. As a result, System 1 can improve the decrease in search accuracy that occurs when RAG technology is used to generate answers to user input questions.

[0069] Furthermore, the correspondence table 323 can be easily maintained by the human resources department 201 via the user interface for editing the correspondence table 323 shown in Figure 2. For example, if the generated responses are reviewed and it is found that the reason a response could not be obtained is that the terminology used in the user-input question was inappropriate, the human resources department 201 can improve the accuracy of keyword searches by editing the correspondence table 323 to add that term as a related term to the official name.

[0070] For example, to improve the search accuracy of the generating AI 400, the HR department 201 could perform maintenance such as tagging related terms in the FAQ data 322 stored in memory 320 or tuning the instructions given to the generating AI 400. However, performing this work on each individual FAQ data 322 becomes increasingly cumbersome as the amount of data increases. Furthermore, tuning the instructions given to the generating AI 400 requires the HR department 201 to have a high level of knowledge about the generating AI 400. From this maintenance perspective, it is beneficial to have the AI ​​determine keywords from terms in user-input questions using the correspondence table 323.

[0071] In step S406, the server 300 instructs the query extension generation AI 401 to generate an inquiry form containing information including "related keywords," "FAQ title," and "questions to be answered" as a preprocessing step. In step S407, the query extension generation AI 401 generates the inquiry form. Figure 6 is an explanatory diagram showing an example of an inquiry form generated by the query extension generation AI 401.

[0072] The instructions (prompts) given to the query extension generation AI 401 may include sentences that define the role, such as, "You are a support representative. Your job is to listen to users' problems as a support contact and create inquiries on the inquiry form." The instructions may also include instructions to filter the approximately 20 keywords found by the full-text search engine 501 down to about 3, and extract relevant keywords.

[0073] The instructions include instructions for generating a text that is expected to contain an answer to a user-inputted question, to be used as the title of the FAQ. The instructions may also include text that provides examples of generated FAQ titles (e.g., "What should I do if I have to take sudden sick leave?", "How do I apply for paid leave?").

[0074] The instructions may include instructions that specify user input questions as questions to be answered and generate questions necessary to solve the problem that user 101 is facing. The instructions may include instructions that generate a maximum number (e.g., three) of questions to be answered, including at least one required question and one optional question.

[0075] Furthermore, the instructions may include instructions to generate questions to confirm the specific details with user 101 if the user input question is unclear.

[0076] In step S408, the query extension generation AI401 sends an inquiry form to the server 300, which includes relevant keywords, the FAQ title, and the question to be answered, generated as text as shown in Figure 6.

[0077] In step S409, the server 300 instructs the hybrid search engine 502 to search the FAQ data 322 using the relevant keywords from the acquired inquiry form for keyword search and the FAQ title for vector search. In step S410, the hybrid search engine 502 searches for matching FAQ data 322 based on the relevant keywords and FAQ title.

[0078] In step S410, not only is a vector search based on the FAQ title performed, but a keyword search is also performed using related keywords obtained by replacing terms in the user-input questions with the official names used in the FAQ data 322. This makes it possible to search the FAQ data 322, which would be difficult to extract using a vector search based on FAQ titles generated in sentences that do not use official names, using keyword search.

[0079] Specifically, for example, if the vector query is a user-input question such as "I want to change the category of my spouse's withholding tax deduction," then simply extracting keywords from this vector query and performing a keyword search may result in the extraction of "spouse" and "withholding tax deduction." However, if there are no terms matching "spouse" or "withholding tax deduction" in the official names used in FAQ data 322, then there is a risk that appropriate FAQ data 322 cannot be obtained through either vector search or keyword search.

[0080] In contrast, as described above, System 1 refers to the keywords, related terms, and explanatory information in the correspondence table 323, replaces "spouse" and "withholding tax deduction" with keywords that are their official names, and converts user-input questions into information that is easily searchable from the FAQ data 322. As a result, compared to directly extracting keywords from user-input questions, System 1 can significantly improve the search accuracy in keyword searches and enjoy the advantages of both vector searches and keyword searches.

[0081] In step S411, the hybrid search engine 502 sends a predetermined number (e.g., 6) of FAQ data 322 to the server 300 as the relevant FAQ data, in order of search ranking.

[0082] In step S412, the server 300 instructs the response / evaluation generation AI 402 to generate a response to present to the user 101 based on a predetermined number of relevant FAQ data items acquired and an inquiry form as shown in Figure 6.

[0083] The instructions given to the answer / evaluation generation AI 402 may include sentences defining the role of the generation AI 400, such as, "You are an inquiry answering assistant. Please answer the question based on the following information." The instructions may also include sentences instructing the AI ​​to generate answers from the relevant FAQ data so that they sound like answers from the HR department 201, to avoid answers that sound like they are based on hearsay or speculation, and to avoid answers that rely on general knowledge or other information outside of the relevant FAQ data.

[0084] The instructions may include that among the questions to be answered, answers to mandatory questions are required. The instructions may also include statements indicating that optional questions are supplementary questions to mandatory questions, and that if an answer to an optional question cannot be obtained from the relevant FAQ data, the question should be ignored without providing an answer. Optional questions are supplementary questions added by System 1 related to user-input questions, and responding to them with "We cannot answer this" would cause discomfort to User 101.

[0085] For example, if the required question is "My overtime hours have exceeded 45 hours. What should I do?" and the optional question is "Please tell me how deemed overtime hours are calculated," then instead of answering User 101 with "If your overtime hours exceed 45 hours, (omitted) you must obtain approval from your supervisor. Note that we could not find detailed information on how deemed overtime hours are calculated from the provided search results," the system should simply be instructed to answer "If your overtime hours exceed 45 hours, (omitted) you must obtain approval from your supervisor."

[0086] Furthermore, the instructions may include instructions to output a predetermined number (e.g., 2) of FAQ data 322 (reference FAQ data) from the given FAQ data that were referenced when generating the answer and served as the basis for the answer.

[0087] Furthermore, in step S412, the server 300 instructs the response / evaluation generation AI 402 to evaluate the generated response.

[0088] Generally, responses generated by AI may contain unreliable information due to hallucination, and countermeasures are necessary. Therefore, this system 1 can provide user 101 with information to determine whether the response is reliable or not by further evaluating the response generated by the response / evaluation generation AI 402.

[0089] The instructions given to the response / evaluation generation AI 402 may include identification information to identify the part of the response to be evaluated, and instructions to perform a partial evaluation of the part to be evaluated. Specifically, the instructions may include, as identification information, an instruction to evaluate only the answers to mandatory questions and not the answers to optional questions. This ensures that the evaluation is not included for the answers to supplementary optional questions intended to improve user satisfaction, and only an evaluation of the answers to the user-input questions themselves is obtained. Furthermore, since the evaluation is only for the part to be evaluated, even if the accuracy of the answers outside the part to be evaluated decreases due to hallucination or other reasons, the results can be obtained in which the accuracy of the answer in the part to be evaluated is appropriately evaluated.

[0090] The instructions given to the answer / evaluation generation AI 402 may include instructions to evaluate the degree of agreement between the answer generated by the answer / evaluation generation AI 402 and the relevant FAQ data on a 5-point scale, with 5 being the highest and 1 being the lowest. For example, the instructions may include assigning a rating of 5 if the relevant FAQ data contains content that is almost identical to the generated answer. Alternatively, the instructions may include assigning a rating of 4 if the relevant FAQ data contains content similar to the generated answer, and the answer to the user-inputted question can be inferred from that content in a common-sense manner. The instructions may include assigning a rating of 3 if the relevant FAQ data contains information related to the generated answer, but the answer is not clear. The instructions may include assigning a rating of 2 if the relevant FAQ data contains some information related to the answer, but the answer is not possible. The instructions may include assigning a rating of 1 if the relevant FAQ data contains no relevant information and the answer is not possible.

[0091] In step S413, the response / evaluation generation AI 402 generates a response and performs an evaluation. In step S414, the response / evaluation generation AI 402 sends the response and evaluation to the server 300.

[0092] In step S415, the server 300 performs processing to display the acquired answers and evaluations on the user terminal 100.

[0093] Here, Figure 7 is a flowchart that specifically explains the response and evaluation display process of step S415 in Figure 4, which is performed by the server 300. Figure 8 is an explanatory diagram showing an example of the response display screen on the user terminal 100. Figure 9 is a diagram showing an example of the display screen on the user terminal 100 when the evaluation is 3 or lower (negative), where (A) is an example of the display of a message indicating that a response is not possible and the evaluation, and (B) is an example of the display of the inquiry data 903.

[0094] In step S701, the server 300 determines whether the acquired evaluation is a rating of 5. If the server 300 determines that the acquired evaluation is a rating of 5 (YES in step S701), in step S702, it sends the acquired answer and evaluation to the user terminal 100 (step S419 in Figure 4). The user terminal 100 displays the answer 801 and evaluation 802 (confidence level of the answer) as shown in Figure 8, for example (step S420). The answer 801 may include not only answers to mandatory questions but also answers to optional questions in natural language. This is expected to improve the user 101's satisfaction with the answer.

[0095] Furthermore, the server 300 transmits the reference FAQ data that served as the basis for the answer, along with its URL. For the user 101's reference, the server 300 also transmits the relevant FAQ data obtained in step S411, along with its URL. As shown in Figure 8, the user terminal 100 can display these FAQ data 322 with links to each FAQ data 322 inserted, as the source Q&A 804 and related Q&A 805 (step S420).

[0096] If the server 300 determines that the acquired evaluation is not a rating of 5 (NO in step S701), in step S703, it determines whether the acquired evaluation is a rating of 4. If the server 300 determines that the acquired evaluation is a rating of 4 (YES in step S703), in step S704, it sends a message to the user terminal 100 prompting it to check the reference FAQ along with the acquired answer and evaluation. That is, in addition to displaying the case where the evaluation is 5, the server 300 sends the above-mentioned message prompting confirmation (step S419).

[0097] As shown in Figure 8, the user terminal 100 displays message 803 along with the answer 801 and evaluation 802 (step S420). Message 803 may be, "The generated AI's answer may contain errors. Please also check the source Q&A." By displaying message 803 along with the answer 801 and evaluation 802, even if the displayed answer lacks sufficient information to answer the user-inputted question, the user 101 can be prompted to obtain a more accurate answer from the source Q&A.

[0098] If the server 300 determines that the acquired evaluation is not a rating of 4 (NO in step S703), in step S705, it instructs the inquiry data generation AI 403 to generate inquiry data, assuming that the acquired evaluation is a rating of 3 or lower and the evaluation is negative (step S416). The inquiry data may be the text of an inquiry that user 101 sends to the human resources department 201 (administrator).

[0099] Inquiry data is recorded in a format pre-set by the Human Resources Department 201, as shown in Figure 9(B), and may include fields such as "Inquiry Subject," "Inquiry Category," and "Inquiry Details." The "Inquiry Subject" is a concise description of the "Inquiry Details." The "Inquiry Category" is a category pre-defined by the Human Resources Department 201, such as "Attendance / Leave." The "Inquiry Details" is the content of the inquiry addressed to the Human Resources Department 201, generated based on an inquiry form or similar.

[0100] The instructions given to the inquiry data generation AI 403 may include instructions to generate an inquiry subject consisting of a concise subject. The instructions may include instructions to select a category appropriate to the inquiry content from a predetermined category. The instructions may include instructions to generate the inquiry content in natural language, as if written by user 101, based on user input questions or inquiry forms.

[0101] The inquiry data generation AI 403 generates inquiry data based on the instructions (step S417) and sends the generated inquiry data to the server 300 (step S418).

[0102] In step S706 (step S419 in Figure 4), the server 300 sends a message indicating that the inquiry cannot be answered, along with an evaluation, and the inquiry data generated by the inquiry data generation AI 403 to the user terminal 100. The user terminal 100 then displays the message indicating that the inquiry cannot be answered, the evaluation, and the inquiry data that it has received (step S420), as shown in Figure 9(A).

[0103] The user terminal 100 displays a response 902 consisting of an evaluation 901 and a message indicating that it is not possible to answer. The user terminal 100 also displays inquiry data 903 along with a message (response 902) prompting user 101 to contact a pre-prepared contact point such as the human resources department 201. The message prompting the user to contact a contact point is, for example, "Please send this information to the contact point," as shown in Figure 9. The inquiry data 903 shown in Figure 9(B) can be displayed, for example, by clicking "here" in the message prompting the user to contact a contact point in Figure 9(A).

[0104] Inquiry data 903 may be generated and displayed as if it were created by user 101, so that user 101 can use it as is to contact the person in charge. User 101 can then make an inquiry to the Human Resources Department 201 by copying the text of inquiry data 903 and pasting it into, for example, an inquiry form prepared in advance by the Human Resources Department 201.

[0105] <3. Addendum> The present invention is not limited to the above embodiments, and various modifications are possible.

[0106] For example, the generation AI 400 and the search engine 500 may be functions provided via a network such as the Internet, or they may be functions implemented within the server 300 by a program owned by the server 300.

[0107] Furthermore, the query extension generation AI 401, the response / evaluation generation AI 402, and the inquiry data generation AI 403 may be implemented using the same generation AI model, or they may each be implemented using different generation AI models.

[0108] The evaluation target portion for which the response / evaluation generation AI 402 performs evaluation may be any part of the response, not just the required question. For example, each sentence of the response may be designated as the evaluation target portion, and a partial evaluation may be performed on the identified evaluation target portion. Furthermore, the evaluation target portion may be identified by the instructions for the evaluation target portion received from the user 101 when the user 101 inputs the question. In addition, the evaluation target portion may be determined by the generation AI 400 itself. [Explanation of symbols]

[0109] 1: System 21: Edit button 22: Delete button 51: Button 100: User terminal 101: User 110: Processor 120: Memory 200: Administrator terminal 201:Human Resources Department 210: Processor 220: Memory 300: Server 310: Processor 320: Memory 321: Computer Program 322: FAQ Data 323: Correspondence Table 400:Generation AI 401: Query Extension Generation AI 402: AI for generating answers and evaluations 403: AI for generating inquiry data 500: Search Engine 501: Full-text search engine 502: Hybrid Search Engine 801:Answer 803: Message 804: Source Q&A 805: Related Q&A 902:Answer 903: Inquiry Data

Claims

1. A computer implementation method that is executed by one or more computers, A generative artificial intelligence generates an answer to a question, and the generative artificial intelligence generates an answer to the question input to the generative artificial intelligence. The artificial intelligence that generates the above-mentioned answers is evaluated, The results of the above evaluation are displayed. This includes, Performing the aforementioned evaluation means Identify the part of the above response that is to be evaluated, Perform a partial evaluation on the identified portion to be evaluated. Including, Computer implementation method.

2. Identifying the portion to be evaluated means Identification information used to identify the part to be evaluated is obtained, Using the aforementioned identification information, the portion of the response to be evaluated is identified. Including, The computer implementation method according to claim 1.

3. Identifying the portion to be evaluated is performed by artificial intelligence that uses the identification information to identify the portion to be evaluated in the response. The computer implementation method according to claim 2.

4. The generation of the aforementioned answer includes generating the aforementioned answer by generating an extended search using the search results for the data to be searched, Performing the aforementioned evaluation includes evaluating the degree of agreement between the response generated by the generative artificial intelligence and the search results. The computer implementation method according to claim 1.

5. The aforementioned search target data includes multiple FAQ data, The aforementioned FAQ data is a combination of questions and answers. The computer implementation method according to claim 4.

6. The aforementioned FAQ data is a combination of questions and answers regarding the personnel system. The computer implementation method according to claim 5.

7. If the result of the evaluation is negative, the system further includes generating inquiry data in a pre-set format based on the content of the question input to the generating artificial intelligence. Claim 1 Computer implementation method.

8. The question input to the aforementioned generating artificial intelligence is generated based on the question input by the user. The following further includes displaying the inquiry data along with a message prompting the user to contact the inquiry destination using the inquiry data: The computer implementation method according to claim 7.

9. A generative artificial intelligence generates an answer to a question, and the generative artificial intelligence generates an answer to the question input to the generative artificial intelligence. The artificial intelligence that generates the above-mentioned answers is evaluated, The results of the above evaluation are displayed. It is configured to perform a process that includes the following: Performing the aforementioned evaluation means Identify the part of the above response that is to be evaluated, Perform a partial evaluation on the identified portion to be evaluated. Including, system.

10. A generative artificial intelligence generates an answer to a question, and the generative artificial intelligence generates an answer to the question input to the generative artificial intelligence. The artificial intelligence that generates the above-mentioned answers is evaluated, The results of the above evaluation are displayed. A computer program that causes a computer to perform a process that includes the following: Performing the aforementioned evaluation means Identify the part of the above response that is to be evaluated, Perform a partial evaluation on the identified portion to be evaluated. Including, Computer program.

Images