Program, method, information processing apparatus, system

Abstract

Provided are a suitable system, program, method, and information processing apparatus for outputting answer contents corresponding to questions. 【Solution means】In an information processing system that stores pairs of common questions (query information) from a user and their answers (conversation cards) (question-answer information), when the conversation log extraction process receives an input of an input string, it searches for at least either the question or the answer included in the question-answer information based on the string by full-text search or the like, and outputs the question-answer information related to the search result in association with the degree of relevance (score) with the input string.

Description

Technical Field

[0001] The present disclosure relates to a program, a method, an information processing apparatus, and a system.

Background Art

[0002] A system (chatbot system) that outputs response contents corresponding to questions in response to online inquiries such as questions from users is known. Patent Document 1 discloses a technique that enables seamless transfer to a responder when a chatbot automatically responds to an inquiry from a user and is then taken over by a human response.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a problem that a suitable system has not been realized. Therefore, the present disclosure has been made to solve the above problems, and an object thereof is to provide a technique for realizing a suitable system.

Means for Solving the Problems

[0005] A program for causing a computer including a processor and a memory unit to execute. The processor performs a log acquisition step of acquiring one or more query information received from a user and conversation log information including one or more conversation cards indicating answers to the one or more query information; a first generation step of generating a first prompt by including the conversation log information acquired in the log acquisition step in an instruction to extract query information; a first transmission step of transmitting the first prompt generated in the first generation step to an artificial intelligence system; a first reception step of receiving output content including the first query information included in the conversation log information in response to the first prompt from the artificial intelligence system; a second generation step of generating a second prompt by including the first query information received in the first reception step and a conversation card group including a plurality of conversation cards in an instruction to extract conversation cards; a second transmission step of transmitting the second prompt generated in the second generation step to the artificial intelligence system; a second reception step of receiving output content including the first conversation card included in the conversation card group in response to the second prompt from the artificial intelligence system; a third generation step of generating a third prompt by including the first query information received in the first reception step, the first conversation card received in the second reception step, and a second conversation card indicating an answer to the first query information in the conversation log information in an instruction to evaluate query information and conversation cards; a third transmission step of transmitting the third prompt generated in the third generation step to the artificial intelligence system; a third reception step of receiving output content including a first evaluation result for the first conversation card and a second evaluation result for the second conversation card in response to the third prompt from the artificial intelligence system; and an improvement proposal step of proposing improvement measures for the conversation card for the first query information based on a combination of the first evaluation result and the second evaluation result.

Effect of the Invention

[0006] According to the present disclosure, a suitable system can be realized.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all the drawings for describing the embodiments, common components are denoted by the same reference numerals, and repeated descriptions are omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Also, not all of the components shown in the embodiments are essential components of the present disclosure. Further, each figure is a schematic diagram and is not necessarily drawn precisely.

[0009] <Configuration of System 1> System 1 in the present disclosure is an information processing system that outputs a character string related to a response sentence in accordance with an input question sentence received from a user. Specifically, System 1 is an information processing system that stores pairs of common questions (query information) from users and their answers (conversation cards) (question-and-answer information). When System 1 receives an input of an input character string, it searches for at least either the question or the answer included in the question-and-answer information based on the character string by full-text search or the like, and can output the question-and-answer information related to the search result in association with the degree of relevance (score) with the input character string. For example, in a bank or the like, it receives question sentences from users regarding account opening, domestic and international money transfers, loan applications, ATMs, how to use cash cards, how to use mobile banking, etc., and provides an information processing service that provides a response sentence (conversation card) for solving the doubts and problems of the users related to the question sentence to the users. The application of the information processing service according to the present disclosure is not limited to banks, and can be applied to any business entity such as public transportation such as railways, airplanes, taxis, accommodation facilities such as hotels and guesthouses, other software services, hospitals, etc. System 1 includes information processing devices of server 10, user terminal 20, administrator terminal 30, and generation AI 50 connected via network N. FIG. 1 is a block diagram showing the functional configuration of System 1. FIG. 2 is a block diagram showing the functional configuration of server 10. FIG. 3 is a block diagram showing the functional configuration of user terminal 20. FIG. 4 is a block diagram showing the functional configuration of administrator terminal 30.

[0010] Each information processing device is composed of a computer including an arithmetic unit and a storage unit. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the server 10, the user terminal 20, the administrator terminal 30, and the generation AI 50, descriptions overlapping with the basic hardware configuration of the computer and the basic functional configuration of the computer described later are omitted.

[0011] <Configuration of Server 10> The server 10 is an information processing device that provides an information processing service (question answering service) for outputting a suitable answer sentence (conversation card) for a question sentence (query information) received from a user. The server 10 includes a storage unit 101 and a control unit 104.

[0012] <Configuration of the Storage Unit 101 of Server 10> The storage unit 101 of the server 10 includes an application program 1011, a user table 1012, a conversation card table 1013, a conversation log table 1014, and a problem query table 1021.

[0013] The application program 1011 is a program for causing the control unit 104 of the server 10 to function as each functional unit. The application program 1011 includes applications such as a web browser application.

[0014] The user table 1012 is a table for storing and managing information of member users (hereinafter referred to as users) who use the service. By registering for use of the service, the information of the user is stored in a new record of the user table 1012. Thereby, the user can use the service according to the present disclosure. The user table 1012 is a table having columns of user ID, user name, and user data with the user ID as the primary key. FIG. 5 is a diagram showing the data structure of the user table 1012.

[0015] The user ID is an item that stores user identification information for identifying a user. The user identification information is an item for which a unique value is set for each user. The user name is an item that stores the user's name. The user name may be set to any string such as a nickname instead of the real name. User data includes the user's personal unique information and attribute information related to the user's characteristics and background. The user's unique information includes the user's unique information such as date of birth (age), gender, etc. The user's attribute information includes information such as the user's educational background (highest level of education, major, graduation year), occupation, work history, interests, place of residence, language, etc.

[0016] The conversation card table 1013 is a table for storing and managing information related to conversation cards (conversation card information). The conversation card table 1013 is a table having columns of card ID, question pattern, and conversation card, with the card ID as the primary key. Figure 6 is a diagram showing the data structure of the conversation card table 1013.

[0017] The card ID is an item that stores card identification information for identifying a conversation card. The card identification information is an item for which a unique value is set for each conversation card information. The question pattern is an item that stores the pattern of the question text (query information) to be presented for a conversation card. Specifically, it is determined whether the query information input from the user corresponds to the question pattern, and if it does, the conversation card is presented to the user. The degree of match (relevance, correspondence, fitness) between the query information and the question pattern may be determined as a score, and it may also be configured to present the conversation card with the highest score to the user. For example, the question pattern stores a character string such as "Please teach me how to open a bank account". The conversation card is an item that stores the answer content (conversation card information) for the query information input from the user. For example, the conversation card includes a character string such as "To open a bank account, please prepare Document 1, Document 2, Document 3, ···".

[0018] The conversation log table 1014 is a table for storing and managing information related to the conversation log (conversation log information). In the present disclosure, for the sake of simplicity, as an example, a configuration in which a conversation card (card ID) is associated with one query data will be described, but the present disclosure is not limited thereto. Specifically, the conversation log includes a conversation card for query information received from the user, and further includes query information additionally received from the user and information on the conversation card for the query information, and may include information on a continuous pair of a question sentence from the user and an answer sentence from the system 1 (also referred to as a conversation rally or a conversation session). That is, the item of query data may include a plurality of pieces of query information, and the card ID may include conversation card identification information for the plurality of pieces of query information. The conversation log table 1014 is a table having columns of a log ID, query data, a card ID, and an extraction flag, with the log ID as the primary key. FIG. 7 is a diagram showing the data structure of the conversation log table 1014.

[0019] The log ID is an item for storing log identification information for identifying a log. The log identification information is an item for which a unique value is set for each piece of log information. The query data is an item for storing a question sentence (query information) received from the user. In addition to a character string such as a question sentence (query information) received from the user, the query information may include input information (selection information) by the user, such as "useful" or "not useful" buttons presented to the user together with the conversation card. That is, the query information may include an evaluation result from the user for the conversation card presented to the user. The card ID is an item for storing card identification information for identifying a conversation card. The extraction flag is an item for storing an extraction flag when extracting conversation log information based on a predetermined condition.

[0020] The problem query table 1021 is a table for storing and managing information related to problem queries (problem query information). Specifically, query information (problem query information) that could not provide an appropriate answer to the query information received from the user is stored from the conversation log information. The problem query table 1021 is a table having columns of a problem query ID, problem query data, conversation card ID, conversation card evaluation result, preferred conversation card ID, preferred conversation card evaluation result, and improvement measures, with the problem query ID as the primary key. FIG. 8 is a diagram showing the data structure of the problem query table 1021.

[0021] The problem query ID is an item for storing problem query identification information for identifying a problem query. The problem query identification information is an item for which a unique value is set for each problem query information. The problem query data is an item for storing query information (problem query information) that could not provide an appropriate answer to the query information received from the user. The conversation card ID is an item for storing conversation card identification information for identifying the conversation card associated with the problem query information in the conversation log table 1014. The conversation card evaluation result is an item for storing the evaluation result of the conversation card specified by the conversation card ID with respect to the problem query information. The preferred conversation card ID is an item for storing preferred conversation card identification information for identifying a preferred conversation card. The preferred conversation card evaluation result is an item for storing the evaluation result of the preferred conversation card with respect to the problem query information. The improvement measure is an item for storing improvement measures for the problem query information.

[0022] <Configuration of the control unit 104 of the server 10> The control unit 104 of the server 10 includes a user registration control unit 1041. The control unit 104 realizes each functional unit by executing the application program 1011 stored in the storage unit 101.

[0023] The user registration control unit 1041 performs a process of storing information of a user who wishes to use the service according to the present disclosure in the user table 1012. The information stored in the user table 1012 is such that the user opens a web page or the like operated by the service provider from an arbitrary information processing terminal, inputs information into a predetermined input form, and transmits it to the server 10. The user registration control unit 1041 stores the received information in a new record of the user table 1012, and the user registration is completed. As a result, the user stored in the user table 1012 can use the service. Prior to the registration of user information by the user registration control unit 1041 in the user table 1012, the service provider may perform a predetermined review to restrict whether the user can use the service. The user ID may be any character string or number that can identify the user, any character string or number desired by the user, or the user registration control unit 1041 may automatically set any character string or number.

[0024] <Configuration of the user terminal 20> The user terminal 20 is an information processing device operated by a user who uses the service. The user terminal 20 may be, for example, a mobile terminal such as a smartphone or a tablet, or a stationary PC (Personal Computer) or a laptop PC. It may also be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch-type terminal. The user terminal 20 includes a storage unit 201, a control unit 204, an input device 206, and an output device 208.

[0025] <Configuration of the storage unit 201 of the user terminal 20> The storage unit 201 of the user terminal 20 stores a user ID 2011 and an application program 2012.

[0026] The user ID 2011 is the user's account ID. The user transmits the user ID 2011 from the user terminal 20 to the server 10. The server 10 identifies the user based on the user ID 2011 and provides the service according to the present disclosure to the user. Note that the user ID 2011 includes information such as a session ID temporarily assigned by the server 10 when identifying the user using the user terminal 20.

[0027] The application program 2012 may be stored in advance in the storage unit 201, or may be configured to be downloaded from a web server or the like operated by a service provider via a communication IF. The application program 2012 includes applications such as a web browser application. The application program 2012 includes an interpreter-type programming language such as JavaScript (registered trademark) that is executed on a web browser application stored in the user terminal 20.

[0028] <Configuration of the control unit 204 of the user terminal 20> The control unit 204 of the user terminal 20 includes an input control unit 2041 and an output control unit 2042. The control unit 204 realizes each functional unit by executing the application program 2012 stored in the storage unit 201.

[0029] <Configuration of the input device 206 of the user terminal 20> The input device 206 of the user terminal 20 includes a camera 2061, a microphone 2062, a position information sensor 2063, a motion sensor 2064, and a touch device 2065.

[0030] <Configuration of the output device 208 of the user terminal 20> The output device 208 of the user terminal 20 includes a display 2081 and a speaker 2082.

[0031] <Configuration of the administrator terminal 30> The administrator terminal 30 is an information processing terminal operated by an employee (administrator) such as an operator of a business that provides the information processing service according to the present disclosure. The administrator terminal 30 may be, for example, a mobile terminal such as a smartphone or a tablet, or a stationary PC (Personal Computer) or a laptop PC. It may also be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch-type terminal. The administrator terminal 30 includes a storage unit 301, a control unit 304, an input device 306, and an output device 308.

[0032] <Configuration of the storage unit 301 of the administrator terminal 30> The storage unit 301 of the administrator terminal 30 includes a user ID 3011 and an application program 3012.

[0033] The user ID 3011 is the account ID of the administrator. The administrator transmits the user ID 3011 from the administrator terminal 30 to the server 10. The server 10 identifies the administrator based on the user ID 3011 and provides the service according to the present disclosure to the administrator. Note that the user ID 3011 includes information such as a session ID temporarily assigned by the server 10 for identifying the user using the administrator terminal 30.

[0034] The application program 3012 may be stored in advance in the storage unit 301, or may be configured to be downloaded from a web server or the like operated by a service provider via a communication IF. The application program 3012 includes an application such as a web browser application. The application program 3012 includes an interpreter-type programming language such as JavaScript (registered trademark) that is executed on a web browser application stored in the administrator terminal 30.

[0035] <Configuration of the control unit 304 of the administrator terminal 30> The control unit 304 of the administrator terminal 30 includes an input control unit 3041 and an output control unit 3042. The control unit 304 realizes each functional unit by executing the application program 3012 stored in the storage unit 301.

[0036] <Configuration of the input device 306 of the administrator terminal 30> The input device 306 of the administrator terminal 30 includes a camera 3061, a microphone 3062, a position information sensor 3063, a motion sensor 3064, and a touch device 3065.

[0037] <Configuration of the output device 308 of the administrator terminal 30> The output device 308 of the administrator terminal 30 includes a display 3081 and a speaker 3082.

[0038] <Configuration of the generation AI 50> The generation AI 50 refers to a large-scale artificial intelligence model used in the field of natural language processing (NLP). These models can understand the patterns of the language used by humans and effectively perform natural language generation (NLG) tasks by learning a large amount of text data (web pages, books, articles, etc.). The generation AI 50 is used in many NLP tasks such as response generation to specific questions, automatic article generation, text summarization, translation, sentiment analysis, etc. It can also be applied to various uses such as education, entertainment, customer service, product development, etc. There are the following types of generation AI 50. In this disclosure, a large language model that mainly outputs text information as output information is described as a type of generation AI 50. ·OpenAI ChatGPT ·Google Bard ·Stable Diffusion ·midjourney Note that the generation AI 50 may also be configured to be realized as a part of the functions of the server 10.

[0039] <Operation of System 1> The following describes each process of System 1. FIG. 9 is a flowchart showing the operation of the conversation log extraction process. FIG. 10 is a flowchart showing the operation of the query extraction process. FIG. 11 is a flowchart showing the operation of the conversation card extraction process. FIG. 12 is a flowchart showing the operation of the improvement measure proposal process.

[0040] <Conversation Log Extraction Process> The conversation log extraction process is a process of extracting conversation logs (conversation rallies, conversation sessions) that satisfy predetermined conditions from among the conversation logs (consisting of conversation cards presented for query information input by the user in the past). In the present disclosure, as an example, a case of particularly extracting a conversation log (fallback conversation log) for which an appropriate conversation card could not be presented for a plurality of query information (inquiries) of the user and for which an input operation of "not useful" was received from the user will be described.

[0041] <Overview of Conversation Log Extraction Process> The conversation log extraction process is a series of processes that acquire a plurality of conversation logs and extract predetermined conversation logs from the conversation logs.

[0042] <Details of Conversation Log Extraction Process> The details of the conversation log extraction process will be described below.

[0043] In step S101, the control unit 104 of the server 10 refers to the conversation log table 1014 and acquires conversation log information including query data and card IDs.

[0044] In step S102, the control unit 104 of the server 10 executes a log acquisition step of acquiring conversation log information composed of one or more pieces of query information received from a user and one or more conversation cards indicating answers to the one or more pieces of query information. Specifically, the control unit 104 of the server 10 analyzes the query data included in the conversation log information, identifies the conversation log information including the query data for which an appropriate conversation card could not be presented to the query data received from the user, and stores a value of True in the extraction flag item of the record of the conversation log information in the conversation log table 1014. For example, the control unit 104 of the server 10 identifies conversation log information (conversation log information in which information indicating that an appropriate conversation card could not be presented to the user is stored) including information indicating "not useful" in the query data. The control unit 104 of the server 10 performs clustering processing on the query data stored in the conversation log table 1014, and may identify the conversation log information included in the cluster similar to the conversation log information in which information indicating that an appropriate conversation card could not be presented to the user is stored as the conversation log information including the query data for which an appropriate conversation card could not be presented. That is, similar conversation log information may be identified as the conversation log information including the query data for which an appropriate conversation card could not be presented in the conversation log information in which information indicating that an appropriate conversation card could not be presented to the user is stored. The similarity can be calculated based on the cosine similarity, Jaccard similarity, doc2vec, LSA, BERT, etc. of the query data included in the conversation log information or the content of the conversation card associated with the query data, or other similarities calculated based on the reciprocal of the Euclidean distance, Manhattan distance, etc. Alternatively, it may be calculated using the generative AI 60.

[0045] <Query Extraction Process> The query extraction process is a process of extracting predetermined query information from one or more conversation logs. In the present disclosure, as an example, an example of extracting query information (inquiries) for which an appropriate conversation card could not be presented will be described.

[0046] <Overview of Query Extraction Process> The query extraction process is a series of processes that acquires a predetermined conversation log, generates a first prompt including the predetermined conversation log, inputs the first prompt to the generation AI 60, receives the output content from the generation AI 60, and extracts first query information from the output content.

[0047] <Details of Query Extraction Process> The details of the query extraction process will be described below.

[0048] In step S301, the control unit 104 of the server 10 executes a conversation log acquisition step of acquiring predetermined conversation log information. Specifically, the control unit 104 of the server 10 refers to the conversation log table 1014 and acquires one or more pieces of conversation log information (conversation log information including query data for which an appropriate conversation card could not be presented for the query data received from the user) in which a True value is stored in the extraction flag item. In the present disclosure, as an example, an example of acquiring predetermined conversation log information extracted by the conversation log extraction process is disclosed, but it is not limited thereto. For example, it may be executed on predetermined conversation log information collected manually. In addition, it is applicable to any one or more pieces of conversation log information. That is, the query extraction process is a process that can be executed independently without executing other processes according to the present disclosure. The control unit 104 of the server 10 does not necessarily need to acquire all of the one or more query data included in the conversation log information. For example, it may extract only one or more query data for which an appropriate conversation card could not be presented for the query data received from the user. Also in this case, it is included in the conversation log information in the query extraction process. That is, the conversation log information does not necessarily need to include the entire query information included in one conversation log (conversation rally, conversation session), and it may use a part of the query information (which may be the query information for which an appropriate conversation card could not be presented) and exclude the other query information.

[0049] In step S302, the control unit 104 of the server 10 executes a first generation step of generating a first prompt by including the conversation log information acquired in the log acquisition step in an instruction to extract predetermined query information. The first generation step executes a step of generating a first prompt by including the conversation log information in an instruction to extract predetermined query information that could not indicate an answer related to an appropriate conversation card from the conversation log. Specifically, the control unit 104 of the server 10 inserts a character string of one or more query data included in the conversation log information acquired in step S301 and a character string of the answer content of the conversation card acquired by searching the card ID item of the conversation card table 1013 based on the card ID stored in association with the query data into the following first instruction at the positions of ${query data 1}, ${conversation card 1}, ${query data 2}, ${conversation card 2}... to generate the first prompt.

[0050] [First Instruction] The interaction between the web service support chatbot (assistant) and the customer (user) is provided. Please output the problems with the chatbot's response. Please output the user's questions for which the chatbot could not give an appropriate answer. User: ${query data 1} Assistant: ${conversation card 1} User: ${Query Data 2} Assistant: ${Conversation Card 2} ···

[0051] In step S303, the control unit 104 of the server 10 executes a first transmission step of transmitting the first prompt generated in the first generation step to the artificial intelligence system. Specifically, the control unit 104 of the server 10 transmits the first prompt generated in step S302 to the API (Application Programming Interface) endpoint provided by the generation AI 60.

[0052] In step S304, the control unit 104 of the server 10 executes a first reception step of receiving output content including first query information included in the conversation log information for the first prompt from the artificial intelligence system. Specifically, the control unit 104 of the server 10 receives a response to the transmitted first prompt from the generation AI 60. The response includes predetermined query data (first query data) that could not indicate an answer related to an appropriate conversation card from among one or more query data input to the first prompt from the conversation log.

[0053] In step S305, the control unit 104 of the server 10 extracts and acquires predetermined query data (first query data) that could not indicate an answer related to an appropriate conversation card from the received response. Thereby, it is possible to extract query information for which an appropriate answer could not be given to the query information received from the user from the conversation log information.

[0054] <Conversation Card Extraction Process> The conversation card extraction process is a process of extracting a predetermined conversation card from a conversation card group composed of a plurality of conversation cards for predetermined query information. In the present disclosure, as an example, an example of extracting the most appropriate conversation card among the conversation cards included in the conversation card group for a specific piece of query information will be described.

[0055] <Overview of Conversation Card Extraction Process> The conversation card extraction process is a series of processes that acquire first query information, acquire a conversation card group, generate a second prompt including the first query information and the conversation card group, input the second prompt to the generation AI 60, receive the output content from the generation AI 60, and extract the first conversation card from the output content.

[0056] <Details of Conversation Card Extraction Process> The details of the conversation card extraction process will be described below.

[0057] In step S501, the control unit 104 of the server 10 executes a query acquisition step of acquiring predetermined query information. Specifically, the control unit 104 of the server 10 acquires the first query information (first query data) extracted in the query extraction process. In the present disclosure, as an example, an example of acquiring the first query information (first query data) extracted by the query extraction process is disclosed, but it is not limited thereto. For example, it may be executed for the first query information manually extracted from the conversation log information. In addition, it is applicable to any one or a plurality of pieces of query information. That is, the conversation card extraction process is a process that can be executed independently without executing other processes according to the present disclosure.

[0058] In step S502, the control unit 104 of the server 10 executes a conversation card group acquisition step of acquiring a conversation card group. Specifically, the control unit 104 of the server 10 acquires one or more conversation card information by referring to the conversation card table 1013. The control unit 104 of the server 10 may acquire all the conversation card information in the conversation card table 1013 as a conversation card group, or may acquire a part of the conversation card information related to the predetermined query information acquired in step S501 as a conversation card group. For example, conversation card information with a score such as a degree of match (degree of relevance, degree of applicability, degree of conformity) calculated by comparing the predetermined query information with the query pattern in the conversation card table 1013 being equal to or greater than a predetermined value may be specified and acquired as the conversation card group. The degree of match can be calculated based on, for example, the cosine similarity, Jaccard similarity, doc2vec, LSA, BERT, etc. of the query data included in the conversation log information or the content of the conversation card associated with the query data, or the reciprocal of other Euclidean distances, Manhattan distances, etc. Alternatively, it may be calculated using the generative AI 60.

[0059] In step S503, the control unit 104 of the server 10 executes a second generation step of generating a second prompt by including, in the instruction to extract a predetermined conversation card, the first query information received in the first reception step and the conversation card group composed of a plurality of conversation cards. The second generation step executes a step of generating a second prompt by including the conversation card group in the instruction to extract a predetermined conversation card suitable for the first query information from the conversation card group. Specifically, the control unit 104 of the server 10 inserts the character string of the first query data acquired in step S501 into the ${first query data} part of the second instruction shown below. The control unit 104 of the server 10 inserts the character strings of the answer contents of the respective conversation cards included in the conversation card group acquired in step S502 into the ${conversation card 1}, ${conversation card 2}, ${conversation card 3}... parts of the second instruction shown below. The control unit 104 of the server 10 thereby generates the second prompt.

[0060] 〔Second Instruction〕 Please output the conversation card suitable for the inquiry sentence from the customer among the following conversation cards. # Inquiry sentence from customer ${First query data} # Conversation card ${Conversation card 1} ${Conversation card 2} ${Conversation card 3} ···

[0061] In step S504, the control unit 104 of the server 10 executes a second transmission step of transmitting the second prompt generated in the second generation step to the artificial intelligence system. Specifically, the control unit 104 of the server 10 transmits the second prompt generated in step S503 to the API (Application Programming Interface) endpoint provided by the generation AI 60.

[0062] In step S505, the control unit 104 of the server 10 executes a second reception step of receiving output content including the first conversation card included in the conversation card group for the second prompt from the artificial intelligence system. Specifically, the control unit 104 of the server 10 receives a response to the transmitted second prompt from the generation AI 60. The response includes information indicating a conversation card (first conversation card) suitable as an answer to the first query data among the conversation card group input to the second prompt.

[0063] In step S506, the control unit 104 of the server 10 extracts and acquires a conversation card (first conversation card) suitable as an answer to the first query data among the conversation card group input to the second prompt from the received response. Thereby, a conversation card (first conversation card) suitable as an answer to the first query data can be extracted from the conversation card group.

[0064] <Improvement measure proposal process> The improvement measure proposal process is a process of proposing improvement measures for predetermined query information. In the present disclosure, as an example, a process of proposing an improvement measure for presenting a more suitable conversation card for query information (inquiries) for which it was not possible to present a particularly appropriate conversation card will be described. Specifically, the improvement measure includes at least one or more of improvement of the response content of the conversation card, improvement of the question pattern associated with the conversation card, and creation of a new conversation card.

[0065] <Outline of the improvement measure proposal process> The improvement measure proposal process is a series of processes that acquire first query information, acquire a first conversation card, acquire a second conversation card, generate a third prompt including the first query information, the first conversation card, and the second conversation card, input the third prompt to the generation AI 60, receive the output content from the generation AI 60, acquire an evaluation result from the output content, and propose an improvement measure based on the evaluation result.

[0066] <Details of the improvement measure proposal process> Details of the improvement measure proposal process will be described below.

[0067] In step S701, the control unit 104 of the server 10 executes a query acquisition step of acquiring predetermined query information. Specifically, the control unit 104 of the server 10 acquires the first query information (first query data) extracted in the query extraction process. In the present disclosure, as an example, an example of acquiring the first query information (first query data) extracted by the query extraction process is disclosed, but it is not limited thereto. For example, it may be executed for the first query information manually extracted from the conversation log information. In addition, it is applicable to any one or a plurality of query information. The improvement measure proposal process is a process that can be executed independently of the query extraction process.

[0068] In step S702, the control unit 104 of the server 10 executes a first conversation card acquisition step of acquiring a predetermined conversation card. Specifically, the control unit 104 of the server 10 acquires the first conversation card extracted in the conversation card extraction process. In the present disclosure, as an example, an example of acquiring the first conversation card extracted by the conversation card extraction process is disclosed, but it is not limited thereto. For example, it may be executed for the first conversation card extracted manually from the conversation card group. In addition, it is applicable to any conversation card. The improvement measure proposal process is a process that can be executed independently of the conversation card extraction process.

[0069] In step S703, the control unit 104 of the server 10 executes a second conversation card acquisition step of acquiring a second conversation card indicating an answer to the first query data. Specifically, the control unit 104 of the server 10 acquires the first query data acquired in step S701 and the card ID (the card ID of the second conversation card indicating the answer to the first query information in the conversation log information) stored in association in the conversation log table 1014. The control unit 104 of the server 10 searches the card ID item of the conversation card table 1013 based on the acquired card ID and acquires a conversation card (second conversation card).

[0070] <Third prompt generation step (first embodiment)> In step S704, the control unit 104 of the server 10 executes a 3A generation step of generating a 3A prompt by including the first query information received in the first reception step and the first conversation card received in the second reception step in an instruction to evaluate query information and a conversation card. The 3A generation step executes a step of generating a 3A prompt by including the first query information and the first conversation card in an instruction to output information indicating the benefit degree indicating the benefit of the conversation card with respect to the query information.

[0071] Specifically, the control unit 104 of the server 10 inserts the character string of the first query data obtained in step S701 into the location of ${first query data} in the following third A instruction. The control unit 104 of the server 10 inserts the character string of the response content of the first conversation card obtained in step S702 into the location of ${conversation card} in the following third A instruction. Thereby, the control unit 104 of the server 10 generates a third A prompt.

[0072] [Third A Instruction] Please output the benefit as the response content to the inquiry sentence from the customer for the following conversation card. The benefit should be output in ranks A, B, and C in descending order of benefit. # Inquiry sentence from customer ${first query data} # Conversation card ${conversation card}

[0073] In step S704, the control unit 104 of the server 10 executes a third B generation step of generating a third B prompt by including the first query information received in the first reception step and the second conversation card indicating the response to the first query information in the conversation log information in the instruction to evaluate the query information and the conversation card. The third B generation step executes a step of generating a third B prompt by including the first query information and the second conversation card in the instruction to output information indicating the benefit indicating the benefit of the conversation card for the query information.

[0074] Specifically, the control unit 104 of the server 10 inserts the character string of the first query data obtained in step S701 into the location of ${first query data} in the third A instruction. The control unit 104 of the server 10 inserts the character string of the response content of the second conversation card obtained in step S703 into the location of ${conversation card} in the following third A instruction. Thereby, the control unit 104 of the server 10 generates a third B prompt.

[0075] In the third prompt generation step (first embodiment), for each of the first conversation card and the second conversation card, two prompts, namely the 3A prompt and the 3B prompt, are generated.

[0076] <Third Prompt Generation Step (Second Embodiment)> In step S704, the control unit 104 of the server 10 executes a 3C generation step of generating a 3C prompt by including the first query information received in the first reception step, the first conversation card received in the second reception step, and the second conversation card indicating an answer to the first query information in the conversation log information in an instruction to evaluate the query information and the conversation card. The 3C generation step executes a step of generating a 3C prompt by including the first query information, the first conversation card, and the second conversation card in an instruction to output information indicating the degree of benefit indicating the benefit of the conversation card for the query information.

[0077] Specifically, the control unit 104 of the server 10 inserts the character string of the first query data acquired in step S701 into the ${first query data} part of the 3C instruction shown below. The control unit 104 of the server 10 inserts the character string of the answer content of the first conversation card acquired in step S702 into the ${conversation card 1} part of the 3C instruction shown below. The character string of the answer content of the second conversation card acquired in step S702 is inserted into the ${conversation card 2} part. Thereby, the control unit 104 of the server 10 generates a 3C prompt.

[0078] 〔3C Instruction〕 For each of the following conversation cards 1 and 2, please output the degree of benefit as the answer content to the inquiry sentence from the customer. The degree of benefit should be output in the ranks of A, B, and C in descending order of the degree of benefit. # Inquiry sentence from the customer ${first query data} # Conversation card 1 ${conversation card 1} # Conversation card 2 Conversation Card 2

[0079] In step S705, the control unit 104 of the server 10 executes a 3A transmission step of transmitting the 3A prompt generated in the 3A generation step to the artificial intelligence system. The control unit 104 of the server 10 executes a 3B transmission step of transmitting the 3B prompt generated in the 3B generation step to the artificial intelligence system. The control unit 104 of the server 10 executes a 3C transmission step of transmitting the 3C prompt generated in the 3C generation step to the artificial intelligence system. Specifically, the control unit 104 of the server 10 transmits the 3A, 3B, and 3C prompts generated in step S704 to the API (Application Programming Interface) endpoint provided by the generation AI 60.

[0080] In step S706, the control unit 104 of the server 10 executes a 3A reception step of receiving output content including a first evaluation result for the first conversation card with respect to the 3A prompt from the artificial intelligence system. The control unit 104 of the server 10 executes a 3B reception step of receiving output content including a second evaluation result for the second conversation card with respect to the 3B prompt from the artificial intelligence system. The control unit 104 of the server 10 executes a 3C reception step of receiving output content including a first evaluation result for the first conversation card and a second evaluation result for the second conversation card with respect to the 3C prompt from the artificial intelligence system. Specifically, the control unit 104 of the server 10 receives responses from the generation AI 60 for each of the transmitted 3A, 3B, and 3C prompts. The responses include evaluation results for the first conversation card (3A prompt, 3C prompt) and evaluation results for the second conversation card (3A prompt, 3B prompt). The response includes strings indicating A, B, and C as the benefit level of the first conversation card (the first benefit level). The response includes strings indicating A, B, and C as the benefit level of the second conversation card (the second benefit level).

[0081] In step S708, the control unit 104 of the server 10 executes an improvement proposal step of proposing improvement measures for the conversation card for the first query information based on the combination of the first evaluation result and the second evaluation result. The improvement proposal step executes a step of proposing improvement measures for the first conversation card or the second conversation card based on the combination of the first evaluation result and the second evaluation result. The improvement proposal step executes a step of proposing at least one of the following improvement measures based on the combination of the first evaluation result and the second evaluation result: improvement of the response content of the second conversation card, improvement of the question pattern associated with the second conversation card, and creation of a new conversation card. Specifically, the control unit 104 of the server 10 proposes the following improvement measures based on the combination of the first benefit level and the second benefit level. For example, based on the combination of the first benefit level and the second benefit level, the following improvement measures for the first conversation card and the second conversation card are proposed. · First benefit level C, second benefit level C: Creation of a new conversation card · First benefit level B, second benefit level C: Improvement of the question pattern associated with the first conversation card or the second conversation card, and improvement of the response content of the first conversation card or the second conversation card · First benefit level A, second benefit level C: Improvement of the question pattern associated with the first conversation card or the second conversation card (review of the question pattern associated with the first conversation card and the second conversation card so that the first conversation card is presented for the first query data) · First benefit level B, second benefit level B: Improvement of the response content of the first conversation card or the second conversation card · First benefit level A, second benefit level B: Improvement of the question pattern associated with the first conversation card or the second conversation card, or improvement of the response content of the first conversation card or the second conversation card · First Benefit Degree A, Second Benefit Degree A: Improvement of the response content of the first conversation card or the second conversation card

[0082] The details of the processing of each improvement measure will be described below.

[0083] <Improvement Measure A (Improvement of the response content of the conversation card)> In step S708, the control unit 104 of the server 10 executes a 4A generation step of generating a 4A prompt by including the first query information and the second conversation card in an instruction to improve the response content of the conversation card. Specifically, the control unit 104 of the server 10 inserts the character string of the first query data obtained in step S701 into the ${first query data} part of the following 4A instruction. The control unit 104 of the server 10 inserts the character string of the response content of the second conversation card obtained in step S703 into the ${conversation card} part of the following 4A instruction. Thereby, the control unit 104 of the server 10 generates a 4A prompt. Note that the character string of the response content of the first conversation card may be inserted into the ${conversation card} part. Also, it may be configured to accept the input of character strings such as a draft text, a correction policy, etc. for improving the content of the conversation card by omitting the inquiry sentence (${first query data}) from the customer.

[0084] 〔4A Instruction〕 Please improve the content of the following conversation card to be suitable for the inquiry sentence from the customer. # Inquiry sentence from the customer ${first query data} # Conversation card ${conversation card}

[0085] In step S708, the control unit 104 of the server 10 executes a 4A transmission step of transmitting the 4A prompt generated in the 4A generation step to the artificial intelligence system. Specifically, the control unit 104 of the server 10 transmits the fourth A prompt generated in step S704 to the API (Application Programming Interface) endpoint provided by the generation AI 60.

[0086] In step S708, the control unit 104 of the server 10 executes a fourth A reception step of receiving, from the artificial intelligence system, output content for improving the response content of the second conversation card for the fourth A prompt. Specifically, the control unit 104 of the server 10 receives, from the generation AI 60, a response for each of the transmitted fourth A prompts. The response includes output content (response content) for improving the response content of the second conversation card.

[0087] In step S708, the improvement proposal step includes a step of proposing improvement measures for improving the response content of the second conversation card received in the fourth A reception step. Specifically, the control unit 104 of the server 10 transmits the received response content to the administrator terminal 30. The control unit 204 of the administrator terminal 30 outputs the received response content to the display 3081 of the administrator terminal 30 and presents it to the administrator. In the present disclosure, the administrator is an administrator such as an employee of an operator who operates the information processing service related to the system 1, and it is assumed that various information stored in the storage unit 201 of the system 1 can be edited by operating the administrator terminal 30. The administrator operates the input device 306 of the administrator terminal 30 to correct the item of the response content of the second conversation card stored in the conversation card table 1013 with reference to the presented response content. Alternatively, the output content output from the generation AI 60 may be used as the response content of the second conversation card. Thereby, when a query similar to the first query data is made by the user, the second conversation card including the corrected response content is presented to the user. A conversation card suitable for the user's query can be presented.

[0088] <Improvement Measure B (Improvement of Association with Query Data of Conversation Card)> In step S708, the control unit 104 of the server 10 executes a 4B generation step of generating a 4B prompt by including the first query information and the second conversation card in an instruction to improve the question pattern associated with the conversation card. Specifically, the control unit 104 of the server 10 inserts the character string of the first query data acquired in step S701 into the location of ${first query data} in the following 4B instruction. The control unit 104 of the server 10 inserts the character string of the answer content of the first conversation card acquired in step S703 into the location of ${conversation card 1} in the following 4B instruction, and inserts the character string related to the question pattern associated with the first conversation card in the conversation card table 1013 into the location of ${question pattern 1}. The control unit 104 of the server 10 inserts the character string of the answer content of the second conversation card acquired in step S703 into the location of ${conversation card 2} in the following 4B instruction, and inserts the character string related to the question pattern associated with the second conversation card in the conversation card table 1013 into the location of ${question pattern 2}.

[0089] 〔4B Instruction〕 For the following inquiry sentence from the customer, the conversation card is identified by referring to the question pattern. In this case, please propose a correction policy for the question pattern so that the first conversation card is selected instead of the second conversation card for the following inquiry sentence from the customer. # Inquiry Sentence from Customer ${first query data} # Conversation Card 1 ${question pattern 1} ${conversation card 1} # Conversation Card 2 ${question pattern 2} ${conversation card 2}

[0090] In step S708, the control unit 104 of the server 10 executes a 4B transmission step of transmitting the 4B prompt generated in the 4B generation step to the artificial intelligence system. Specifically, the control unit 104 of the server 10 transmits the 4B prompt generated in step S704 to the API (Application Programming Interface) endpoint provided by the generation AI 60.

[0091] In step S708, the control unit 104 of the server 10 executes a 4B reception step of receiving, from the artificial intelligence system, output content for improving the question pattern associated with the second conversation card for the 4B prompt. Specifically, the control unit 104 of the server 10 receives, from the generation AI 60, a response for each of the transmitted 4B prompts. The response includes a correction policy (improvement policy) for the question pattern 1 of conversation card 1 or the question pattern 2 of conversation card 2.

[0092] In step S708, the improvement proposal step includes a step of proposing improvement measures for improving the question pattern associated with the second conversation card received in the 4B reception step. Specifically, the control unit 104 of the server 10 transmits the received response content to the administrator terminal 30. The control unit 304 of the administrator terminal 30 outputs the received correction policy to the display 3081 of the administrator terminal 30 and presents it to the administrator. The administrator operates the input device 306 of the administrator terminal 30 to correct the question pattern items of the first conversation card or the second conversation card stored in the conversation card table 1013 with reference to the presented correction policy. Alternatively, the output content output from the generation AI 60 may be used as the question pattern for each of the first conversation card and the second conversation card. As a result, when a user makes an inquiry similar to the first query data, instead of the second conversation card, the first conversation card, which is considered to be more suitable response content for the first query data, is presented to the user. It is possible to present the first conversation card that is more suitable for the inquiry from the user.

[0093] <Improvement Measure C (Creation of New Conversation Card)> In step S708, the control unit 104 of the server 10 executes a fourth C generation step of generating a fourth C prompt by including the first query information in an instruction to create a new conversation card from the query information. Specifically, the control unit 104 of the server 10 inserts the character string of the first query data obtained in step S701 into the location of ${the first query data} in the following fourth C instruction. The control unit 104 of the server 10 inserts, into the location of ${reference information} in the following fourth A instruction, document information suitable for creating response content for the first query data, such as the first conversation card, the second conversation card, one or more conversation cards stored in the other conversation card table 1013, the manual, and the FAQ. Note that the control unit 104 of the server 10 may insert document information related to the first query data among the conversation card, the manual, the FAQ, etc. into the location of ${reference information}. The relevant document information may be specified based on the similarity calculated using cosine similarity, Jaccard similarity, doc2vec, LSA, BERT, etc., or other similarity of string vectors such as the reciprocal of the Euclidean distance and the Manhattan distance, or the relevant document data related to the first query data may be specified using the generation AI 60. Alternatively, it may be specified using the generation AI 60. The fourth C instruction may be an instruction to output only at least one of the response content and the question pattern and not output the other.

[0094] 〔Fourth C Instruction〕 Please create appropriate response content for the following inquiries from customers. Also, when outputting the response content, please output the question pattern indicating the pattern of the inquiry received from the user. # Inquiry from customer ${First query data} # Reference information ${Reference information}

[0095] In step S708, the control unit 104 of the server 10 executes a 4C transmission step of transmitting the 4C prompt generated in the 4C generation step to the artificial intelligence system. Specifically, the control unit 104 of the server 10 transmits the 4C prompt generated in step S704 to the API (Application Programming Interface) endpoint provided by the generation AI60.

[0096] In step S708, the control unit 104 of the server 10 executes a 4C reception step of receiving output content including a new conversation card for the 4C prompt from the artificial intelligence system. Specifically, the control unit 104 of the server 10 receives a response for each of the transmitted 4C prompts from the generation AI60. The response includes conversation card information including a question pattern and response content (conversation card information). Note that the response may include only one of the question pattern or the conversation card (response content) and not the other.

[0097] In step S708, the improvement proposal step includes a step of proposing improvement measures including a new conversation card received in the 4C reception step. Specifically, the control unit 104 of the server 10 transmits the received question pattern and conversation card to the administrator terminal 30. The control unit 304 of the administrator terminal 30 outputs the received question pattern and conversation card to the display 3081 of the administrator terminal 30 and presents them to the administrator. The administrator operates the input device 306 of the administrator terminal 30 to store the presented question patterns and conversation cards as records of new conversation card information in the conversation card table 1013. In addition, the administrator may modify the items of the received question pattern or conversation card (answer content), and store the modified content as a record of new conversation card information in the conversation card table 1013.

[0098] <Basic Hardware Configuration of Computer> FIG. 13 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main memory device 902, an auxiliary storage device 903, and a communication IF 991 (Interface). These are electrically connected to each other by a communication bus 921.

[0099] The processor 901 is hardware for executing an instruction set described in a program. The processor 901 is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0100] The main memory device 902 is for temporarily storing programs and data processed by programs and the like. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0101] The auxiliary storage device 903 is a storage device for storing data and programs. For example, it is a flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, or the like.

[0102] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards. The network is composed of various mobile communication systems such as the Internet, LAN, wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks (such as Wi-Fi (registered trademark)) that can be connected to the Internet by a predetermined access point, etc. When connecting wirelessly, communication protocols such as Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. are included. When connecting by wire, the network also includes those directly connected by a USB (Universal Serial Bus) cable, etc.

[0103] Note that all or part of each hardware configuration can be provided in a distributed manner in a plurality of computers 90, and the computers 90 can be virtually realized by connecting them to each other via a network. In this way, the computer 90 is a concept that includes not only a single housing or a computer 90 housed in a case, but also a virtualized computer system.

[0104] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration (Figure 13) of the computer 90 will be described. The computer includes at least functional units of a control unit, a storage unit, and a communication unit.

[0105] Note that the functional units included in the computer 90 can also be realized by distributing all or part of each functional unit in a plurality of computers 90 interconnected by a network. The computer 90 is a concept that includes not only a single computer 90, but also a virtualized computer system.

[0106] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903 and expanding them in the main storage device 902, and executing processing according to the programs. The control unit can realize functional units that perform various information processes according to the type of program. Thereby, the computer is realized as an information processing device that performs information processing.

[0107] The functions realized by the components described in this specification may be implemented in circuitry or processing circuitry including a general-purpose processor, a specific-purpose processor, an integrated circuit, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), a conventional circuit, and / or a combination thereof, which are programmed to realize the described functions. The processor includes transistors and other circuits and is regarded as circuitry or processing circuitry. The processor may be a programmed processor that executes a program stored in a memory. In this specification, circuitry, unit, and means are hardware programmed to realize the described functions or hardware that executes them. The hardware may be any hardware disclosed in this specification or any hardware known as being programmed or executing to realize the described functions. When the hardware is a processor regarded as the circuitry type, the circuitry, means, or unit is a combination of hardware and software used to configure the hardware and / or the processor.

[0108] The memory unit is realized by the main memory device 902 and the auxiliary memory device 903. The memory unit stores data, various programs, and various databases. Also, the processor 901 can secure a storage area corresponding to the memory unit in the main memory device 902 or the auxiliary memory device 903 according to a program. Further, the control unit can cause the processor 901 to execute addition, update, and deletion processing of the data stored in the memory unit according to various programs.

[0109] The database refers to a relational database and is for managing by associating with each other a table in a tabular form structurally defined by rows and columns, and a data set called a master. In a database, a table is called a table, a master, a column of a table is called a column, and a row of a table is called a record. In a relational database, relationships between tables and masters can be set and associated. Normally, a column serving as a primary key for uniquely identifying a record is set for each table and each master, but setting a primary key for a column is not essential. The control unit can cause the processor 901 to execute addition, deletion, and update of records in a specific table and master stored in the memory unit according to various programs. Also, by storing data, various programs, and various databases in the memory unit, the information processing apparatus and the information processing system according to the present disclosure can be regarded as being manufactured.

[0110] Note that the database and the master in the present disclosure may include any data structure (such as a list, a dictionary, an associative array, an object, etc.) in which information is structurally defined. The data structure shall also include data that can be regarded as a data structure by combining data with functions, classes, methods, etc. described in any programming language.

[0111] The communication unit is realized by the communication IF991. The communication unit realizes the function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input it to the control unit. The control unit can cause the processor 901 to execute information processing on the received information according to various programs. Also, the communication unit can transmit the information output from the control unit to other computers 90.

[0112] <Supplementary Note> The matters described in each of the above embodiments are appended below.

[0113] (Supplementary Note 1) A program for causing a computer including a processor and a storage unit to execute, the processor includes: a log acquisition step (S102) of acquiring conversation log information including one or more query information received from a user and one or more conversation cards indicating answers to the one or more query information; a first generation step (S302) of generating a first prompt by including the conversation log information acquired in the log acquisition step in an instruction to extract query information; a first transmission step (S303) of transmitting the first prompt generated in the first generation step to an artificial intelligence system; a first reception step (S304) of receiving output content including the first query information included in the conversation log information in response to the first prompt from the artificial intelligence system; a second generation step (S503) of generating a second prompt by including the first query information received in the first reception step and a conversation card group including a plurality of conversation cards in an instruction to extract conversation cards; a second transmission step (S504) of transmitting the second prompt generated in the second generation step to the artificial intelligence system; a second reception step (S505) of receiving output content including the first conversation card included in the conversation card group in response to the second prompt from the artificial intelligence system; a third generation step (S704) of generating a third prompt by including the first query information received in the first reception step, the first conversation card received in the second reception step, and a second conversation card indicating an answer to the first query information in the conversation log information in an instruction to evaluate query information and conversation cards; a third transmission step (S705) of transmitting the third prompt generated in the third generation step to the artificial intelligence system; a third reception step (S706) of receiving output content including a first evaluation result for the first conversation card and a second evaluation result for the second conversation card in response to the third prompt from the artificial intelligence system; and an improvement proposal step (S708) of proposing improvement measures for the conversation card for the first query information based on a combination of the first evaluation result and the second evaluation result. Accordingly, predetermined query information can be extracted from conversation log information including query information received from a user and conversation cards indicating answers to the query information.

[0114] (Appendix 2) A program for causing a computer including a processor and a storage unit to execute: a log acquisition step (S102) in which the processor acquires conversation log information including one or more pieces of query information input from a user and one or more conversation cards indicating answers to the one or more pieces of query information; and a first generation step (S302) in which a first prompt is generated by including the conversation log information in an instruction to extract query information. Accordingly, predetermined query information can be extracted from conversation log information including query information received from a user and conversation cards indicating answers to the query information.

[0115] (Appendix 3) The program according to Appendix 2, wherein the first generation step (S302) is a step of generating a first prompt by including the conversation log information in an instruction to extract query information for which an answer indicating an appropriate conversation card cannot be shown from the conversation log. Accordingly, query information for which an appropriate answer could not be given to the query information received from the user can be extracted from the conversation log information.

[0116] (Appendix 4) The program according to Appendix 2, wherein the processor executes a first transmission step (S303) of transmitting the first prompt generated in the first generation step to an artificial intelligence system, and a first reception step (S304) of receiving output content including the first query information included in the conversation log information in response to the first prompt from the artificial intelligence system. Accordingly, predetermined query information can be extracted from conversation log information including query information received from a user and conversation cards indicating answers to the query information.

[0117] (Appendix 5) A second generation step (S503) in which the processor generates a second prompt by including the first query information received in the first reception step and a conversation card group composed of a plurality of conversation cards in an instruction to extract a conversation card, and the program described in Appendix 4 that executes the step. Thereby, a predetermined conversation card corresponding to the first query information can be extracted from the conversation card group.

[0118] (Appendix 6) The second generation step (S503) is a step of generating a second prompt by including the conversation card group in an instruction to extract a conversation card suitable for the first query information from the conversation card group, and the program described in Appendix 5. Thereby, a predetermined conversation card suitable for the first query information (for example, optimal as an answer to the first query information) can be extracted from the conversation card group.

[0119] (Appendix 7) A second transmission step (S504) in which the processor transmits the second prompt generated in the second generation step to the artificial intelligence system, and a second reception step (S505) in which the processor receives output content including the first conversation card included in the conversation card group for the second prompt from the artificial intelligence system, and the program described in Appendix 5 that executes the step. Thereby, a predetermined conversation card corresponding to the first query information can be extracted from the conversation card group.

[0120] (Appendix 8) The 3A generation step (S704) in which the processor generates a 3A prompt by including, in the instruction to evaluate query information and the conversation card, the first query information received in the first reception step and the first conversation card received in the second reception step; and the 3B generation step (S704) in which the processor generates a 3B prompt by including, in the instruction to evaluate query information and the conversation card, the first query information received in the first reception step and the second conversation card indicating an answer to the first query information in the conversation log information. The program according to Appendix 7 executes these steps. As a result, the evaluation results of the first query information and the first conversation card can be obtained. The evaluation results of the first query information and the second conversation card can be obtained.

[0121] (Appendix 9) The 3A generation step (S704) is a step of generating a 3A prompt by including the first query information and the first conversation card in the instruction to output information indicating the degree of benefit indicating the benefit of the conversation card for the query information. The 3B generation step (S704) is a step of generating a 3B prompt by including the first query information and the second conversation card in the instruction to output information indicating the degree of benefit indicating the benefit of the conversation card for the query information. The program according to Appendix 8. As a result, the degree of benefit of the first conversation card for the first query information can be obtained. The degree of benefit of the second conversation card for the first query information can be obtained.

[0122] (Appendix 10) A 3A transmission step (S705) in which the processor transmits the 3A prompt generated in the 3A generation step to the artificial intelligence system, a 3B transmission step (S705) in which the processor transmits the 3B prompt generated in the 3B generation step to the artificial intelligence system, a 3A reception step (S706) in which the processor receives output content including a first evaluation result for the first conversation card with respect to the 3A prompt from the artificial intelligence system, and a 3B reception step (S706) in which the processor receives output content including a second evaluation result for the second conversation card with respect to the 3B prompt from the artificial intelligence system, and the program according to Supplementary Note 8 that executes these steps. Thereby, the evaluation results of the first query information and the first conversation card can be obtained. The evaluation results of the first query information and the second conversation card can be obtained.

[0123] (Supplementary Note 11) A 3C generation step (S704) in which the processor generates a 3C prompt by including the first query information received in the first reception step, the first conversation card received in the second reception step, and the second conversation card indicating an answer to the first query information in the conversation log information in an instruction to evaluate the query information and the conversation card, and the program according to Supplementary Note 7 that executes this step. Thereby, the evaluation results of the first query information and the first conversation card can be obtained. The evaluation results of the first query information and the second conversation card can be obtained.

[0124] (Supplementary Note 12) The 3C generation step (S704) is a step of generating a 3C prompt by including the first query information, the first conversation card, and the second conversation card in an instruction to output information indicating a benefit degree indicating the benefit of the conversation card with respect to the query information, and the program according to Supplementary Note 11. Thereby, the benefit degree of the first conversation card with respect to the first query information can be obtained. The benefit degree of the second conversation card with respect to the first query information can be obtained.

[0125] (Supplementary Note 13) A 3C transmission step (S705) in which the processor transmits the 3C prompt generated in the 3C generation step to the artificial intelligence system, and a 3C reception step (S706) in which the processor receives output content including a first evaluation result for the first conversation card and a second evaluation result for the second conversation card in response to the 3C prompt from the artificial intelligence system, which are executed by the program described in Appendix 11. Thereby, the evaluation results of the first query information and the first conversation card can be obtained. The evaluation results of the first query information and the second conversation card can be obtained.

[0126] (Appendix 14) A program described in Appendix 10 or 13, in which the processor executes an improvement proposal step (S708) of proposing improvement measures for the conversation card for the first query information based on the combination of the first evaluation result and the second evaluation result. Thereby, it is possible to obtain a proposal for improvement measures for the conversation card for query information (predetermined query information) for which an appropriate answer could not be given.

[0127] (Appendix 15) The improvement proposal step (S708) is a step of proposing improvement measures for the first conversation card or the second conversation card based on the combination of the first evaluation result and the second evaluation result, which is the program described in Appendix 14. Thereby, it is possible to obtain a proposal for improvement measures for the first conversation card or the second conversation card for query information (predetermined query information) for which an appropriate answer could not be given.

[0128] (Appendix 16) The improvement proposal step (S708) is a step of proposing at least one of the improvement measures of improving the answer content of the second conversation card, improving the question pattern associated with the second conversation card, and creating a new conversation card based on the combination of the first evaluation result and the second evaluation result, which is the program described in Appendix 14. As a result, it is possible to obtain a proposal for improvement measures for query information (predetermined query information) for which an appropriate answer could not be given.

[0129] (Appendix 17) A fourth A generation step (S708) in which the processor generates a fourth A prompt by including the first query information and the second conversation card in an instruction to improve the answer content of the conversation card, a fourth A transmission step (S708) in which the fourth A prompt generated in the fourth A generation step is transmitted to the artificial intelligence system, and a fourth A reception step (S708) in which output content for improving the answer content of the second conversation card for the fourth A prompt is received from the artificial intelligence system are executed, and the improvement proposal step (S708) includes a step of proposing improvement measures for improving the answer content of the second conversation card received in the fourth A reception step. The program described in Appendix 16. As a result, it is possible to obtain a proposal for improving the answer content of the second conversation card.

[0130] (Appendix 18) A fourth B generation step (S708) in which the processor generates a fourth B prompt by including the first query information and the second conversation card in an instruction to improve the question pattern associated with the conversation card, a fourth B transmission step (S708) in which the fourth B prompt generated in the fourth B generation step is transmitted to the artificial intelligence system, and a fourth B reception step (S708) in which output content for improving the question pattern associated with the second conversation card for the fourth B prompt is received from the artificial intelligence system are executed, and the improvement proposal step (S708) includes a step of proposing improvement measures for improving the question pattern associated with the second conversation card received in the fourth B reception step. The program described in Appendix 16. As a result, it is possible to obtain a proposal for improving the question pattern associated with the second conversation card.

[0131] (Appendix 19) A fourth C generation step (S708) in which the processor generates a fourth C prompt by including the first query information in an instruction to create a new conversation card from the query information; a fourth C transmission step (S708) in which the fourth C prompt generated in the fourth C generation step is transmitted to the artificial intelligence system; and a fourth C reception step (S708) in which output content including a new conversation card for the fourth C prompt is received from the artificial intelligence system, and the improvement proposal step (S708) includes a step of proposing improvement measures including the new conversation card received in the fourth C reception step. The program according to Supplementary Note 16. Thereby, it is possible to obtain a proposal for a suitable new conversation card (answer content) suitable for the first query information.

[0132] (Supplementary Note 20) A method executed by an information processing apparatus including a processor and a storage unit, the method including the processor executing all steps executed in the invention according to any one of Supplementary Notes 2 to 13. Thereby, predetermined query information can be extracted from conversation log information including query information received from a user and a conversation card indicating an answer to the query information.

[0133] (Supplementary Note 21) An information processing apparatus including a processor and a storage unit, the processor executing all steps executed in the invention according to any one of Supplementary Notes 2 to 13. Thereby, predetermined query information can be extracted from conversation log information including query information received from a user and a conversation card indicating an answer to the query information.

[0134] (Supplementary Note 22) A system including means for executing all steps executed in the invention according to any one of Supplementary Notes 2 to 13. Accordingly, predetermined query information can be extracted from conversation log information including query information received from a user and a conversation card indicating an answer to the query information.

Explanation of Signs

[0135] 1 System, 10 Server, 101 Storage Unit, 104 Control Unit, 106 Input Device, 108 Output Device, 20 User Terminal, 201 Storage Unit, 204 Control Unit, 206 Input Device, 208 Output Device, 30 Administrator Terminal, 301 Storage Unit, 304 Control Unit, 306 Input Device, 308 Output Device, 50 Generation AI, 501 Storage Unit, 504 Control Unit, 506 Input Device, 508 Output Device

Claims

1. A program to be executed by a computer having a processor and a storage unit, The processor, a log acquisition step of acquiring conversation log information including one or more pieces of query information inputted from a user and one or more conversation cards indicating answers to the one or more pieces of query information; a first generating step of generating a first prompt by including the conversation log information acquired in the log acquiring step in an instruction to extract query information from the conversation log information; a first sending step of sending the first prompt generated in the first generating step to an artificial intelligence system; a first receiving step of receiving an output content including first query information included in the conversation log information in response to the first prompt from the artificial intelligence system; a second generating step of generating a second prompt by including the first query information received in the first receiving step and a conversation card group consisting of a plurality of conversation cards in an instruction to extract a conversation card; a second sending step of sending the second prompt generated in the second generating step to an artificial intelligence system; a second receiving step of receiving from the artificial intelligence system an output content in response to the second prompt, the output content including a first conversation card included in the group of conversation cards; a third generating step of generating a third prompt by including in an instruction to evaluate the query information and the conversation card the first query information received in the first receiving step, the first conversation card received in the second receiving step, and a second conversation card indicating an answer to the first query information in the conversation log information; a third sending step of sending the third prompt generated in the third generating step to an artificial intelligence system; a third receiving step of receiving output content from the artificial intelligence system, the output content including a first evaluation result for the first conversation card and a second evaluation result for the second conversation card in response to the third prompt; an improvement suggestion step of suggesting an improvement measure for the phrase card for the first query information based on a combination of the first evaluation result and the second evaluation result; A program that executes.

2. A program to be executed by a computer having a processor and a storage unit, The processor, a log acquisition step of acquiring conversation log information including one or more pieces of query information inputted from a user and one or more conversation cards indicating answers to the one or more pieces of query information; a first generating step of generating a first prompt by including the conversation log information acquired in the log acquiring step in an instruction to extract query information from the conversation log information; A program that executes.

3. the first generating step is a step of generating the first prompt by including the conversation log information in an instruction to extract the query information for which an answer to an appropriate conversation card cannot be indicated from the conversation log; The program according to claim 2.

4. The processor, a first sending step of sending the first prompt generated in the first generating step to an artificial intelligence system; a first receiving step of receiving an output content including first query information included in the conversation log information in response to the first prompt from the artificial intelligence system; Execute The program according to claim 2.

5. The processor, a second generating step of generating a second prompt by including the first query information received in the first receiving step and a conversation card group consisting of a plurality of conversation cards in an instruction to extract a conversation card; Execute The program according to claim 4.

6. the second generating step is a step of generating the second prompt by including the conversation card group in an instruction to extract a conversation card suitable for the first query information from the conversation card group; The program according to claim 5.

7. The processor, a second sending step of sending the second prompt generated in the second generating step to an artificial intelligence system; a second receiving step of receiving from the artificial intelligence system an output content in response to the second prompt, the output content including a first conversation card included in the group of conversation cards; Execute The program according to claim 5.

8. The processor, a third A generating step of generating a third A prompt by including the first query information received in the first receiving step and the first conversation card received in the second receiving step in an instruction to evaluate the query information and the conversation card; a third-B generating step of generating a third-B prompt by including the first query information received in the first receiving step and a second conversation card indicating an answer to the first query information in the conversation log information in an instruction to evaluate the query information and the conversation card; Execute The program according to claim 7.

9. The third A generation step is a step of generating the third A prompt by including the first query information and the first conversation card in an instruction to output information indicating a usefulness of the conversation card for the query information. The third B generation step is a step of generating the third B prompt by including the first query information and the second conversation card in an instruction to output information indicating a usefulness of the conversation card for the query information. The program according to claim 8.

10. The processor, a 3A transmission step of transmitting the 3A prompt generated in the 3A generation step to an artificial intelligence system; a 3B transmission step of transmitting the 3B prompt generated in the 3B generation step to an artificial intelligence system; a 3A receiving step of receiving output content including a first evaluation result for the first conversation card in response to the 3A prompt from the artificial intelligence system; a third B receiving step of receiving output content including a second evaluation result for the second conversation card in response to the third B prompt from the artificial intelligence system; Execute The program according to claim 8.

11. The processor, a third C generation step of generating a third C prompt by including in an instruction to evaluate the query information and the conversation card the first query information received in the first receiving step, the first conversation card received in the second receiving step, and a second conversation card indicating an answer to the first query information in the conversation log information; Execute The program according to claim 7.

12. The third C generation step is a step of generating the third C prompt by including the first query information, the first conversation card, and the second conversation card in an instruction to output information indicating a usefulness of the conversation card for the query information. The program according to claim 11.

13. The processor, a third-C transmission step of transmitting the third-C prompt generated in the third-C generation step to an artificial intelligence system; a third C receiving step of receiving output contents including a first evaluation result for the first conversation card and a second evaluation result for the second conversation card in response to the third C prompt from the artificial intelligence system; Execute The program according to claim 11.

14. The processor, an improvement suggestion step of suggesting an improvement measure for the phrase card for the first query information based on a combination of the first evaluation result and the second evaluation result; Execute 14. The program according to claim 10 or 13.

15. the improvement suggestion step is a step of proposing an improvement measure for the first conversation card or the second conversation card based on a combination of the first evaluation result and the second evaluation result; The program according to claim 14.

16. The improvement suggestion step is based on a combination of the first evaluation result and the second evaluation result, Improving the answer content of the second conversation card; and refining the question pattern associated with the second phrase card; and Creating new conversation cards and A step of proposing at least one improvement measure. The program according to claim 14.

17. The processor, a fourth A generating step of generating a fourth A prompt by including the first query information and the second conversation card in an instruction for improving the answer content of the conversation card; a fourth A transmission step of transmitting the fourth A prompt generated in the fourth A generation step to an artificial intelligence system; a fourth A receiving step of receiving from the artificial intelligence system an output content that improves the answer content of the second conversation card in response to the fourth A prompt; Run The improvement suggestion step includes: and a step of proposing an improvement measure for improving the answer content of the second conversation card received in the fourth A receiving step. The program according to claim 16.

18. The processor, generating a fourth prompt by including the first query information and the second phrase card in an instruction to improve a question pattern associated with the phrase card; a fourth-B transmission step of transmitting the fourth-B prompt generated in the fourth-B generation step to an artificial intelligence system; a fourth B receiving step of receiving from the artificial intelligence system an output content for the fourth B prompt that improves a question pattern associated with the second conversation card; Run The improvement suggestion step includes: and a step of proposing an improvement measure for improving the question pattern associated with the second conversation card received in the fourth B receiving step. The program according to claim 16.

19. The processor, a fourth C generation step of generating a fourth C prompt by including the first query information in an instruction to create a new conversation card from query information; a fourth C transmission step of transmitting the fourth C prompt generated in the fourth C generation step to an artificial intelligence system; a fourth C receiving step of receiving output content including a new conversation card in response to the fourth C prompt from the artificial intelligence system; Run The improvement suggestion step includes: Proposing an improvement measure including the new conversation card received in the fourth C receiving step, The program according to claim 16.

20. A method executed by an information processing device including a processor and a storage unit, the processor executing all of the steps executed in the invention according to any one of claims 2 to 13.

21. 14. An information processing device comprising a processor and a storage unit, wherein the processor executes all of the steps executed in the invention according to any one of claims 2 to 13.

22. A system comprising means for executing all the steps performed in any one of claims 2 to 13.

Images