Information processing device, information processing system, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing system, an information processing method, and a program for easily presenting a method for improving the accuracy of response to a system administrator of a chatbot.SOLUTION: In a chat system 100, an information processing apparatus includes: a display control unit which causes a terminal device connected over a communication network to display a screen including a list of improvement methods for question / answer information referred in outputting output answer data to input question data; a receiving unit which receives, from the terminal device, a predetermined improvement method selected from the list of the improvement methods displayed on the screen of the terminal device; a request unit which issues a request to a large language model of a server device to generate an improvement idea according to the predetermined improvement method received by the receiving unit; and an output unit which outputs, to the terminal device, the improvement idea generated in the large language model based on the request.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] In recent years, chatbot systems have become known that search for question and answer information stored in a database and answer questions posed by users. Specifically, for example, a technology is known in which questions and answers are collected from conversations between users, stored in a database, and the user's utterance is matched with a question template to extract candidate answers corresponding to the questions from the database. Summary of the Invention [Problem to be solved by the invention]

[0003] In chatbot systems, it is important to improve the accuracy of answers to questions. Therefore, chatbot system administrators often carefully examine the content of the question and answer information and edit it so that the questions and answers match. In this case, the chatbot system administrator must personally determine how to improve the accuracy of the answers, which is time-consuming.

[0004] The disclosed technology was developed in consideration of the above circumstances, and aims to easily present chatbot system administrators with improvement methods for increasing the accuracy of responses. [Means for solving the problem]

[0005] The disclosed technology is an information processing device having a display control unit that causes a terminal device to display a screen including a list of improvement methods for question answer information that was referenced when outputting output answer data for input question data, a reception unit that receives from the terminal device a predetermined improvement method selected from the list of improvement methods displayed on the screen of the terminal device, a request unit that requests a large-scale language model to generate an improvement plan according to the predetermined improvement method received by the reception unit, and an output unit that outputs the improvement plan generated by the large-scale language model to the terminal device based on the request. [Effects of the Invention]

[0006] The purpose is to easily present chatbot system administrators with ways to improve the accuracy of responses. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of a system configuration of a chat system. [Figure 2] FIG. 2 illustrates an example of a hardware configuration of an information processing device. [Figure 3] FIG. 2 illustrates an example of a hardware configuration of a terminal device. [Figure 4] FIG. 10 illustrates an example of a chat history storage unit. [Figure 5] FIG. 2 illustrates an example of a question and answer storage unit. [Figure 6] FIG. 10 is a diagram illustrating an example of a first synonym storage unit. [Figure 7] FIG. 10 is a diagram illustrating an example of a second synonym storage unit. [Figure 8] FIG. 10 illustrates an example of an analysis result storage unit. [Figure 9] FIG. 10 illustrates an example of an improvement proposal storage unit. [Figure 10] FIG. 2 illustrates an example of a document storage unit. [Figure 11] FIG. 2 illustrates an example of a user information storage unit. [Figure 12]FIG. 2 is a diagram illustrating the functions of each device in the chat system. [Figure 13] FIG. 10 is a first sequence diagram showing the operation of the chat system. [Figure 14] 10 is a flowchart illustrating processing by the information processing device. [Figure 15] FIG. 10 is a first diagram showing an example of a chat screen displayed on the terminal device. [Figure 16] FIG. 10 is a second sequence diagram illustrating the operation of the chat system. [Figure 17] 10 is a flowchart illustrating processing by an analysis unit of the information processing device. [Figure 18] 10 is a flowchart illustrating processing by a screen generation unit of the information processing device. [Figure 19] FIG. 10 is a first diagram showing an example of a management screen displayed on a terminal device. [Figure 20] FIG. 10 is a diagram illustrating an example of a proposal list screen displayed on the terminal device. [Figure 21] FIG. 10 is a diagram illustrating an example of a proposal details screen displayed on the terminal device. [Figure 22] FIG. 10 is a third sequence diagram illustrating the operation of the chat system. [Figure 23] FIG. 1 is a first diagram showing an example of a prompt. [Figure 24] FIG. 10 is a first diagram showing an example of a display of an improvement plan on a terminal device. [Figure 25] FIG. 10 is a second diagram showing an example of a prompt. [Figure 26] FIG. 10 is a second diagram showing an example of how improvement proposals are displayed on the terminal device. [Figure 27] FIG. 10 is a third diagram showing an example of a prompt. [Figure 28] FIG. 3 is a third diagram showing an example of a display of an improvement plan on a terminal device; [Figure 29] FIG. 4 is a fourth diagram showing an example of displaying an improvement plan on a terminal device; [Figure 30] FIG. 10 is a second diagram showing an example of a chat screen displayed on the terminal device. [Figure 31]FIG. 10 is a third diagram showing an example of a chat screen displayed on the terminal device. [Figure 32] FIG. 4 is a fourth diagram showing an example of a chat screen displayed on the terminal device. DETAILED DESCRIPTION OF THE INVENTION

[0008] The present embodiment will be described below with reference to the drawings: Fig. 1 is a diagram showing an example of the system configuration of a chat system.

[0009] The chat system 100 of this embodiment is an example of an information processing system that includes an information processing device 200, a first storage device 300, a second storage device 400, an information processing device 200, a terminal device 600, and a terminal device 700, each of which is connected via a communication network 1.

[0010] In the chat system 100, the information processing device 200, the first storage device 300, the second storage device 400, and the information processing device 200 provide a chatbot service to a user of a terminal device 600.

[0011] The first storage device 300 includes a chat history storage unit 310 , a question and answer storage unit 320 , a first synonym storage unit 330 , a second synonym storage unit 340 , an analysis result storage unit 350 , and an improvement suggestion storage unit 360 .

[0012] The chat history storage unit 310 stores chat history information that associates questions input from the terminal device 600 with answers output to the terminal device 600. The chat history information may be managed for each tenant that introduces the chat system 100. The tenant in this embodiment may be an organization such as a company, or may be the organization to which a user who uses the chat system 100 belongs.

[0013] The question and answer storage unit 320 stores question and answer information in which questions and answers are associated with each other. In other words, the question and answer storage unit 320 is an example of a question and answer information group in which question data indicating a question is associated with answer data indicating an answer. Note that the question and answer information is information that is stored in advance in the question and answer storage unit 320, and may be stored for each tenant that introduces the chat system 100. In the following description, the question data included in the question and answer information will be referred to as registered question data, and the answer data included in the question and answer information will be referred to as registered answer data.

[0014] The first synonym storage unit 330 stores first synonym information that associates keywords (words) included in the registered question data and the registered answer data with synonyms of each keyword. In other words, the first synonym storage unit 330 is an example of a group of synonym information that associates keywords (words) included in the registered question data and the registered answer data with synonyms of each keyword.

[0015] The second synonym storage unit 340 stores second synonym information that associates various words with synonyms. The second synonym storage unit 340 may be, for example, a general-purpose database made public on the Internet, and stores more synonyms than the synonyms stored in the first synonym storage unit 330. In other words, the second synonym information stored in the second synonym storage unit 340 contains more information than the first synonym information stored in the first synonym storage unit 330.

[0016] The analysis result storage unit 350 stores analysis result information that is the result of analyzing the chat history information stored in the chat history storage unit 310 in order to identify question and answer information that should be improved. The improvement proposal storage unit 360 stores improvement proposal information that is referenced when presenting methods for improving the registered question data and registered answer data stored in the question and answer storage unit 320.

[0017] In addition, the first synonym storage unit 330, the second synonym storage unit 340, the analysis result storage unit 350, and the improvement proposal storage unit 360 of this embodiment may each be stored for each tenant that introduces the chat system 100.

[0018] The second storage device 400 has a document storage unit 410 and a user information storage unit 420. The document storage unit 410 stores user-specific information that is not publicly available.

[0019] A user of chat system 100 may be, for example, a tenant, in which case the user-specific information is tenant-specific information. A user of chat system 100 may also be an individual user, in which case the user-specific information is user-specific information.

[0020] In the following description, user-specific information will be referred to as in-house information, and in-house information will be taken as an example of user-specific information.

[0021] In the present embodiment, internal company information may be stored for each tenant in the document storage unit 410. The user information storage unit 420 stores, for each tenant, user information relating to users who belong to the tenant.

[0022] The information processing device 200 has a large-scale language model storage unit 510 and realizes a generative AI (Artificial Intelligence). The generative AI generates text data according to an input prompt, such as ChatGPT, and outputs the text data to the information processing device 200. Note that the prompt in this embodiment is text data for expressing an instruction or question to the generative AI realized by the information processing device 200. The large-scale language model will be described in detail later.

[0023] The terminal device 600 is mainly used by users who use the chat system 100. The terminal device 700 is mainly used by an administrator who manages the chat system 100. In this embodiment, an administrator of the chat system 100 may exist for each tenant that has introduced the chat system 100. In other words, each storage unit of the first storage device 300 and the second storage device 400 is managed for each tenant by the tenant administrator who is the user of the terminal device 700.

[0024] Each of the terminal devices 600 and 700 may be, for example, a smartphone or a tablet terminal.

[0025] In the chat system 100 of this embodiment, when a user inputs a question to the terminal device 600, the terminal device 600 transmits question data indicating the question to the information processing device 200.

[0026] When a question is input from the terminal device 600, the information processing device 200 of this embodiment references the question and answer storage unit 320 of the first storage device 300 and outputs an answer to the question to the terminal device 600. In the following description, data indicating the question input from the terminal device 600 is referred to as input question data. Also, in the following description, answer data output from the information processing device 200 to the terminal device 600 as an answer to the input question is referred to as output answer data.

[0027] Furthermore, the information processing device 200 of this embodiment associates the input question data with the output answer data to generate chat history information, and stores the information in the chat history storage unit 310 of the first storage device 300. In this embodiment, the chat history information may include information indicating an evaluation of the output answer data by the terminal device 600.

[0028] The information processing device 200 of this embodiment analyzes the chat history information stored in the chat history storage unit 310, identifies question and answer information that should be improved from the question and answer information stored in the question and answer storage unit 320, and identifies a method for improving the question and answer information, and displays the identified information on the terminal device 700.

[0029] Furthermore, when a method for improving answer information is selected on the terminal device 700, the information processing device 200 of this embodiment outputs a prompt requesting the large-scale language model included in the information processing device 200 to generate a specific improvement plan. Then, when the information processing device 200 acquires the specific improvement plan from the information processing device 200, the information processing device 200 displays the improvement plan on the terminal device 700.

[0030] In this embodiment, the question and answer information to be improved and the improvement method are identified and presented to the administrator of the chat system 100, and specific improvement proposals are presented to the administrator for the question and answer information selected.

[0031] Therefore, in this embodiment, the administrator of the chat system 100 does not need to identify question and answer information that needs to be improved, or to search for ways to improve the accuracy of answers.

[0032] 1, the information processing device 200, the first storage device 300, and the second storage device 400 are each separate devices, but this is not limiting. The first storage device 300 and the second storage device 400 may be included in the information processing device 200.

[0033] In addition, in FIG. 1, the information processing device 200 may be realized by one or more information processing devices, and may be realized by a plurality of information processing devices.

[0034] Next, an example of the hardware configuration of each device included in the chat system 100 of this embodiment will be described.

[0035] In this embodiment, the information processing device 200, the first storage device 300, the second storage device 400, and the information processing device 200 are general-purpose computers, and therefore, in FIG. 2, the hardware configuration of the information processing device 200 will be described as an example.

[0036] In this embodiment, the terminal devices 600 and 700 are smartphones, and in FIG. 3, the hardware configuration of the terminal device 600 will be described as an example.

[0037] Fig. 2 is a diagram showing an example of the hardware configuration of an information processing device. The information processing device 200 shown in Fig. 2 is constructed by a computer (information processing device), and includes a CPU 201, a ROM 202, a RAM 203, an HD 204, an HDD (Hard Disk Drive) controller 205, a display 206, an external device connection I / F (Interface) 208, a network I / F 209, a bus line 210, a keyboard 211, a pointing device 212, a DVD-RW (Digital Versatile Disk Rewritable) drive 214, and a media I / F 216, as shown in Fig. 2.

[0038] Of these, the CPU 201 controls the overall operation of the information processing device 200. The ROM 202 stores programs used to drive the CPU 201, such as an IPL. The RAM 203 is used as a work area for the CPU 501. The HD 204 stores various data, such as programs. The HDD controller 205 controls the reading and writing of various data from and to the HD 204 under the control of the CPU 501. The display 206 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 208 is an interface for connecting various external devices. In this case, the external devices are, for example, USB (Universal Serial Bus) memories, printers, etc. The network I / F 209 is an interface for data communication using a communication network. The bus line 210 is an address bus, a data bus, etc. for electrically connecting the components, such as the CPU 201, shown in FIG. 2.

[0039] The keyboard 211 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 212 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 214 controls reading and writing of various data from and to a DVD-RW 213, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 216 controls reading and writing (storing) of data from and to a recording medium 215, such as a flash memory.

[0040] 3 is a diagram showing an example of the hardware configuration of a terminal device 700 according to this embodiment. The terminal device 700 includes a CPU 701, a ROM 702, a RAM 703, an EEPROM 704, a CMOS sensor 705, an image sensor I / F 706, an acceleration / direction sensor 707, a media I / F 709, and a GPS receiver 711.

[0041] Of these, the CPU 701 is an arithmetic processing unit that controls the overall operation of the terminal device 700. The ROM 702 stores the CPU 701 and programs used to drive the CPU 701, such as the IPL. The RAM 703 is used as a work area for the CPU 701. The EEPROM 704 reads or writes various data, such as smartphone programs, under the control of the CPU 701. The ROM 702, RAM 703, and EEPROM 704 are examples of storage devices of the terminal device 700.

[0042] The CMOS (Complementary Metal Oxide Semiconductor) sensor 705 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) and obtains image data under the control of the CPU 701. Note that instead of a CMOS sensor, an imaging means such as a CCD (Charge Coupled Device) sensor may also be used.

[0043] The imaging element I / F 706 is a circuit that controls the driving of the CMOS sensor 705. The acceleration / azimuth sensor 707 is a variety of sensors, such as an electronic magnetic compass that detects geomagnetism, a gyrocompass, and an acceleration sensor. The media I / F 709 controls the reading and writing (storage) of data from and to a recording medium 708, such as a flash memory. The GPS receiver 711 receives GPS signals from GPS satellites.

[0044] The terminal device 700 also includes a long-distance communication circuit 712, an antenna 712a of the long-distance communication circuit 712, a CMOS sensor 713, an image sensor I / F 714, a microphone 715, a speaker 716, an audio input / output I / F 717, a display 718, an external device connection I / F (Interface) 719, a short-distance communication circuit 720, an antenna 720a of the short-distance communication circuit 720, and a touch panel 721.

[0045] Of these, the long-distance communication circuit 712 is a circuit that communicates with other devices via a communication network. The CMOS sensor 713 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 701. The image sensor I / F 714 is a circuit that controls the driving of the CMOS sensor 713. The microphone 715 is a built-in circuit that converts sound into an electrical signal. The speaker 716 is a built-in circuit that converts an electrical signal into physical vibrations to produce sound such as music or voice. The sound input / output I / F 717 is a circuit that processes the input and output of sound signals between the microphone 715 and the speaker 716 under the control of the CPU 701.

[0046] The display 718 is a type of display means such as a liquid crystal display or organic electroluminescence (EL) display that displays an image of a subject, various icons, etc. The external device connection I / F 719 is an interface for connecting various external devices. The short-range communication circuit 720 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 721 is a type of input means that allows a user to operate the terminal device 700 by pressing the display 718. The display 718 is an example of a display unit that the terminal device 700 has.

[0047] 3 is an example of the terminal device 700, and the hardware configuration of the terminal device 700 is not limited to this. The hardware configuration of the terminal device 700 may be the configuration shown in FIG. 2. The hardware configuration of the terminal device 600 may be the same as that of the terminal device 700, and therefore a description thereof will be omitted.

[0048] Next, each storage unit of the first storage device 300 will be described with reference to Figures 4 to 9. Figure 4 is a diagram showing an example of a chat history storage unit.

[0049] The chat history storage unit 310 of this embodiment includes information items such as tenant ID, conversation ID, conversation date and time, input question data, output answer data, category, question and answer ID, and user rating, and the item "conversation ID" is associated with other items.

[0050] The value of the item "Tenant ID" is identification information for identifying the tenant that manages the chat history storage unit 310. The value of the item "Conversation ID" indicates identification information that identifies chat history information. The value of the item "Conversation Date and Time" indicates the date and time when the chat took place. The value of the item "Input Question Data" indicates text data (input question data) that indicates a question input from the terminal device 600. The value of the item "Output Response Data" is text data (output response data) that indicates a response output by the information processing device 200 to the terminal device 600.

[0051] The value of the item "Category" indicates the category associated with the registered answer data output as output answer data in the question and answer storage unit 320. The value of the item "Question and Answer ID" indicates the question and answer ID associated with the output answer data in the question and answer storage unit 320. The value of the item "User Rating" indicates the rating entered in the terminal device 600 for the answer indicated by the output answer data.

[0052] 5 is a diagram illustrating an example of a question and answer storage unit. The question and answer storage unit 320 of this embodiment includes information items such as tenant ID, question and answer ID, update date and time, category, registered question data, registered answer data, and question paraphrase, and the item "question and answer ID" is associated with other items. The question and answer information of this embodiment is information in which the value of the item "question and answer ID" is associated with the values ​​of the other items.

[0053] The value of the item "Tenant ID" is identification information for identifying the tenant that manages the question and answer storage unit 320. The value of the item "Question and Answer ID" indicates identification information that identifies the question and answer information. The value of the item "Update Date and Time" indicates the date and time when the question and answer information was updated. The value of the item "Category" indicates the category of the question indicated by the registered question data. The value of the item "Registered Question Data" is text data (registered question data) indicating the registered question, and the value of the item "Registered Answer Data" is text data (registered answer data) indicating the registered answer. The item "Question Rephrasing" indicates another pattern of the registered question data.

[0054] 6 is a diagram illustrating an example of a first synonym storage unit. The first synonym information stored in the first synonym storage unit 330 of this embodiment includes, as information items, a tenant ID, a synonym ID, an update date and time, a keyword, and a synonym, and the item "synonym ID" is associated with other items. The first synonym information of this embodiment is information in which the value of the item "synonym ID" is associated with the values ​​of the other items. The first synonym information may be stored in the first synonym storage unit 330 in advance.

[0055] The value of the item "Tenant ID" is identification information for identifying the tenant that manages the first synonym storage unit 330. The value of the item "Synonym ID" is identification information for identifying the first synonym information. The value of the item "Update Date and Time" indicates the date and time when the first synonym information was updated. The item "Keyword" is various keywords, and the value of the item "Synonym" is a synonym of the keyword.

[0056] FIG. 7 is a diagram illustrating an example of a second synonym storage unit. The second synonym information stored in the second synonym storage unit 340 of this embodiment includes, as information items, a tenant ID, a synonym ID, an update date and time, a keyword, and a synonym, and the item "synonym ID" is associated with other items. The second synonym information of this embodiment is information in which the value of the item "synonym ID" is associated with the values ​​of the other items. The second synonym information may be stored in the second synonym storage unit 340 in advance.

[0057] The value of the item "Tenant ID" is identification information for identifying the tenant that manages the second synonym storage unit 340. The value of the item "Synonym ID" is identification information for identifying the second synonym information. The value of the item "Update Date and Time" indicates the date and time when the first synonym information was updated. The item "Keyword" is various keywords, and the value of the item "Synonym" is a synonym of the keyword. Note that in this embodiment, the second synonym storage unit 340 is managed for each tenant, but this is not limited to this. The second synonym storage unit 340 may be shared by multiple tenants.

[0058] 8 is a diagram illustrating an example of an analysis result storage unit. The analysis result information stored in the analysis result storage unit 350 includes, as information items, a tenant ID, a conversation ID, input question data, an analysis result, an improvement method type, an appropriate question and answer ID, an appropriate question data, a reference question and answer ID, a reference question data, a user rating, and a group ID, and the items "tenant ID" and "conversation ID" are associated with other items. The analysis result information of this embodiment is information in the analysis result storage unit 350 in which the value of the item "conversation ID" is associated with other items.

[0059] The value of the item "Tenant ID" is identification information for identifying the tenant that manages the analysis result storage unit 350. The value of the item "Conversation ID" indicates identification information that identifies the chat history information to be analyzed. The value of the item "Input Question Data" is the value of the item "Input Question Data" in the chat history information to be analyzed. The value of the item "Analysis Result" indicates the analysis result of the chat history information to be analyzed. Details of the analysis of chat history information will be described later.

[0060] The item "Improvement method type" indicates the type of improvement method for the answer to question information referenced in the chat indicated by the chat history information to be analyzed. In other words, the item "Improvement method type" indicates how to improve the answer to question information referenced in the chat indicated by the chat history information to be analyzed.

[0061] The value of the item "appropriate answer to question ID" indicates identification information of the answer to question information to be referenced for the input question data. Note that the value of the item "appropriate answer to question ID" is the question answer ID of the answer to question information identified by the information processing device 200 by referring to the second synonym storage unit 340.

[0062] The value of the item "appropriate question data" is the value of the item "input question data" included in the answer to question information identified by the value of the item "appropriate question and answer ID".

[0063] The value of the item "Referenced Question / Answer ID" is the question / answer ID of the question / answer information referenced when the information processing device 200 identifies the output answer data for the input question data. In other words, the value of the item "Referenced Question / Answer ID" is the question / answer ID of the question / answer information identified by the information processing device 200 with reference to the first synonym storage unit 330, and is the question / answer ID that identifies the first question / answer information described below.

[0064] The value of the item "Reference Question Data" is the value of the item "Input Question Data" included in the question answer information identified by the value of the item "Reference Question Answer ID." The value of the item "User Rating" is the user rating associated with the value of the item "Conversation ID" in the chat history storage unit 310.

[0065] The value of the "Group ID" item is identification information that is assigned when analysis result information whose input question data have a similarity equal to or greater than a threshold is grouped together. In other words, the value of the "Group ID" item is identification information that identifies a group of analysis result information whose input question data (text data) are similar.

[0066] 9 is a diagram illustrating an example of the improvement proposal storage unit 360. The improvement proposal information stored in the improvement proposal storage unit 360 includes information items associated with tenant IDs, analysis results, and improvement method types.

[0067] The value of the item "Tenant ID" is identification information for identifying the tenant that manages the improvement proposal storage unit 360. The value of the item "Analysis Result" indicates the pattern of the analysis results of the chat history information to be analyzed. The item "Improvement Method Type" indicates the method of improving the question and answer information associated with the pattern of the analysis results. Note that in this embodiment, the improvement proposal storage unit 360 is managed for each tenant, but this is not limited to this. The improvement proposal storage unit 360 may be shared by multiple tenants.

[0068] Next, each storage unit of the second storage device 400 will be described with reference to FIGS.

[0069] Fig. 10 is a diagram illustrating an example of a document storage unit. In the document storage unit 410 of this embodiment, internal company information may be stored for each tenant and managed for each tenant. Fig. 10 illustrates an example of internal company information managed by a tenant identified by tenant ID "11."

[0070] In the document storage unit 410 of this embodiment, the internal company information is divided into predetermined units and each divided piece of information is stored. In the following description, each divided piece of internal company information is called a chunk. In the document storage unit 410, the chunk may be stored in association with identification information that identifies the chunk.

[0071] 10, one page of internal information in an internal document is defined as a chunk, and the page number is used as identification information to identify the chunk. Note that a chunk is not limited to one page of internal information, and may be, for example, each paragraph or each chapter in an internal document.

[0072] 11 is a diagram illustrating an example of the user information storage unit 420. The user information stored in the user information storage unit 420 of this embodiment is in-house information stored for each tenant, and may be managed for each tenant.

[0073] The user information in this embodiment includes information items such as a tenant ID, a user ID, a user name, and an email address. The value of the item "tenant ID" is identification information that identifies the tenant to which the user identified by the user ID belongs. The value of the item "user ID" is identification information that identifies the user of the terminal device 600. The item "user name" indicates the name of the user identified by the user ID, and the value of the item "email address" indicates the email address of the user identified by the user ID. In this embodiment, the user information storage unit 420 is stored in the second storage device 400, but the user information storage unit 420 does not have to be provided in the second storage device 400; it is sufficient if only the document storage unit 410 is provided.

[0074] Next, the functional configuration of each device included in the chat system 100 of this embodiment will be described with reference to Fig. 12. Fig. 12 is a diagram illustrating the functions of each device included in the chat system.

[0075] First, a description will be given of the functional configuration of the information processing device 200. The information processing device 200 of this embodiment includes a reception unit 251, a chat processing unit 252, a search request unit 253, an information acquisition unit 254, an analysis unit 255, a prompt generation unit 256, a request unit 257, an output unit 258, and a screen generation unit 259.

[0076] The receiving unit 251 receives various inputs to the information processing device 200. Specifically, the receiving unit 251 receives input of input question data from the terminal device 600. The receiving unit 251 also receives various inputs from the terminal device 700 in response to operations.

[0077] When the chat processing unit 252 receives input of input question data from the terminal device 600, it references the question and answer storage unit 320 and the first synonym storage unit 330 and outputs output answer data to the terminal device 600. Furthermore, the chat processing unit 252 associates the input question data input from the terminal device 600 with the output answer data output to the terminal device 600 and stores them in the chat history storage unit 310. Note that the input question data and the output answer data may each be text data or audio data.

[0078] The search request unit 253 issues a search request to the second storage device 400. Specifically, the search request unit 253 sends to the second storage device 400 a search request for the document storage unit 410 using the input question data as a search key.

[0079] The information acquisition unit 254 acquires, from the second storage device 400, in-house information that is a search result of the document storage unit 410 in response to a search request from the search request unit 253. The information acquisition unit 254 also acquires chat history information for a specified period from the chat history storage unit 310.

[0080] Analysis unit 255 analyzes whether the content of the output answer data to the input question data included in the answer-to-question information is appropriate, based on the chat history information acquired by information acquisition unit 254. More specifically, analysis unit 255 analyzes whether the content of the output answer data output to the input question data was appropriate. Details of the processing by analysis unit 255 will be described later.

[0081] The prompt generation unit 256 generates a prompt to be output to the information processing device 200. For example, when the prompt generation unit 256 acquires information to be included in the prompt, it generates a prompt in a predetermined format that includes the acquired information.

[0082] The request unit 257 transmits the prompt generated by the prompt generation unit 256 to the information processing device. 200. In other words, the request unit 257 makes a prompt-based request to the large-scale language model 511. In this embodiment, making a prompt-based request to the large-scale language model 511 may be expressed as sending a prompt generated by the prompt generation unit 255 to the server device 500.

[0083] The output unit 258 outputs various types of information from the information processing device 200. Specifically, for example, the output unit 258 outputs information indicating the analysis result by the analysis unit 255 to the terminal device 700. Also, for example, the output unit 258 outputs output data of the large-scale language model 511 acquired from the information processing device 200 to the terminal device 700. Also, the output unit 258 outputs screen data for displaying various screens to the terminal device 700.

[0084] The output unit 258 also outputs chat history information indicating the history of chats conducted by the chat processing unit 252 to the first storage device 300, and causes the chat history information to be stored in the chat history storage unit 310. The output unit 258 also outputs question and answer information to the first storage device 300, and causes the question and answer information to be stored in the question and answer storage unit 320 of the first storage device 300. The output unit 258 also outputs analysis result information indicating the results of the analysis by the analysis unit 255 to the first storage device 300, and causes the analysis result storage unit 350 to store the analysis result information.

[0085] The screen generation unit 259 generates screen data for displaying a screen on the terminal device 700. Specifically, the screen generation unit 259 generates screen data for displaying a screen for proposing a method for improving answer to question information on the terminal device 700.

[0086] Next, the functions of the first storage device 300 will be described. The first storage device 300 includes a chat history storage unit 310, a question and answer storage unit 320, a first synonym storage unit 330, a second synonym storage unit 340, an analysis result storage unit 350, an improvement proposal storage unit 360, a search unit 370, and a storage unit 380. The search unit 370 searches the chat history storage unit 310 or the question and answer storage unit 320 in response to a search request from the information processing device 200, and transmits the search results to the information processing device 200. When input question data is input and output answer data is output in the information processing device 200, the storage unit 380 stores chat history information including the input question data and the output answer data in the chat history storage unit 310.

[0087] Next, the functions of the second storage device 400 will be described. The second storage device 400 includes a document storage unit 410, a user information storage unit 420, a search unit 430, and a storage unit 440. The search unit 430 searches the document storage unit 410 in response to a search request from the information processing device 200, and transmits the search results to the information processing device 200. When an administrator of the chat system 100 or the like performs an operation to upload in-house information, the storage unit 440 stores the new in-house information in the document storage unit 410.

[0088] Next, we will explain the functions of the information processing device 200. The information processing device 200 has a large-scale language model storage unit 510, an input unit 520, and an output unit 530. The large-scale language model storage unit 510 stores a large-scale language model 511 that realizes a generative AI.

[0089] The large-scale language model 511 of this embodiment is a computer language model that is generated by executing a learning process using a huge amount of unlabeled text as learning data and is composed of an artificial neural network with a large number of parameters. The large-scale language model 511 can capture much of the syntax and meaning of human language by being sufficiently trained using a context learning technique such as next sentence prediction, which understands the context by determining whether sentence 1 and sentence 2 are consecutive, or a masked language model, which understands the context by masking words in a sentence and predicting the masked words from the words before and after them.

[0090] The input unit 520 inputs a prompt transmitted from the information processing device 200 to the large-scale language model 511. The output unit 530 outputs (transmits) output data output from the large-scale language model 511 to the information processing device 200. In other words, the output unit 530 transmits output response data output from the large-scale language model 511 to the information processing device 200.

[0091] Next, a description will be given of the functions of the terminal devices 600 and 700. The terminal devices 600 and 700 each include a communication control unit 630 and 730, a reception unit 640 and 740, and a display control unit 650 and 750.

[0092] The communication control units 630, 730 control communication between the terminal devices 600, 700, respectively, and the information processing device 200. The reception units 640, 740 receive input to the terminal devices 600, 700. The display control units 650, 750 may be realized by a browser or the like, and control the display on the terminal devices 600, 700.

[0093] Next, the operation of the chat system 100 of this embodiment will be described with reference to Fig. 13. Fig. 13 is a first sequence diagram showing the operation of the chat system. Fig. 13 shows the operation of the chat system 100 when the information processing device 200 outputs answer output data in response to input question data input from the terminal device 600.

[0094] In the chat system 100 of the embodiment, when the terminal device 600 receives a question input by a user on the chat screen displayed on the terminal device 600 (step S1301), it transmits input question data indicating the question to the information processing device 200 (step S1302).

[0095] When the information processing device 200 receives the input of the input question data, the chat processing unit 252 refers to the question and answer storage unit 320 and the first synonym storage unit 330 to identify the registered answer data corresponding to the input question data (step S1303). The details of the process of step S1303 will be described later.

[0096] Next, the information processing device 200 transmits the identified registered answer data to the terminal device 600 as output answer data (step S1304). Next, the information processing device 200 stores chat history information in which the input question data and the output answer data output to the terminal device 600 are associated with each other in the chat history storage unit 310 (step S1305).

[0097] Upon receiving the output response data, the terminal device 600 displays the received output response data on the chat screen (step S1306). Subsequently, the terminal device 600 accepts input of a user evaluation of the content of the output response data from the user (step S1307), and transmits information indicating the input user evaluation to the information processing device 200 (step S1308).

[0098] When the information processing device 200 receives the information indicating the evaluation, the information processing device 200 stores the user evaluation together with the chat history information stored in the chat history storage unit 310 in step S1305 (step S1309).

[0099] Next, details of the process of step S1303 in Fig. 13 will be described with reference to Fig. 14. Fig. 14 is a flowchart illustrating the process of the information processing device.

[0100] The information processing device 200 of this embodiment acquires input question data corresponding to a question sentence input from the terminal device 600 by the chat processing unit 252 (step S1401).

[0101] Next, the chat processing unit 252 executes the processes from step S1403 onwards for each piece of answer-to-question information stored in the question and answer storage unit 320 (step S1402).

[0102] The chat processing unit 252 determines whether or not first synonyms corresponding to the registered question data and keywords included in the registered answer data included in the answer to question information are set in the first synonym storage unit 330 (step S1403).

[0103] The keywords contained in the registered question data and the registered answer data are, for example, extracted keywords (character strings) extracted from each text data by performing morphological analysis or the like on the text data that is the registered question data and the text data that is the registered question data.

[0104] In this embodiment, when first synonym information whose value in the item "keyword" matches the extracted keyword is stored in the first synonym storage unit 330, the first synonym corresponding to the keyword contained in the registered question data and the registered answer data is set in the first synonym storage unit 330.

[0105] In step S1403, if a first synonym is set, the chat processing unit 252 calculates the similarity between the registered question data and the registered answer data and the input question data. Furthermore, the chat processing unit 252 replaces the extracted keyword included in the input question data with the first synonym corresponding to the extracted keyword, and calculates the similarity between the registered question data and the registered question data (step S1404).

[0106] In this embodiment, instead of calculating the first similarity between the question data and the answer data after replacing the keywords of the input question data with the corresponding first synonyms, the results of morphological analysis of the input question data may be added to the word list.

[0107] Furthermore, in step S1404, if there are a plurality of first synonyms associated with the extracted keyword, the chat processing unit 252 performs the same process for each of the first synonyms.

[0108] If the first synonym is not set in step S1403, the chat processing unit 252 calculates the similarity between the registered question data and the registered answer data and the input question data (step S1405).

[0109] In the following description of this embodiment, the similarity calculated by the chat processing unit 252 will be referred to as the first similarity.

[0110] After executing the processes from step S1403 to step S1405 for all answer to question information stored in the question and answer storage unit 320 (step S1406), the chat processing unit 252 determines whether there is any answer to question information for which the first similarity calculated in the processes up to step S1406 exceeds the first similarity threshold (step S1407).

[0111] If there is no corresponding answer to question information in step S1407, the chat processing unit 252 identifies the text data "Answer not found" as the output answer data for the input question data (step S1408), and ends the process.

[0112] In step S1407, if there is corresponding answer to question information, the chat processing unit 252 identifies the answer to question information with the highest first similarity, specifies the registered question data included in the identified answer to question information as output answer data for the input question data (step S1409), and terminates the process.

[0113] In this embodiment, the chat processing unit 252 may treat multiple pieces of answer information as the answer. Specifically, if there is answer information whose first similarity is equal to or greater than the first similarity threshold, the chat processing unit 252 identifies the corresponding answer information as the answer. Furthermore, if there is answer information whose first similarity is less than the first similarity threshold and equal to or less than the second similarity threshold, the chat processing unit 252 identifies the answer information as the answer. Furthermore, if there is multiple pieces of answer information whose first similarity is equal to or greater than the second similarity threshold, the chat processing unit 252 identifies the corresponding answer information as the answer. Details of the second similarity will be described later.

[0114] In the following description of this embodiment, the answer-to-question information identified by the chat processing unit 252 will be referred to as first answer-to-question information.

[0115] Next, a screen displayed on the terminal device 600 in step S1306 of Fig. 13 will be described with reference to Fig. 15. Fig. 15 is a first diagram showing an example of a chat screen displayed on the terminal device.

[0116] 15 is an example of a chat screen displayed on the terminal device 600. The screen 110 includes a display area 112 and an input field 113. The display area 112 displays an exchange between the user and the server device 500. The input field 113 is where a question from the user is entered.

[0117] 15, display area 112 includes display areas 112a, 112b, and 112c. Display area 112a displays input question data entered in input field 113. Display area 112b displays output answer data identified by chat processing unit 252. Display area 112c displays an input field for user evaluation of the output answer data.

[0118] In this embodiment, as described above, the input question data input from the terminal device 600, the output answer data output to the terminal device 600, and the user's evaluation of the output answer data of the terminal device 600 are acquired and stored in the chat history storage unit 310 as chat history information.

[0119] In this embodiment, chat history information stored in chat history storage unit 310 is analyzed in this manner, and answer-to-question information that should be improved is identified based on the analysis results, and an improvement method is proposed.

[0120] Next, a process for proposing a method for improving answer-to-question information performed by the information processing device 200 of this embodiment will be described with reference to FIGS.

[0121] 16 is a second sequence diagram illustrating the operation of the chat system 100 when the information processing device 200 proposes to the terminal device 700 answer-to-question information to be improved and a method for improving it.

[0122] In the chat system 100 of this embodiment, the terminal device 700 receives, via the receiving unit 740, an input of a setting for a period for collecting chat history information to be analyzed and a setting for a time for analyzing the chat history information (step S1601). Subsequently, the terminal device 700 transmits, via the communication control unit 730, information indicating the set period and time to the information processing device 200 (step S1602).

[0123] In this embodiment, in step S1601, the terminal device 700 may display a management screen for managing the question and answer information stored in the question and answer storage unit 320.

[0124] This management screen may be displayed on the terminal device 700, for example, when the user of the terminal device 700 logs in to the chat system 100 based on the tenant ID of the organization to which the user belongs and the user's own user ID.

[0125] In addition, in this embodiment, a setting of a period for tallying the chat history information to be analyzed and a setting of a time for analyzing the chat history information may be input on the management screen displayed on the terminal device 700.

[0126] When the information processing device 200 receives input of information indicating a period and a time through the receiving unit 251, the information processing device 200 stores this information in the analyzing unit 255 (step S1603).

[0127] The processes from step S1601 to step S1603 may be executed at a timing independent of the processes from step S1604 to step S1608, which will be described later.

[0128] Furthermore, the processes from step S1601 to step S1603 do not have to be performed. In that case, the processes from step S1604 to step S1608 may be performed periodically, and the information processing device 200 automatically performs the processes from step S1604 onwards, which will be described later, when it is time to perform analysis.

[0129] The information processing device 200 causes the information acquisition unit 254 to transmit to the first storage device 300 a request to acquire the question and answer information stored in the question and answer storage unit 320 and the chat history information collected during the set period stored in the chat history storage unit 310 (step S1604). The first storage device 300 accepts the acquisition request and acquires the chat history information collected during the specified period (step S1605).

[0130] Next, the first storage device 300 transmits the question and answer information stored in the question and answer storage unit 320 and the acquired chat history information to the information processing device 200 (step S1606).

[0131] When the information processing device 200 acquires the question and answer information and chat history information using the information acquisition unit 254, the analysis unit 255 performs analysis by referring to the chat history information, the question and answer information, and the second synonym storage unit 340 (step S1607). Specifically, the analysis unit 255 analyzes whether the content of the output answer data for the input question data included in the chat history information was appropriate. Details of the processing of step S1607 will be described later.

[0132] Next, the information processing device 200 transmits analysis result information indicating the analysis result by the analysis unit 255 to the first storage device 300 via the output unit 258 (step S1608). The first storage device 300 stores the received analysis result information in the analysis result storage unit 350 via the storage unit 380 (step S1609).

[0133] In this embodiment, in the manner described above, analysis result information is stored in analysis result storage unit 350. Next, an operation of presenting to information processing device 200 a method for improving answer to question information will be described.

[0134] In the chat system 100, the terminal device 700 receives, via the receiving unit 740, an operation to instruct display of a proposal list screen for question and answer information (step S1610). Subsequently, the terminal device 700 transmits a display request for the proposal list screen to the information processing device 200 (step S1611).

[0135] At this time, the terminal device 700 may display a list screen of question answer information, and when an operation is performed on the list screen of question answer information to instruct the display of a proposal list, the terminal device 700 sends a request to display the proposal list screen to the information processing device 200.

[0136] The information processing device 200 uses the screen generation unit 259 to refer to the analysis result storage unit 350 of the first storage device 300 (steps S1612 and S1613) and generates screen data for displaying a list of improvement method proposals (step S1614).

[0137] Here, the processing of step S1614 will be described. In step S1612, the information processing device 200 causes the analysis unit 255 to refer to the analysis result storage unit 350 and calculate the similarity between the text data indicating the input question data included in the analysis result storage unit 350. Next, the information acquisition unit 254 groups the text data having a similarity equal to or greater than a predetermined threshold into one group, and assigns the same group ID to the group in the analysis result storage unit 350. Then, the screen generation unit 259 generates screen data for the proposal list screen so that the text data that is the value of the input question data is displayed for each group ID.

[0138] Specifically, the screen generating unit 259 identifies analysis result information that has not been classified into a group in the analysis result storage unit 350, and calculates the similarity between the value of the item "input question data" included in the identified analysis result information and the item "input question data" for each of the other analysis result information. Next, the screen generating unit 259 classifies, from among the other analysis result information, analysis result information whose similarity is equal to or greater than a predetermined threshold into the same group as the identified analysis result information.

[0139] By repeating this process, screen generator 259 groups each piece of analysis result information stored in analysis result storage unit 350.

[0140] When the screen generating unit 259 of the information processing device 200 generates the screen data of the proposal list screen, the information processing device 200 transmits an instruction to display the proposal list screen to the terminal device 700 (step S1615).

[0141] The terminal device 700 receives the display instruction and causes the display control unit 750 to display the proposal list screen (step S1616). When the terminal device 700 receives the selection of a specific group on the proposal list screen (step S1617), it transmits a display request for a proposal detail screen of the selected group to the information processing device 200 (step S1618).

[0142] Upon receiving this display request, the information processing device 200 refers to the analysis result storage unit 350 (steps S1619 and S1620) and generates screen data for displaying a proposal details screen for the selected group (step S1621). Details of the processing in step S1621 will be described later.

[0143] Next, the information processing device 200 transmits an instruction to display a proposal details screen to the terminal device 700 (step S1622).

[0144] Upon receiving the display instruction, the terminal device 700 determines the display mode of the proposal details screen by the display control unit 750 (step S1623), and displays the proposal details screen (step S1624).

[0145] Next, the processing of the analysis unit 255 and the processing of the screen generation unit 259 of the information processing device 200 will be described with reference to FIGS.

[0146] 17 is a flowchart illustrating the process of the analysis unit of the information processing device, showing details of the process of step S1607 in FIG.

[0147] The analysis unit 255 of the information processing device 200 of this embodiment executes the processes from step S1702 onwards for each piece of chat history information acquired by the information acquisition unit 254 (step S1701).

[0148] The analysis unit 255 acquires text data (input question data) that is the value of the item "input question data" included in the chat history information (step S1702).

[0149] Next, the analysis unit 255 acquires the text data (registered question data) that is the value of the item "registered question data" of each piece of question answer information acquired by the information acquisition unit 254, and the text data (registered answer data) that is the value of the item "registered answer data" (step S1703).

[0150] Next, the analysis unit 255 refers to the second synonym storage unit 340 of the first storage device 300 and determines whether or not a second synonym corresponding to the extracted keyword extracted from each text data acquired in step S1703 has been set (step S1704).

[0151] In this embodiment, when second synonym information whose value in the item "keyword" matches the extracted keyword is stored in the second synonym storage unit 340, second synonyms corresponding to the keywords contained in the registered question data and the registered answer data are set in the second synonym storage unit 340.

[0152] If a second synonym is set in step S1704, the analysis unit 255 calculates the similarity between the registered question data and the registered answer data and the input question data acquired from the chat history information. Furthermore, the analysis unit 255 replaces the extracted keyword included in the input question data acquired from the chat history information with the second synonym corresponding to the extracted keyword, and calculates the similarity between the registered question data and the registered answer data (step S1705).

[0153] In step S1705, if there are multiple second synonyms associated with the extracted keyword, the analysis unit 255 performs the same process for each second synonym.

[0154] If a second synonym is not set in step S1704, the analysis unit 255 calculates the similarity between the registered question data and registered answer data and the input question data acquired from the chat history information (step S1706).

[0155] In the following description of this embodiment, the similarity calculated by the analysis unit 255 will be referred to as the second similarity.

[0156] Next, after calculating the second similarity between all answer to question information and the input question data, the analysis unit 255 determines whether there is any answer to question information whose second similarity exceeds the second similarity threshold (step S1707).

[0157] If there is no corresponding answer to question information in step S1707, the analysis unit 255 determines as an analysis result that there is no answer to question information containing appropriate answer data corresponding to the input question data acquired in step S1702 in the question and answer storage unit 320 (step S1708). Next, the analysis unit 255 references the improvement proposal storage unit 360 of the first storage device 300, selects the improvement type "Add new answer to question" corresponding to the value of the item "Analysis result" that is "No appropriate answer to question information", associates the analysis result and the improvement type with the conversation ID of the chat history information acquired in step S1701, stores them in the analysis result storage unit 350 of the first storage device 300 (step S1709), and proceeds to step S1722, which will be described later.

[0158] If there is corresponding answer to question information in step S1707, the analysis unit 255 identifies the answer to question information with the highest second similarity and acquires it as the second answer to question information (step S1710).

[0159] Next, the analysis unit 255 acquires the value of the item "reference question answer ID" included in the chat history information acquired in step S1701 (step S1711). Next, the analysis unit 255 determines whether or not the value of the item "reference question answer ID" has been acquired in step S1711 (step S1712).

[0160] In other words, the analysis unit 255 determines whether the chat history information includes a reference question answer ID that identifies the first question answer information that the chat processing unit 252 referenced when outputting the output answer data to the terminal device 600.

[0161] If a corresponding reference question and answer ID cannot be obtained in step S1712, the analysis unit 255 determines as an analysis result that answer to question information other than the answer to question information identified by the chat processing unit 252 based on the first similarity is appropriate (step S1713).

[0162] Next, the analysis unit 255 refers to the improvement proposal storage unit 360, selects the improvement type "Improvement of question and answer matching" for which the value of the item "Analysis result" corresponds to "Question and answer information other than the question and answer information used in the answer is appropriate", associates the analysis result and the improvement type with the conversation ID of the chat history information acquired in step S1701, stores them in the analysis result storage unit 350 of the first storage device 300 (step S1714), and proceeds to step S1722, which will be described later.

[0163] In step S1712, if the corresponding reference question answer ID is acquired, the analysis unit 255 determines whether the first question answer information identified by the reference question answer ID is the same as the second question answer information acquired in step S1710 (step S1715).

[0164] If the first answer to question information and the second answer to question information do not match in step S1715, analysis section 255 proceeds to step S1713.

[0165] If the answer to the first question information and the answer to the second question information match in step S1715, analysis unit 255 obtains the value of the item "user evaluation" in the chat history information obtained in step S1701 (step S1716).

[0166] Next, in step S1716, the analysis unit 255 determines whether the user rating is "unsatisfied" (step S1717). If the user rating is "unsatisfied" in step S1717, the analysis unit 255 determines, as an analysis result, that the answer to question information used in the answer is inappropriate or the user rating is "unsatisfied" (step S1718).

[0167] Next, the analysis unit 255 refers to the improvement proposal storage unit 360 of the first storage device 300, selects the improvement type "Consider revising the answer or resolving the dissatisfaction" corresponding to the value of the item "analysis result" that is "The question and answer information used in the answer is inappropriate, or the user evaluation is unsatisfactory," associates the analysis result and the improvement type with the conversation ID of the chat history information acquired in step S1701, stores them in the analysis result storage unit 350 of the first storage device 300 (step S1714), and proceeds to step S1722 described below.

[0168] In step S1717, if the user evaluation is not "unsatisfactory," the analysis unit 255 determines the analysis result as "other" (step S1720).

[0169] Next, the analysis unit 255 refers to the improvement proposal storage unit 360 of the first storage device 300, selects the improvement type "No improvement method proposed" for which the value of the item "Analysis result" corresponds to "Other", associates the analysis result and the improvement type with the conversation ID of the chat history information acquired in step S1701, stores them in the analysis result storage unit 350 of the first storage device 300 (step S1721), and proceeds to step S1722, which will be described later.

[0170] The analysis unit 255 executes the processes from step S1701 to step S1722 for all chat history information collected by the information acquisition unit 254 (step S1722), and then ends the process.

[0171] In this embodiment, the analysis unit 255 executes the process of FIG. 17 to identify, as the question and answer information to be improved, the question and answer information corresponding to the question and answer ID included in the chat history information for which an improvement type other than “no improvement method proposed” has been selected from the chat history information.

[0172] In the present embodiment, the information processing device 200 includes the second synonym storage unit 340, but this is not limiting. The information processing device 200 may determine whether the sentences are similar by vectorizing the sentences and calculating the distance between the sentences. In this way, it is possible to determine whether the input question data and the registered question data are similar without using the second synonym storage unit 340.

[0173] 18 is a flowchart illustrating the process of the screen generation unit of the information processing device, showing details of the process of step S1621 in FIG.

[0174] The screen generation unit 259 of the information processing device 200 refers to the analysis result storage unit 350 of the first storage device 300 and obtains analysis result information including the group ID assigned to the group selected on the proposal list screen (step S1801).

[0175] Next, the screen generating unit 259 performs the processes from step S1803 onwards for each piece of acquired analysis result information (step S1802).

[0176] The screen generating unit 259 acquires the value of the item "improvement type" from the analysis result information (step S1803). Subsequently, the screen generating unit 259 determines whether the value of the item "improvement type" is "no improvement method proposed" (step S1804).

[0177] In step S1804, if the value of the item "improvement type" is "no improvement method proposed", the screen generating unit 259 does not display buttons indicating improvement methods for the question and answer information corresponding to the reference question and answer ID included in the acquired analysis result information in the screen data for displaying the proposal details screen (step S1805), and proceeds to step S1813, which will be described later. Note that, if the value of the item "improvement type" is "no improvement method proposed", the screen generating unit 259 does not need to display the acquired analysis result information on the proposal details screen.

[0178] In step S1804, if the value of the item "Improvement type" is other than "No improvement method proposed" and is "Add new question and answer" (step S1806), the screen generation unit 259 highlights the "Add new question and answer" button, displays the "Improve question and answer matching" button normally, and hides the "Edit question and answer that seem appropriate" button in the screen data for displaying the proposal details screen (step S1807), and proceeds to step S1810 described below.

[0179] In step S1804, if the value of the item "Improvement type" is other than "No improvement method proposed" and is "Improvement of question and answer matching" (step S1806), the screen generation unit 259 displays the "Add new question and answer" button normally, the "Improve question and answer matching" button highlighted, and the "Edit question and answer that seem appropriate" button highlighted in the screen data for displaying the proposal details screen (step S1808), and proceeds to step S1810 described below.

[0180] In step S1804, if the value of the item "Improvement type" is other than "No improvement method proposed" and "Consideration of modifying answer or resolving dissatisfaction" (step S1806), the screen generation unit 259 displays the "Add new question and answer" button normally, the "Improve question and answer matching" button normally, the "Edit question and answer that seems appropriate" button non-display, and the "Edit question and answer used in answer" button highlighted in the screen data for displaying the proposal details screen (step S1809), and proceeds to step S1813 described below.

[0181] Following steps S1807 and S1808, the screen generating unit 259 determines whether or not a reference question and answer ID has been acquired from the analysis result information acquired in step S1801 (step S1810).

[0182] If a reference answer to question ID is not acquired in step S1810, the screen generating unit 259 hides the "Edit answer to question used in answer" button (step S1811), and proceeds to step S1813, which will be described later.

[0183] In step S1810, if a reference answer / question ID is acquired, the screen generating unit 259 displays the "Edit answer / question used in answer" button in the normal display mode (step S1812), and proceeds to step S1813, which will be described later.

[0184] Next, the screen generating unit 259 executes the processes from step S1802 to step S1812 for each piece of analysis result information acquired in step S1801, and then ends the process.

[0185] In this manner, the screen generating unit 259 of this embodiment generates screen data for displaying the proposal details screen.

[0186] In the present embodiment, the process of displaying the proposal details screen shown in Fig. 18 is executed by the screen generation unit 259 of the information processing device 200. However, this is not limiting. The information processing device 200 of the present embodiment may distribute a program for displaying the proposal details screen to the terminal device 700. In this case, the process of displaying the proposal details screen shown in Fig. 18 may be executed in the terminal device 700.

[0187] In addition, in this embodiment, the operation buttons that are to be hidden may be displayed in a grayed-out state in which acceptance of operations is prohibited, instead of being hidden.

[0188] 19 to 21, a description will now be given of an example of a management screen for answer to question information displayed on the terminal device 700. Fig. 19 is a first diagram showing an example of the management screen displayed on the terminal device.

[0189] The screen 150 shown in FIG. 19 is an example of a list screen of answer-to-question information, and may be displayed on the terminal device 700 in step S1601 of FIG.

[0190] The information processing device 200 of this embodiment, for example, when it receives an operation on the terminal device 700 instructing the display of the screen 150, may generate screen data for displaying the screen 150 by referring to the question and answer storage unit 320 in the first storage device 300 that is associated with the tenant ID of the tenant to which the user of the terminal device 700 belongs.

[0191] Screen 150 includes a display area 151 and operation buttons 152. Display area 151 displays a list of question and answer information stored in question and answer storage unit 320. Operation button 152 is an operation button for displaying question and answer information to be improved and an improvement method.

[0192] Furthermore, when answer-to-question information displayed in the display area 151 is selected, the screen 150 assumes that the answer-to-question information to be edited has been selected.

[0193] 20 is a diagram showing an example of a proposal list screen displayed on the terminal device 700. A screen 140 shown in FIG. 20 is an example of the proposal list screen displayed on the terminal device 700 in step S1616 of FIG.

[0194] 20 includes display areas 141 and 142. The period during which the chat history information was collected is displayed as the analysis period in the display area 141. The display area 142 displays a list of input question data included in the chat history information for each group.

[0195] In the example of FIG. 20, input question data 142a, 142b, 142c, 142d, 142e, etc. included in chat history information classified into group 1 are displayed in display area 142.

[0196] In this embodiment, when a specific group is selected in the display area 142, the screen 140 transitions to a proposal details screen for each group.

[0197] Fig. 21 is a diagram showing an example of a proposal details screen displayed on a terminal device. The screen 130 shown in Fig. 21 is an example of a proposal list screen displayed on the terminal device 700 in step S1624 of Fig. 16. The screen 130 is also an example of a proposal details screen when group 1 is selected in the display area 142 of the screen 140.

[0198] The screen 130 has a display area 131, in which chat history information classified into group 1, corresponding analysis result information, and buttons corresponding to improvement types are displayed in association with each other.

[0199] Specifically, display area 131 includes display areas 132, 133, 134, 135, 136, 137, etc. Display area 132 displays a portion of the analysis result information including text data 142a shown in Fig. 20, and buttons 132a and 132b.

[0200] Here, in the analysis result information corresponding to the text data 142a, the improvement type is "Add new question and answer" and there is no reference question and answer ID (see FIG. 8). Therefore, in the display area 132, the "Add new question and answer" button 132a is highlighted, the "Improve question and answer matching" button 132b is displayed normally, and the "Edit appropriate question and answer" button and the "Edit question and answer used in answer" button are hidden.

[0201] Also displayed in display area 133 are a portion of the analysis result information including text data 142b shown in FIG. 20, and buttons 133a, 133b, and 133c.

[0202] Here, in the analysis result information corresponding to the text data 142b, the improvement type is "add new question and answer" and a reference question and answer ID exists (see FIG. 8). Therefore, in the display area 133, the "add new question and answer" button 133a is highlighted, the "edit question and answer used in answer" button 133b and the "improve question and answer matching" button 133c are normally displayed, and the "edit question and answer that seems appropriate" button is hidden.

[0203] Also displayed in display area 134 are a portion of the analysis result information including text data 142c shown in FIG. 20, and buttons 134a, 134b, and 134c.

[0204] Here, in the analysis result information corresponding to the text data 142c, the value of the item "Improvement type" is other than "No improvement method proposed" and is "Consider modifying the answer or resolving the dissatisfaction" (see FIG. 8). Therefore, in the display area 134, the "Add new question and answer" button 134a and the "Improve question and answer matching" button 134c are normally displayed, the "Edit question and answer used in the answer" button 134b is highlighted, and the "Edit question and answer considered appropriate" button is hidden.

[0205] Also displayed in display area 135 are a portion of the analysis result information including text data 142d shown in FIG. 20, and buttons 135a, 135b, and 135c. Here, in the analysis result information corresponding to the text data 142d, the value of the item "Improvement type" is other than "No improvement method proposed" and is "Improvement of question and answer matching" (see FIG. 8). Therefore, in the display area 135, the "Add new question and answer" button 135a is normally displayed, the "Edit question and answer that seems appropriate" button 135b and the "Improve question and answer matching" button 135c are highlighted, and the "Edit question and answer used in answer" button is hidden.

[0206] Also displayed in display area 136 are a portion of the analysis result information including text data 142e shown in FIG. 20, and buttons 136a, 136b, 136c, and 136d. Here, in the analysis result information corresponding to text data 142e, the value of the item "Improvement type" is other than "No improvement method proposed" and is "Improvement of question and answer matching" (see FIG. 8). Therefore, in display area 135, "Add new question and answer" button 136a and "Edit question and answer used in answer" button 136b are normally displayed, and "Edit question and answer considered appropriate" button 136b and "Improve question and answer matching" button 136d are highlighted.

[0207] Also, in the display area 137, a part of the analysis result information including the text data 142f shown in FIG. 20 is displayed.

[0208] Here, in the analysis result information corresponding to text data 142f, the value of the item "improvement type" is "no improvement method proposed" (see FIG. 8). Therefore, in display area 137, a button proposing an improvement method corresponding to text data 142f is not displayed.

[0209] In this way, the information processing device 200 of this embodiment refers to chat history information indicating the history of chats conducted with the terminal device 600, the second synonym memory unit 340 storing a group of general synonyms, and user evaluations, and analyzes whether the content of the output response data output by the information processing device 200 to the terminal device 600 is appropriate.

[0210] Then, the information processing device 200 causes the terminal device 700 to display a proposal details screen showing improvement methods recommended for each piece of answering information for which improvement is required. Furthermore, the information processing device 200 of this embodiment causes a list of improvement methods for each piece of answering information to be displayed as a list of operation buttons on the proposal details screen.

[0211] Therefore, according to this embodiment, when editing question answer information, the administrator simply selects the operation button corresponding to the improvement method displayed on the proposal details screen, allowing the administrator to edit question answer information efficiently and easily.

[0212] Furthermore, when an operation button corresponding to an improvement method is selected on the proposal details screen displayed on the terminal device 700, the information processing device 200 of this embodiment causes the terminal device 700 to display a specific improvement proposal corresponding to the improvement method indicated by the selected operation button.

[0213] The following describes the operation of the chat system 100 when an operation button corresponding to an improvement method is selected on the proposal details screen displayed on the terminal device 700.

[0214] Fig. 22 is a third sequence diagram illustrating the operation of the chat system. In Fig. 22, it is assumed that the screen 130 (proposal detail screen) shown in Fig. 21 is displayed on the display 718 of the terminal device 700.

[0215] The terminal device 700 receives, via the receiving unit 740, an operation to select an improvement method corresponding to the analysis result information on the screen 130 (step S2201).

[0216] The following operations from step S2202 to step S2210 indicate the operations when the "Add new question and answer" button is selected in step S2201.

[0217] The terminal device 700 transmits a notification indicating that the "Add new question and answer" button has been selected to the information processing device 200 (step S2202).

[0218] When the information processing device 200 receives this notification via the receiving unit 251, the search request unit 253 sends a search request for internal company information that can be a source of information to the document storage unit 410 of the second storage device 400 (step S2203). At this time, the search request includes the input question data included in the analysis result information. In other words, the search request unit 253 sends a search request based on the input question data included in the analysis result information to the second storage device 400.

[0219] When the second storage device 400 receives the search request, the search unit 430 searches the document storage unit 410 and acquires chunks that are search results (step S2204). Subsequently, the second storage device 400 transmits the chunks acquired as search results to the information processing device 200 (step S2205).

[0220] The information processing device 200 acquires chunks of search results by the information acquisition unit 254. The prompt generating unit 256 generates a prompt requesting the generation of answer data to the input question data using the chunk of search results (internal information) as an information source (step S2206). Subsequently, the information processing device 200 transmits the generated prompt to the server device 500 by the requesting unit 257 (step S2207). The prompt generated in step S2206 will be described in detail later.

[0221] When the server device 500 receives this prompt, the input unit 520 inputs the prompt to the large-scale language model 511, and the output unit 530 transmits output data output from the large-scale language model 511 to the information processing device 200 (step S2208).

[0222] When the information processing device 200 acquires the output data, the output unit 258 outputs the output response data included in the acquired output data to the terminal device 700 (step S2209). The terminal device 700 causes the display control unit 750 to display the response data included in the output data as an improvement proposal on the display 718 (step S2210).

[0223] The operations from step S2211 to step S2214 below are the operations performed when the "Improve question and answer matching" button is selected in step S2201. Note that question and answer matching in this embodiment refers to matching between input question data and registered answer data included in question and answer information identified based on the input question data.

[0224] In this embodiment, by providing multiple patterns of registered question data included in the answer to question information, it becomes easier to identify the answer to question information corresponding to the input question data from the answer to question information stored in the question and answer storage unit 320.

[0225] The terminal device 700 transmits a notification indicating that the "Improve question and answer matching" button has been selected to the information processing device 200 (step S2211).

[0226] When the information processing device 200 receives this notification via the reception unit 251, the prompt generation unit 256 generates a prompt requesting the generation of new registered question data that paraphrases the expression of the registered question data corresponding to the input question data (step S2212).

[0227] The alternative registered question data, which is a rephrasing of the registered question data, is text data in which the content indicated by the registered question data is expressed in a different way.

[0228] Subsequently, information processing device 200 causes request unit 257 to transmit the generated prompt to server device 500 (step S2213). Details of the prompt generated in step S2212 will be described later.

[0229] When the server device 500 receives this prompt, the input unit 520 inputs the prompt to the large-scale language model 511, and the output unit 530 transmits output data output from the large-scale language model 511 to the information processing device 200 (step S2214).

[0230] When the information processing device 200 acquires the output data, the output unit 258 outputs the different registered question data, which is included in the acquired output data and in which the expression has been rephrased, as an improvement proposal to the terminal device 700 (step S2215). The terminal device 700 causes the display control unit 750 to display the improvement proposal including the different registered question data in which the expression has been rephrased on the display 718 (step S2216).

[0231] The following operations from step S2217 to step S2225 indicate the operations when the "Edit appropriate question and answer" button or the "Edit question and answer used in answer" button is selected in step S2201.

[0232] The terminal device 700 transmits to the information processing device 200 a notification indicating that the "Edit appropriate answer to question" button or the "Edit answer to question used in answer" button has been selected (step S2217).

[0233] The operations from step S2218 to step S2220 in FIG. 22 are the same as the operations from step S2203 to step S2205, and therefore the explanation will be omitted.

[0234] Next, the information processing device 200 causes the prompt generating unit 256 to generate a prompt requesting analysis of question and answer information including registered question data corresponding to the input question data (step S2221).

[0235] Subsequently, information processing device 200 causes request unit 257 to transmit the generated prompt to server device 500 (step S2222). Details of the prompt generated in step S2222 will be described later.

[0236] When the server device 500 receives this prompt, the input unit 520 inputs the prompt to the large-scale language model 511, and the output unit 530 transmits output data output from the large-scale language model 511 to the information processing device 200 (step S2223).

[0237] When the information processing device 200 acquires the output data, the output unit 258 outputs the analysis results included in the acquired output data as an improvement proposal to the terminal device 700 (step S2224). The terminal device 700 causes the display control unit 750 to display the improvement proposal including the analysis results of the question and answer information on the display 718 (step S2225).

[0238] Next, a prompt generated by the prompt generating unit 256 of the information processing device 200 and a display example of output data of the large-scale language model 511 in response to the prompt will be described.

[0239] Fig. 23 is a first diagram showing an example of a prompt. Prompt 10 shown in Fig. 23 is an example of a prompt generated by prompt generation unit 256 in step S2206 of Fig. 22. In other words, prompt 10 shown in Fig. 23 is a prompt that requests the generation of answer data to input question data using chunks (company information) as an information source.

[0240] Furthermore, the prompt 10 shown in FIG. 23 is an example of a prompt that is generated when the “Add new question and answer” button 132a displayed in the display area 132 on the screen 130, which is the proposal details screen shown in FIG. 21, is selected.

[0241] The prompt 10 includes text data 10a, 10b, and 10c. The text data 10a is text data that indicates to the large-scale language model 511 that the role of the large-scale language model 511 is to generate answer data for input question data, and requests that an answer be generated based on chunks of search results.

[0242] Text data 10b is text data indicating input question data. When an operation button is selected on screen 130, the prompt generation unit 256 of this embodiment refers to the analysis result storage unit 350 of the first storage device 300 and identifies the analysis result information corresponding to the selected operation button. Then, the prompt generation unit 256 sets the input question data included in the identified analysis result information as text data 10b.

[0243] In the example of Figure 23, the input question data included in the analysis result information corresponding to the "Add new question and answer" button 132a is "Product introduction case," so the text data 10b is text data indicating that the input question data is "Product introduction case."

[0244] The text data 10c is an information source used to generate answer data. Here, the information source is a chunk included in the company information. In this embodiment, the information source included in the prompt is not all of the company information stored in the document storage unit 410, but is a chunk of the company information acquired as a search result. This makes it possible to request the generation of answer data while satisfying the token restrictions in the large-scale language model 511.

[0245] Next, with reference to FIG. 24, a display example of an improvement plan including output data from the large-scale language model 511 will be described.

[0246] 24 is a first diagram showing an example of a display of an improvement plan on a terminal device, which is displayed on the display 718 of the terminal device 700 in step S2210 of FIG.

[0247] 24 includes display areas 161 and 162, and operation buttons 163 and 164. Display area 161 displays the input question data included in the analysis result information corresponding to "Add new answer to question" button 132a.

[0248] Display area 162 displays the input question data and corresponding answer data displayed in display area 161. Operation button 163 is an operation button for terminating the display of screen 160. Operation button 164 is an operation button for storing the input question data and answer data displayed in display areas 161 and 162 in question and answer storage unit 320 as new question and answer information.

[0249] In this embodiment, by displaying the input question data in correspondence with the answer data generated by the large-scale language model 511, the user of the terminal device 700 (the administrator of the chat system 100) can confirm the contents of the question data and answer data that will be newly registered as question data and answer data.

[0250] Furthermore, in the display area 162 of this embodiment, the input question data displayed in the display area 161 and the answer data displayed in the display area 162 can be edited (corrected).

[0251] The input question data and answer data displayed in the display areas 161 and 162 are text data. In this embodiment, the user of the terminal device 700 can correct the content of the question data and answer data to be newly registered question data and registered answer data by editing the input question data and answer data displayed in the display areas 161 and 162, respectively.

[0252] In this embodiment, the output data from the large-scale language model 511 may include information other than the answer to the input question data.

[0253] Specifically, for example, the answer data may include, in addition to the answer corresponding to the input question data, information indicating contact information for the user of the terminal device 600 in case of further inquiries. Furthermore, the output data from the large-scale language model 511 may include, in addition to the answer corresponding to the input question data, a link for displaying information related to the content indicated by the input question data and the answer data.

[0254] 24, information 162a indicating contact information and a link 162b are displayed in the display area 162. In this embodiment, the information 162a indicating contact information and the link 162b may be included in the output data output from the large-scale language model 511, or may be input by the user of the terminal device 700 in an editing operation.

[0255] When the operation button 164 is operated on the screen 160 of the terminal device 700, the information processing device 200 of this embodiment associates the input question data displayed in the display area 161 with the answer data displayed in the display area 162 to generate question and answer information. Then, the information processing device 200 outputs the question and answer information to the first storage device 300 via the output unit 258, and stores the information in the question and answer storage unit 320 of the first storage device 300.

[0256] In this manner, in this embodiment, new question and answer information can be added to the question and answer storage unit 320. Therefore, in this embodiment, the administrator of the chat system 100 does not need to create and register new question and answer information himself, thereby reducing the burden on the administrator.

[0257] Fig. 25 is a second diagram showing an example of a prompt. Prompt 20 shown in Fig. 25 is an example of a prompt generated by prompt generation unit 256 in step S2212 of Fig. 22. In other words, prompt 20 shown in Fig. 25 is a prompt requesting the generation of new registered question data that paraphrases the expression of the registered question data corresponding to the input question data.

[0258] Furthermore, the prompt 20 shown in FIG. 25 is an example of a prompt that is generated when the “Improve question and answer matching” button 135c displayed in the display area 132 on the screen 130, which is the proposal details screen shown in FIG. 21, is selected.

[0259] The prompt 20 includes text data 20a, 20b, 20c, and 20d. The text data 20a is text data that indicates to the large-scale language model 511 that the role of the large-scale language model 511 is to generate answer data to the input question data, and that a proposal of another registered question data that paraphrases the expression of the registered question data corresponding to the input question data is requested.

[0260] The text data 20b is text data indicating the input question data included in the analysis result information corresponding to the "Improve question and answer matching" button 135c.

[0261] The text data 20c is text data indicating output response data output to the terminal device 600 in response to the input question data. The output response data indicated by the text data 20c is acquired from chat history information identified by a conversation ID included in the analysis result information corresponding to the "Improve question and answer matching" button 135c.

[0262] The text data 20d is text data indicating an answer that is believed to be correct for the question indicated by the input question data. The text data 20d is registered answer data included in the question and answer information identified by the "appropriate question and answer ID" included in the analysis result information corresponding to the "Improve question and answer matching" button 135c.

[0263] In the example of Figure 25, the input question data included in the analysis result information corresponding to the "Improve question and answer matching" button 135c is "Product list," so the text data 20b is text data indicating that the input question data is "Product list."

[0264] Furthermore, the text data 20c is the output response data "No response found" obtained from the chat history information identified by the conversation ID included in the analysis result information corresponding to the "Improve question and answer matching" button 135c.

[0265] In addition, the text data 20d is the registered answer data "The product list is..." contained in the question and answer information identified by the "Appropriate Question and Answer ID" included in the analysis result information corresponding to the "Improve Question and Answer Matching" button 135c.

[0266] Next, with reference to FIG. 26, a display example of an improvement plan including output data from the large-scale language model 511 will be described.

[0267] 26 is a second diagram showing an example of the display of the improvement plan on the terminal device, which is displayed on the display 718 of the terminal device 700 in step S2216 of FIG.

[0268] 26 includes display areas 171 and 172, and operation buttons 173 and 174. Display area 171 displays input question data included in the analysis result information corresponding to "Improve question and answer matching" button 135c.

[0269] Display area 172 displays question data that is a paraphrase of the registered answer data corresponding to the input question data displayed in display area 171. In other words, display area 172 displays registered question data of a different pattern that changes the expression of the registered answer data corresponding to the input question data displayed in display area 171.

[0270] Operation button 173 is an operation button for ending the display of screen 170. Operation button 174 is an operation button for storing the question data displayed in display area 172 in question and answer storage unit 320.

[0271] In this embodiment, when the operation button 174 is operated on the terminal device 700, the information processing device 200 stores the question data displayed in the display area 172 in the question and answer storage unit 320 of the first storage device 300 as another pattern of registered question data corresponding to the registered answer data displayed as text data 20d.

[0272] In addition, in this embodiment, the question data displayed in the display area 172 can be edited. In this embodiment, the user of the terminal device 700 can edit the question data displayed in the display area 172 to correct the content of the question data.

[0273] In this way, in this embodiment, different patterns that rephrase the registered question data corresponding to the input question data are generated in the large-scale language model 511. Therefore, in this embodiment, the administrator of the chat system 100 does not need to consider multiple patterns of registered question data, and the red color is reduced.

[0274] Fig. 27 is a third diagram showing an example of a prompt. Prompt 30 shown in Fig. 27 is an example of a prompt generated by prompt generation unit 256 in step S2221 of Fig. 22. In other words, prompt 30 shown in Fig. 27 is a prompt requesting analysis of question and answer information including registered question data corresponding to input question data.

[0275] Furthermore, the prompt 30 shown in FIG. 27 is an example of a prompt that is generated when the “Edit question and answer used in the answer” button 134b displayed in the display area 134 or the “Edit question and answer that seems appropriate” button 135b displayed in the display area 135 is selected on the screen 130, which is the proposal details screen shown in FIG. 21.

[0276] The prompt 30 includes text data 30a, 30b, 30c, 30d, and 30e. The text data 30a includes text data indicating to the large-scale language model 511 that the role of the large-scale language model 511 is to generate answer data to input question data, that the user's evaluation of the output answer data is dissatisfactory, and that the reason for the user's dissatisfaction is either "1. The content of the answer is insufficient or inaccurate" or "2. The content of the answer is appropriate, but the user's evaluation is dissatisfactory."

[0277] The text data 30b is text data indicating a request for an analysis of the cause of the user's dissatisfaction with the output response data, and the procedure for the analysis.

[0278] Specifically, the text data 30b indicates that the analysis procedure involves identifying whether the reason for the user's dissatisfaction in the evaluation is "1" or "2" as described above, proposing an improvement plan if the cause is "1," and proposing the reason for the user's dissatisfaction if the cause is "2."

[0279] The text data 30c indicates the input question data included in the analysis result information corresponding to the selected operation button.

[0280] The text data 30d indicates output response data in the chat history information identified by the conversation ID included in the analysis result information corresponding to the selected operation button.

[0281] The text data 30e indicates the information source used for the analysis. Specifically, the text data 30d is a chunk obtained from the document storage unit 410.

[0282] Next, a display example of an improvement plan including output data from the large-scale language model 511 will be described with reference to Fig. 28. Fig. 28 is a third diagram showing a display example of an improvement plan on a terminal device.

[0283] Screen 180 shown in Figure 28 is an example of a screen that displays improvement suggestions when the reason for a user's dissatisfaction in the large-scale language model 511 is determined to be "1. The content of the answer is insufficient or inaccurate."

[0284] Screen 180 includes display areas 181 and 182, and operation buttons 183 and 184. Display area 181 displays input question data included in the analysis result information corresponding to the operation button selected on screen 130 shown in FIG.

[0285] The input question data and corresponding answer data are displayed in the display area 181. The answer data displayed in the display area 182 is the answer data generated by the large-scale language model 511 in response to the input of the prompt 30.

[0286] Operation button 183 is an operation button for terminating the display of screen 180. Operation button 184 is an operation button for storing the input question data and answer data displayed in display areas 181 and 182 in question and answer storage unit 320 as new question and answer information.

[0287] Furthermore, in the display area 182 of this embodiment, the input question data displayed in the display area 181 and the answer data displayed in the display area 182 can be edited (corrected).

[0288] The input question data and answer data displayed in the display areas 181 and 182 are text data. In this embodiment, the user of the terminal device 700 can edit the input question data and answer data displayed in the display areas 181 and 182, respectively, to modify the question and answer information referenced when specifying the output answer data.

[0289] In addition to the answer corresponding to the input question data, the display area 182 also displays information 182a indicating contact information for further inquiries by the user of the terminal device 600 and a link 182b. In this embodiment, the information 182a indicating contact information and the link 182b may be included in the output data output from the large-scale language model 511, or may be input by the user of the terminal device 700 in an editing operation.

[0290] In this embodiment, when the operation button 184 is operated on the screen 180, the input question data displayed in the display area 181 and the answer data displayed in the display area 182 are linked and stored in the question and answer storage unit 320 via the information processing device 200.

[0291] In other words, when the operation button 184 is operated on the terminal device 700, the information processing device 200 of this embodiment overwrites the content displayed in the display areas 181 and 182 with the question and answer information identified by the reference question and answer ID included in the analysis result information stored in the question and answer storage unit 320.

[0292] In this manner, the present embodiment allows modification of the question and answer information in the question and answer storage unit 320. Therefore, in this embodiment, the administrator of the chat system 100 does not need to come up with suggestions for modifying the question and answer information himself, thereby reducing the burden on the administrator.

[0293] Next, another example of displaying an improvement plan including output data from large-scale language model 511 will be described with reference to Fig. 29. Fig. 29 is a fourth diagram showing an example of displaying an improvement plan on a terminal device.

[0294] Screen 190 shown in Figure 29 is an example of a screen that displays improvement suggestions when the reason for the user's dissatisfaction in the large-scale language model 511 is determined to be "2. The answer content is appropriate but the user's evaluation is dissatisfied."

[0295] Screen 190 includes display areas 191 and 192, and operation buttons 193 and 194. Display area 191 displays a message indicating that the answer to question information used in the answer is appropriate but the user's evaluation is unsatisfactory.

[0296] Display area 192 displays information indicating the result of estimation by large-scale language model 511 of the reason why the user is dissatisfied.

[0297] The operation button 193 is an operation button for ending the display of the screen 190. The operation button 194 is an operation button for transitioning the screen 190 to the screen 130 (proposal detail screen) shown in FIG.

[0298] In this manner, in this embodiment, the administrator of the chat system 100 can understand whether the answer to question information is appropriate and the reason why the user is dissatisfied with the evaluation of the output answer data.

[0299] In addition, in the chat system 100 of this embodiment, the information processing device 200 may send a link to the terminal device 600 in response to input question data received from the terminal device 600 to display information related to the content indicated by the input question data and answer data.

[0300] Below, examples of displays on the terminal device 600 will be described with reference to Fig. 30 to Fig. 32. Fig. 30 is a second diagram showing an example of a chat screen displayed on the terminal device.

[0301] 30 includes display areas 121a, 121b, and 121d. The display area 121a displays input question data input by the user of the terminal device 600. The display area 121b displays output answer data from the chat system 100, as well as text data 121b-1 and a link 121b-2.

[0302] The text data 121b-1 includes information indicating contact information for the user of the terminal device 600 in case of further inquiries. The information indicating contact information may be, for example, an email address or a telephone number. This contact information may be stored in the user information storage unit 420, for example. Specifically, the user information storage unit 420 may store information indicating the work responsibilities of each user in association with the user ID.

[0303] When transmitting output response data to the terminal device 600, the information processing device 200 may obtain the email address of the user in charge of the task corresponding to the question indicated by the input question data, and transmit the email address to the terminal device 600 together with the output response data.

[0304] The link 121b-2 is a link for displaying information related to the content indicated by the input question data or the output answer data.

[0305] In this embodiment, by notifying the user of the terminal device 600 of contact information for further inquiries and information related to the question and answer along with the output answer data, it is possible to provide the user with an answer that will satisfy them.

[0306] In the display area 121d, a message notifying that the link 121b-2 displayed in the display area 121b has been sent to the email address of the user of the terminal device 600 is displayed.

[0307] In this case, the user ID of the user of the terminal device 600 may be input to the information processing device 200 together with the input question data, and the information processing device 200 may refer to the user information storage unit 420 and acquire an email address identified from the tenant ID and the user ID. Then, when transmitting output response data to the terminal device 600, the information processing device 200 may transmit the link 121b-2 together with the output response data to the identified email address, and then cause the terminal device 600 to display the display area 121d.

[0308] In the example of Figure 30, by sending link 121b-2 to the user's email address to display information related to the user's inquiry, the user can refer to the information related to the inquiry even after finishing using the chat system 100.

[0309] In this embodiment, by notifying the user of the terminal device 600 of contact information for further inquiries and information related to the question and answer along with the output answer data, it is possible to provide the user with an answer that will satisfy them.

[0310] Furthermore, the information processing device 200 of this embodiment may send a notification to the contact information included in the text data 121b-1, indicating that an inquiry has been received from the user of the terminal device 600. By sending such a notification, it is possible to prompt the user who is the contact information when making an inquiry to make preparations for responding to further inquiries.

[0311] Furthermore, in this embodiment, a notification indicating that an inquiry has been received from the user of the terminal device 600 may be sent to the administrator of the chat system 100. Specifically, the information processing device 200 may send a notification indicating that an inquiry has been received from the user of the terminal device 600 to the terminal device 700.

[0312] In this embodiment, by doing so, the administrator of the chat system 100 can understand what kind of questions users are asking the chat system 100 .

[0313] Fig. 31 is a third diagram showing an example of a chat screen displayed on a terminal device. Screen 121B shown in Fig. 31 includes display areas 121a and 121e. Display area 121e includes text data 121b-1 indicating contact information for the user of terminal device 600 in case of further inquiries, and a message 121f.

[0314] Message 121f is a message indicating that a link for displaying information related to the user's inquiry has been sent to the user's email address.

[0315] In this way, in the example of FIG. 31, a link for displaying information related to the user's inquiry can be notified to the user separately from the output response data.

[0316] Fig. 32 is a fourth diagram showing an example of a chat screen displayed on a terminal device. Screen 121C shown in Fig. 32 is an example of a screen displayed on terminal device 600 when the user information storage unit 420 of second storage device 400 does not store the user's email address.

[0317] Screen 121C includes display areas 121a, 121b, 121g, 121h, and 121i. Display area 121g displays a message inquiring about the destination of link 121b-2. Display area 121h displays the email address entered by the user. Display area 121i displays a message notifying the user that link 121b-2 has been sent to the email address entered by the user.

[0318] In the example of FIG. 32, even if the user's email address is not stored in the user information storage unit 420, the link 121b-2 can be sent to the user if the user so desires.

[0319] The functions of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to execute each of the above-described functions.

[0320] Additionally, the devices described in each embodiment represent only one of multiple computing environments for implementing the embodiments disclosed herein.

[0321] In one embodiment, information processing apparatus 200 includes multiple computing devices, such as a server cluster, configured to communicate with each other over any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein. Similarly, information processing apparatus 200 may include multiple computing devices configured to communicate with each other.

[0322] Furthermore, the information processing device 200 can be configured to share the disclosed processing steps in various combinations. For example, a process performed by a predetermined unit can be executed by the information processing device 200. Similarly, the function of a predetermined unit can be executed by the information processing device 200. Furthermore, in the information processing device 200, each element may be integrated into a single server device or may be separated into multiple devices.

[0323] The information processing device 200 may be any device that has a communication function. The information processing device 200 may be, for example, a PJ (Projector), an IWB (Interactive White Board: a white board with an electronic blackboard function that allows mutual communication), an output device such as digital signage, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, medical equipment, a network home appliance, an automobile (Connected Car), a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

[0324] For example, aspects of the present invention are as follows. <1> a display control unit that causes the terminal device to display a screen including a list of improvement methods for the answer to question information that was referenced when outputting the output answer data for the input question data; a receiving unit that receives, from the terminal device, a predetermined improvement method selected from the list of improvement methods displayed on a screen of the terminal device; a request unit that requests a large-scale language model to generate an improvement plan according to the predetermined improvement method received by the receiving unit; an output unit that outputs an improvement plan generated by the large-scale language model to the terminal device based on the request. <2> When adding new answer information is selected as the improvement method, The request unit a request for generating answer data to the input question data from a large-scale language model, the request including the input question data; The output unit outputting the answer data generated by the large-scale language model to the terminal device in association with the input question data; <1> The information processing device described. <3> the request for generating response data includes user-specific information to be used in generating the response data; <1> or <2> The information processing device described. <4> When the improvement method is selected to improve the matching between the input question data and the registered answer data included in the question and answer information, The request unit requesting the large-scale language model to generate registered question data in which the registered question data included in the question and answer information is expressed differently; The output unit the registered question data with the different expressions generated by the large-scale language model is output to the terminal device in association with the input question data; <1> ~ <3> 10. The information processing device according to claim 9, wherein: <5> As the improvement method, when editing the question and answer information is selected, The request unit requesting the large-scale language model to analyze the cause of the user's dissatisfaction with the output response data; The output unit outputting data corresponding to the analysis results generated by the large-scale language model to the terminal device; <1> ~ <4> The information processing device described. <6> If the cause is that the content of the output response data is insufficient, The output unit outputting the answer data generated by the large-scale language model to the terminal device in association with the input question data; <5> The information processing device described. <7> When the cause is that the content of the output response data is appropriate but the user's evaluation is dissatisfied, The output unit outputting to the terminal device a result of estimating the reason why the user's evaluation is dissatisfied, the result being generated by the large-scale language model; <5> or <6> The information processing device described. <8> An information processing system including an information processing device and a terminal device that communicates with the information processing device, The information processing device includes: a display control unit that causes the terminal device to display a screen including a list of improvement methods for the answer to question information that was referenced when outputting the output answer data for the input question data; a receiving unit that receives, from the terminal device, a predetermined improvement method selected from the list of improvement methods displayed on a screen of the terminal device; a request unit that requests a large-scale language model to generate an improvement plan according to the predetermined improvement method received by the receiving unit; an output unit that outputs an improvement plan generated by a large-scale language model based on the request to the terminal device; The terminal device An information processing system having a display control unit that displays the improvement proposal. <9> An information processing method including an information processing device and a terminal device that communicates with the information processing device, The information processing device, displaying on the terminal device a screen including a list of improvement methods for the answer to question information referred to when outputting the output answer data for the input question data; receiving, from the terminal device, a predetermined improvement method selected from the list of improvement methods displayed on the screen of the terminal device; requesting a large-scale language model to generate an improvement plan according to the received predetermined improvement method; outputting the improvement plan generated by the large-scale language model to the terminal device; The terminal device, An information processing method for displaying the improvement proposal. <10> displaying on the terminal device a screen including a list of improvement methods for the answer to question information referred to when outputting the output answer data for the input question data; receiving, from the terminal device, a predetermined improvement method selected from the list of improvement methods displayed on the screen of the terminal device; requesting a large-scale language model to generate an improvement plan according to the received predetermined improvement method; An information processing program that causes an information processing device to execute a process of outputting the improvement plan generated by the large-scale language model to a terminal device.

[0325] Although the present invention has been described above based on the embodiments, the present invention is not limited to the requirements shown in the above embodiments. These requirements can be changed without departing from the spirit of the present invention, and can be appropriately determined depending on the application form. [Explanation of symbols]

[0326] 100 Chat System 200 Information processing device 251 Reception 252 Chat processing section 253 Search Request Section 254 Information Acquisition Department 255 Analysis Department 256 Prompt Generation Unit 257 Request part 258 output section 259 Screen generation section 300 First Storage Device 310 Chat history memory section 320 Question and answer storage section 400 Secondary Storage Device 410 Document Storage Unit 420 User information storage unit 500 Server device 511 Large-scale Language Models 600, 700 terminal equipment [Prior art documents] [Patent documents]

[0327] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-339314

Claims

1. a display control unit that causes the terminal device to display a screen including a list of improvement methods for the answer to question information that was referenced when outputting the output answer data for the input question data; a receiving unit that receives, from the terminal device, a predetermined improvement method selected from the list of improvement methods displayed on a screen of the terminal device; a request unit that requests a large-scale language model to generate an improvement plan according to the predetermined improvement method received by the receiving unit; an output unit that outputs an improvement plan generated by the large-scale language model to the terminal device based on the request.

2. When adding new answer information is selected as the improvement method, The request unit a request for generating answer data to the input question data from a large-scale language model, the request including the input question data; The output unit The information processing apparatus according to claim 1 , wherein the answer data generated by the large-scale language model is output to the terminal device in association with the input question data.

3. The information processing apparatus according to claim 2 , wherein the request for generating the response data includes user-specific information used in generating the response data.

4. When the improvement method is selected to improve the matching between the input question data and the registered answer data included in the question and answer information, The request unit requesting the large-scale language model to generate registered question data in which the registered question data included in the question and answer information is expressed differently; The output unit The information processing apparatus according to claim 3 , wherein the registered question data with the different expressions generated by the large-scale language model is output to the terminal device in association with the input question data.

5. As the improvement method, when editing the question and answer information is selected, The request unit requesting the large-scale language model to analyze the cause of the user's dissatisfaction with the output response data; The output unit The information processing apparatus according to claim 1 , wherein data corresponding to the analysis results generated by the large-scale language model is output to the terminal device.

6. If the cause is that the content of the output response data is insufficient, The output unit The information processing apparatus according to claim 5 , wherein the answer data generated by the large-scale language model is output to the terminal device in association with the input question data.

7. When the cause is that the content of the output response data is appropriate but the user's evaluation is dissatisfied, The output unit The information processing apparatus according to claim 5 , wherein the result of estimating the reason why the user's evaluation is dissatisfied, which is generated by the large-scale language model, is output to the terminal device.

8. An information processing system including an information processing device and a terminal device that communicates with the information processing device, The information processing device includes: a display control unit that causes the terminal device to display a screen including a list of improvement methods for the answer to question information that was referenced when outputting the output answer data for the input question data; a receiving unit that receives, from the terminal device, a predetermined improvement method selected from the list of improvement methods displayed on a screen of the terminal device; a request unit that requests a large-scale language model to generate an improvement plan according to the predetermined improvement method received by the receiving unit; an output unit that outputs an improvement plan generated by a large-scale language model based on the request to the terminal device; The terminal device An information processing system having a display control unit that displays the improvement proposal.

9. An information processing method including an information processing device and a terminal device that communicates with the information processing device, The information processing device, displaying on the terminal device a screen including a list of improvement methods for the answer to question information referred to when outputting the output answer data for the input question data; receiving, from the terminal device, a predetermined improvement method selected from the list of improvement methods displayed on the screen of the terminal device; requesting a large-scale language model to generate an improvement plan according to the received predetermined improvement method; outputting the improvement plan generated by the large-scale language model to the terminal device; The terminal device, An information processing method for displaying the improvement proposal.

10. displaying on the terminal device a screen including a list of improvement methods for the answer to question information referred to when outputting the output answer data for the input question data; receiving, from the terminal device, a predetermined improvement method selected from the list of improvement methods displayed on the screen of the terminal device; requesting a large-scale language model to generate an improvement plan according to the received predetermined improvement method; An information processing program that causes an information processing device to execute a process of outputting the improvement plan generated by the large-scale language model to a terminal device.

Images