Information processing equipment, information processing system, information processing method

Abstract

Reduce the effort required to create question-and-answer pairs. [Means for solving the problem] The system accepts input question data and an evaluation of the answer data output in response to the question data. Upon receiving input question data, a large-scale language model generates answer data using user-specific information as the information source and outputs it to a first terminal device. First chat history information, including the question data, the answer data output to the first terminal device, and the evaluation of the answer data, is stored in a first storage device. A prompt is generated requesting the large-scale language model to generate a pair of question data and answer data based on the answer data included in the first chat history information whose evaluation satisfies a first condition and the question data corresponding to said answer data. The pair of question data and answer data generated by the large-scale language model is output to a second terminal device.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, and an information processing method.

Background Art

[0002] In recent years, for each user (business operator) and / or for each dataset, knowledge data for generating an appropriate answer sentence for a question sentence is recorded, and based on the search result of the knowledge data, A chatbot system that transmits the generated answer information to a user terminal is known. Also, in this chatbot system, it is known that the knowledge data includes Frequently Asked Questions (FAQ) in a form where questions and answers are paired.

Summary of the Invention

Problems to be Solved by the Invention

[0003] When searching for a pair of a question and an answer as in the above-described conventional technology, the administrator of the chatbot system needs to create a large number of pairs of questions and answers, which is time-consuming.

[0004] The disclosed technology has been made in view of the above circumstances, and aims to reduce the labor of creating pairs of questions and answers in a chatbot system.

Means for Solving the Problems

[0005] The disclosed technology includes: a reception unit that receives input question data and an evaluation of answer data output in response to the question data; a first chat processing unit that, upon receiving input question data, outputs answer data generated by a large-scale language model using user-specific information as an information source to a first terminal device; a storage control unit that stores first chat history information, including the question data, the answer data output by the first chat processing unit, and an evaluation of the answer data output by the first chat processing unit, in a first storage device; a prompt generation unit that generates a prompt requesting the large-scale language model to generate a pair of question data and answer data based on the answer data included in the first chat history information, specifically answer data whose evaluation satisfies a first condition, and the corresponding question data; and an output unit that outputs the pair of question data and answer data generated by the large-scale language model to a second terminal device. [Effects of the Invention]

[0006] This reduces the effort required to create question-and-answer pairs in chatbot systems. [Brief explanation of the drawing]

[0007] [Figure 1] This is a diagram showing an example of the system configuration of a chatbot system. [Figure 2] This figure shows an example of the hardware configuration of an information processing device. [Figure 3] This figure shows an example of the hardware configuration of a terminal device. [Figure 4] This is a diagram showing an example of a chat history storage unit. [Figure 5] This figure shows an example of a question and answer memory unit. [Figure 6] This figure shows an example of the first synonym memory section. [Figure 7] This figure shows an example of the second synonym memory section. [Figure 8] This figure shows an example of an analysis result storage unit. [Figure 9]This figure shows an example of an improvement suggestion storage unit. [Figure 10] This figure shows an example of a registration candidate storage unit. [Figure 11] This figure shows an example of a document storage unit. [Figure 12] This figure shows an example of a user information storage unit. [Figure 13] This diagram illustrates the functions of each component in the chatbot system. [Figure 14] This is the first sequence diagram illustrating the operation of the chatbot system. [Figure 15] This figure shows an example of how the chat screen looks. [Figure 16] This is the second sequence diagram illustrating the operation of the chatbot system. [Figure 17] This is the first flowchart explaining the processing performed by the information processing device. [Figure 18] This is the third sequence diagram illustrating the operation of the chatbot system. [Figure 19] This figure shows an example of a prompt generated by an information processing device. [Figure 20] This is the fourth sequence diagram illustrating the operation of the chatbot system. [Figure 21] This is the fifth sequence diagram illustrating the operation of the chatbot system. [Figure 22] This is the sixth sequence diagram illustrating the operation of the chatbot system. [Figure 23] This is the second flowchart explaining the processing performed by the information processing device. [Figure 24] This is the third flowchart explaining the processing performed by the information processing device. [Figure 25] The first figure shows an example of a management screen displayed on a terminal device. [Figure 26] This figure shows an example of a suggestion list screen displayed on a terminal device. [Figure 27] This figure shows an example of a proposal details screen displayed on a terminal device. [Figure 28] This is the seventh sequence diagram for explaining the operation of the chatbot system. [Figure 29] This is a diagram showing an example of an additional screen for question and answer information displayed on the terminal device. [Figure 30] This is a diagram showing an example of a management screen for in-company information displayed on the terminal device.

Mode for Carrying Out the Invention

[0008] Hereinafter, this embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the system configuration of the chatbot system.

[0009] The chatbot system 100 of this embodiment includes an information processing device 200, a first storage device 300, a second storage device 400, an information processing device 200, a terminal device 600, and a terminal device 700, and is an example of an information processing system in which each is connected via a communication network 1.

[0010] In the chatbot system 100, the information processing device 200, the first storage device 300, the second storage device 400, and the information processing device 200 provide a chatbot service to the user of the terminal device 600.

[0011] The first storage device 300 has a chat history storage unit 310, a question and answer storage unit 320, a first synonym storage unit 330, a second synonym storage unit 340, an analysis result storage unit 350, an improvement proposal storage unit 360, and a registration candidate storage unit 365.

[0012] The chat history storage unit 310 stores chat history information in which the questions input from the terminal device 600 are associated with the answers output to the terminal device 600. Note that the chat history information may be managed for each tenant that introduces the chatbot system 100. The tenant in this embodiment may be an organization such as a company, and may be the place of affiliation to which the user who uses the chatbot system 100 belongs.

[0013] The question-answer storage unit 320 stores question-answer information, in which questions and answers are associated. In other words, the question-answer storage unit 320 is an example of a group of question-answer information in which question data representing a question and answer data representing an answer are associated. The question-answer information is information that is stored in the question-answer storage unit 320 in advance, and may be stored for each tenant that installs the chatbot system 100. In the following explanation, the question data included in the question-answer information will be referred to as registered question data, and the answer data included in the question-answer information will be referred to as registered answer data.

[0014] The first synonym memory unit 330 stores first synonym information, which associates keywords (words) contained in the registered question data and registered answer data with synonyms for each keyword. In other words, the first synonym memory unit 330 is an example of a group of synonym information that associates keywords (words) contained in the registered question data and registered answer data with synonyms for each keyword.

[0015] The second synonym memory unit 340 stores second synonym information, which associates various words with their synonyms. The second synonym memory unit 340 may be, for example, a general-purpose database publicly available on the internet, and it stores more synonyms than the first synonym memory unit 330. In other words, the second synonym information stored in the second synonym memory unit 340 contains more information than the first synonym information stored in the first synonym memory unit 330.

[0016] The analysis result storage unit 350 stores analysis result information, which is the result of analyzing the chat history information stored in the chat history storage unit 310 in order to identify question and answer information that needs improvement. The improvement suggestion storage unit 360 stores improvement suggestion information that is referenced when presenting improvement methods for the registered question data and registered answer data stored in the question and answer storage unit 320.

[0017] Furthermore, the first synonym storage unit 330, the second synonym storage unit 340, the analysis result storage unit 350, and the improvement suggestion storage unit 360 of this embodiment may each be stored for each tenant that installs the chatbot system 100.

[0018] The registration candidate storage unit 365 stores combinations of question data and answer data that are candidates for question-answer information. In the following description, the information indicating the combination of question data and answer data stored in the registration candidate storage unit 365 is referred to as registration candidate information. The registration candidate information in this embodiment is information generated by periodic processing that is performed periodically in the chatbot system 100.

[0019] The second storage device 400 includes a document storage unit 410 and a user information storage unit 420. The document storage unit 410 stores user-specific information that is not publicly available.

[0020] A user of chatbot system 100 may be, for example, a tenant, in which case user-specific information will be tenant-specific information. Alternatively, a user of chatbot system 100 may be an individual user, in which case user-specific information will be user-specific information. In the following explanation, user-specific information will be referred to as internal company information, and internal company information will be used as an example of user-specific information.

[0021] Internal information refers to data managed by an organization consisting of multiple members, and may include data stored in a database managed by that organization. The organization may include, for example, a for-profit organization such as a company, or a non-profit organization such as a hospital, school, or cram school. As an example, internal information managed by a hospital may include patient information such as medical records. In this case, the chatbot system 100 can generate response data including a diagnosis based on the patient information. As another example, internal information managed by a school or cram school may include information regarding the learning status or academic performance of users such as students. In this case, the chatbot system 100 can generate response data including appropriate practice problems or video lessons based on the learning status or academic performance information.

[0022] In this embodiment, the document storage unit 410 may store internal company information for each tenant. The user information storage unit 420 stores user information about users belonging to each tenant.

[0023] The information processing device 200 has a large-scale language model storage unit 510 and implements generative AI (Artificial Intelligence). The generative AI generates text data in response to input prompts, such as ChatGPT, and outputs it to the information processing device 200. In this embodiment, the prompts are text data that represent instructions or questions to the generative AI implemented by the information processing device 200. Details of the large-scale language model will be described later.

[0024] Terminal device 600 is mainly used by users of the chatbot system 100. Terminal device 700 is mainly used by administrators who manage the chatbot system 100. In this embodiment, there may be an administrator for the chatbot system 100 for each tenant that has installed the chatbot system 100. In other words, each storage unit of the first storage device 300 and the second storage device 400 is managed for each tenant by the tenant administrator who is a user of terminal device 700.

[0025] Each of the terminal devices 600 and 700 may be, for example, a smartphone or a tablet. Also, although terminal device 600 and terminal device 700 are shown as separate terminal devices in Figure 1, terminal device 600 and terminal device 700 may be the same terminal device. Terminal device 600 is an example of a first terminal device among the multiple terminal devices included in the chatbot system 100, and terminal device 700 is an example of a second terminal device.

[0026] In the chatbot system 100 of this embodiment, when a user inputs a question to the terminal device 600, which is the first terminal device, the terminal device 600 transmits question data indicating the question to the information processing device 200.

[0027] In this embodiment, when a question is input from a terminal device 600, the information processing device 200 refers to the question answer storage unit 320 of the first storage device 300 and outputs the answer to the question to the terminal device 600, which is the first terminal device.

[0028] Furthermore, when a question is input from the terminal device 600, the information processing device 200 of this embodiment generates a prompt requesting the large-scale language model to generate answer data using the company information stored in the document storage unit 410 of the second storage device 400 as the information source, and sends it to the server device 500. The information processing device 200 then outputs the answer data output from the server device 500 to the terminal device 600.

[0029] In the following explanation, data representing the questions entered from the terminal device 600 will be referred to as input question data. Similarly, in the following explanation, the answer data output from the information processing device 200 to the terminal device 600 as the answer to the entered question will be referred to as output answer data.

[0030] Furthermore, the information processing device 200 of this embodiment associates input question data with output answer data to create chat history information, which is then stored in the chat history storage unit 310 of the first storage device 300. In this embodiment, this chat history information may include information indicating the evaluation of the output answer data by the terminal device 600.

[0031] Furthermore, the information processing device 200 in this embodiment generates a prompt requesting the generation of registration candidate information based on output response data that has received high ratings from the user of the terminal device 600 among the output response data included in the chat history information, and sends it to the server device 500. The information processing device 200 then obtains the question data and answer data pairs generated by the large-scale language model from the server device 500 and stores them in the registration candidate storage unit 365 as registration candidate information. The information processing device 200 in this embodiment performs this process periodically as a periodic process.

[0032] Furthermore, the information processing device 200 analyzes the chat history information stored in the chat history storage unit 310, identifies question and answer information that needs improvement and methods for improving the question and answer information from the question and answer information stored in the question and answer storage unit 320, and displays them on the terminal device 700, which acts as a second terminal device.

[0033] Furthermore, in this embodiment, when the terminal device 700, which serves as a second terminal device, is instructed to add new question answer information, the information processing device 200 displays the registration candidate information stored in the registration candidate storage unit 365 on the terminal device 700 as an improvement method.

[0034] In this embodiment, the system identifies question and answer information that needs improvement and methods for improvement, presents them to the administrator of the chatbot system 100, and when the administrator is instructed to add new question and answer information as an improvement method, the system presents the administrator with candidate registration information that includes pairs of question data and answer data generated by the large-scale language model.

[0035] Therefore, in this embodiment, the administrator of the chatbot system 100 does not need to create new question-and-answer information themselves, and the effort required to create question-and-answer pairs can be reduced. Furthermore, in this embodiment, registration candidate information is generated using highly-rated output answer data, so highly accurate question-and-answer information can be generated.

[0036] Furthermore, the information processing device 200 of this embodiment periodically performs a process to identify, as a periodic process, the internal company information stored in the document storage unit 410 of the second storage device 400 that was the source of low-rated output response data. In addition, the information processing device 200 periodically performs a process to identify, as a periodic process, the internal company information stored in the document storage unit 410 of the second storage device 400 that has not been updated for a predetermined period of time.

[0037] The information processing device 200 then notifies the user of the terminal device 700, who is the administrator of the chatbot system 100, that internal company information identified through these periodic processes exists.

[0038] In this embodiment, by performing such periodic processing, the administrator of the chatbot system 100 can be made aware of the existence of unreliable internal company information. Therefore, according to this embodiment, the burden on the administrator of the chatbot system 100 in managing internal company information can be reduced. Furthermore, in this embodiment, the reliability of the internal company information stored in the document storage unit 410 can be increased, and the accuracy of the response data using the internal company information stored in the document storage unit 410 as the information source can be improved.

[0039] In the example shown in Figure 1, the information processing device 200, the first storage device 300, and the second storage device 400 are each represented as separate devices, but the system is not limited to this. The first storage device 300 and the second storage device 400 may be included in the information processing device 200.

[0040] Furthermore, in Figure 1, the information processing device 200 may be implemented by one or more information processing devices, or it may be implemented by multiple information processing devices.

[0041] Next, an example of the hardware configuration of each device in the chatbot system 100 of this embodiment will be described.

[0042] In this embodiment, the information processing device 200, the first storage device 300, the second storage device 400, and the information processing device 200 are all general-purpose computers. Therefore, Figure 2 describes the hardware configuration of the information processing device 200 as an example.

[0043] In this embodiment, terminal devices 600 and 700 are smartphones, and Figure 3 illustrates the hardware configuration of terminal device 600 as an example.

[0044] Figure 2 shows an example of the hardware configuration of an information processing device. The information processing device 200 shown in Figure 2 is built by a computer (information processing device) and includes a CPU 201, ROM 202, RAM 203, HD 204, HDD (Hard Disk Drive) controller 205, display 206, external device connection I / F (Interface) 208, network I / F 209, bus line 210, keyboard 211, pointing device 212, DVD-RW (Digital Versatile Disk Rewritable) drive 214, and media I / F 216.

[0045] Of these, the CPU 201 controls the operation of the entire information processing unit 200. The ROM 202 stores programs used to drive the CPU 201, such as IPL. The RAM 203 is used as the work area for the CPU 501. The HD 204 stores various data such as programs. The HDD controller 205 controls the reading or writing of various data to the HD 204 according to the control of the CPU 501. The display 206 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 208 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 209 is an interface for data communication using a communication network. The bus line 210 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 201 shown in Figure 2.

[0046] The keyboard 211 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 212 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 214 controls the reading or writing of various data to the DVD-RW 213, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media I / F 216 controls the reading or writing (storage) of data to the recording medium 215, such as flash memory.

[0047] Figure 3 shows an example of the hardware configuration of a terminal device. The terminal device 700 in this embodiment includes a CPU 701, ROM 702, RAM 703, EEPROM 704, CMOS sensor 705, image sensor I / F 706, acceleration / direction sensor 707, media I / F 709, and GPS receiver 711.

[0048] Of these, the CPU 701 is an arithmetic processing unit that controls the operation of the entire terminal device 700. The ROM 702 stores programs used to drive the CPU 701, such as the CPU 701 and the IPL. The RAM 703 is used as the work area for the CPU 701. The EEPROM 704 reads or writes various data, such as smartphone programs, according to the control of the CPU 701. The ROM 702, RAM 703, and EEPROM 704 are examples of the storage devices of the terminal device 700.

[0049] The CMOS (Complementary Metal Oxide Semiconductor) sensor 705 is a type of built-in imaging device that captures an image of a subject (mainly a self-portrait) and obtains image data according to the control of the CPU 701. Note that an imaging device such as a CCD (Charge Coupled Device) sensor may also be used instead of a CMOS sensor.

[0050] The image sensor interface 706 is a circuit that controls the driving of the CMOS sensor 705. The acceleration / direction sensor 707 is a variety of sensors, such as an electronic magnetic compass, gyrocompass, and acceleration sensor that detect the Earth's magnetic field. The media interface 709 controls the reading or writing (storage) of data to or from the recording medium 708, such as flash memory. The GPS receiver 711 receives GPS signals from GPS satellites.

[0051] Furthermore, the terminal device 700 includes a long-range communication circuit 712, an antenna 712a for the long-range communication circuit 712, a CMOS sensor 713, an image sensor interface 714, a microphone 715, a speaker 716, an audio input / output interface 717, a display 718, an external device connection interface 719, a short-range communication circuit 720, an antenna 720a for the short-range communication circuit 720, and a touch panel 721.

[0052] Of these, the long-distance communication circuit 712 is a circuit that communicates with other devices via a communication network. The CMOS sensor 713 is a type of built-in imaging means that captures an image of a subject and obtains image data according to the control of the CPU 701. The image sensor interface 714 is a circuit that controls the driving of the CMOS sensor 713. The microphone 715 is a built-in circuit that converts sound into electrical signals. The speaker 716 is a built-in circuit that converts electrical signals into physical vibrations to produce sounds such as music and speech. The sound input / output interface 717 is a circuit that processes the input and output of sound signals between the microphone 715 and the speaker 716 according to the control of the CPU 701.

[0053] The display 718 is a type of display means, such as a liquid crystal or organic EL (Electro-Luminescence), that displays images of the subject and various icons. The external device connection I / F 719 is an interface for connecting various external devices. The short-range communication circuit 720 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 721 is a type of input means that allows the user to operate the terminal device 700 by pressing the display 718. The display 718 is an example of a display unit of the terminal device 700.

[0054] Note that the hardware configuration shown in Figure 3 is just one example of terminal device 700, and the hardware configuration of terminal device 700 is not limited to this. The hardware configuration of terminal device 700 may also be the configuration shown in Figure 2. Furthermore, the hardware configuration of terminal device 600 may be the same as that of terminal device 700, so its explanation is omitted.

[0055] Next, with reference to Figures 4 to 10, the storage units of the first storage device 300 will be described. Figure 4 shows an example of a chat history storage unit.

[0056] The chat history storage unit 310 of this embodiment includes tenant ID, conversation ID, conversation date and time, input question data, output answer data, category, question answer ID, user rating, and answer source information as information items, with the item "conversation ID" being associated with the other items.

[0057] The value of the item "Tenant ID" is identification information for identifying the tenant that manages the chat history storage unit 310. The value of the item "Conversation ID" indicates identification information that identifies the chat history information. The value of the item "Conversation Date and Time" indicates the date and time the chat took place. The value of the item "Input Question Data" indicates text data (input question data) showing the question entered from the terminal device 600. The value of the item "Output Response Data" is text data (output response data) showing the response output by the information processing device 200 to the terminal device 600.

[0058] The value of the item "Category" indicates the category associated with the registered answer data output as output answer data in the question answer storage unit 320. The value of the item "Question Answer ID" indicates the question answer ID associated with the output answer data in the question answer storage unit 320. The value of the item "User Evaluation" indicates the evaluation entered in the terminal device 600 for the answer shown in the output answer data.

[0059] The value of the item "Answer Source" indicates the source of the output answer data. Specifically, for example, if the output answer data is text data based on question answer information stored in the question answer storage unit 320, the value of the item "Answer Source" will be "Question Answer Storage Unit". If the output answer data is internal company information stored in the document storage unit 410, the value of the item "Answer Source" will be information that identifies the internal company information used as the source of the output answer data. Note that the information that identifies the internal company information may be a link or the like indicating the location where the relevant internal company information is stored.

[0060] Figure 5 shows an example of a question and answer storage unit. The question and answer storage unit 320 of this embodiment includes tenant ID, question and answer ID, update date and time, category, registered question data, registered answer data, and question paraphrasing as information items, with the item "question and answer ID" being associated with the other items. The question and answer information of this embodiment is information in which the value of the item "question and answer ID" is associated with the values ​​of the other items.

[0061] The value of the "Tenant ID" field is identification information used to identify the tenant managing the question answer storage unit 320. The value of the "Question Answer ID" field indicates identification information that identifies the question answer information. The value of the "Update Date and Time" field indicates the date and time when the question answer information was updated. The value of the "Category" field indicates the category of the question indicated by the registered question data. The value of the "Registered Question Data" field is text data (registered question data) indicating the registered question, and the value of the "Registered Answer Data" field is text data (registered answer data) indicating the registered answer. The "Question Paraphrasing" field indicates an alternative pattern of the registered question data.

[0062] Figure 6 shows an example of the first synonym storage unit. The first synonym information stored in the first synonym storage unit 330 of this embodiment includes tenant ID, synonym ID, update date and time, keyword, and synonym as information items, with the item "synonym ID" being associated with the other items. The first synonym information of this embodiment is information in which the value of the item "synonym ID" is associated with the values ​​of the other items. The first synonym information may be stored in the first synonym storage unit 330 in advance.

[0063] The value of the item "Tenant ID" is identification information for identifying the tenant that manages the first synonym storage unit 330. The value of the item "Synonym ID" is identification information for identifying the first synonym information. The value of the item "Update Date and Time" indicates the date and time when the first synonym information was updated. The item "Keyword" is a keyword, and the value of the item "Synonym" is a synonym for the keyword.

[0064] Figure 7 shows an example of a second synonym storage unit. The second synonym information stored in the second synonym storage unit 340 of this embodiment includes tenant ID, synonym ID, update date and time, keyword, and synonym as information items, with the item "synonym ID" being associated with the other items. The second synonym information of this embodiment is information in which the value of the item "synonym ID" is associated with the values ​​of the other items. The second synonym information may be stored in the second synonym storage unit 340 in advance.

[0065] The value of the item "Tenant ID" is identification information for identifying the tenant that manages the second synonym storage unit 340. The value of the item "Synonym ID" is identification information for identifying the second synonym information. The value of the item "Update Date and Time" indicates the date and time when the first synonym information was updated. The item "Keyword" is various keywords, and the value of the item "Synonym" is a synonym of the keyword. In this embodiment, the second synonym storage unit 340 is managed for each tenant, but it is not limited to this. The second synonym storage unit 340 may be shared by multiple tenants.

[0066] Figure 8 shows an example of an analysis result storage unit. The analysis result information stored in the analysis result storage unit 350 includes the following information items: tenant ID, conversation ID, input question data, analysis result, improvement method type, appropriate question answer ID, appropriate question data, reference question answer ID, reference question data, user evaluation, and group ID. The items "tenant ID" and "conversation ID" are associated with the other items. In this embodiment, the analysis result information in the analysis result storage unit 350 is information in which the value of the item "conversation ID" is associated with the other items.

[0067] Furthermore, the analysis results information of this embodiment shows the results of analyzing chat history information where the value of the item "Answer Source" in the chat history information is set to "Question and Answer Storage Unit". In other words, the analysis results information of this embodiment shows the results of analyzing chat history information when the registered answer data stored in the question and answer storage unit 320 is output to the terminal device 600 as output answer data.

[0068] The value of the item "Tenant ID" is identification information used to identify the tenant managing the analysis result storage unit 350. The value of the item "Conversation ID" indicates identification information that identifies the chat history information to be analyzed. The value of the item "Input Question Data" is the value of the item "Input Question Data" in the chat history information to be analyzed. The value of the item "Analysis Result" indicates the analysis result of the chat history information to be analyzed. Details of the analysis of chat history information will be described later.

[0069] The "Improvement Method Type" item indicates the type of improvement method used for the question-and-answer information referenced in the chat history data analyzed. In other words, the "Improvement Method Type" item indicates how the question-and-answer information referenced in the chat history data analyzed was improved.

[0070] The value of the item "Appropriate Question Answer ID" indicates the identification information of the question answer information to be referenced for the input question data. The value of the item "Appropriate Question Answer ID" is the question answer ID of the question answer information identified by the information processing device 200 by referring to the second synonym storage unit 340.

[0071] The value of the item "Appropriate Question Data" is the value of the item "Input Question Data" included in the question and answer information identified by the value of the item "Appropriate Question and Answer ID".

[0072] The value of the item "Reference Question Answer ID" is the question answer ID of the question answer information that the information processing device 200 referenced when identifying the output answer data for the input question data. In other words, the value of the item "Reference Question Answer ID" is the question answer ID of the question answer information that the information processing device 200 referenced the first synonym storage unit 330 to identify, and is the question answer ID that identifies the first question answer information described later.

[0073] The value of the item "Reference Question Data" is the value of the item "Input Question Data" included in the question and answer information identified by the value of the item "Reference Question Answer ID". The value of the item "User Rating" is the user rating associated with the value of the item "Conversation ID" in the chat history storage unit 310.

[0074] The value of the "Group ID" field is an identifier assigned when analysis results information whose similarity to the input question data exceeds a certain threshold is grouped together. In other words, the value of the "Group ID" field is an identifier that identifies groups of analysis results information whose text data, which is the input question data, is similar.

[0075] Figure 9 shows an example of an improvement suggestion storage unit. The improvement suggestion information stored in the improvement suggestion storage unit 360 is associated with the following information items: tenant ID, analysis result, and improvement method type.

[0076] The value of the item "Tenant ID" is identification information for identifying the tenant that manages the improvement suggestion storage unit 360. The value of the item "Analysis Result" indicates the pattern of the analysis result of the chat history information being analyzed. The item "Improvement Method Type" indicates the improvement method for the question answer information associated with the analysis result pattern. In this embodiment, the improvement suggestion storage unit 360 is managed per tenant, but is not limited to this. The improvement suggestion storage unit 360 may be shared by multiple tenants.

[0077] Figure 10 shows an example of a registration candidate storage unit. The registration candidate information stored in the registration candidate storage unit 365 includes information items such as tenant ID, registration candidate ID, question data, answer data, and category, with the item "tenant ID" being associated with the other items.

[0078] The value of the item "Registration Candidate ID" is identification information used to identify registration candidate information generated in the large-scale language model. In other words, the value of the item "Registration Candidate ID" is information that identifies the pair of question data and answer data generated by the large-scale language model. The registration candidate ID may be assigned when the registration candidate information is generated, or when the registration candidate information is stored in the registration candidate storage unit 365. The value of the item "Category" indicates the category associated with the pair of question data and answer data.

[0079] Furthermore, the registration candidate information may include information such as the date and time the registration candidate information was generated. The information items included in the registration candidate information are not limited to the example shown in Figure 10.

[0080] Next, with reference to Figures 11 and 12, the storage units of the second storage device 400 will be described.

[0081] Figure 11 shows an example of a document storage unit. In the document storage unit 410 of this embodiment, internal company information is stored for each tenant and may be managed on a tenant-by-tenant basis. Figure 11 shows an example of internal company information managed in a tenant identified by tenant ID "11".

[0082] In this embodiment, the document storage unit 410 divides the company information into predetermined units and stores each divided piece of information separately. In the following description, each divided piece of company information is referred to as a chunk. In the document storage unit 410, chunks may be stored in association with identification information that identifies the chunk. In this embodiment, information indicating the date and time the company information in the document storage unit 410 was updated may be associated with information that identifies the company information.

[0083] In Figure 11, the file name of the internal company information, which serves as information to identify the internal company information, is associated with the update date and time of the internal company information, the chunk ID that identifies the chunk, and the chunk itself. In the example in Figure 11, one page of internal company information within the internal company document indicated by the internal company information is considered a chunk, and the page number is used as the chunk ID that identifies the chunk.

[0084] Furthermore, a chunk is not limited to a single page of internal company information; for example, it could be a paragraph within an internal document, or even a chapter.

[0085] Figure 12 shows an example of a user information storage unit. The user information stored in the user information storage unit 420 of this embodiment may be managed separately for each tenant, as it contains company information for each tenant.

[0086] The user information in this embodiment includes the following information items: tenant ID, user ID, username, email address, etc. The value of the "tenant ID" item is identification information that identifies the tenant to which the user identified by the user ID belongs. The value of the "user ID" item is identification information that identifies the user of the terminal device 600. The "user name" item indicates the name of the user identified by the user ID, and the value of the "email address" item indicates the email address of the user identified by the user ID. In this embodiment, the user information storage unit 420 is stored in the second storage device 400, but the user information storage unit 420 does not have to be provided in the second storage device 400; it is sufficient if only the document storage unit 410 is provided.

[0087] Next, with reference to Figure 13, the functional configuration of each device included in the chatbot system 100 of this embodiment will be described. Figure 13 is a diagram illustrating the functions of each device in the chatbot system.

[0088] First, the functional configuration of the information processing device 200 will be described. The information processing device 200 in this embodiment includes an information acquisition unit 250, a reception unit 251, a first chat processing unit 252, a second chat processing unit 253, a storage control unit 254, an analysis unit 255, a prompt generation unit 256, an output unit 257, a registration candidate generation unit 258, an evaluation and determination unit 259, an update timing determination unit 260, and a screen generation unit 261.

[0089] The information acquisition unit 250 acquires various types of information from the first storage device 300, the second storage device 400, or the information processing device 200, according to the processing performed by the device.

[0090] The reception unit 251 receives various inputs to the information processing device 200. Specifically, the reception unit 251 receives input of evaluations of output answer data for questions indicated by input question data from the terminal device 600. The reception unit 251 also receives various inputs from the terminal device 700 according to the operation.

[0091] When the first chat processing unit 252 receives input question data from the terminal device 600, it causes the large-scale language model to generate response data using the company information stored in the document storage unit 410 of the second storage device 400 as the source of information. The first chat processing unit 252 then outputs the response data generated by the large-scale language model as output response data to the terminal device 600.

[0092] Furthermore, the first chat processing unit 252 associates the input question data received from the terminal device 600 with the output response data output to the terminal device 600 to create chat history information, which is then stored in the chat history storage unit 310 of the first storage device 300. At this time, the first chat processing unit 252 may include information identifying the source of the output response data in the chat history information. The chat history information stored in the chat history storage unit 310 by the first chat processing unit 252 is sometimes referred to as the first chat history information. Details of the processing of the first chat processing unit 252 will be described later.

[0093] When the second chat processing unit 253 receives input question data from the terminal device 600, it refers to the question-answer storage unit 320 and the first synonym storage unit 330 and outputs the output answer data to the terminal device 600. The second chat processing unit 253 also associates the input question data received from the terminal device 600 with the output answer data output to the terminal device 600 and stores it in the chat history storage unit 310. At this time, the second chat processing unit 253 may include information in the chat history information indicating that the source of the output answer data is the question-answer information stored in the question-answer storage unit 320. The chat history information stored in the chat history storage unit 310 by the second chat processing unit 253 is sometimes referred to as the second chat history information. Details of the processing of the second chat processing unit 253 will be described later.

[0094] In this embodiment, the input question data and output answer data may be text data or audio data, respectively.

[0095] The storage control unit 254 controls the storage of information received by the reception unit 251. Specifically, the storage control unit 254 stores information indicating the evaluation of the output response data received by the reception unit 251 from the terminal device 600 in the chat history storage unit 310 as part of the chat history information. The storage control unit 254 also stores registration candidate information, which will be described later, in the registration candidate storage unit 365, and stores the registration candidate information in the question answer storage unit 320 as part of the question answer information in response to the operation of the terminal device 700.

[0096] The analysis unit 255 analyzes whether the content of the output answer data corresponding to the input question data stored in the question answer information stored in the question answer information storage unit 320 is appropriate, based on the second chat history information stored in the chat history storage unit 310. Details of the processing of the analysis unit 255 will be described later.

[0097] The prompt generation unit 256 generates a prompt to be sent to the server device 500. Specifically, the prompt generation unit 256 generates a prompt that will be sent to the server device 500 during processing by the first chat processing unit 252.

[0098] Furthermore, the prompt generation unit 256 generates a prompt requesting the generation of a new set of question data and answer data based on the output response data with a high rating among the output response data included in the first chat history information, and the input question data associated with this output response data.

[0099] The output unit 257 outputs various types of information to the outside of the information processing device 200. Specifically, the output unit 257 outputs prompts generated by the prompt generation unit 256 to the server device 500. The output unit 257 also outputs screen data generated by the screen generation unit 261 to the terminal device 600 and terminal device 700.

[0100] The registration candidate generation unit 258 uses the first chat history information from the chat history information stored in the chat history storage unit 310 to generate registration candidate information that will be used as candidate question answer information to be stored in the question answer storage unit 320. The registration candidate information generated by the registration candidate generation unit 258 is stored in the registration candidate storage unit 365 by the storage control unit 254.

[0101] The evaluation determination unit 259 determines whether the output response data included in the first and second chat history information stored in the chat history storage unit 310 is output response data with a high evaluation or output response data with a low evaluation. Output response data with a high evaluation is an example of output response data that satisfies the first condition. Output response data with a low evaluation is an example of output response data that satisfies the second condition.

[0102] Output response data that satisfies the first condition is output response data associated with chat history information that indicates the user was satisfied with the content of the output response data. More specifically, output response data with a high rating may be output response data in which the value of the "User Rating" item in the chat history information is set to "Satisfied".

[0103] Output response data that satisfies the second condition is output response data associated with information in the chat history that indicates the user is dissatisfied with the content of the output response data. More specifically, output response data with a low rating may be output response data in which the value of the "User Rating" item in the chat history information is set to "Dissatisfied".

[0104] Furthermore, the evaluation determination unit 259 determines the evaluation of the internal company information that served as the source of the output response data, based on the evaluation of the output response data. Specifically, for example, if a predetermined number of output response data from which a certain internal company information was used as the source have low evaluations, the evaluation determination unit 259 determines that the evaluation of this internal company information is low. An example of internal company information that has a low evaluation is internal company information that satisfies the third condition.

[0105] The update timing determination unit 260 determines, for each piece of company information stored in the document storage unit 410, whether a predetermined period has elapsed since the last update of that piece of company information. The predetermined period may be set in advance.

[0106] The screen generation unit 261 generates screen data for displaying screens on terminal devices 600 and 700. Specifically, the screen generation unit 261 generates screen data for displaying a chat screen on terminal device 600. In addition, the screen generation unit 261 generates screen data for displaying on terminal device 700 screens for suggesting ways to improve the question and answer information stored in the question and answer storage unit 320, and screens for managing company information stored in the document storage unit 410.

[0107] Next, the functions of the first storage device 300 will be described. The first storage device 300 includes a chat history storage unit 310, a question and answer storage unit 320, a first synonym storage unit 330, a second synonym storage unit 340, an analysis result storage unit 350, an improvement suggestion storage unit 360, a registration candidate storage unit 365, a search unit 370, and a storage unit 380. The search unit 370 searches the chat history storage unit 310 and the question and answer storage unit 320 in response to a search request from the information processing device 200, and transmits the search results to the information processing device 200. When input question data is input and output answer data is output in the information processing device 200, the storage unit 380 stores the chat history information, including the input question data and output answer data, in the chat history storage unit 310.

[0108] Next, the functions of the second storage device 400 will be described. The second storage device 400 includes a document storage unit 410, a user information storage unit 420, a search unit 430, and a storage unit 440. The search unit 430 searches the document storage unit 410 in response to a search request from the information processing device 200 and transmits the search results to the information processing device 200. The storage unit 440 stores new company information in the document storage unit 410 when an operation to upload company information is performed by the administrator of the chatbot system 100 or the like.

[0109] Next, the functions of the server device 500 will be described. The server device 500 has a large-scale language model storage unit 510, an input unit 520, and an output unit 530. The large-scale language model storage unit 510 stores a large-scale language model 511 that realizes generative AI.

[0110] The large-scale language model 511 of this embodiment is a computer language model that is generated by performing a training process using a vast amount of unlabeled text as training data and consists of an artificial neural network with many parameters. The large-scale language model 511 can capture much of the syntax and meaning of human language by being sufficiently trained with context-learning techniques such as next sentence prediction, which understands the context by determining whether sentence 1 and sentence 2 are consecutive, and a masked language model, which understands the context by masking words in a sentence and predicting the masked words from the words before and after them.

[0111] The input unit 520 inputs prompts sent from the information processing device 200 to the large-scale language model 511. The output unit 530 outputs (sends) the output data output from the large-scale language model 511 to the information processing device 200. In other words, the output unit 530 sends the output response data output from the large-scale language model 511 to the information processing device 200.

[0112] Next, the functions of terminal devices 600 and 700 will be described. Each of terminal devices 600 and 700 includes a communication control unit 630, 730, an input receiving unit 640, 740, and a display control unit 650, 750.

[0113] Communication control units 630 and 730 control communication between terminal devices 600 and 700, respectively, and the information processing device 200. Input receiving units 640 and 740 receive input to terminal devices 600 and 700. Display control units 650 and 750, which may be implemented by a browser or the like, control the display on terminal devices 600 and 700.

[0114] Next, we will explain the operation of the chatbot system 100 when the first chat processing unit 252 outputs the response data to the terminal device 600, and the operation when the second chat processing unit 253 outputs the response data to the terminal device 600.

[0115] First, we will explain the operation of the chatbot system 100 when the first chat processing unit 252 outputs the response data to the terminal device 600.

[0116] Figure 14 is a first sequence diagram showing the operation of the chatbot system. In the chatbot system 100, when the terminal device 600 receives a question input via the input receiving unit 640, it transmits the question data to the information processing device 200 (step S1401). The question data transmitted here may include the tenant ID of the tenant to which the user of the terminal device 600 belongs.

[0117] When the information processing device 200 receives the input of question data from the reception unit 251, the first chat processing unit 252 sends a search request including the tenant ID to the first storage device 300 (step S1402).

[0118] When the first storage device 300 receives this search request, the search unit 370 searches the chat history storage unit 310 (step S1403).

[0119] The processing of the search unit 370 will now be explained. The search unit 370 identifies chat history information containing the tenant ID included in the search request from the chat history information stored in the chat history storage unit 310, and obtains chat history information related to the question and content indicated by the question data as search results from the identified chat history information.

[0120] Specifically, the search unit 370 identifies chat history information including the tenant ID included in the search request, and from the identified chat history information, it obtains the chat history information that continued up until just before the question data was entered in step S1401.

[0121] The chat history information that continued until just before the question data was entered in step S1401 is chat history information in which the time between the time the question data was entered and the time the output response data was output immediately before the question data was entered is within a certain period of time. In this embodiment, if the next question data is entered within a certain period of time after the output response data has been output, the series of questions and answers are assumed to be responses concerning a common inquiry.

[0122] In this embodiment, by acquiring chat history information in this way, it is possible to obtain the chat history information related to the question indicated by the question data.

[0123] When the search unit 370 of the first storage device 300 searches the chat history storage unit 310, it sends the search results to the information processing device 200, and the information acquisition unit 250 of the information processing device 200 acquires the chat history information (step S1404).

[0124] Here, the first chat processing unit 252 of the information processing device 200 performs the following steps S1405 to S1407 when chat history information is obtained from the first storage device 300 as a search result.

[0125] In addition, chat history information is acquired when, before the question data entered in step S1401 is entered, question data related to the question indicated by the question data entered in step S1401 is entered, and corresponding output answer data is output. In this case, the chat history storage unit 310 stores chat history information including this question data and output answer data.

[0126] Furthermore, if the first chat processing unit 252 of the information processing device 200 does not obtain chat history information from the first storage device 300 as a search result, it proceeds to step S1408. The case in which chat history information is not obtained is when the question data entered in step S1401 indicates the first question in the inquiry.

[0127] The first chat processing unit 252 of the information processing device 200 causes the prompt generation unit 256 to generate a prompt requesting the generation of a query for searching the document storage unit 410 (step S1405). At this time, the prompt generation unit 256 generates a prompt that includes the question data entered in step S1401 and the acquired chat history information, and requests the generation of a query based on the question data and the chat history information.

[0128] Next, the information processing device 200 sends a prompt to the server device 500 via its output unit 257, requesting the generation of a query (step S1406).

[0129] The server device 500 inputs a prompt to the large-scale language model 511 via the input unit 520, and sends the output data output from the large-scale language model 511 as a query to the information processing device 200 via the output unit 530 (step S1407).

[0130] The information processing device 200, via its first chat processing unit 252, sends a request to the second storage device 400 to search for internal company information that can serve as a source of information when generating output response data for question data (step S1408).

[0131] Specifically, if the first chat processing unit 252 has obtained a query in step S1407, it sends a search request based on the query to the second storage device 400. If the first chat processing unit 252 has not obtained a query, that is, if it has not obtained chat history information in step S1404, it sends a search request including the question data entered in step S1401 to the second storage device 400.

[0132] When the second storage device 400 receives a search request, the search unit 430 searches the document storage unit 410 (step S1409) and transmits the company information of the search results to the information processing device 200 (step S1410).

[0133] When the information acquisition unit 250 of the information processing device 200 acquires the company information of the search results, the prompt generation unit 256 generates a prompt requesting the generation of an answer to the question data using the company information of the search results as the information source (step S1411), and the output unit 257 transmits the generated prompt to the server device 500 (step S1412).

[0134] When the server device 500 receives this prompt, the input unit 520 inputs the prompt to the large-scale language model 511, and the output unit 530 transmits the output data output from the large-scale language model 511 as response output data to the information processing device 200 (step S1413).

[0135] The first chat processing unit 252 of the information processing device 200, upon acquiring output response data, outputs the acquired output response data to the terminal device 600 (step S1414). The terminal device 600, via the display control unit 650, displays the output response data on the display 618 (step S1415). At this time, the display control unit 650 displays the output response data on the display 618 in association with the question data.

[0136] Furthermore, the storage control unit 254 of the information processing device 200 sends a request to the first storage device 300 to save chat history information, which includes the question data received in step S1401 and the output response data output to the terminal device 600 in step S1412 (step S1416). Here, the request to save chat history information includes the tenant ID of the tenant to which the user of the terminal device 600 belongs.

[0137] In the first storage device 300, the storage unit 380 stores the new chat history information in the chat history storage unit 310 (step S1417). Specifically, the storage unit 380 adds a new conversation ID and date and time to the question data, output answer data, and tenant ID acquired from the information processing device 200, and stores them in the chat history storage unit 310 as chat history information. In this embodiment, the chat history storage unit 310 is updated in this way.

[0138] Furthermore, when the terminal device 600 receives input for an evaluation of the output response data displayed on the display 618 (step S1418), it transmits information indicating the result of the input evaluation to the information processing device 200 (step S1419).

[0139] When the information processing device 200 receives information indicating the result of the evaluation of the output response data from the receiving unit 251, the storage control unit 254 sends a storage request to the first storage device 300, along with the information indicating the evaluation result, to include the information indicating the evaluation result as part of the chat history information including the output response data (step S1420). When the first storage device 300 receives the storage request, it stores the information indicating the result of the evaluation of the output response data in the chat history storage unit 310 (step S1421).

[0140] The evaluation of output response data in this embodiment will be described below with reference to Figure 15. Figure 15 is a diagram showing an example of the chat screen display.

[0141] The chat screen 110 shown in Figure 15 is an example of a screen displayed on the terminal device 600 in step S1415 of Figure 14.

[0142] The chat screen 110 includes display areas 111, 112, and 113. Display area 111 displays input question data entered by the user of the terminal device 600. Display area 112 includes display areas 112a and 112b. Display area 112a displays output response data acquired by the first chat processing unit 252. Display area 112b displays information that identifies the internal company information that serves as the source of the output response data. Specifically, display area 112b may display a link indicating the storage location of the internal company information that serves as the source of the output response data displayed in display area 112a.

[0143] Display area 113 displays icon images 113a and 113b for inputting an evaluation of the output response data displayed in display area 112a. In this embodiment, if icon image 113a is selected, it means that the output response data has been given a high evaluation (satisfied), and if icon image 113b is selected, it means that the output response data has been given a low evaluation (dissatisfied).

[0144] Next, we will describe the operation of the chatbot system 100 when the second chat processing unit 253 outputs the response data to the terminal device 600. Figure 16 is a second sequence diagram showing the operation of the chatbot system.

[0145] In the chatbot system 100 of this embodiment, when the input receiving unit 640 of the terminal device 600 receives a question input from the user on the chat screen displayed on the terminal device 600 (step S1601), it transmits input question data indicating the question to the information processing device 200 (step S1602).

[0146] When the information processing device 200 receives input question data, the second chat processing unit 253 refers to the question answer storage unit 320 and the first synonym storage unit 330 to identify registered answer data corresponding to the input question data (step S1603). Details of the process in step S1603 will be described later.

[0147] Next, the second chat processing unit 253 of the information processing device 200 transmits the identified registered response data to the terminal device 600 as output response data (step S1604). Subsequently, the information processing device 200, via its storage control unit 254, sends a request to the first storage device 300 to store chat history information, which associates the input question data with the output response data output to the terminal device 600 (step S1606), and stores the chat history information in the chat history storage unit 310 (step S1607).

[0148] When terminal device 600 receives output response data, it displays the received output response data on the chat screen (step S1608). Subsequently, terminal device 600 receives input from the user regarding their evaluation of the content of the output response data (step S1609) and transmits information indicating the user's evaluation result to information processing device 200 (step S1610).

[0149] When the information processing device 200 receives information indicating the result of the evaluation of the output response data from the receiving unit 251, the storage control unit 254 sends a storage request to the first storage device 300, along with the information indicating the evaluation result, to include the information indicating the evaluation result as part of the chat history information including the output response data (step S1611). When the first storage device 300 receives the storage request, it stores the information indicating the result of the evaluation of the output response data in the chat history storage unit 310 (step S1612).

[0150] Next, with reference to Figure 17, the details of the process in step S1603 of Figure 16 are shown. Figure 17 is a first flowchart illustrating the process of the information processing device.

[0151] In this embodiment, the information processing device 200 acquires the input question data corresponding to the question text entered from the terminal device 600 by the second chat processing unit 253 (step S1701).

[0152] Next, the second chat processing unit 253 executes the processing from step S1703 onward for each question and answer information stored in the question and answer storage unit 320 (step S1702).

[0153] The second chat processing unit 253 determines whether the first synonyms corresponding to the registered question data and registered answer data included in the question and answer information are set in the first synonym storage unit 330 (step S1703).

[0154] Keywords included in the registered question data and registered answer data are, for example, extracted keywords (strings) obtained by performing morphological analysis or similar methods on each of the text data that constitute the registered question data.

[0155] In this embodiment, if the first synonym storage unit 330 stores first synonym information in which the value of the item "keyword" matches the extracted keyword, then the first synonym corresponding to the keyword included in the registered question data and registered answer data is set in the first synonym storage unit 330.

[0156] In step S1703, if a first synonym is set, the second chat processing unit 253 calculates the similarity between the registered question data and registered answer data and the input question data. Furthermore, the second chat processing unit 253 replaces the extracted keyword contained in the input question data with the first synonym corresponding to this extracted keyword and calculates the similarity between the registered question data and the input question data (step S1704).

[0157] In this embodiment, instead of calculating the first similarity between the question data and answer data after replacing the keywords in the input question data with corresponding first synonyms, the results of morphological analysis of the input question data may be added to a list of words.

[0158] Furthermore, in step S1704, if there are multiple first synonyms associated with the extracted keyword, the second chat processing unit 253 performs the same processing for each first synonym.

[0159] In step S1703, if the first synonym is not set, the second chat processing unit 253 calculates the similarity between the registered question data and registered answer data and the input question data (step S1705).

[0160] In the following description of this embodiment, the similarity calculated by the second chat processing unit 253 will be referred to as the first similarity.

[0161] The second chat processing unit 253 performs the processing from step S1703 to step S1705 for all question answer information stored in the question answer storage unit 320 (step S1706), and then determines whether or not there is any question answer information whose first similarity calculated in the processing up to step S1706 exceeds the first similarity threshold (step S1707).

[0162] In step S1707, if no corresponding question-answer information exists, the second chat processing unit 253 identifies the text data "No answer found" as the output answer data for the input question data (step S1708) and terminates the process.

[0163] In step S1707, if relevant question and answer information exists, the second chat processing unit 253 identifies the question and answer information with the highest similarity to the first, identifies the registered question data included in the identified question and answer information as the output answer data for the input question data (step S1709), and terminates the process.

[0164] In this embodiment, the second chat processing unit 253 may select multiple question-answer information as answers. Specifically, if there is question-answer information with a first similarity score equal to or greater than the first similarity threshold, the second chat processing unit 253 identifies the relevant question-answer information as the answer. Furthermore, if there is question-answer information with a similarity score less than the first similarity threshold and equal to or less than the second similarity threshold, the second chat processing unit 253 identifies that question-answer information as the answer. Furthermore, if there are multiple question-answer information items with a similarity score equal to or greater than the second similarity threshold, the second chat processing unit 253 identifies the relevant question-answer information as the answer. Details of the second similarity score will be described later.

[0165] In the following description of this embodiment, the question and answer information identified by the second chat processing unit 253 will be referred to as the first question and answer information.

[0166] Next, we will describe the periodic processing performed in the chatbot system 100 of this embodiment. First, with reference to Figures 18 and 19, we will describe the process of generating registration candidate information and storing it in the registration candidate storage unit 365. The process shown in Figure 18 is one of the periodic processes that are performed periodically in the chatbot system 100.

[0167] Figure 18 is a third sequence diagram illustrating the operation of the chatbot system. In the chatbot system 100 of this embodiment, the process shown in Figure 18 is executed periodically. More specifically, the chatbot system 100 may, for example, execute the process shown in Figure 18 every 24 hours.

[0168] In the chatbot system 100, the information processing device 200, via the registration candidate generation unit 258, requests the chat history storage unit 310 of the first storage device 300 to acquire chat history information (step S1801).

[0169] This acquisition request is an acquisition request that requests the acquisition of the first chat history information from the chat history information stored in the chat history storage unit 310.

[0170] The first storage device 300 receives this acquisition request, acquires the first chat history information from the chat history information stored in the chat history storage unit 310 (step S1802), and transmits it to the information processing device 200 (step S1803).

[0171] When the information processing device 200 acquires first chat history information using the information acquisition unit 250, the registration candidate generation unit 258 extracts chat history information from the first chat history information that has a high evaluation for the output response data (step S1804).

[0172] In other words, the registration candidate generation unit 258 extracts chat history information from the acquired first chat history information that contains output response data that satisfies the first condition. Specifically, the registration candidate generation unit 258 extracts chat history information from the acquired first chat history information in which the value of the item "User Rating" is "Satisfied".

[0173] Next, the registration candidate generation unit 258 causes the prompt generation unit 256 to generate a prompt requesting the generation of a pair of question data and answer data based on the output answer data extracted in step S1804 and the input question data associated with this output answer data (step S1805).

[0174] Specifically, when the prompt generation unit 256 extracts output response data with a high rating, it refers to the chat history information containing this output response data and identifies the input question data associated with this output response data in the chat history information. Then, the prompt generation unit 256 generates a prompt that includes the output response data with a high rating and the input question data.

[0175] In other words, the prompt generation unit 256 generates a prompt that requests the generation of a new set of question data and answer data based on a highly rated set of output answer data and input question data in the first chat history information. Details of the prompt generated in step S1805 will be described later.

[0176] Next, the information processing device 200 sends the generated prompt to the server device 500 via the output unit 257 (step S1806).

[0177] When the server device 500 receives this prompt, the input unit 520 inputs the prompt to the large-scale language model 511, and the output unit 530 transmits the question data and answer data set output from the large-scale language model 511 to the information processing device 200 (step S1807).

[0178] When the registration candidate generation unit 258 of the information processing device 200 receives a pair of question data and answer data from the server device 500, it generates registration candidate information by assigning a registration candidate ID to the acquired pair of question data and answer data (step S1808). Subsequently, the information processing device 200, via the storage control unit 254, sends a request to the first storage device 300 to store the registration candidate information along with the registration candidate information (step S1809).

[0179] The first storage device 300 receives a storage request and stores the registration candidate information in the registration candidate storage unit 365 (step S1810).

[0180] In this embodiment, registration candidate information is generated, which includes pairs of question data and answer data to be stored in the question answer storage unit 320 as question answer information. The registration candidate information stored in the registration candidate storage unit 365 is displayed on a screen for suggesting methods to improve the question answer information. Details of the display example of the registration candidate information will be described later.

[0181] Now, with reference to Figure 19, the prompt generated in step S1805 will be described. Figure 19 is a diagram showing an example of a prompt generated by the information processing device.

[0182] The prompt 120 shown in Figure 19 is generated in step S1805 in Figure 18 and sent to the server device 500 in step S1806.

[0183] Prompt 120 includes text data 121, 122, 123, and 124. Text data 121 includes a request to generate candidate question data and answer data pairs for question answer information to be stored in the question answer storage unit 320, based on highly rated output answer data and the input question data associated with this output answer data. Text data 121 also includes a request to select a category to associate with the generated question data and answer data pairs.

[0184] Text data 122 contains information indicating the character count constraints for the generated question data and answer data. Text data 123 contains candidate categories to associate with the generated question data and answer data pairs. Text data 124 contains highly rated output answer data and the input question data associated with the output answer data.

[0185] In this embodiment, such prompts 120 are periodically sent to the server device 500, and pairs of question data and answer data generated by the large-scale language model 511 are obtained. This makes it possible to use the pairs of question data and answer data generated by the large-scale language model 511 when adding new question and answer information to the question and answer storage unit 320.

[0186] Furthermore, in this embodiment, the large-scale language model 511 is requested to generate pairs of question data and answer data based on output answer data and input question data that have received high user ratings. Therefore, according to this embodiment, when adding new question and answer information, it is possible to add question and answer information that is likely to receive a high rating from the user.

[0187] In this embodiment, if multiple chat history records with high evaluations of the output response data are extracted in step S1804, text data similar to text data 123 and 124 may be generated for each of the multiple chat history records and included in prompt 120.

[0188] Furthermore, in this embodiment, when generating a prompt, in addition to the first chat history information, the chat history information from the second chat history information that has a high evaluation for the output response data may also be used.

[0189] In this case, the prompt generation unit 256 extracts question and answer information with a high evaluation of the output response data from the second chat history information. Next, the prompt generation unit 256 identifies question and answer information from the extracted question and answer information that is similar to the input question data and registered question data included in the first chat history information with a high evaluation of the output response data. The prompt generation unit 256 may then include the registered question data and registered answer data included in the identified question and answer information in the prompt.

[0190] By doing so, the question and answer data pairs generated by the large-scale language model 511 can be made to be highly satisfactory to users.

[0191] Next, referring to Figure 20, we will explain the process of identifying the internal company information that served as the source of the low-rated output response data, from among the internal company information stored in the document storage unit 410 of the second storage device 400.

[0192] Figure 20 is a fourth sequence diagram illustrating the operation of the chatbot system. The process shown in Figure 20 is one of the periodic processes that are executed periodically in the chatbot system 100.

[0193] The process from step S2001 to step S2003 in Figure 20 is the same as the process from step S1801 to step S1803 in Figure 18, so the explanation is omitted.

[0194] In step S2003, the information processing device 200 acquires chat history information using the information acquisition unit 250. The evaluation and judgment unit 259 then extracts chat history information with low evaluation of output response data from the acquired chat history information, and classifies the extracted chat history information according to the company information contained in the chat history information (step S2004).

[0195] In other words, the evaluation and judgment unit 259 extracts chat history information from the first chat history information that includes output response data that satisfies the second condition, and classifies the extracted chat history information according to the internal company information contained in the chat history information.

[0196] Next, the evaluation and determination unit 259 determines the evaluation of each piece of internal information (step S2005). In other words, the evaluation and determination unit 259 determines whether the evaluation of each piece of internal information satisfies the third condition.

[0197] Specifically, the evaluation determination unit 259 determines, for each piece of internal information, whether the number of output response data with a low evaluation (output response data that satisfies the second condition) among the total number of output response data that use this internal information as a source is equal to or greater than a predetermined number.

[0198] The evaluation unit 259 then determines that the internal company information is internal company information with a low evaluation (internal company information that satisfies the third condition) if the number of output response data with a low evaluation (output response data that satisfies the second condition) among the number of output response data that use this internal company information as an information source is equal to or greater than a predetermined number.

[0199] In step S2005, if the evaluation of the internal company information is determined to be low, the evaluation determination unit 259 transmits the evaluation result for this internal company information to the first storage device 300 (step S2006). The evaluation result transmitted to the first storage device 300 may include information that identifies the relevant internal company information.

[0200] When the first storage device 300 receives the evaluation result, it associates the evaluation result with the internal company information that received a low evaluation in the internal company information stored in the document storage unit 410 (step S2007), and stores the evaluation result in the document storage unit 410 (step S2008).

[0201] In this embodiment, by periodically performing a process to evaluate internal company information, it is possible to identify unreliable internal company information.

[0202] The evaluation results of internal company information may be displayed on a screen for managing internal company information stored in the document storage unit 410. Details of an example of how the evaluation results of internal company information are displayed will be described later.

[0203] Next, referring to Figure 21, we will explain the process of identifying company information stored in the document storage unit 410 of the second storage device 400 for which a predetermined period has elapsed since the last update.

[0204] Figure 21 is a fifth sequence diagram illustrating the operation of the chatbot system. The process shown in Figure 21 is one of the periodic processes that are executed periodically in the chatbot system 100.

[0205] In the chatbot system 100, the information processing device 200, specifically the update timing determination unit 260, sends a search request to the second storage device 400 to identify company information stored in the document storage unit 410 for which a predetermined period has elapsed since the last update (step S2101).

[0206] The second storage device 400 searches the document storage unit 410 in response to a search request (step S2102) and transmits the search results to the information processing device 200 (step S2103).

[0207] The information processing device 200 sends a request to the second storage device 400 for additional information indicating that a predetermined period has elapsed since the last update of the company information identified by the search results from the second storage device 400 (step S2104).

[0208] The second storage device 400 receives a request and stores information indicating that the company information extracted by the search in step S2102 from the company information stored in the document storage unit 410 has not been updated for a predetermined period of time (step S2105).

[0209] In this embodiment, by periodically performing a process to identify internal company information that has not been updated for a predetermined period, it is possible to suppress the generation of output response data based on outdated information. In other words, in this embodiment, it is possible to increase the probability that output response data will be generated based on the latest internal company information.

[0210] Furthermore, information indicating that the information has not been updated for a specified period may be displayed on the screen used to manage internal company information stored in the document storage unit 410. Details of examples of how to display information indicating that the information has not been updated for a specified period will be described later.

[0211] Next, referring to Figures 22 to 24, we will describe the process for improving question answer information using the information processing device 200 of this embodiment.

[0212] Figure 22 is a sixth sequence diagram illustrating the operation of the chatbot system. Figure 22 shows the operation of the chatbot system 100 when the information processing device 200 proposes question answer information that should be improved and methods for improvement to the terminal device 700.

[0213] In the chatbot system 100 of this embodiment, the terminal device 700 receives input from the input receiving unit 740 for setting the period for which the chat history information to be analyzed will be aggregated and setting the time for which the chat history information will be analyzed (step S2201). Subsequently, the terminal device 700 transmits information indicating the set period and time to the information processing device 200 via the communication control unit 730 (step S2202).

[0214] In this embodiment, in step S2201, a management screen for managing the question and answer information stored in the question and answer storage unit 320 may be displayed on the terminal device 700.

[0215] This management screen may be displayed on terminal device 700, for example, when a user of terminal device 700 logs into the chatbot system 100 based on the tenant ID of the organization to which they belong and their own user ID.

[0216] Furthermore, in this embodiment, the management screen displayed on the terminal device 700 may input the period for aggregating the second chat history information to be analyzed, and the time for analyzing the second chat history information.

[0217] When the information processing device 200 receives input of information indicating a period and time from the reception unit 251, it causes the analysis unit 255 to store this information (step S2203).

[0218] The information processing device 200, via the information acquisition unit 250, sends a request to the first storage device 300 to acquire the question and answer information stored in the question and answer storage unit 320 and the second chat history information collected during a set period (step S2204). The first storage device 300 accepts the acquisition request and acquires the second chat history information collected during the specified period from the chat history information stored in the chat history storage unit 310 (step S2205).

[0219] Next, the first storage device 300 transmits the question and answer information stored in the question and answer storage unit 320 and the acquired second chat history information to the information processing device 200 (step S2206).

[0220] When the information processing device 200 acquires question and answer information and second chat history information using the information acquisition unit 250, the analysis unit 255 performs an analysis by referring to the second chat history information, the question and answer information, and the second synonym storage unit 340 (step S2207). Specifically, the analysis unit 255 analyzes whether the content of the output answer data for the input question data included in the second chat history information was appropriate or not. Details of the process in step S2207 will be described later.

[0221] Next, the information processing device 200 transmits analysis result information, which indicates the analysis results from the analysis unit 255, to the first storage device 300 via the output unit 257 (step S2208). The first storage device 300 stores the received analysis result information in the analysis result storage unit 350 via the storage unit 380 (step S2209).

[0222] In this embodiment, the analysis result information is stored in the analysis result storage unit 350 as described above.

[0223] Note that the processing in steps S2201 to S2203 may be executed at a timing independent of the processing in steps S2204 to S2209, which will be described later.

[0224] Furthermore, the processes from steps S2201 to S2203 may be omitted. In that case, the processes from steps S2204 to S2209 may be executed periodically, and the information processing device 200 will automatically execute the processes from step S2204 onwards, as described later, when it is time to perform the analysis.

[0225] In this embodiment, by periodically executing the processing of the analysis unit 255 in step S2207, after the question data and answer data pairs included in the registration candidate information are added to the question answer storage unit 320 as question answer information, the administrator of the chatbot system 100 can be notified whether or not the added question answer information is appropriate.

[0226] Next, we will explain the operation of presenting methods for improving question answer information to the information processing device 200.

[0227] In the chatbot system 100, the terminal device 700 receives an operation from the input reception unit 740 to instruct the display of a list of suggestions for question and answer information (step S2210). Subsequently, the terminal device 700 sends a request to the information processing device 200 to display the list of suggestions (step S2211).

[0228] At this time, the terminal device 700 may display a list screen of question and answer information, and when an operation to instruct the display of the proposal list is performed on the question and answer information list screen, the terminal device 700 sends a request to display the proposal list screen to the information processing device 200.

[0229] The information processing device 200, using the screen generation unit 261, refers to the analysis result storage unit 350 of the first storage device 300 (steps S2212, 2213) and generates screen data for displaying a list of suggested improvement methods (step S2214).

[0230] The process in step S2214 will now be explained. In step S2212, the information processing device 200 uses the analysis unit 255 to refer to the analysis result storage unit 350 and calculates the similarity between text data representing input question data contained in the analysis result storage unit 350. Next, the information acquisition unit 250 groups together text data whose similarity is equal to or greater than a predetermined threshold and assigns the same group ID to them in the analysis result storage unit 350. Then, the screen generation unit 261 generates screen data for the suggestion list screen so that the text data representing the input question data is displayed for each group ID.

[0231] Specifically, the screen generation unit 261 identifies analysis result information that has not been classified into a group in the analysis result storage unit 350, and calculates the similarity between the value of the item "input question data" included in the identified analysis result information and the other analysis result information. Next, the screen generation unit 261 classifies the analysis result information from the other analysis result information whose similarity is above a predetermined threshold into the same group as the identified analysis result information.

[0232] The screen generation unit 261 repeats this process to group each of the analysis result information stored in the analysis result storage unit 350.

[0233] When the information processing device 200 generates screen data for the proposal list screen using the screen generation unit 261, it sends a display instruction for the proposal list screen to the terminal device 700 (step S2215).

[0234] The terminal device 700 receives a display instruction and the display control unit 750 displays the proposal list screen (step S2216). When the terminal device 700 receives a selection of a specific group on the proposal list screen (step S2217), it sends a request to the information processing device 200 to display the proposal details screen for the selected group (step S2218).

[0235] When the information processing device 200 receives this display request, it refers to the analysis result storage unit 350 (steps S2219, S2220) and generates screen data for displaying the proposal details screen for the selected group (step S2221). Details of the process in step S2221 will be described later.

[0236] Next, the information processing device 200 sends an instruction to the terminal device 700 to display the proposal details screen (step S2222).

[0237] The terminal device 700 receives a display instruction, and the display control unit 750 determines the display mode of the proposal details screen (step S2223), and displays the proposal details screen (step S2224).

[0238] Next, we will explain the processing of the analysis unit 255 and the screen generation unit 261 of the information processing device 200.

[0239] Figure 23 is a second flowchart illustrating the processing of the information processing device. Figure 23 shows the details of the process in step S2207 of Figure 22.

[0240] In this embodiment, the analysis unit 255 of the information processing device 200 executes the processing from step S5602 onward for each second chat history information acquired by the information acquisition unit 250 (step S2301).

[0241] The analysis unit 255 obtains the text data (input question data), which is the value of the item "input question data" included in the chat history information (step S2302).

[0242] Next, the analysis unit 255 obtains the text data (registered question data) which is the value of the item "registered question data" in the question and answer information acquired by the information acquisition unit 250, and the text data (registered answer data) which is the value of the item "registered answer data" (step S2303).

[0243] Next, the analysis unit 255 refers to the second synonym storage unit 340 of the first storage device 300 and determines whether a second synonym corresponding to the extracted keyword extracted from each text data obtained in step S2303 is set (step S2304).

[0244] In this embodiment, if the second synonym storage unit 340 stores second synonym information where the value of the item "keyword" matches the extracted keyword, then the second synonym corresponding to the keyword included in the registered question data and registered answer data is set in the second synonym storage unit 340.

[0245] In step S2304, if a second synonym is set, the analysis unit 255 calculates the similarity between the registered question data and registered answer data and the input question data obtained from the second chat history information. Furthermore, the analysis unit 255 replaces the extracted keyword contained in the input question data obtained from the second chat history information with the second synonym corresponding to this extracted keyword and calculates the similarity between this and the registered question data and registered answer data (step S2305).

[0246] Furthermore, in step S2305, if there are multiple secondary synonyms associated with the extracted keyword, the analysis unit 255 performs the same processing for each secondary synonym.

[0247] In step S2304, if a second synonym is not set, the analysis unit 255 calculates the similarity between the registered question data and registered answer data and the input question data obtained from the second chat history information (step S2306).

[0248] In the following description of this embodiment, the similarity calculated by the analysis unit 255 will be referred to as the second similarity.

[0249] Next, the analysis unit 255 calculates a second similarity between all question answer information and the input question data, and then determines whether or not there is any question answer information where the second similarity exceeds the second similarity threshold (step S2307).

[0250] In step S2307, if no relevant question-answer information exists, the analysis unit 255 determines, as an analysis result, that no question-answer information containing appropriate answer data corresponding to the input question data obtained in step S2302 exists in the question-answer storage unit 320 (step S2308). Next, the analysis unit 255 refers to the improvement suggestion storage unit 360 of the first storage device 300, selects the improvement type "Add new question-answer" which corresponds to the value of the item "Analysis Result" being "No appropriate question-answer information exists", associates the analysis result and the improvement type with the conversation ID of the second chat history information obtained in step S1701, and stores them in the analysis result storage unit 350 of the first storage device 300 (step S2309), and proceeds to step S2322 described later.

[0251] In step S2307, if relevant question-answer information exists, the analysis unit 255 identifies the second question-answer information with the highest similarity and acquires it as the second question-answer information (step S2310).

[0252] Next, the analysis unit 255 obtains the value of the item "Reference Question Answer ID" included in the second chat history information obtained in step S2301 (step S2311). Subsequently, the analysis unit 255 determines whether or not the value of the item "Reference Question Answer ID" was obtained in step S2311 (step S2312).

[0253] In other words, the analysis unit 255 determines whether the second chat history information includes a reference question answer ID that identifies the first question answer information that the second chat processing unit 253 referenced when outputting the output response data to the terminal device 600.

[0254] In step S2312, if the corresponding reference question answer ID cannot be obtained, the analysis unit 255 determines, as an analysis result, that question answer information other than the question answer information identified by the second chat processing unit 253 based on the first similarity is appropriate (step S2313).

[0255] Next, the analysis unit 255 refers to the improvement suggestion storage unit 360 and selects the improvement type "Improvement of question-answer matching" which corresponds to the value of the item "Analysis Result" being "Different question-answer information than the one used in the response is appropriate." The analysis result and improvement type are then associated with the conversation ID of the second chat history information obtained in step S2301 and stored in the analysis result storage unit 350 of the first storage device 300 (step S2314), and the process proceeds to step S2322, which will be described later.

[0256] In step S2312, if the corresponding reference question answer ID is obtained, the analysis unit 255 determines whether the first question answer information identified by the reference question answer ID is the same as the second question answer information obtained in step S2310 (step S2315).

[0257] If, in step S2315, the first question answer information and the second question answer information do not match, the analysis unit 255 proceeds to step S2313.

[0258] In step S2315, if the first question answer information and the second question answer information match, the analysis unit 255 obtains the value of the item "User Rating" in the second chat history information obtained in step S2301 (step S2316).

[0259] Next, in step S2316, the analysis unit 255 has the evaluation determination unit 259 determine whether the user evaluation is "unsatisfactory" or not (step S2317). In other words, the analysis unit 255 has the evaluation determination unit 259 determine whether the evaluation of the output response data included in the second chat history information satisfies the second condition.

[0260] In step S2317, if the user evaluation is "unsatisfied," that is, if the evaluation of the output response data is low, the analysis unit 255 determines, as an analysis result, that the question answer information used in the response is inappropriate or that the user evaluation is "unsatisfied" (step S2318).

[0261] Next, the analysis unit 255 refers to the improvement suggestion storage unit 360 of the first storage device 300, selects the improvement type "Consider correcting the answer or resolving the dissatisfaction" which corresponds to the value of the item "Analysis Result" being "The question answer information used in the response is inappropriate or the user evaluation is unsatisfactory", associates the analysis result and the improvement type with the conversation ID of the second chat history information obtained in step S2301, and stores them in the analysis result storage unit 350 of the first storage device 300 (step S2314), and proceeds to step S2322 described later.

[0262] In step S2317, if the user evaluation is not "dissatisfied," the analysis unit 255 determines the analysis result to be "other" (step S2320).

[0263] Next, the analysis unit 255 refers to the improvement suggestion storage unit 360 of the first storage device 300, selects the improvement type "No improvement method suggested" which corresponds to the value of the item "Analysis Result" as "Other", associates the analysis result and improvement type with the conversation ID of the second chat history information obtained in step S2301, stores them in the analysis result storage unit 350 of the first storage device 300 (step S2321), and proceeds to step S2322, which will be described later.

[0264] The analysis unit 255 performs the processes from step S2301 to step S2322 (step S2322) on all chat history information collected by the information acquisition unit 250, and then terminates the process.

[0265] In this embodiment, the analysis unit 255 performs the processing shown in Figure 23, thereby identifying the question and answer information corresponding to the question and answer ID included in the chat history information from the second chat history information for which an improvement type other than "no improvement method proposed" has been selected as the question and answer information that should be improved.

[0266] In other words, the analysis unit 255, through the processing shown in Figure 23, identifies question and answer information that should be improved from the chat history information stored in the chat history storage unit 310, specifically from the chat history information that includes output response data satisfying the second condition.

[0267] In this embodiment, the information processing device 200 is assumed to have a second synonym storage unit 340, but it is not limited to this. The information processing device 200 may determine whether sentences are similar or not by vectorizing the sentences and calculating the distance between them. In this way, it is possible to determine whether input question data and registered question data are similar or not without using the second synonym storage unit 340.

[0268] Figure 24 is a third flowchart illustrating the processing of the information processing device. Figure 24 shows the details of the process in step S2321 of Figure 23.

[0269] The screen generation unit 261 of the information processing device 200 refers to the analysis result storage unit 350 of the first storage device 300 and obtains analysis result information including the group ID assigned to the group selected on the proposal list screen (step S2401).

[0270] Next, the screen generation unit 261 processes each of the acquired analysis result information from step S2403 onward (step S2402).

[0271] The screen generation unit 261 obtains the value of the item "Improvement Type" from the analysis result information (step S2403). Next, the screen generation unit 261 determines whether the value of the item "Improvement Type" is "No improvement method proposed" (step S2404).

[0272] In step S2404, when the value of the item "Improvement Type" is "No proposal for improvement method", the screen generation unit 261 does not display the button indicating the improvement method for the question and answer information corresponding to the reference question and answer ID included in the obtained analysis result information in the screen data for displaying the proposal details screen (step S2405), and proceeds to step S2413 described later. Note that the screen generation unit 261 does not necessarily need to display the obtained analysis result information on the proposal details screen when the value of the item "Improvement Type" is "No proposal for improvement method".

[0273] In step S2404, when the value of the item "Improvement Type" is other than "No proposal for improvement method" and is "New addition of question and answer" (step S2406), the screen generation unit 261 highlights the "New addition of question and answer" button, normally displays the "Improvement of question and answer matching" button, and does not display the "Editing of question and answer considered appropriate" button in the screen data for displaying the proposal details screen (step S2407), and proceeds to step S2410 described later.

[0274] In step S2404, when the value of the item "Improvement Type" is other than "No proposal for improvement method" and is "Improvement of question and answer matching" (step S2406), the screen generation unit 261 normally displays the "New addition of question and answer" button, highlights the "Improvement of question and answer matching" button, and highlights the "Editing of question and answer considered appropriate" button in the screen data for displaying the proposal details screen (step S2408), and proceeds to step S2410 described later.

[0275] In step S2404, when the value of the item "Improvement Type" is other than "No proposal for improvement method" and is "Consider revision of answer or resolution of dissatisfaction" (step S2406), the screen generation unit 261 normally displays the "New addition of question and answer" button, normally displays the "Improvement of question and answer matching" button, does not display the "Editing of question and answer considered appropriate" button, and highlights the "Editing of question and answer used in answer" button in the screen data for displaying the proposal details screen (step S2409), and proceeds to step S2413 described later.

[0276] Following step S2407 and step S2408, the screen generation unit 261 determines whether a reference question and answer ID has been obtained from the analysis result information acquired in step S2401 (step S2410).

[0277] In step S2410, if the reference question and answer ID has not been obtained, the screen generation unit 261 hides the "Edit Question and Answer Used in Answer" button (step S2411) and proceeds to step S2413 described below.

[0278] In step S2410, if the reference question and answer ID has been obtained, the screen generation unit 261 normally displays the "Edit Question and Answer Used in Answer" button (step S2412) and proceeds to step S2413 described below.

[0279] Subsequently, the screen generation unit 261 executes the processes from step S2402 to step S2412 for each of the analysis result information acquired in step S2401, and ends the process.

[0280] As described above, the screen generation unit 261 of the present embodiment generates screen data for displaying the proposal details screen.

[0281] In the present embodiment, the process of displaying the proposal details screen shown in FIG. 24 is executed by the screen generation unit 261 of the information processing apparatus 200, but it is not limited to this. The information processing apparatus 200 of the present embodiment may distribute a program for displaying the proposal details screen to the terminal device 700. In that case, the process of displaying the proposal details screen shown in FIG. 24 may be executed in the terminal device 700.

[0282] In the present embodiment, the operation button that is hidden may be displayed in a grayed-out state where operation acceptance is prohibited instead of being hidden.

[0283] Next, with reference to Figures 25 to 27, an example of the display of the management screen for question and answer information shown on the terminal device 700 will be described. Figure 25 is the first diagram showing an example of the management screen displayed on the terminal device.

[0284] The screen 150 shown in Figure 25 is an example of a list screen for question and answer information, and may be displayed on the terminal device 700 in step S2201 of Figure 22.

[0285] In this embodiment, for example, when the information processing device 200 receives an operation from the terminal device 700 to instruct the display of screen 150, the first storage device 300 may generate screen data for displaying screen 150 by referring to the question answer storage unit 320 associated with the tenant ID of the tenant to which the user of the terminal device 700 belongs.

[0286] Screen 150 includes a display area 151 and operation buttons 152 and 153. The display area 151 displays a list of question and answer information stored in the question and answer storage unit 320.

[0287] Operation button 152 is used to display question and answer information that needs improvement, along with methods for improvement. Operation button 153 is used to display a management screen for managing company information stored in the document storage unit 410.

[0288] Furthermore, when a question and answer information displayed in the display area 151 is selected, screen 150 will indicate that the question and answer information to be edited has been selected.

[0289] Figure 26 shows an example of a proposal list screen displayed on a terminal device. Screen 140 shown in Figure 26 is an example of a proposal list screen displayed on the terminal device 700 in step S2216 of Figure 22.

[0290] The screen 140 shown in Figure 26 includes display areas 141 and 142. Display area 141 displays the period during which the second chat history information was collected as the analysis period. Display area 142 displays a list of input question data included in the second chat history information for each group.

[0291] In the example shown in Figure 26, the display area 142 shows the input question data 142a, 142b, 142c, 142d, 142e, etc., included in the second chat history information classified as group 1.

[0292] In this embodiment, when a specific group is selected in the display area 142, screen 140 transitions to a detailed suggestion screen for that group.

[0293] Figure 27 shows an example of a proposal details screen displayed on a terminal device. Screen 130 shown in Figure 27 is an example of a proposal list screen displayed on the terminal device 700 in step S2224 of Figure 22. Screen 130 is also an example of a proposal details screen when group 1 is selected in the display area 142 of screen 140.

[0294] Screen 130 has a display area 131, in which second chat history information classified into group 1, corresponding analysis result information, and buttons corresponding to improvement types are displayed in association.

[0295] Specifically, display area 131 includes display areas 132, 133, 134, 135, 136, 137, etc. Display area 132 displays a portion of the analysis results information, including the text data 142a shown in Figure 26, as well as buttons 132a and 132b.

[0296] Here, in the analysis result information corresponding to the text data 142a, the improvement type is "New addition of question and answer", and there is no reference question and answer ID (see Fig. 8). Therefore, in the display area 132, the "New addition of question and answer" button 132a is highlighted, the "Improvement of question and answer matching" button 132b is normally displayed, and the "Editing of question and answer considered appropriate" button and the "Editing of question and answer used in the answer" button are not displayed.

[0297] Also, in the display area 133, a part of the analysis result information including the text data 142b shown in Fig. 27 and the buttons 133a, 133b, and 133c are displayed.

[0298] Here, in the analysis result information corresponding to the text data 142b, the improvement type is "New addition of question and answer", and there is a reference question and answer ID (see Fig. 8). Therefore, in the display area 133, the "New addition of question and answer" button 133a is highlighted, the "Editing of question and answer used in the answer" button 133b and the "Improvement of question and answer matching" button 133c are normally displayed, and the "Editing of question and answer considered appropriate" button is not displayed.

[0299] Also, in the display area 134, a part of the analysis result information including the text data 142c shown in Fig. 26 and the buttons 134a, 134b, and 134c are displayed.

[0300] Here, in the analysis result information corresponding to the text data 142c, the value of the item "Improvement type" is other than "No proposal for improvement method", and is "Consider answering correction or dissatisfaction resolution" (see Fig. 8). Therefore, in the display area 134, the "New addition of question and answer" button 134a and the "Improvement of question and answer matching" button 134c are normally displayed, the "Editing of question and answer used in the answer" button 134b is highlighted, and the "Editing of question and answer considered appropriate" button is not displayed.

[0301] Also, in the display area 135, a part of the analysis result information including the text data 142d shown in Fig. 26 and the buttons 135a, 135b, and 135c are displayed.

[0302] Here, in the analysis results information corresponding to the text data 142d, the value of the item "Improvement Type" is something other than "No suggested improvement method," and is "Improvement of question-answer matching" (see Figure 8). Therefore, in the display area 135, the "Add new question-answer" button 135a is displayed normally, the "Edit appropriate question-answer" button 135b and the "Improve question-answer matching" button 135c are highlighted, and the "Edit question-answer used in the answer" button is hidden.

[0303] Furthermore, the display area 136 shows a portion of the analysis results information, including the text data 142e shown in Figure 26, as well as buttons 136a, 136b, 136c, and 136d. Here, in the analysis results information corresponding to the text data 142e, the value of the item "Improvement Type" is something other than "No suggested improvement method," and is "Improvement of question-answer matching" (see Figure 8). Therefore, in the display area 135, the "Add new question-answer" button 136a and the "Edit question-answer used in the answer" button 136b are displayed as usual, and the "Edit appropriate question-answer" button 136b and the "Improve question-answer matching" button 136d are highlighted.

[0304] Furthermore, the display area 137 shows a portion of the analysis results information, including the text data 142f shown in Figure 26.

[0305] In this case, the value of the item "Improvement Type" in the analysis result information corresponding to the text data 142f is "No improvement method suggested" (see Figure 8). Therefore, in the display area 137, the button that suggests an improvement method corresponding to the text data 142f is not displayed.

[0306] In this embodiment, the information processing device 200 analyzes whether the content of the output response data output by the information processing device 200 to the terminal device 600 is appropriate, by referring to the second chat history information, the second synonym storage unit 340 which stores a general-purpose group of synonyms, and user evaluations.

[0307] In other words, the information processing device 200 of this embodiment can identify question-answer information with a low evaluation of the answer data from among the question-answer information referenced as an information source, and notify the administrator of the chatbot system 100.

[0308] The information processing device 200 then displays a detailed suggestion screen on the terminal device 700, showing recommended improvement methods for each question and answer information that requires improvement. Furthermore, in this embodiment, the information processing device 200 displays a list of improvement methods for each question and answer information as a list of operation buttons on the detailed suggestion screen.

[0309] Therefore, according to this embodiment, when an administrator edits question and answer information, they only need to select the improvement method displayed on the proposal details screen and the corresponding operation button, allowing for efficient and easy editing of the question and answer information.

[0310] Furthermore, in this embodiment, when an improvement method and corresponding operation button are selected on the proposal details screen displayed on the terminal device 700, the information processing device 200 displays specific improvement proposals on the terminal device 700 that correspond to the improvement method indicated by the selected operation button.

[0311] The following describes the operation of the chatbot system 100 when an improvement method and the corresponding operation button are selected on the suggestion details screen displayed on the terminal device 700.

[0312] Figure 28 is the seventh sequence diagram illustrating the operation of the chatbot system. Figure 28 shows the operation when "Add a new question answer" is selected as the improvement method on screen 130 shown in Figure 27.

[0313] In the chatbot system 100, when the terminal device 700 selects "Add a new question answer" on the screen 130 (step S2801), it sends a notification to the information processing device 200 indicating that the "Add a new question answer" button has been selected (step S2802).

[0314] When the information processing device 200 receives this notification via the receiving unit 251, the output unit 257 sends a search request for candidate registration information, which will be a candidate for question answer information, to the registration candidate storage unit 365 of the first storage device 300 (step S2803).

[0315] The search request here includes the question data contained in the question answer information for which an improvement method was selected in step S2801.

[0316] When the first storage device 300 receives a search request, it searches the registration candidate storage unit 365 using the query data included in the search request as a search key (step S2804). The first storage device 300 then sends the registration candidate information extracted as a search result to the information processing device 200. Send (step S2805). In the search in the first storage device 300, candidate registration information containing question data that matches or is similar to the question data included in the search request may be extracted as search results.

[0317] When the information processing device 200 receives registration candidate information from the first storage device 300, the screen generation unit 261 generates screen data for an additional screen of question and answer information, including the question data and answer data pairs and categories included in the registration candidate information, and the output unit 257 transmits a display instruction for the screen data to the terminal device 700 (step S2806).

[0318] The terminal device 700 receives a command to display an additional screen and displays an additional screen for question and answer information (step S2807). Details of the additional screen will be described later.

[0319] Next, the terminal device 700 receives, on the additional screen, the response data for the question data included in the registration candidate information and corrections to the category from the administrator of the chatbot system 100 (step S2808).

[0320] Next, the terminal device 700 transmits the question data and answer data pairs with the correction results to the information processing device 200 (step S2809). If the question data and answer data pairs included in the registered candidate information have not been corrected, the terminal device 700 may send a notification to that effect.

[0321] When the information processing device 200 receives the correction results, the storage control unit 254 sends a request to the first storage device 300 to store new question and answer information, which includes the corrected question and answer data pair and category (step S2810). The storage request sent to the first storage device 300 here includes the new question and answer information.

[0322] The first storage device 300 receives this storage request and stores the question and answer information included in the storage request in the question and answer storage unit 320 (step S2811).

[0323] The following describes the screen for adding question and answer information, referring to Figure 29. Figure 29 is a diagram showing an example of the screen for adding question and answer information displayed on the terminal device.

[0324] The additional screen 160 shown in Figure 29 is an example of a screen displayed on the terminal device 700 in step S2807 of Figure 28.

[0325] The additional screen 160 includes a display area 161, a selection field 162, display areas 163 and 164, and an operating element 165.

[0326] The display area 161 shows a message indicating that pairs of question data and answer data were generated using highly-rated output response data obtained in the past through a search of internal company information using the generative AI implemented by the large-scale language model 511.

[0327] The selection field 162 displays the categories included in the registration candidate information. In this embodiment, depending on the user's operation of the terminal device 700, the selection field 162 may display a list of categories that correspond to the question data and answer data pairs displayed in the display areas 163 and 164. Alternatively, the selection field 162 may display the category selected by the user of the terminal device 700 from the list.

[0328] Display area 163 displays the question data included in the registration candidate information, and display area 164 displays the answer data included in the registration candidate information.

[0329] The operating member 165 is an operating member for storing the category selected in the selection field 162 and the pairs of question data and answer data displayed in the display areas 163 and 164 as question and answer information.

[0330] The additional screen 160 shown in Figure 29 is an additional screen that is displayed on the terminal device 700 when the "Add New Question and Answer" button 132a is selected on the screen 130 shown in Figure 27.

[0331] In this case, when the "Add New Question and Answer" button 132a is selected on the terminal device 700, the information processing device 200 sends a search request to the first storage device 300 that includes the "Add New Question and Answer" button 132a and the corresponding input question data, which is "Product Implementation Case Studies".

[0332] The first storage device 300 searches the registration candidate storage unit 365 using "product implementation examples" as the search key, and transmits the search results to the information processing device 200, which include a pair of question data "Regarding product implementation examples" and answer data "Product A has been implemented in the past," along with registration candidate information including the category "Product."

[0333] The information processing device 200 displays the category "Product" in the selection field 162, the question data "Regarding product implementation examples" in the display area 163, and the answer data "Product A has been implemented in the past" in the display area 164, and displays this additional screen 160 on the terminal device 700.

[0334] Furthermore, when the operating member 165 is selected on the additional screen 160, the terminal device 700 sends a request to the first storage device 300 to store question and answer information, which includes the question data "Regarding product implementation examples" and the answer data "Product A has been implemented in the past," along with the category "Product."

[0335] Thus, according to this embodiment, when the "Add New Question and Answer" button is selected on the screen 130 displayed on the terminal device 700, an additional screen suggesting new question and answer information is displayed on the terminal device 700. Therefore, the user of the terminal device 700 does not need to think of or input new question and answer information themselves, and the effort required to create question and answer information in the chatbot system 100 can be reduced.

[0336] Next, with reference to Figure 30, we will describe the screen for managing company information stored in the document storage unit 410.

[0337] Figure 30 shows an example of a management screen for internal company information displayed on a terminal device. The management screen 140A shown in Figure 30 may be displayed on the terminal device 700 when, for example, the operation button 153 is operated on the screen 150 shown in Figure 25.

[0338] The management screen 140A includes a display area 143. The display area 143 displays a list of company information stored in the document storage unit 410.

[0339] Furthermore, on the management screen 140A, information indicating internal company information that has been determined to have a low evaluation by the evaluation judgment unit 259, and internal company information that has not been updated for a predetermined period, as identified by the update timing determination unit 260, is displayed along with a list of internal company information.

[0340] In the management screen 140A, icon images 144a and 144b are displayed in association with the company information whose file names are displayed in display area 144, from the list of company information displayed in display area 143.

[0341] Icon image 144a indicates that the internal company information has been judged to have a low evaluation, while icon image 144b indicates that the information has not been updated for a specified period of time.

[0342] Therefore, according to the management screen 140A, it can be seen that the internal information file named "Attendance Management" has a low rating and has not been updated for a specified period.

[0343] Thus, in this embodiment, users of the terminal device 700 can visually identify internal company information stored in the document storage unit 410 that has a low rating or has not been updated for a predetermined period. Therefore, according to this embodiment, it is possible to support the administrator of the chatbot system 100 in managing the internal company information stored in the document storage unit 410.

[0344] In this embodiment, if there is internal information with a low evaluation or internal information that has not been updated for a predetermined period, that information is displayed on the internal information management screen, but this is not limited to this. In this embodiment, for example, if internal information with a low evaluation is identified during periodic processing, the terminal device 700 may be notified of this fact each time. Also, if internal information that has not been updated for a predetermined period is identified during periodic processing, the terminal device 700 may be notified of this fact each time.

[0345] By doing so, the internal company information stored in the document storage unit 410 can always be considered highly reliable internal company information.

[0346] Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" in this specification includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0347] Furthermore, the apparatus described in each embodiment represents only one of several computing environments for carrying out the embodiments disclosed herein.

[0348] In one embodiment, the information processing device 200 includes a plurality of computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein. Similarly, the information processing device 200 may include a plurality of computing devices configured to communicate with each other.

[0349] Furthermore, the information processing device 200 can be configured to share the disclosed processing steps in various combinations. For example, a process performed by a predetermined unit may be performed by the information processing device 200. Similarly, the functions of a predetermined unit may be performed by the information processing device 200. Also, each element of the information processing device 200 may be combined into a single server device or divided among multiple devices.

[0350] The information processing device 200 may be any device equipped with communication capabilities. For example, the information processing device 200 may be an output device such as a PJ (Projector), IWB (Interactive White Board: an electronic whiteboard with the ability to communicate with each other), or digital signage; a HUD (Head Up Display) device; industrial machinery; imaging devices; sound collection devices; medical equipment; networked home appliances; automobiles (Connected Cars); notebook PCs (Personal Computers); mobile phones; smartphones; tablet terminals; game consoles; PDAs (Personal Digital Assistants); digital cameras; wearable PCs; or desktop PCs.

[0351] Examples of the present invention are as follows: <1> A reception unit that receives input of question data and an evaluation of the response data output in response to the question data, Upon receiving the aforementioned question data, a first chat processing unit outputs response data generated by a large-scale language model using user-specific information as the information source to a first terminal device. A storage control unit stores first chat history information, including the question data, the answer data output by the first chat processing unit, and an evaluation of the answer data output by the first chat processing unit, in a first storage device. A prompt generation unit generates a prompt that requests the large-scale language model to generate a set of question data and answer data based on the answer data included in the first chat history information, specifically the answer data whose evaluation satisfies the first condition, and the question data corresponding to the answer data. An information processing device having an output unit that outputs the pair of question data and answer data generated by the large-scale language model to a second terminal device. <2> The first storage device stores question and answer information that associates pre-created registered question data with registered answer data. The storage control unit, In response to the operation of the second terminal device, the pair of question data and answer data displayed on the second terminal device is stored in the first storage device as part of the question and answer information. <1> The information processing device described. <3> The storage control unit, The registration candidate information, including the question data and answer data pairs generated by the large-scale language model, is stored in the first storage device. <2> The information processing device described. <4> The system includes a second chat processing unit that searches for the question and answer information stored in the first storage device based on the question data received by the reception unit, and outputs the answer data associated with the question data to the first terminal device. The storage control unit, The first storage device stores second chat history information, which includes the question data, the answer data output by the second chat processing unit, and the evaluation of the answer data output by the second chat processing unit. The output unit is, The second terminal device outputs a notification indicating that, among the response data included in the second chat history information, the response data whose evaluation satisfies the second condition. <2> or <3> The information processing device described. <5> The device has a screen generation unit that generates screen data to display a screen containing a list of improvement methods for the question answer information stored in the first storage device, The output unit is, The screen data is output to the second terminal device. When a method for adding question-answer information is selected on the screen including the list of improvement methods displayed on the second terminal device, a search request for registration candidate information using the question data included in the question-answer information as the search key is output to the first storage device. The question data and answer data pairs included in the registration candidate information obtained from the first storage device are output to the second terminal device. <2> ~ <4> An information processing device as described in any one of the following items. <6> The system includes an evaluation determination unit that determines whether the evaluation of the user-specific information stored in the second storage device satisfies the third condition, The evaluation and determination unit is, Based on the chat history information from the first chat history information, which includes response data whose evaluation satisfies the second condition, it is determined whether the evaluation of the user-specific information satisfies the third condition. The output unit is, The evaluation determines that the third condition is met, and a notification indicating user-specific information is output to the second terminal device. <1> ~ <5> An information processing device as described in any one of the following items. <7> It has an update timing determination unit that determines whether a predetermined period has elapsed since the last update date of the user-specific information stored in the second storage device, The output unit is, A notification indicating user-specific information for which a predetermined period has elapsed since the last update date is output to the second terminal device. <1> ~ <6> An information processing device as described in any one of the following items. <8> An information processing system including an information processing device and a terminal device that communicates with the information processing device, The aforementioned information processing device is A reception unit that receives input of question data and an evaluation of the response data output in response to the question data, Upon receiving the aforementioned question data, a first chat processing unit outputs response data generated by a large-scale language model using user-specific information as the information source to a first terminal device among the terminal devices. A storage control unit stores first chat history information, including the question data, the answer data output by the first chat processing unit, and an evaluation of the answer data output by the first chat processing unit, in a first storage device. A prompt generation unit generates a prompt that requests the large-scale language model to generate a set of question data and answer data based on the answer data included in the first chat history information, where the answer data satisfies the first condition and the corresponding question data. The device has an output unit that outputs the pair of question data and answer data generated by the large-scale language model to a second terminal device among the terminal devices, The second terminal device is, An information processing system having a display control unit that displays a screen containing the pair of question data and answer data generated by the large-scale language model. <9> An information processing method comprising an information processing device and a terminal device that communicates with the information processing device, The aforementioned information processing device The system accepts the input of question data and the evaluation of the response data output in response to the said question data. Upon receiving the aforementioned question data, the response data generated by the large-scale language model using user-specific information as the information source is output to the first terminal device among the terminal devices. The first chat history information, which includes the question data, the response data output to the first terminal device, and the evaluation of the response data output to the first terminal device, is stored in the first storage device. A prompt is generated that requests the large-scale language model to generate a set of question data and answer data based on the answer data included in the first chat history information, where the evaluation satisfies the first condition, and the question data corresponding to the answer data. The pair of question data and answer data generated by the large-scale language model is output to the second terminal device of the terminal device. The second terminal device is An information processing method for displaying a screen containing the question data and answer data pairs generated by the large-scale language model.

[0352] Although the present invention has been described above based on various embodiments, the present invention is not limited to the requirements shown in the above embodiments. These points can be modified as long as they do not impair the spirit of the present invention, and can be appropriately determined according to their application. [Explanation of Symbols]

[0353] 100 Chatbot Systems 200 Information Processing Devices 250 Information Acquisition Department 251 Reception Department 252 First Chat Processing Unit 253 Second Chat Processing Unit 254 Storage Control Unit 255 Analysis Department 256 Prompt generation unit 257 Output section 258 Registration Candidate Generation Unit 259 Evaluation and Judgment Unit 260 Update timing determination section 261 Screen generation section 300 First memory device 310 Chat history storage unit 320 Question and answer storage section 400 Second storage device 410 Document Storage Unit 420 User Information Storage Unit 500 Server Devices 511 Large-scale language models 600, 700 terminal devices [Prior art documents] [Patent Documents]

[0354] [Patent Document 1] Patent No. 7504531

Claims

1. A reception unit that receives input of question data and an evaluation of the response data output in response to the question data, Upon receiving the aforementioned question data, a first chat processing unit outputs response data generated by a large-scale language model using user-specific information as the information source to a first terminal device. A storage control unit stores first chat history information, including the question data, the answer data output by the first chat processing unit, and an evaluation of the answer data output by the first chat processing unit, in a first storage device. A prompt generation unit generates a prompt that requests the large-scale language model to generate a set of question data and answer data based on the answer data included in the first chat history information, specifically answer data whose evaluation satisfies the first condition, and the corresponding question data. An information processing device having an output unit that outputs the pair of question data and answer data generated by the large-scale language model to a second terminal device.

2. The first storage device stores question and answer information that associates pre-created registered question data with registered answer data. The storage control unit, The information processing apparatus according to claim 1, wherein, in response to the operation of the second terminal device, the pair of question data and answer data displayed on the second terminal device is stored in the first storage device as part of the question and answer information.

3. The storage control unit, The information processing apparatus according to claim 2, wherein registration candidate information including the question data and answer data pairs generated by the large-scale language model is stored in the first storage device.

4. The system includes a second chat processing unit that searches for the question and answer information stored in the first storage device based on the question data received by the reception unit, and outputs the answer data associated with the question data to the first terminal device. The storage control unit, The first storage device stores second chat history information, which includes the question data, the answer data output by the second chat processing unit, and the evaluation of the answer data output by the second chat processing unit. The output unit is, The information processing apparatus according to claim 2, which outputs a notification to the second terminal device indicating that, among the response data included in the second chat history information, the response data whose evaluation satisfies the second condition.

5. The device has a screen generation unit that generates screen data to display a screen containing a list of improvement methods for the question answer information stored in the first storage device, The output unit is, The screen data is output to the second terminal device. When a method for adding question-answer information is selected on the screen including the list of improvement methods displayed on the second terminal device, a search request for registration candidate information using the question data included in the question-answer information as the search key is output to the first storage device. The information processing apparatus according to claim 3, which outputs a pair of question data and answer data included in the registration candidate information obtained from the first storage device to the second terminal device.

6. The system includes an evaluation determination unit that determines whether the evaluation of the user-specific information stored in the second storage device satisfies the third condition, The evaluation and determination unit is, Based on the chat history information from the first chat history information, which includes response data whose evaluation satisfies the second condition, it is determined whether the evaluation of the user-specific information satisfies the third condition. The output unit is, The information processing apparatus according to claim 1, which outputs a notification to the second terminal device indicating user-specific information for which the evaluation has been determined to satisfy the third condition.

7. It has an update timing determination unit that determines whether a predetermined period has elapsed since the last update date of the user-specific information stored in the second storage device, The output unit is, The information processing apparatus according to claim 1, which outputs a notification to the second terminal device indicating user-specific information for which a predetermined period has elapsed since the last update date.

8. An information processing system including an information processing device and a terminal device that communicates with the information processing device, The aforementioned information processing device is A reception unit that receives input of question data and an evaluation of the response data output in response to the question data, Upon receiving the aforementioned question data, a first chat processing unit outputs response data generated by a large-scale language model using user-specific information as the information source to a first terminal device among the terminal devices. A storage control unit stores first chat history information, including the question data, the answer data output by the first chat processing unit, and an evaluation of the answer data output by the first chat processing unit, in a first storage device. A prompt generation unit generates a prompt that requests the large-scale language model to generate a set of question data and answer data based on the answer data included in the first chat history information, specifically the answer data whose evaluation satisfies the first condition, and the question data corresponding to the answer data. The device has an output unit that outputs the pair of question data and answer data generated by the large-scale language model to a second terminal device among the terminal devices, The second terminal device is, An information processing system having a display control unit that displays a screen containing the pair of question data and answer data generated by the large-scale language model.

9. An information processing method comprising an information processing device and a terminal device that communicates with the information processing device, The aforementioned information processing device The system accepts the input of question data and the evaluation of the response data output in response to the said question data. Upon receiving the aforementioned question data, the response data generated by the large-scale language model using user-specific information as the information source is output to the first terminal device among the terminal devices. The first chat history information, which includes the question data, the response data output to the first terminal device, and the evaluation of the response data output to the first terminal device, is stored in the first storage device. A prompt is generated that requests the large-scale language model to generate a set of question data and answer data based on the answer data included in the first chat history information whose evaluation satisfies the first condition, and the question data corresponding to the answer data. The pair of question data and answer data generated by the large-scale language model is output to the second terminal device of the terminal device. The second terminal device is An information processing method for displaying a screen containing the question data and answer data pairs generated by the large-scale language model.

Images