Information processing device, information processing method, and program

Abstract

In a dialogue system, the system will be able to modify the know-how information stored in the database based on the user's responses. [Solution] The information processing device includes an acquisition unit and a control unit. The acquisition unit acquires questions from the user. When the control unit acquires a question about know-how entered by the user from the acquisition unit, it acquires one or more pieces of know-how information from a database that stores multiple pieces of know-how information according to the content of the question. The control unit displays an answer to the question based on the specific piece of know-how information contained in the one or more pieces of know-how information. Furthermore, the control unit determines whether or not the specific piece of know-how information needs to be modified based on the user's reaction to the answer.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] A technique has been proposed for generating a chatbot specialized for a given domain by using a large language model based on documents in the domain (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a dialogue system that outputs an answer to a question, it is conceivable to obtain and display one or more pieces of know-how information that can be an answer to the input question from a database. However, if incorrect know-how information is stored in the database, it may not always be possible to give a correct answer.

[0005] In view of such circumstances, an object of the present disclosure is to enable the correction of know-how information stored in a database in a dialogue system.

Means for Solving the Problems

[0006] An information processing apparatus according to one embodiment of the present disclosure is characterized by comprising: an acquisition unit that acquires a question from a user; and a control unit that, upon acquiring a question concerning know-how entered by the user by the acquisition unit, acquires one or more pieces of know-how information from a database storing multiple pieces of know-how information according to the content of the question; displays an answer to the question based on a specific piece of know-how information included in the one or more pieces of know-how information; and determines whether or not the specific piece of know-how information needs to be modified based on the user's reaction to the answer.

[0007] An information processing method according to one embodiment of the present disclosure is an information processing method executed by an information processing device, which includes, when a user inputs a question regarding know-how, obtaining one or more pieces of know-how information from a database storing multiple pieces of know-how information according to the content of the question, displaying an answer to the question based on a specific piece of know-how information included in the one or more pieces of know-how information, and determining whether or not the specific piece of know-how information needs to be modified based on the user's reaction to the answer.

[0008] A program according to one embodiment of the present disclosure causes a computer to perform the following processes when a user inputs a question about know-how: obtaining one or more pieces of know-how information from a database storing multiple pieces of know-how information according to the content of the question; displaying an answer to the question based on a specific piece of know-how information included in the one or more pieces of know-how information; and determining whether or not the specific piece of know-how information needs to be modified based on the user's reaction to the answer. [Effects of the Invention]

[0009] According to this disclosure, a dialogue system can be configured to modify know-how information stored in a database based on user responses. [Brief explanation of the drawing]

[0010] [Figure 1]This is a block diagram showing a schematic configuration of a system including an information processing device according to one embodiment of the present disclosure. [Figure 2] Figure 1 is a sequence diagram showing the information processing flow of the system. [Figure 3] This figure shows an example of a screen displayed on a user's terminal. [Figure 4] This figure shows an example of a screen displayed on a user's terminal. [Figure 5] This figure shows an example of a screen displayed on a user's terminal. [Figure 6] Figure 1 is a flowchart of the decision process executed by the control unit of the information processing device. [Modes for carrying out the invention]

[0011] Hereinafter, one embodiment of this disclosure will be described with reference to the drawings.

[0012] (Overall system overview) System 1 according to one embodiment of this disclosure includes, as shown in Figure 1, an information processing device 10, a user terminal 20, a Large Language Models (LLM) server 30, a know-how database 40, and a user database 50. The information processing device 10, the user terminal 20, the LLM server 30, the know-how database 40, and the user database 50 are each connected to each other so as to be able to communicate with one another by any communication means, including wired communication and wireless communication.

[0013] In System 1, users can use user terminal 20 to ask questions about know-how in a free-form style. Know-how refers to specialized knowledge, techniques, and methods. Know-how can be rephrased as knowledge. In the following, information related to know-how may be referred to as know-how information.

[0014] The information processing device 10 receives a question from the user terminal 20 and retrieves a set of know-how information related to the question from the know-how database 40. The know-how information retrieved by the information processing device 10 from the know-how database includes one or more pieces of know-how information. The information processing device 10 generates an answer based on the set of know-how information retrieved from the know-how database 40 and sends it to the user terminal 20. The information processing device 10 exchanges information with the user via the user terminal 20 in a text or voice dialogue format. The information processing device 10 may have the functionality of a so-called "chatbot".

[0015] The information processing device 10 may receive an answer to a question from the LLM server 30 by sending the user's question and the know-how information obtained from the know-how database 40 to the LLM server 30. In other words, the information processing device 10 provides the functionality of RAG (Retrieval Augmented Generation). RAG is a technology that improves the accuracy of answers by combining text generation using a large-scale language model with retrieval of external information.

[0016] The LLM server 30 generates answers to questions obtained from the information processing device 10 based on a set of know-how information related to the content of the questions obtained from the information processing device 10. The LLM server 30 can extract the necessary information from the set of know-how information and generate answers using easy-to-understand language that corresponds to the questions.

[0017] The information processing device 10 can transmit a response to the user terminal 20 and obtain a reaction from the user to the response. The user may be able to input a reaction to the response as an evaluation such as "good" or "bad" for the response. The information processing device 10 determines whether it is necessary to correct the know-how information that is the basis for the response to the user's question based on the user's reaction. The information processing device 10 can accept a proposal for the correction content from a user who has evaluated that the correction of the know-how information is necessary. When the information processing device 10 determines that the correction of the know-how information is necessary, it may correct the corresponding know-how information in the know-how database 40 based on the correction content proposed by the user so far.

[0018] Each component of system 1 will be described below.

[0019] (Information processing device) The information processing device 10 is a computer including, for example, a PC (Personal Computer) and a workstation. Note that the information processing device 10 is not limited to these, and may be any general-purpose electronic device, or may be another electronic device dedicated to system 1. The information processing device 10 may be a server belonging to a cloud computing system or other computing system. The information processing device 10 includes a control unit 11, a communication unit 12, and a storage unit 13.

[0020] The control unit 11 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (central processing unit) or GPU (graphics processing unit), or a dedicated processor specialized for specific processing. The dedicated circuit is, for example, an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit). The control unit 11 executes processes related to the operation of the information processing device 10 while controlling each part of the information processing device 10.

[0021] The control unit 11 may include each component of a user interaction unit 11a, an information search unit 11b, an instruction text generation unit 11c, a modification necessity determination unit 11d, and a data update unit 11e. Each of these components may be a hardware module or a software module. The control unit 11 can execute the processes described as the functions of each component below. In the following, the processes executed by each component may be described as the processes executed by the control unit 11.

[0022] The user interaction unit 11a realizes an interactive interface with the user via the user terminal 20. The user interaction unit 11a mediates the interaction between the user and the LLM server 30. The user interaction unit 11a receives questions from the user and presents answers to the questions to the user. The user interaction unit 11a asks the user to evaluate the answer and obtains the evaluation of the answer from the user. The evaluation result of the answer by the user may be selected from two-stage evaluation results: a first evaluation result indicating that the evaluation is high and a second evaluation result indicating that the evaluation is low. The evaluation result of the answer by the user is included in the user's reaction to the answer.

[0023] The information search unit 11b instructs the know-how database 40 to search for information corresponding to the user's question acquired from the user terminal 20 by the user interaction unit 11a. The information search unit 11b acquires know-how information related to the user's question from the know-how database 40. For example, when the user's question is related to the operation method of a specific machine, the information search unit 11b acquires know-how information including information on the operation manual of the machine and information registered by other users within the organization who have expertise related to the machine. The information acquired by the information search unit 11b may include a lot of information related to the user's question.

[0024] The instruction generation unit 11c generates an instruction for the LLM server 30. An instruction for the LLM server 30 is called a "prompt". The instruction generation unit 11c generates an instruction that includes a user's question, know-how information acquired by the information retrieval unit 11b, and an instruction to output the answer to the question along with identification information that identifies the know-how information used in the answer. By adding the know-how information to the user's question and sending it to the LLM server 30, the control unit 11 makes it easier to obtain an appropriate answer from the LLM server 30 based on the know-how information. Furthermore, by causing the LLM server 30 to output identification information that identifies the know-how information used in the answer, the control unit 11 can confirm which of the one or more pieces of know-how information transmitted by the LLM server 30 was used to generate the answer.

[0025] The correction necessity determination unit 11d processes the evaluation results of the user's response obtained by the user interaction unit 11a and determines whether the know-how information used in the response needs to be corrected. The correction necessity determination unit 11 may determine that correction of the specific know-how information is necessary if the user's evaluation of that specific know-how information is a second evaluation result indicating a low evaluation result. The correction necessity determination unit 11d may determine whether correction of the know-how information is necessary by statistically processing the evaluation results of multiple users for the specific know-how. Statistical processing includes calculating the proportion of the second evaluation result included in the user's evaluation results.

[0026] When the correction necessity determination unit 11d obtains a second evaluation result indicating a low evaluation from the user, it may search the user database 50 to obtain proficiency information indicating the proficiency level of the user who performed the evaluation. When the proficiency information meets predetermined requirements, the correction necessity determination unit 11d determines that correction is necessary for the specific know-how information. The predetermined requirements are requirements indicating a high level of proficiency for the user. The predetermined requirements include, for example, that the user's years of experience in the field related to the know-how information are at least a predetermined number of years (e.g., 5 or 10 years). The predetermined requirements also include, for example, that the user is certified as an expert in the field related to the know-how information.

[0027] The data update unit 11e updates the know-how information when the correction necessity determination unit 11d determines that correction is necessary for that know-how information. The data update unit 11e may update the data based on the correction content of the know-how information obtained from the user by the user interaction unit 11a. For example, the data update unit 11e may update the know-how information by other means. For example, the data update unit 11e searches the user database 50 for users with high skills in the field related to the know-how information that has been determined to require correction. The data update unit 11e may send a notification to the highly skilled user requesting them to correct the know-how information that has been determined to require correction.

[0028] The communication unit 12 includes an external communication interface for communicating with the user terminal 20, the LLM server 30, the know-how database 40, and the user database 50. The communication unit 12 may communicate with each component via a communication network such as the Internet. The communication interface may be either a wired or wireless communication interface. In the case of wired communication, the communication interface may be, for example, a LAN (Local Area Network) interface or a USB (Universal Serial Bus) interface. In the case of wireless communication, the communication interface may be, for example, an interface compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation), or an interface compatible with short-range wireless communication such as Bluetooth (registered trademark). The communication unit 12 receives data used in the operation of the information processing device 10 and transmits data obtained by the operation of the information processing device 10 to the outside. The communication unit 12 functions as an acquisition unit that obtains user questions from the user terminal 20.

[0029] The storage unit 13 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or at least two combinations thereof. The semiconductor memory is, for example, RAM (random access memory) or ROM (read-only memory). The RAM is, for example, SRAM (static random access memory) or DRAM (dynamic random access memory). The ROM is, for example, EEPROM (electrically erasable programmable read-only memory). The storage unit 13 functions, for example, as a main memory, auxiliary memory, or cache memory. The storage unit 13 stores programs and data used for the operation of the information processing device 10, and data obtained by the operation of the information processing device 10.

[0030] The memory unit 13 may store the number of evaluations and evaluation results for each piece of know-how information stored in the know-how database 40, based on the user's evaluation. The memory unit 13 may also store candidate sentences for correction and / or replacement based on the user's correction suggestions for each piece of know-how information that the user has evaluated.

[0031] Although the information processing device 10 in Figure 1 of this embodiment does not show an input unit and an output unit, it may further include an input unit and an output unit. That is, in addition to receiving (inputting) and transmitting (outputting) information via the communication unit 12, the information processing device 10 may also perform information input and output using its own input unit and output unit.

[0032] The functions of the information processing device 10 are realized by executing a program relating to the information processing method of this embodiment on a processor corresponding to the control unit 11. In other words, the functions of the information processing device 10 are realized by software. The program causes the computer to perform the operations of the information processing device 10, thereby causing the computer to function as the information processing device 10. That is, the computer functions as the information processing device 10 by performing the operations of the information processing device 10 according to the program.

[0033] In this embodiment, the program can be recorded on a computer-readable recording medium. The computer-readable recording medium includes non-temporary computer-readable media, such as magnetic recording devices, optical discs, magneto-optical recording media, or semiconductor memory. The program can be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs (digital versatile discs) or CD-ROMs (compact disc read-only memory) on which the program is recorded. Alternatively, the program may be distributed by storing it on the storage of an external server and transmitting it from the external server to other computers. The program may also be provided as a program product.

[0034] (User terminal) The user terminal 20 is an information device such as a computer used by the user. The user terminal 20 includes general-purpose devices and dedicated devices. General-purpose devices include PCs, tablet terminals, and smartphones. The user terminal includes a processor, a communication interface, memory, and a display. The display may be a touch panel display that also functions as an input device. The user terminal 20 may further include input / output devices such as a keyboard, microphone, speaker, and camera.

[0035] The user terminal 20 can send and receive information with the information processing device 10 via a communication interface. The user terminal 20 works in conjunction with the information processing device 10 to present information to the user. The information presented to the user is displayed on the user terminal 20's screen. The user terminal 20 may function as a client of the information processing device 10 when the information processing device 10 is operating as a server. Each function of the user terminal 20 may be executed by application software installed on the user terminal 20. The processor of the user terminal 20 may control each part of the user terminal 20 and execute processing on the user terminal 20 according to the application software.

[0036] The user terminal 20 can authenticate the user by the user's login name, password, and / or an image acquired from a camera, or by other means. The user terminal 20 can display text and images on its display based on information received from the information processing device 10. The user terminal 20 receives text information input from a touch panel, keyboard, or other input means and transmits it to the information processing device 10. The user terminal 20 may be configured to perform speech recognition processing and speech synthesis processing. The user terminal 20 may convert the text information received from the information processing device 10 into speech and output it from a speaker. The user terminal 20 may recognize the user's voice input from a microphone and convert it into text information. The user terminal 20 may transmit the input text information to the information processing device 10.

[0037] (LLM server) The LLM server 30 is a general-purpose or dedicated computer. The LLM server 30 is equipped with a large-scale language model. The large-scale language model is a natural language processing model generated by learning from a large amount of text data. The large-scale language model can be used for various natural language processing applications.

[0038] The LLM server 30 performs natural language processing on the instruction text obtained from the information processing device 10 and analyzes the instruction content contained in the instruction text. In accordance with the instruction text, the LLM server 30 extracts the necessary specific know-how information from the know-how information contained in the instruction text in response to the question in the instruction text, and generates an answer using that know-how information. The LLM server 30 can perform analysis and summarization of specific know-how information in order to answer the question. In accordance with the instruction text, the LLM server 30 returns to the information processing device 10 the answer to the question contained in the instruction text and identification information that identifies the specific know-how information used to generate the answer to the question. The identification information is information that can identify each piece of know-how information stored in the know-how database 40.

[0039] (Know-how database) The know-how database 40 is a database for storing know-how information. The know-how database 40 is, for example, a vector database, but is not limited to this. The know-how database may also be other databases such as a hierarchical database, a network database, an RDB (Relational Database), or an ODB (Object Database). Know-how information includes information obtained based on practical experience. In this disclosure, know-how information includes information stored within organizations such as companies, local governments, and research institutions. However, know-how information is not limited to information used in a closed environment within an organization, and may also include information made public outside the organization.

[0040] The know-how information stored in the know-how database 40 may include internal manuals, user-registered professional information, and information on past successful methods. The know-how information may be indexed in various ways. The know-how database 40 may include information that has been converted from the text of the know-how information into numerical vectors. The know-how information may be classified and organized by field and / or application. The know-how information may be searchable and / or extracted by field, application, or free keywords. Each piece of know-how information included in the know-how database may be managed with identification information to identify the know-how information.

[0041] (User database) The user database 50 manages information about users who use System 1 and / or members of organizations that use System 1. The user database 50 may manage users' login names and passwords.

[0042] The user database 50 stores the skill level of each user who queries know-how information using the user terminal 20. For each user, the user database 50 stores the user ID, password, area of ​​expertise, and proficiency information indicating the level of expertise in that area. Proficiency information is indicated, for example, by the number of years of experience in the area of ​​expertise. The user database 50 may also store contribution information indicating each user's contribution to registering know-how in the know-how database. For example, when a user registers their own know-how in the know-how database 40, contribution information is added. Also, when a user makes a suggestion to modify know-how information via the information processing device 10, contribution information is added. If the contribution information is high, the user may be given benefits within their organization. This provides an incentive for users to contribute to the accumulation and updating of know-how information in the know-how database 40.

[0043] The know-how database 40 and the user database 50 do not need to be separate components from the information processing device 10. The know-how database 40 and the user database 50 may be included in the storage unit 13 of the information processing device 10. Also, the know-how database 40 and the user database 50 may be installed on the same hardware. The know-how database 40 may be distributed across multiple computers within the organization.

[0044] (Information processing performed by the system) Based on Figure 2, the information processing flow performed by System 1 will be explained.

[0045] Prior to the process shown in Figure 2, the user logs in to the user terminal 20 by entering their login name and password. The user terminal 20 may further ensure the user's authenticity through biometric authentication, including fingerprint authentication using a fingerprint sensor and facial recognition using a camera.

[0046] First, the user terminal 20 displays a message on its screen indicating that it is accepting questions from the user. The user then enters a question into the user terminal 20. The question may be entered via keyboard, touch panel, or microphone. Figure 3 shows an example of what is displayed on the terminal 20's screen. The user enters a question such as, "Please tell me how to do...", "What temperature is appropriate for...", or "Please tell me what precautions to take when creating...". The format of the question is not limited to these examples and may be any form of text.

[0047] As shown in Figure 2, the question entered by the user into the user terminal 20 is sent to the information processing device 10. The information processing device 10 queries the know-how database 40 for know-how information highly relevant to the question. The know-how database 40 searches for know-how information based on the question and sends one or more pieces of know-how information related to the question to the information processing device 10. The one or more pieces of know-how information sent from the know-how database 40 to the information processing device 10 include a set of information extracted from manuals, reports, documents registered as know-how information by other users, etc., related to the question. The know-how database 40 may be searched in any way. For example, the know-how database 40 may be a vector database and know-how information may be searched based on the similarity to the question. Alternatively, for example, the information processing device 10 may detect keywords from the question and search the know-how database 40 using those keywords.

[0048] When the information processing device 10 obtains know-how information from the know-how database 40, it generates a prompt for the LLM server 30. The prompt includes a question, one or more pieces of know-how information obtained from the know-how database, and an instruction to output an answer to the question along with information identifying the know-how information used in that answer. The information processing device 10 sends the generated prompt to the LLM server 30.

[0049] The LLM server 30 generates an answer to a question based on the instruction. In generating the answer, the LLM server 30 uses one or more pieces of know-how information contained in the instruction. If the one or more pieces of know-how information contained in the instruction are sentences consisting of multiple sentences, the LLM server 30 analyzes the content of those sentences and identifies the parts necessary for generating the answer. The LLM server 30 transmits the answer and identification information that identifies the specific pieces of know-how information used to generate the answer to the information processing device 10. The identification information may be, for example, information that can identify the know-how information at the paragraph, sentence, or phrase level of the information contained in the know-how database 40. The identification information may be assigned to each piece of know-how information in advance when one or more pieces of know-how information are obtained from the know-how database 40.

[0050] When the information processing device 10 receives a response from the LLM server 30, it sends the response, excluding the identification information, to the user terminal 20 and displays it on the user terminal 20's display. At this time, the information processing device 10 also displays a question on the user terminal 20's display asking for the user's evaluation of the reliability of the response. Figure 4 shows an example of the display on the user terminal 20's display. In this example, the user terminal 20 displays a question asking whether the response was good or not. The user can input a "good" or "bad" evaluation of the response by pressing the "yes" or "no" button on the touch panel display, or by speaking "yes" or "no," etc. The user evaluates the response as "good" if they think the content of the response is good. The user evaluates the response as "bad" if they think the content of the response is questionable or contains errors. "Good" is the first evaluation result, indicating a high evaluation. "Bad" is the second evaluation result, indicating a low evaluation.

[0051] In Figure 2, the area enclosed by the dashed line shows the processing when the user inputs a second evaluation result in response to the answer displayed on the user terminal 20. In this case, the information processing device 10 displays a question on the user terminal 20 asking for a suggestion for correction, as shown in Figure 5. The user can then input a suggestion for correction in any format. The information processing device 10 sends the correction suggestion entered by the user to the LLM server 30, along with instructions to convert it into a know-how sentence in a format suitable for publication in the know-how database 40. The LLM server 30 deletes unnecessary parts and makes necessary changes to the user's correction suggestion. For example, if the user makes a correction suggestion such as "I think 30 minutes is better than 15 minutes for heating...", the LLM server 30 generates a know-how sentence such as "Heat... for 30 minutes". The information processing device 10 may store the know-how sentence in the storage unit 13 as a candidate sentence for correcting and / or replacing know-how information. If the user inputs a first evaluation result in response to the answer, the processing in the area enclosed by the dashed line is not performed.

[0052] Next, the information processing device 10 performs a determination process to determine whether or not the know-how information used in the answer needs to be modified. As an example, the determination process is performed by the control unit 11 of the information processing device 10 according to the flowchart in Figure 6.

[0053] The control unit 11 acquires evaluation information from the user terminal 20 via the communication unit 12 (S01). This process corresponds to the "evaluation response" from the user terminal 20 to the information processing device 10 in Figure 2. If the control unit 11 receives a second evaluation result, the processing shown in the dashed line in Figure 2 takes place between S01 and S02.

[0054] The control unit 11 adds 1 to the user rating count of the know-how information identified by the identification information. The user rating count indicates the cumulative number of times that a user has evaluated each piece of know-how information.

[0055] The control unit 11 terminates the judgment process if the user's evaluation result is the first evaluation result, "Good" (S03: No). If the user's evaluation result is the second evaluation result, "Bad" (S03: Yes), the control unit 11 adds 1 to the number of "Bad" evaluations. The number of "Bad" evaluations is the number of second evaluation results. The "Bad" evaluations indicate the cumulative number of times the user has given a "Bad" evaluation to each piece of know-how information. After S04, the control unit 11 proceeds to S05.

[0056] The control unit 11 searches the user database 50, obtains proficiency information indicating the proficiency level of the evaluated user, and determines whether the proficiency information meets predetermined requirements (S05). The proficiency information is represented, for example, by the number of years of work experience in the field related to the know-how information. The predetermined requirements are, for example, that the number of years of work experience is 5 years or more or 10 years or more. If the proficiency information meets the predetermined conditions (S05: Yes), the control unit 11 proceeds to process S08. If the proficiency information does not meet the predetermined conditions (S05: No), the control unit 11 proceeds to process S06.

[0057] In S06, the control unit 11 determines whether the number of user evaluations for a specific piece of know-how information evaluated by the user is above a predetermined threshold. The predetermined threshold can be set to any integer of 2 or more, such as 5, 10, or 20. If the number of user evaluations is above the predetermined threshold (S06: Yes), the control unit 11 proceeds to S07. If the number of user evaluations is below the predetermined threshold (S06: No), the control unit 11 terminates the determination process.

[0058] In S07, the control unit 11 calculates the percentage of "bad" ratings for specific know-how information relative to the total number of user ratings. If the percentage of "bad" ratings is above a predetermined percentage (S07: Yes), the control unit 11 proceeds to S08. If the percentage of "bad" ratings is below the predetermined percentage (S07: No), the control unit 11 terminates the determination process.

[0059] In step S08, the control unit 11 decides to modify the know-how information.

[0060] In other words, the control unit 11 decides to modify the know-how information if a highly skilled user gives a "bad" rating, regardless of the ratings of other users. If a less skilled general user gives a "bad" rating, the control unit 11 decides to modify the know-how information if the number of user ratings is above a predetermined threshold and the proportion of "bad" ratings is above a predetermined percentage.

[0061] The judgment process shown in Figure 6 is just one example. The judgment process can be performed using various procedures other than those shown in Figure 6.

[0062] As shown in Figure 2, if the determination process determines that the know-how information needs to be modified, the information processing device 10 updates the know-how information stored in the know-how database 40. Updating the know-how information includes rewriting and replacing the know-how information. The control unit 11 may modify or replace specific know-how information used to create the answer in the know-how database 40 using candidate statements for modification and / or replacement stored in the storage unit 13.

[0063] As described above, according to this embodiment, the information processing device determines whether or not to correct the specific know-how information used in answering a question, based on the user's response to the user's answer. This reduces the occurrence of the inconvenience of incorrect know-how information remaining stored in the know-how database 40.

[0064] In the above embodiment, the evaluation results of the responses obtained from the user are used as the user's reaction, so it is possible to obtain highly accurate user reactions regarding the quality of the responses. Furthermore, since the user makes a two-stage evaluation of "good" and "bad," it is easy for the user to select an evaluation and easy to analyze the evaluation results.

[0065] Furthermore, the control unit 11 determines whether or not to revise specific know-how information by statistically processing evaluation results obtained from multiple users, thus enabling the determination of whether or not to revise know-how information based on objective criteria. In particular, the control unit determines that revision of know-how information is necessary when it obtains evaluations from a number of users exceeding a predetermined threshold and when a predetermined percentage or more of users give a "bad" evaluation, thus enabling the determination of whether or not to revise based on the reactions of multiple users.

[0066] For example, if a user notices that the know-how information displayed as an answer contains an error, they can suggest a correction to the know-how information. The information processing device 10 can update the know-how information in the know-how database based on the user's correction suggestion. This allows for the quick and efficient correction of know-how information containing errors.

[0067] The information processing device 10 determines that corrections to know-how information are necessary when the proficiency information of a user who has given a "bad" rating meets predetermined requirements, thus enabling the rapid utilization of the knowledge of highly skilled users regarding know-how information. Furthermore, by using the number of years of experience in the field of know-how information as a measure of user proficiency, the user's proficiency can be objectively evaluated.

[0068] Furthermore, the information processing device 10 is configured to input a question into the LLM server 30, and to add identification information of the know-how information used to the answer obtained, as well as to obtain an evaluation of the answer from the user. This allows the know-how information in the know-how database 40 to be corrected if the know-how information contains errors or other issues. In this way, in a system that combines a large-scale language model system and RAG, it becomes possible to update the information in the know-how database, which is external information from the perspective of the large-scale language model system.

[0069] In the above embodiment, the information processing device 10, upon receiving a question from the user terminal 20, searched the know-how database 40 to obtain know-how information, added the know-how information to the question, and sent it to the LLM server 30. However, the flow of information processing executed by System 1 is not limited to this form. For example, the information processing device 10, upon receiving a question from the user terminal 20, may be configured to send the question to the LLM server 30 and receive a request for information necessary for the answer from the LLM server 30. The information processing device 10 may search the know-how database 40 for the know-how information requested by the LLM server 30 and send it to the LLM server 30.

[0070] Before receiving a question from the user at the user terminal 20, the information processing device 10 may ask the user a question to narrow down the area of ​​the question. By narrowing down the area of ​​the question, the accuracy of the know-how information that the information processing device 10 can obtain from the know-how database 40 can be improved.

[0071] In the above embodiment, users evaluated their answers to questions using a two-tiered system: "good" and "bad." However, the evaluation method is not limited to two tiers; a multi-tiered system including three or five tiers may be adopted. Users may also evaluate answers using a score from 0 to 100. These evaluation results may be analyzed using statistical methods such as the mean and median.

[0072] When the information processing device 10 needs to modify know-how information, instead of making modifications based on user modification suggestions, it may search the user database 50 to find users with high expertise in the field of know-how information and send a message requesting modification of the know-how information.

[0073] In the above embodiment, the user's response was defined as the evaluation result of the answer to the question entered by the user. However, the user's response is not limited to this. The user terminal 20 is equipped with a camera, and the user terminal 20 or the information processing device 10 may analyze the user's facial expressions and eye movements as part of the user's response. For example, the user terminal 20 may read the user's confidence level in the answer from the user's facial expressions. Also, for example, the user terminal 20 is equipped with a timer, and the time from receiving an answer to a question until the user makes the next input may be measured as part of the user's response. If the user's response remains paused for a long time, it is possible that the user has doubts about the answer.

[0074] It should be noted that the present invention is not limited to the embodiments described above, and numerous modifications or alterations are possible. For example, the functions included in each means, step, etc., can be rearranged in a way that is not logically contradictory, and multiple means or steps, etc., can be combined into one or divided.

[0075] Some embodiments of the present disclosure are described below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Note 1] A unit that retrieves questions from the user, When the acquisition unit acquires a question about know-how entered by the user, the control unit performs a process that includes acquiring one or more pieces of know-how information from a database storing multiple pieces of know-how information according to the content of the question, displaying an answer to the question based on the specific piece of know-how information included in the one or more pieces of know-how information, and determining whether or not the specific piece of know-how information needs to be modified based on the user's reaction to the answer. An information processing device equipped with the following features. [Note 2] The information processing apparatus described in Appendix 1, wherein the control unit takes the evaluation result for the answer obtained from the user as the user's response to the answer. [Note 3] The information processing device described in Appendix 2, wherein the evaluation result is selected from two levels of evaluation results: a first evaluation result indicating a high evaluation, and a second evaluation result indicating a low evaluation. [Note 4] The information processing device described in Appendix 3, wherein the control unit, when the evaluation result is the second evaluation result, further accepts a proposal for the modification of the know-how information, and when it determines in the determination of whether modification is necessary that modification is necessary, modifies the specific know-how information based on the proposed modification. [Note 5] The information processing apparatus according to Appendix 3 or 4, wherein the control unit determines whether or not the specific know-how information needs to be modified by statistically processing the evaluation results obtained from multiple users. [Note 6] The information processing apparatus according to Appendix 5, wherein the control unit obtains a number of evaluations from users equal to or greater than a predetermined threshold, and determines that correction is necessary for one or more of the know-how information when the proportion of the second evaluation results is equal to or greater than a predetermined proportion. [Note 7] The information processing device according to any one of the appendices 3 to 6, wherein the control unit obtains a second evaluation result from the user, obtains proficiency information indicating the user's level of expertise, and determines that correction is necessary to the specific know-how information when the proficiency information meets predetermined requirements. [Note 8] The information processing device described in Appendix 7, wherein the proficiency information is the number of years of work in a field related to the know-how information, and when the number of years of work is equal to or greater than a predetermined number of years, the proficiency information satisfies the predetermined requirements. [Note 9] The information processing apparatus according to any one of the appendices 1 to 8, wherein the control unit inputs an instruction statement to a large-scale language model system, which includes an instruction to output the question, one or more pieces of know-how information, and the answer to the question together with information identifying the specific know-how information used in the answer, thereby acquiring the answer to the question and information identifying the specific know-how information. [Note 10] An information processing method performed by an information processing device, When a user enters a question about know-how, one or more pieces of know-how information are retrieved from a database that stores multiple pieces of know-how information, depending on the content of the question. Displaying the answer to the question based on specific know-how information contained in one or more of the aforementioned know-how information, and Based on the user's response to the aforementioned answer, a determination is made as to whether or not the specific know-how information needs to be modified. Methods that include... [Note 11] The method described in Appendix 10, wherein the evaluation result for the response obtained from the user is defined as the user's response to the response. [Note 12] The method described in Appendix 11, wherein the evaluation result is selected from two levels of evaluation results: a first evaluation result indicating a high evaluation and a second evaluation result indicating a low evaluation. [Note 13] If the evaluation result is the second evaluation result, further suggestions for modifications to the know-how information will be accepted. The method described in Appendix 12, further comprising modifying the specific know-how information based on the proposed modification content if it is determined that modification is necessary in the determination of whether modification is necessary. [Note 14] The method according to Appendix 12 or 13, which determines whether or not the particular know-how information needs to be modified by statistically processing the evaluation results obtained from multiple users. [Note 15] The method according to Appendix 14, wherein, when a number of evaluations from the users exceeding a predetermined threshold is obtained, and the proportion of the second evaluation results exceeds a predetermined proportion, it is determined that modifications are necessary for one or more of the know-how information. [Note 16] The method according to any one of the appendices 12 to 15, wherein when a second evaluation result is obtained from the user, proficiency information indicating the user's level of expertise is obtained, and when the proficiency information meets predetermined requirements, it is determined that modifications to the specific know-how information are necessary. [Note 17] The method described in Appendix 16, wherein the proficiency information is the number of years of work in the field related to the know-how information, and when the number of years of work is equal to or greater than a predetermined number of years, the proficiency information satisfies the predetermined requirements. [Note 18] The method according to any one of the appendices 10 to 17, wherein an instruction statement including the question, one or more pieces of know-how information, and an instruction to output the answer to the question together with information identifying the specific know-how information used in the answer is input into a large-scale language model system, thereby obtaining the answer to the question and information identifying the specific know-how information. [Note 19] On the computer, When a user enters a question about know-how, one or more pieces of know-how information are retrieved from a database that stores multiple pieces of know-how information, depending on the content of the question. Displaying the answer to the question based on specific know-how information contained in one or more of the aforementioned know-how information, and Based on the user's response to the aforementioned answer, a determination is made as to whether or not the specific know-how information needs to be modified. A program that performs a process that includes this. [Note 20] The program described in Appendix 19, which further causes the computer to perform a process of obtaining an answer to a question and information identifying the specific know-how information by inputting an instruction statement into a large-scale language model system, the instruction statement including an instruction to output the question, one or more pieces of know-how information, and the answer to the question together with information identifying the specific know-how information used in the answer. [Explanation of symbols]

[0076] 10 Information Processing Devices 11 Control Unit 11a User Interaction Section 11b Information Retrieval Department 11c Directive sentence generator 11d Correction necessity determination section 11e Data Update Unit 12 Communications Department (Acquisition Department) 13 Storage section 20 User Terminals 30 Large-Scale Language Model (LLM) Servers 40 Know-how Database 50 User Databases

Claims

1. A unit that retrieves questions from the user, When the acquisition unit acquires a question about know-how entered by the user, the control unit performs a process that includes acquiring one or more pieces of know-how information from a database storing multiple pieces of know-how information according to the content of the question, displaying an answer to the question based on the specific piece of know-how information included in the one or more pieces of know-how information, and determining whether or not the specific piece of know-how information needs to be modified based on the user's reaction to the answer. An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, wherein the control unit takes the evaluation result for the answer obtained from the user as the user's response to the answer.

3. The information processing apparatus according to claim 2, wherein the evaluation result is selected from two evaluation results: a first evaluation result indicating a high evaluation, and a second evaluation result indicating a low evaluation.

4. The information processing apparatus according to claim 3, wherein the control unit, when the evaluation result is the second evaluation result, further receives a proposal for the content of the modification of the know-how information, and when it is determined in the determination of whether modification is necessary that modification is necessary, modifies the specific know-how information based on the proposed content of modification.

5. The information processing apparatus according to claim 3, wherein the control unit determines whether or not the specific know-how information needs to be modified by statistically processing the evaluation results obtained from multiple users.

6. The information processing apparatus according to claim 5, wherein the control unit obtains a number of evaluations from users equal to or greater than a predetermined threshold, and determines that it is necessary to modify one or more of the know-how information when the proportion of the second evaluation results is equal to or greater than a predetermined proportion.

7. The information processing apparatus according to claim 3, wherein the control unit, upon obtaining a second evaluation result from the user, obtains proficiency information indicating the user's level of expertise, and determines that correction is necessary to the specific know-how information when the proficiency information satisfies predetermined requirements.

8. The information processing apparatus according to claim 7, wherein the proficiency information is the number of years of work in a field related to the know-how information, and the proficiency information satisfies the predetermined requirements when the number of years of work is equal to or greater than a predetermined number of years.

9. The information processing apparatus according to claim 1, wherein the control unit inputs an instruction statement to a large-scale language model system, which includes an instruction to output the question, one or more pieces of know-how information, and the answer to the question together with information identifying the specific know-how information used in the answer, thereby obtaining the answer to the question and information identifying the specific know-how information.

10. An information processing method performed by an information processing device, When a user enters a question about know-how, one or more pieces of know-how information are retrieved from a database that stores multiple pieces of know-how information, depending on the content of the question. Displaying the answer to the question based on specific know-how information contained in one or more of the aforementioned know-how information, and Based on the user's response to the aforementioned answer, a determination is made as to whether or not the specific know-how information needs to be modified. Methods that include...

11. The method according to claim 10, wherein the evaluation result for the response obtained from the user is defined as the user's response to the response.

12. The method according to claim 11, wherein the evaluation result is selected from two evaluation results: a first evaluation result indicating a high evaluation and a second evaluation result indicating a low evaluation.

13. If the evaluation result is the second evaluation result, further suggestions for modifications to the know-how information will be accepted. The method according to claim 12, further comprising modifying the specific know-how information based on the proposed modification content if it is determined that modification is necessary in the determination of whether modification is necessary.

14. The method according to claim 12, wherein the need for modification of the specific know-how information is determined by statistically processing the evaluation results obtained from multiple users.

15. The method according to claim 14, wherein, when a number of evaluations from the users is obtained that is greater than or equal to a predetermined threshold, and the proportion of the second evaluation results is greater than or equal to a predetermined proportion, it is determined that modifications are necessary for one or more of the know-how information.

16. The method according to claim 12, wherein when a second evaluation result is obtained from the user, proficiency information indicating the user's level of expertise is obtained, and when the proficiency information satisfies predetermined requirements, it is determined that modifications to the specific know-how information are necessary.

17. The method according to claim 16, wherein the proficiency information is the number of years of work in a field related to the know-how information, and the proficiency information satisfies the predetermined requirements when the number of years of work is equal to or greater than a predetermined number of years.

18. The method according to claim 10, wherein an instruction statement including the question, one or more pieces of know-how information, and an instruction to output the answer to the question together with information identifying the specific know-how information used in the answer is input into a large-scale language model system, thereby obtaining the answer to the question and information identifying the specific know-how information.

19. On the computer, When a user enters a question about know-how, one or more pieces of know-how information are retrieved from a database that stores multiple pieces of know-how information, depending on the content of the question. Displaying the answer to the question based on specific know-how information contained in one or more of the aforementioned know-how information, and Based on the user's response to the aforementioned answer, a determination is made as to whether or not the specific know-how information needs to be modified. A program that performs a process that includes this.

20. The program according to claim 19, further comprising inputting an instruction statement into a large-scale language model system, which includes an instruction to output the question, one or more pieces of know-how information, and the answer to the question together with information identifying the specific know-how information used in the answer, thereby causing the computer to perform a process of obtaining the answer to the question and information identifying the specific know-how information.

Images