Information processing device, information processing method, and program

Abstract

To enhance the quality of response data to question data, when acquiring the response data utilizing a large-scale language model.SOLUTION: A management server 1 is an information processing device which can acquire response data to question data input to a language model server 3 from the language model server 3. The management server 1 includes: a prior knowledge narrowing unit 62 for extracting a first character string "rabbit" associated with LLM determination 305 "unknown", from each term included in the question data, in prior knowledge DB 300 for holding the LLM determination 305, a term 303, and a description 304 thereof in association with each other; and a provision information generation unit 63 and a response information management unit 64 for inputting a prompt 730 including the first character string "rabbit" and the description 304 associated with the first character string "rabbit" in the question data to the language model server 3, and acquiring the response data to the question data from the language model server 3.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, there has been remarkable progress in machine learning technology, and services that generate answers to user inquiries using so-called large language models that are generated by general-purpose learning using large amounts of information are beginning to be provided (e.g., Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "Natural Language Commanding via Program Synthesis", [online], [Retrieved July 4, 2023], Internet (URL: https: / / arxiv.org / abs / 2306.03460) Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned conventional technology, it is possible to obtain answer data to question data by using a large-scale language model. However, since the large-scale language model is a model trained with general-purpose training data, it is often the case that so-called in-house terminology, technical terminology, the latest terminology, etc. have not been trained. Therefore, if an inquiry contains unknown words or terms, the accuracy of the answer may be extremely degraded.

[0005] An object of the present invention is to improve the quality of answer data when answer data to question data is acquired using a large-scale language model. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device capable of obtaining answer data to question data input into a large-scale language model from the large-scale language model, the information processing device including: an extraction unit that extracts a first string from among the strings included in the question data, the first string being associated with judgment information that does not reach the criterion, in a database that stores judgment information indicating whether knowledge about a specific string in the large-scale language model has reached a certain standard, and the string and explanatory information for the string in association with each other; and a control unit that inputs input data that includes the first string and the explanatory information associated with the first string in the question data into the large-scale language model to obtain answer data to the question data from the large-scale language model; an information processing method that includes each of the processes; and a program that causes a computer to execute each of the processes. Effect of the Invention

[0007] According to the present invention, when answer data to question data is acquired using a large-scale language model, the quality of the answer data can be improved. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a management server that configures an information processing system according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram illustrating an example of the functional configuration of a management server, a user terminal, and a language model server that configure an information processing system according to an embodiment of the present invention. [Figure 4] FIG. 13 is an image diagram for explaining an example of the content of prior knowledge in the information processing system according to the embodiment of the present invention. [Diagram 5] FIG. 1 is an image diagram for explaining an example of a hierarchical structure of prior knowledge in an information processing system according to an embodiment of the present invention. [Figure 6] FIG. 11 is a diagram showing an example of an image displayed on a user terminal constituting an information processing system according to an embodiment of the present invention. [Figure 7] FIG. 13 is a diagram showing an example of an image displayed on a management server constituting an information processing system according to an embodiment of the present invention. [Figure 8] 7 is a diagram showing an example of an image displayed on a user terminal constituting an information processing system according to an embodiment of the present invention, which is an example different from the example of FIG. 6. FIG. [Figure 9] 11 is a diagram showing an example of the flow of various processes executed when confirming prior knowledge in a management server constituting an information processing system according to an embodiment of the present invention. FIG. [Figure 10] FIG. 2 is a diagram showing an example of the flow of various processes executed at the time of an inquiry or request in a management server, a user terminal, and a language model server that constitute an information processing system according to an embodiment of the present invention. [Figure 11] FIG. 2 is a diagram illustrating an example of the configuration of a business knowledge DB and a prior knowledge DB. [Figure 12] FIG. 13 is a diagram illustrating an example of determining whether an LLM is known or unknown. [Figure 13] FIG. 13 is a diagram illustrating an example of determining whether an LLM is known or unknown. [Figure 14] 13 is a flowchart illustrating an example of a known / unknown determination process. [Figure 15] FIG. 13 is a diagram showing a process for creating a knowledge graph for each term in a glossary. [Figure 16] FIG. 13 is a diagram showing a process for creating a knowledge graph for each term in a glossary. [Figure 17] FIG. 13 is a diagram showing a display example when a knowledge graph is displayed. [Figure 18] FIG. 13 is a diagram showing a process for creating a knowledge graph for each term in a glossary. [Figure 19] FIG. 13 is a diagram showing a process for creating a knowledge graph for each term in a glossary. [Figure 20] FIG. 13 is a diagram showing a display example when a knowledge graph is displayed. [Figure 21]13 is a flowchart illustrating an example of a prior knowledge generation process. [Figure 22] FIG. 13 is a diagram showing an example of outputting an answer to a question. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0010] [Example of information processing system configuration] FIG. 1 is a diagram showing an example of a configuration of an information processing system according to an embodiment of the present invention.

[0011] 1 includes a management server 1, a user terminal 2, and a language model server 3. The management server 1, the user terminal 2, and the language model server 3 are connected to each other via a predetermined network N such as the Internet. Note that the network N is not an essential component, and for example, NFC (Near Field Communication), Bluetooth (registered trademark), LAN (Local Area Network), etc. may be used.

[0012] The management server 1 is configured with a general-purpose PC (Personal Computer) or the like, and is managed by an administrator of this system or the like. The management server 1 acquires, for example, information related to an inquiry or request from a user, and transmits various information required for generating a response to the language model server 3. The management server 1 also acquires information related to a response to the inquiry or request transmitted from the language model server 3, and transmits the information to the user terminal 2.

[0013] The user terminal 2 is configured with a general-purpose PC, a smartphone, or the like, and is used by a user who wishes to receive a service to which this system is applied. The user terminal 2 transmits, for example, information related to an inquiry or request input by the user to the management server 1. The user terminal 2 also obtains information related to a response to the inquiry or request transmitted from the management server 1, and displays the information on the output unit 130 (see FIG. 3) or the like.

[0014] The language model server 3 is managed by an administrator of various large-scale language models.

[0015] Here, the large-scale language model is a trained model trained from a large amount of information across various fields and languages, and typically includes GPT (Generative Pre-Trained Transformer) (e.g., GPT-3.5, GPT-4, GPT-4 Turbo, GPT-4o (Omni)), BERT (Bidirectional Encoder Representations from Transformers), Transformer-XL, etc. Note that the examples of large-scale language models mentioned here are merely illustrative, and are intended to broadly include various language models that may be developed and used in the future.

[0016] <Hardware configuration> FIG. 2 is a diagram showing an example of a hardware configuration constituting a management server of an information processing system according to an embodiment of the present invention.

[0017] As shown in FIG. 2, the management server 1 includes a control unit 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a memory unit 18, a communication unit 19, and a drive 20.

[0018] The control unit 11 is composed of a microcomputer including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a semiconductor memory, and executes various processes according to a program recorded in the ROM 12 or a program loaded from the memory unit 18 to the RAM 13.

[0019] The RAM 13 also stores information necessary for the control unit 11 to execute various processes as appropriate.

[0020] The control unit 11, ROM 12, and RAM 13 are connected to one another via a bus 14. An input / output interface 15 is also connected to this bus 14. An output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20 are connected to the input / output interface 15.

[0021] The output unit 16 is composed of various liquid crystal displays, speakers, etc., and outputs various information as images and sounds.

[0022] The input unit 17 is composed of a keyboard, a mouse, etc., and is used to input various information.

[0023] The storage unit 18 is configured with a hard disk drive (HDD), a solid state drive (SSD), etc., and stores various data. In this embodiment, for example, various information including various programs and various databases is stored.

[0024] The communication unit 19 controls communications with other devices via a network N including the Internet.

[0025] The drive 20 is provided as necessary. Removable media 31, which may be a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is appropriately mounted in the drive 20. The program read from the removable media 31 by the drive 20 is installed in the storage unit 18 as necessary. The removable media 31 can also store various data stored in the storage unit 18 in the same manner as the storage unit 18.

[0026] The hardware configurations of the user terminal 2 and the language model server 3 can be basically the same as the hardware configuration of the above-mentioned management server 1, so a description thereof will be omitted.

[0027] FIG. 3 is a diagram showing an example of the functional configuration of a management server, a user terminal, and a language model server that constitute an information processing system according to an embodiment of the present invention.

[0028] As shown in FIG. 3, the control unit 11 of the management server 1 executes various programs, etc., so that an input information acquisition unit 60, a prior knowledge management unit 61, a prior knowledge narrowing down unit 62, a provision information generation unit 63, and a response information management unit 64 function.

[0029] Furthermore, a prior knowledge DB (Data Base) 300 is provided in one area of ​​the storage unit 18. The prior knowledge DB 300 stores a result of confirmation as to whether a target word is known or unknown in a predetermined language model, together with information indicating the relationship between the respective words. Details of the information stored in the prior knowledge DB 300 will be described later with reference to, for example, Figs. 4 and 5. Furthermore, the prior knowledge DB 300 and the business knowledge DB 350 will be described in detail with reference to Fig. 11.

[0030] The input information acquisition unit 60 of the management server 1 acquires text information or voice information related to an inquiry or request.

[0031] Specifically, the input information acquisition unit 60 acquires information relating to the contents of an inquiry or request transmitted from the user terminal 2 (hereinafter referred to as “input information”) via the communication unit 19.

[0032] The prior knowledge management unit 61 of the management server 1 executes various processes related to prior knowledge that is confirmed before an inquiry or request from a user. The prior knowledge management unit 61 includes a prior knowledge confirmation unit 80 and a hierarchical management unit 81.

[0033] The prior knowledge checking unit 80 of the management server 1 acquires information regarding checking whether or not knowledge regarding a predetermined word in a predetermined language model reaches a certain standard.

[0034] Specifically, the prior knowledge confirmation unit 80 generates, for example, information for making an inquiry or request for a predetermined word in advance (hereinafter referred to as "prior knowledge information"), and transmits the prior knowledge information to the language model server 3. Here, the predetermined word is, for example, any word such as industry jargon or technical terminology used by the administrator of the management server 1 in the business of the company. Then, the prior knowledge confirmation unit 80 obtains information on the answer to the prior knowledge information generated by the language model server 3, and, for example, if the content of the answer meets a certain standard, determines the word to be known, and if the content does not meet the certain standard, determines the word to be unknown.

[0035] The criteria for judgment by the prior knowledge verification unit 80 can be determined at the discretion of the administrator of this system, but can be, for example, by asking a question such as "Please give a concise explanation of the word XX in about 100 characters," and having a person (with knowledge in that field) evaluate the answer and enter the result, or by using a system that automatically evaluates the content of summaries, etc.

[0036] The hierarchical management unit 81 of the management server 1 holds the relationships between the target words as a hierarchical structure, and manages the confirmation results of the words together with the relationships between the words.

[0037] Specifically, the hierarchical management unit 81 generates relationships between each of the target words. The hierarchical management unit 81 also hierarchically manages the results of the confirmation of each word by the prior knowledge confirmation unit 80 and the relationships between each of the words. The hierarchical management unit 81 also stores various types of information that it manages in the prior knowledge DB 300 as prior knowledge information.

[0038] The prior knowledge narrowing-down unit 62 of the management server 1 determines word candidates (hereinafter referred to as "provided word candidates") to be provided to the language model server 3 as part of information to be provided, which will be described later, based on the input information acquired by the input information acquisition unit 60 and the contents of the prior knowledge information stored in the prior knowledge DB 300. In addition, the prior knowledge narrowing-down unit 62 adds words in a higher hierarchy than the words determined as the provided word candidates to the provided word candidates.

[0039] The provided information generating unit 63 of the management server 1 generates information (hereinafter referred to as "provided information") to be provided by the management server 1 to the language model server 3, based on various information such as the result of narrowing down by the prior knowledge narrowing down unit 62 and the input information acquired by the input information acquiring unit 60, in order to improve the accuracy of responses to inquiries or requests. The provided information generating unit 63 also transmits the generated provided information to the language model server 3. The specific contents and images of the provided information generated by the provided information generating unit 63 will be described later with reference to FIG. 7 etc.

[0040] The response information management unit 64 of the management server 1 acquires information about a response to an inquiry or request based on the text information or voice information.

[0041] Specifically, the answer information management unit 64 acquires information about an answer to an inquiry or request transmitted from the language model server 3 (hereinafter referred to as “answer information”) via the communication unit 19. In addition, the answer information management unit 64 transmits the acquired answer information to the user terminal 2.

[0042] As shown in FIG. 3, in the control unit 100 of the user terminal 2, an input information management unit 120 and a response information management unit 121 function by executing various programs and the like.

[0043] The input information management unit 120 of the user terminal 2 receives input information related to an inquiry or request from a user. The input information management unit 120 also transmits the received input information to the management server 1.

[0044] The answer information management unit 121 of the user terminal 2 acquires the answer information transmitted from the management server 1 via the communication unit 110. In addition, the answer information management unit 121 displays the acquired answer information on the output unit 130 of the user terminal 2.

[0045] As shown in FIG. 3, in the control unit 160 of the language model server 3, an provided information acquiring unit 180 and a response information generating unit 181 function by executing various programs and the like.

[0046] The provided information etc. acquiring unit 180 of the language model server 3 acquires the provided information transmitted from the management server 1 via the communication unit 170. In addition, the provided information etc. acquiring unit 180 acquires the prior knowledge information transmitted from the management server 1 via the communication unit 170.

[0047] The answer information generation unit 181 of the language model server 3 generates information about an answer to an inquiry or request from a user (hereinafter referred to as "answer information") based on the provided information acquired by the provided information acquisition unit 180. The answer information generation unit 181 transmits the generated answer information to the management server 1.

[0048] Furthermore, the answer information generating unit 181 generates information on an answer to the prior knowledge information acquired by the provided information etc. acquiring unit 180, and transmits the information to the management server 1.

[0049] FIG. 4 is an image diagram for explaining an example of prior knowledge related to the information processing system according to one embodiment of the present invention.

[0050] Here, a supplementary explanation about the large-scale language model is provided. As described above, the large-scale language model is a trained model trained using general-purpose information, so that answers to general terms and words tend to be highly accurate. On the other hand, as described above, learning of in-house information, specialized information, the latest information, etc. tends to be insufficient, making it difficult to provide answers with high accuracy. Therefore, if the large-scale language model is to provide answers with high accuracy to, for example, specialized inquiries or requests or inquiries or requests regarding the latest trends, etc., it is necessary to devise a way to provide the large-scale language model with information that tends to be insufficient.

[0051] In the example of Figure 4, the word ID, category, explanation, and LLM judgment (the result of the language model's judgment as to whether the word is known or unknown) are displayed for the words Rabbit, Rabbit (3rd generation), RB3-A, RB3-B, RB3-C, Ecotoku car, and compact car.

[0052] Specifically, in the example of Figure 4, Rabbit is a unique term used mainly within the company, and the explanation displayed is "Rabbit is a compact car produced and sold by A Motors."

[0053] In addition, in the example of FIG. 4, Rabbit is displayed as unknown in the LLM judgment, which indicates that the answer provided by the large-scale language model for Rabbit did not meet a certain standard, as judged by the above-mentioned prior knowledge verification unit 80.

[0054] In the example of Figure 4, Rabbit (3rd generation) is a unique term used mainly within the company, and the explanation displayed is "The current Rabbit has been on sale since 2023. It is an eco-friendly vehicle." Also, in the example of Figure 4, Rabbit (3rd generation) is unknown in the LLM judgment.

[0055] In the example of Figure 4, RB3-A is a unique term used mainly within the company, and the explanation is displayed as "This is the highest grade of Rabbit (3rd generation)." Also, RB3-A is unknown in LLM judgment.

[0056] In the example of Figure 4, RB3-B is a unique term used mainly within the company, and the explanation is displayed as "It is an intermediate grade of Rabbit (3rd generation)." Also, RB3B is unknown in LLM judgment.

[0057] In the example of Figure 4, RB3-C is a unique term used mainly within the company, and the explanation is displayed as "It is a low-cost grade of Rabbit (3rd generation)." Also, RB3C is unknown in the LLM judgment.

[0058] In the example of Figure 4, Ecotoku vehicle is a term mainly used throughout the industry, and the explanation displayed is "a general term for vehicles that meet environmental performance S standard. The subsidy rate is 50%." Also, Ecotoku vehicle is unknown in LLM judgment.

[0059] On the other hand, in the example of Figure 4, compact car is a term mainly used throughout the industry, and the explanation displayed is "one of the shapes of passenger cars. There is no clear definition, but it is often used for small cars with an overall length of less than 4,400 mm." Also, compact cars are known in the LLM judgment.

[0060] The names and various explanations of words such as Rabbit, Rabbit (3rd generation), RB3-A, RB3-B, RB3-C, Eco-Toku car, and Compact car are fictitious contents created for the purpose of explaining this embodiment. The names and definitions of words to which this system is actually applied, the results of LLM judgment, etc. are at the discretion of the administrator of this system, etc.

[0061] Furthermore, the present system can manage such prior knowledge in a hierarchical manner. Fig. 5 is an image diagram for explaining an example of a hierarchical structure of prior knowledge in an information processing system according to an embodiment of the present invention.

[0062] In the example of FIG. 5, the relationships between the words Rabbit 313, Rabbit (3rd generation) 314, RB3-A (315), RB3-B (316), RB3-C (317), Eco-friendly car 312, and Compact car 311 shown in FIG. 4 are structurally displayed. Specifically, as shown in FIG. 5, Rabbit (3rd generation) 314 corresponds to a superordinate word of RB3-A (315), RB3-B (316), and RB3-C (317). Rabbit 313 corresponds to a superordinate word of Rabbit (3rd generation) 314. Eco-friendly car 312 corresponds to a superordinate word of Rabbit 313. Compact car 311 corresponds to a superordinate word of Rabbit 313. In this way, the system manages each word in a hierarchical manner, thereby making it possible to narrow down prior knowledge. This allows the system to efficiently send only prior knowledge necessary for the answer to the large-scale language model.

[0063] Next, an example of a method in which a user actually makes an inquiry or request using the user terminal 2 will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of an image displayed on a user terminal constituting an information processing system according to one embodiment of the present invention.

[0064] In the example of Fig. 6, a standard message "Please enter your question" is displayed, along with an input message entered by the user, "Please tell me what options are available for Rabbit's audio." In this way, the user can enter any message using the user terminal 2 and confirm the content of the inquiry or request by, for example, pressing the confirm button 131 in Fig. 6.

[0065] In response to this, an example of a specific process performed by the management server 1 in response to such an inquiry or request will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of an image displayed on the management server constituting an information processing system according to one embodiment of the present invention.

[0066] In this embodiment, the RB3-A specification has a description of options on page 10, and this information can be used to generate a sufficient response to the inquiry or request shown in FIG.

[0067] In the example of Figure 7, in response to the inquiry or request in Figure 6, "Please tell me what options are available for Rabbit's audio," specific information regarding each of "#Work instructions" 731, "#Pre-knowledge" 732, "#Reference information" 733, and "#Inquiry content" 734 is displayed as provided information to be sent to the language model server 3.

[0068] The information provided as "#work instructions" 731 includes supplementary instructions that can contribute to improving the accuracy of the answer by the language model server 3, instructions for improving readability for the user, and the like.

[0069] Specifically, in the example in Figure 7, the following work instruction is displayed: "Please answer the inquiry based on the following prior knowledge and product specifications. Finally, please indicate the document that you found helpful in the following format: Reference document: <Document name> If you cannot find information that corresponds to the inquiry from the product specifications, please reply, "Could not find any relevant information."

[0070] The information provided as "#prior knowledge" 732 is necessary to improve the accuracy of the answer of the language model server 3, and includes information on unknown words to the language model server 3. Specifically, in the example of Fig. 7, explanations on the words "rabbit," "rabbit (third generation)," "RB3-A," and "Eco-Toku car" are displayed as prior knowledge.

[0071] An important point is that the present system does not provide information on all unknown words as prior knowledge to the language model server 3, but can narrow down the prior knowledge to be provided. For example, in response to an inquiry or request such as "Please tell me what options are available for Rabbit's audio" in FIG. 6, the present system extracts the word "rabbit" as a candidate for a word that needs to be provided as prior knowledge. As shown in FIG. 4, the word "rabbit" is an unknown word to the language model server 3. In addition, the present system extracts the word "RB3-A" as a candidate for a word that needs to be provided as prior knowledge because the specifications for RB3-A include a description of options. Similarly, the word "RB3-A" is an unknown word to the language model server 3.

[0072] 5, the word "rabbit (3rd generation)" 314 is a superordinate word of the word "RB3-A (315)". The words "eco-friendly car" 312 and "compact car" 311 are superordinate words of the word "rabbit" 313. In general, when the meaning of a word corresponding to a subordinate concept is unknown, it is considered difficult to generate an accurate answer without knowing the meaning of the word corresponding to the superior concept. Therefore, of the superordinate words of each word, "rabbit (3rd generation)" 314 and "eco-friendly car" 312, and "compact car" 311, which are unknown to the language model server 3, are added as candidates for words that need to be provided as prior knowledge.

[0073] On the other hand, RB3-B (316) and RB3-C (317) are words that are subordinate concepts to rabbit 313, and are not superordinate words in relation to any of the candidate words that need to be provided as prior knowledge. Therefore, the present system does not provide information on the words RB3-B (316) and RB3-C (317) as prior knowledge to the language model server 3. By narrowing down the prior knowledge in this way, the present system efficiently provides only necessary information as prior knowledge to the language model server 3.

[0074] The information provided as "#Reference information" 733 includes supplemental information to be referred to depending on the contents of the inquiry or request. In the example of Fig. 7, since the 10th page of the RB3-A specification describes options, information on the RB3-A specification is provided as reference information.

[0075] The information provided as "#inquiry content" 734 includes information on the content of the inquiry or request entered by the user. In the example of Fig. 7, the content of the inquiry or request "Please tell me what options are available for Rabbit's audio" in Fig. 6 is provided as the inquiry content. Note that the work instructions, prior knowledge, reference information, and information on the inquiry content shown in Fig. 7 may all be included in the provided information.

[0076] Next, the contents of the generated answer 740 will be briefly described with reference to Fig. 8. Fig. 8 is a diagram showing an example of an image displayed on a user terminal constituting an information processing system according to an embodiment of the present invention, and is a diagram showing an example different from the example of Fig. 6.

[0077] In the example of FIG. 8, the content of the inquiry or request by the user, "Please tell me what options are available for Rabbit's audio." is displayed, and the following is displayed as a response 740 to the inquiry or request: The following audio options are available for Rabbit: (1) Genuine navigation system A Automobile 100,000 yen This is a car navigation system with a 4-inch monitor. (2) Audio A Car 50,000 Audio only system (3) High-grade set Y Audio 200,000 yen Comes as a set with Y company speaker system. Reference document: RB3-A Product Specification p10 is displayed as an answer 740. Answer information regarding such an answer 740 is generated by the language model server 3 and is ultimately displayed on the output unit 130 of the user terminal 2.

[0078] FIG. 9 is a diagram showing an example of the flow of various processes executed when confirming prior knowledge in the management server 1 constituting the information processing system according to one embodiment of the present invention.

[0079] In step S1, the prior knowledge confirmation unit 80 generates prior knowledge information for making an inquiry or request regarding a predetermined word, and transmits the prior knowledge information to the language model server 3.

[0080] In step S2, the prior knowledge confirmation unit 80 obtains an answer to the prior knowledge information generated by the language model server 3, and if the content of the answer meets a certain criterion, it determines the word to be known, and if the content of the answer does not meet the certain criterion, it determines the word to be unknown.

[0081] In step S3, the hierarchical management unit 81 generates relationships between each of the target words.

[0082] In step S4, the hierarchical management unit 81 hierarchically manages the results of the verification performed by the prior knowledge verification unit 80 for each word and the relationships between the words.

[0083] In step S5, the hierarchical management unit 81 stores the various types of information managed by it as prior knowledge information in the prior knowledge DB 300. This ends the prior knowledge confirmation process.

[0084] FIG. 10 is a diagram showing an example of the flow of various processes executed at the time of an inquiry or request in the management server 1, the user terminal 2, and the language model server 3 that constitute an information processing system according to one embodiment of the present invention.

[0085] In step S21, the input information management unit 120 of the user terminal 2 accepts input information related to an inquiry or request from a user.

[0086] In step S22, the input information management unit 120 of the user terminal 2 transmits the received input information to the management server 1.

[0087] In step S41, the input information acquisition unit 60 of the management server 1 acquires, via the communication unit 19, input information relating to the contents of an inquiry or request transmitted from the user terminal 2.

[0088] In step S42, the prior knowledge narrowing-down unit 62 of the management server 1 determines provided word candidates to be provided to the language model server 3 as part of the provided information, based on the input information acquired by the input information acquisition unit 60 and the contents of the prior knowledge information stored in the prior knowledge DB 300.

[0089] In step S43, the prior knowledge narrowing unit 62 of the management server 1 adds words in a higher hierarchy than the words determined as provided word candidates to the provided word candidates.

[0090] In step S44, the information to be provided generation unit 63 of the management server 1 generates information to be provided by the management server 1 to the language model server 3 based on various information such as the results of the narrowing down by the prior knowledge narrowing down unit 62 and the input information acquired by the input information acquisition unit 60, in order to improve the accuracy of responses to inquiries or requests.

[0091] In step S 45 , the provision information generating unit 63 of the management server 1 transmits the generated provision information to the language model server 3 .

[0092] In step S 61 , the provided information etc. acquiring unit 180 of the language model server 3 acquires the provided information transmitted from the management server 1 via the communication unit 170 .

[0093] In step S62, the answer information generating unit 181 of the language model server 3 generates answer information to the inquiry or request from the user, based on the provided information acquired by the provided information acquiring unit 180.

[0094] In step S 63 , the answer information generation unit 181 of the language model server 3 transmits the generated answer information to the management server 1 .

[0095] In step S46, the answer information management unit 64 of the management server 1 acquires, via the communication unit 19, answer information regarding the answer to the inquiry or request transmitted from the language model server 3.

[0096] In step S47, the answer information management unit 64 of the management server 1 transmits the acquired answer information to the user terminal 2.

[0097] In step S23, the answer information management unit 121 of the user terminal 2 acquires the answer information transmitted from the management server 1 via the communication unit 110.

[0098] In step S24, the answer information management unit 121 of the user terminal 2 displays the acquired answer information on the output unit 130 of the user terminal 2. This completes various processes executed at the time of the inquiry or request in the management server, the user terminal, and the language model server.

[0099] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope that can achieve the object of the present invention are included in the present invention.

[0100] [Other embodiments] Here, we will briefly supplement the techniques used when responding to inquiries or requests using large-scale language models, etc.

[0101] Generally, when responding to inquiries or requests using large-scale language models, a search method called similarity search is used. There are several similarity search methods, including vector-based methods that calculate a numerical representation of a vector called embedding for each divided document and use it as a search index to perform a search, and keyword-based methods such as n-gram, TF-IDF, and BM25. This system can perform various searches by employing any method, including these similarity search methods.

[0102] Although not described in the above embodiment, an inquiry or request does not simply mean a question, but broadly includes requests, suggestions, opinions, etc. Specifically, the above-mentioned concept of an inquiry or request includes a wide range of concepts, such as requests for transcription of audio information or summarization of contents, requests for creating minutes, requests for support in creating catchy slogans or news releases, requests for proofreading of manuals, requests for content analysis of product reviews, etc. In this case, the system may simply transmit instruction information, etc. to the large-scale language model without searching for documents, etc.

[0103] In the above embodiment, the case where there is only one type of large-scale language model has been described as an example, but this is not limited thereto. The present system may be provided by using multiple types of large-scale language models. In this case, for example, the present system may acquire and manage prior knowledge for each type of large-scale language model.

[0104] In the above embodiment (particularly the embodiment in FIG. 5), the system has been described as managing prior knowledge in a hierarchical manner, but this is not limited thereto. The system may manage prior knowledge in a simple structure such as a hyperlink, for example, without hierarchical arrangement.

[0105] In the above embodiment, the present system has been described as using only superordinate or subordinate concepts when managing prior knowledge, but this is not limited thereto. The present system may manage prior knowledge using any concept that expresses the relationship between words, such as constituents, non-constituents, attributes, cognates, etc.

[0106] In the above embodiment, the present system responds to inquiries or requests related to automobiles, but this is an example and is not limited to this. The present system can respond to various inquiries or requests, regardless of the industry or type, such as manufacturing, service, logistics, and administrative procedures.

[0107] Specifically, for example, the present system can be applied in the following manner.

[0108] For example, the present system can be used to aid mutual understanding between clients and users in applications such as advertising and finance.

[0109] For example, the present system can be used in applications such as the pharmaceutical industry and pharmacy operations to assist in business communications with employees who are not licensed as pharmacists.

[0110] For example, in applications in the academic field, the system can assist in the preparation of documents related to research grant applications and grant funding. In addition, the system can be used to assist users in searching for and writing papers in the academic field.

[0111] For example, when applied to customs and quarantine operations, the system can assist with investigating application contents and preparing application documents.

[0112] For example, when applied to administrative fields, the system can assist in the investigation and confirmation of bills.

[0113] For example, the present system can assist in the handover of work between employees in any industry or the like.

[0114] Although the above embodiment has been described simply, one of the advantages of the present system is that it improves the quality of answers even in a finite processing environment such as a large-scale language model by providing the large-scale language model or the like with necessary and sufficient prior knowledge for answering an inquiry or request. Even if the processing environment such as a large-scale language model is relatively large, the present system can improve the calculation efficiency by providing the prior knowledge, thereby reducing the calculation time and cost.

[0115] In the above embodiment, the inquiry, request, and answer are all described as being input or output as text information, but this is not limited thereto. The system may obtain inquiry or request information as voice information, and may output answer information as voice information.

[0116] Furthermore, the above-described series of processes can be executed by hardware or software.

[0117] In other words, the functional configurations in FIG. 3 and the like are merely examples and are not particularly limited.

[0118] That is, it is sufficient that the information processing system is provided with a function capable of executing the above-mentioned series of processes as a whole, and the type of functional block used to realize this function is not limited to the example in Fig. 3 etc. Furthermore, the location of the functional block is not limited to the example in Fig. 3 etc. and may be arbitrary.

[0119] Furthermore, one functional block may be configured as a single piece of hardware, may be configured as a single piece of software, or may be configured as a combination of both.

[0120] In the above embodiment, the various hardware components of the system (e.g., the management server 1, the user terminal 2, and the language model server 3) are described as existing as a single hardware device, but this is not limited thereto. The various hardware components of the system may be configured as multiple hardware devices.

[0121] Furthermore, the various hardware components of this system may be configured to execute various processes via the Web, etc., as in a so-called cloud-type system, and send the results of the processes to the various hardware components.

[0122] Furthermore, when the series of processes is executed by software, the program constituting the software is installed into a computer or the like from a network or a recording medium.

[0123] The computer may be a computer that is built into dedicated hardware, or a computer that is capable of executing various functions by installing various programs.

[0124] That is, for example, any computer, any mobile terminal such as a smartphone, etc. may be freely adopted for the various hardware in the above-described embodiments.

[0125] Furthermore, the recording medium containing such a program may not only be constituted by a removable medium (not shown) provided separately from the device main body in order to provide the program to a user, etc., but may also be constituted by a recording medium, etc. that is provided to the user in a state in which it is pre-installed in the device main body.

[0126] In this specification, the steps of describing the program to be recorded on the recording medium may be executed in parallel or individually, as well as in chronological order. Also, some of the steps described in the above embodiment may be omitted.

[0127] In addition, in this specification, the term "system" refers to an overall device that is composed of a plurality of devices, a plurality of means, etc.

[0128] Even when these other embodiments are adopted, the effects of the present embodiment are exhibited. In addition, the present embodiment can be appropriately combined with other embodiments, and other embodiments can be appropriately combined with each other.

[0129] [Configuration Example of This Embodiment] In other words, it is sufficient for an information processing device to which the present invention is applied to have the following configuration, and various embodiments can be adopted.

[0130] That is, a first aspect of the present invention is an information processing system that can be used to generate a response to a specified inquiry or request, comprising: input information acquisition means (e.g., input information acquisition unit 60) for acquiring text information or voice information related to the inquiry or request; confirmation information acquisition means (e.g., prior knowledge confirmation unit 80) for acquiring information related to confirmation of whether knowledge related to a specified word in a specified language model has reached a certain standard; confirmation information management means (e.g., hierarchical management unit 81) for managing each of the confirmation results of the words; and answer information acquisition means (e.g., answer information management unit 64) for acquiring information related to the answer to the inquiry or request based on the text information or voice information.

[0131] In the first aspect, the answer information acquiring means may acquire information about the answer generated by the language model.

[0132] In the first aspect, the confirmation information management means may manage each of the results of the confirmation of the words, including relationships between the words.

[0133] In addition, in the first aspect, the confirmation information management means may hold the relationships between each of the words as a hierarchical structure, and manage each of the confirmation results for each of the words together with the relationships between each of the words.

[0134] Furthermore, a second aspect of the present invention is an information processing method executed by a computer that can be used to generate an answer to a specified inquiry or request, the information processing method including: an input information acquisition step of acquiring text information or voice information related to the inquiry or request; a confirmation information acquisition step of acquiring information related to confirmation as to whether knowledge related to a specified word in a specified language model has reached a certain standard; a confirmation information management step of managing each of the confirmation results of the word; and an answer information acquisition step of acquiring information related to the answer to the inquiry or request based on the text information or the voice information.

[0135] Furthermore, a third aspect of the present invention is sufficient as long as it is a program that causes a computer that can be used to generate an answer to a specified inquiry or request to execute processing including an input information acquisition step of acquiring text information or voice information related to the inquiry or request, a confirmation information acquisition step of acquiring information related to confirmation as to whether knowledge related to a specified word in a specified language model has reached a certain standard, a confirmation information management step of managing each of the confirmation results of the word, and an answer information acquisition step of acquiring information related to the answer to the inquiry or request based on the text information or voice information.

[0136] [Background and Problems of the Present Embodiment] In recent years, there has been remarkable progress in machine learning technology, and services that generate answers to user inquiries using so-called large language models that are generated by general-purpose learning using large amounts of information are beginning to be provided (e.g., Non-Patent Document 1).

[0137] Conventionally, when responding to inquiries or questions using a chatbot or the like, responses to the questions have been prepared in advance and the response to the question that is closest to the content of the inquiry has been provided. However, the method of preparing responses to inquiries in advance has the problem that the preparation requires a large load in the first place, and it is difficult to provide a correct response to an inquiry that has not been responded to before.

[0138] In response to this, a method that has been attempted in recent years is to utilize so-called large-scale language models, etc. In this method, instead of preparing answers to queries, documents are divided into pieces of a certain size or less (these divided documents are called "chunks"), and by applying techniques such as similarity search to the chunks, it is possible to generate answers with a reasonable degree of accuracy without preparing answers in advance.

[0139] For example, Non-Patent Document 1 discloses a method for utilizing a large-scale language model not only for text-to-text conversion but also with different APIs (see Non-Patent Document 1).

[0140] However, since large-scale language models are models trained using general-purpose training data, they have not yet learned so-called in-house terminology, technical terms, the latest terms, etc. Therefore, if an inquiry contains unknown words or terms, the accuracy of the answer may be extremely degraded. Furthermore, the information (prompts) that can be input (sent) to a large-scale language model is also limited.

[0141] The present embodiment has been made in view of the above circumstances, and aims to provide a technique that can improve the quality of responses to a given inquiry or request.

[0142] [About large-scale language models] Here, the large-scale language model (LLM) is an example of an AI (Artificial Intelligence) model (for example, a machine learning model generated by machine learning). Note that the learning shown in this embodiment means finding a regularity behind a large amount of data based on the data. Also, the AI ​​model generated by the learning shown in this embodiment is generated by various learning algorithms. For example, ChatGPT, Bard, Llama (Large Language Model Meta AI), Gemini, Claude, etc. can be used as the LLM. Note that these are only examples, and other AI models may be used. Also, in this embodiment, the LLM realized by the language model server 3 will be described as LLM 500 (see FIG. 12, etc.).

[0143] [Business knowledge and prior knowledge] Next, the domain knowledge (e.g., industry knowledge), business knowledge, and prior knowledge used in this embodiment will be described. In general, the LLM 500 retains common knowledge, but often does not have specific knowledge (e.g., company-specific knowledge, industry-specific knowledge). For example, in a highly recognized field such as an automobile company, the LLM 500 may have a certain degree of concepts, but the LLM 500 often does not have knowledge of fields that are not general (e.g., manufacturing equipment for a specific industry, etc.).

[0144] Furthermore, even if the LLM500 has knowledge, it is assumed that the knowledge will remain at the time of pre-learning, and therefore the LLM500 will not be able to keep up with the latest trends, such as new concepts and regulatory changes. For this reason, in inquiry response work based on specific knowledge (e.g., specialized documents), it is necessary to provide the LLM500 with domain knowledge (e.g., industry knowledge). In this embodiment, the domain knowledge (e.g., prior knowledge) that needs to be provided to the LLM500 is selected and extracted, making it possible to appropriately pass on the prior knowledge required to the LLM500 and improve the quality of responses.

[0145] Here, business knowledge means content that is specifically known (recognized and understood) to a specific organization. For example, business knowledge means content derived from various sentences and various terms described in documents such as design documents, specifications, and manuals used in the business of the specific organization. The specific organization is, for example, a company. Furthermore, the industry corresponding to the specific organization means the automobile industry if the specific organization is, for example, an automobile manufacturing or sales company.

[0146] Prior knowledge refers to content known to obtain business knowledge, content known to supplement business knowledge (e.g., content that is recognized or understood), etc. For example, prior knowledge refers to the content of various sentences, various terms, etc. that supplement knowledge when reading documents such as design documents, specifications, and manuals used in business in an industry corresponding to a specific organization. In other words, prior knowledge refers to knowledge necessary to interpret internal documents such as design documents that are passed to LLM500 as business knowledge. For example, prior knowledge is assumed to be something like industry-specific knowledge, a glossary of in-house product names, abbreviations, etc. It should be noted that LLM500 is a large-scale language model and corresponds to language model server 3 (see FIG. 1, etc.).

[0147] [Example of business knowledge DB and prior knowledge DB configuration] FIG. 11 is a diagram showing an example of the configuration of the business knowledge DB350 and the prior knowledge DB300 stored in the storage unit 18 (see FIG. 3). FIG. 11(A) shows an example of the configuration of the business knowledge DB350, and FIG. 11(B) shows an example of the configuration of the prior knowledge DB300. FIG. 11(C) shows a simplified version of the contents described on page 10 of the specification (RB3-A) 351. Note that, although FIGS. 3 and 11 show an example in which the business knowledge DB350 and the prior knowledge DB300 are stored in the management server 1 (storage unit 18), they may be stored in an external device of the management server 1, and the management server 1 may obtain the contents of each DB from the external device as necessary. FIG. 11 also shows an example in which knowledge related to a business corresponding to an industry that manufactures and sells automobiles, etc. is set as business knowledge, and knowledge mainly used in A Automobile, which manufactures and sells automobiles, etc., is set as prior knowledge.

[0148] The business knowledge DB 350 stores various pieces of information corresponding to the business knowledge described above. For example, glossary 531, glossary 551, specification (RB-A) 351, etc. are stored. Glossary 531 and glossary 551 contain various terms and explanations of each term related to automobiles manufactured and sold by A Motors. An example of the contents of glossary 531 is shown in FIG. 15. An example of the contents of glossary 551 is shown in FIG. 18.

[0149] Specification (RB3-A) 351 is the specification for RB3-A, which is the highest grade of Rabbit (third generation) manufactured and sold by Motors A. For example, as shown in FIG. 11(C), the 10th page of specification (RB3-A) 351 describes options.

[0150] Note that the prior knowledge DB 300 shown in Fig. 11(B) differs from the configuration shown in Fig. 4 in that it further includes src_id 306 and dst_id 307. That is, the prior knowledge DB 300 is a database in which identification information 301, category 302, term 303, explanation 304, LLM judgment 305, src_id 306, and dst_id 307 are stored in association with each other.

[0151] Identification information 301 is identification information (e.g., ID) for identifying each word. Category 302 is information indicating where each word is mainly used. Fig. 11(B) shows the case where each word is a specific term mainly used within a company (indicating "in-house") and the case where each word is a term mainly used within an industry (indicating "industry").

[0152] The term 303 is a character string corresponding to prior knowledge. The character strings stored in the term 303 are special, technical, etc., and therefore will be described using the word "term", but they may be called words, character strings, etc. instead of terms. A character string means a sequence of characters such as letters, numbers, symbols, and codes.

[0153] Description 304 is a sentence for explaining the term stored in terminology 303. This sentence can be created and stored by a user based on an existing document such as a glossary for explaining the term stored in terminology 303. Note that each explanation may be stored using a sentence creation technique that can automatically generate an explanation of each term included in a document such as a glossary based on each sentence, each character, etc. included in the document. Note that an example of automatically generating information for description 304 using LLM 500 will be described in detail with reference to FIGS. 15 to 21.

[0154] LLM Decision 305 is information indicating the result of a decision as to whether the term stored in Terms 303 is known or unknown in LLM 500. As described above, the information stored in LLM Decision 305 may be stored manually using the results of confirmation by an expert, or may be stored automatically using LLM 500. An example of automatically generating the information for LLM Decision 305 using LLM 500 will be described in detail with reference to FIGS. 12 to 14.

[0155] The src_id306 and the dst_id307 are hierarchical structure information used when each term is managed in a tree structure. The src_id306 means a source ID, and the dst_id307 means a destination ID. That is, among the terms stored in the term 303, the identification information 301 of the term linked to the lower side is stored in the src_id306, and the identification information 301 of the term linked to the upper side is stored in the dst_id307. For example, as shown in FIG. 5, the term linked to the lower side of the rabbit 313 is the rabbit (third generation) 314, so the identification information 301 "2" of the rabbit (third generation) is stored in the src_id306 corresponding to the term 303 "rabbit" of the identification information 301 "1" shown in FIG. 11(B). Similarly, as shown in FIG. 5, the terms linked to the upper side of Rabbit 313 are Compact Car 311 and Ecotoku Car 312, so in dst_id 307 corresponding to term 303 “Rabbit” of identification information 301 “1” shown in FIG. 11(B), the identification information 301 “7” of compact car and the identification information 301 “6” of Ecotoku car are stored.

[0156] In this way, by dividing knowledge into business knowledge and prior knowledge, it is possible to limit the necessary information regarding the business knowledge to be provided to the LLM 500. Note that, although an example of two levels, business knowledge and prior knowledge, is shown in Fig. 11, multiple levels of three or more levels may also be used. Also, by making the prior knowledge hierarchical as described above, it is possible to appropriately provide the minimum necessary prior knowledge to the LLM 500.

[0157] It is also possible to use an LLM with advanced processing capabilities but only general knowledge, and an LLM with a small network size that has learned knowledge in a specialized field. In this case, it is preferable to prepare an item for LLM judgment 305 for each LLM to be used. This makes it possible to efficiently apply the same domain knowledge to multiple LLMs.

[0158] [Example of generating prior knowledge] FIG. 4 and other figures show an example in which confirmation of whether a target term (term 303) is known or unknown in the LLM is performed based on human evaluation. For example, for a candidate's prior knowledge (e.g., the target term "rabbit"), a question such as "Please give a concise explanation of the word rabbit in about 100 characters" is asked to the LLM 500 in advance, and an output result (explanation of the target term "rabbit") is obtained from the LLM 500. In this case, no background information other than the above-mentioned question is given to the LLM 500, and only the knowledge of the LLM 500 is queried.

[0159] Next, the expert checks the prior knowledge (e.g., the target term "rabbit") and the output result from LLM500 (explanation of the target term "rabbit"). In this case, it is checked whether the explanation (output result from LLM500) is appropriate or requires only minor modification. If the explanation (output result from LLM500) is appropriate or requires only minor modification, the output result from LLM500 (explanation of the target term "rabbit") is adopted, and the LLM decision 305 (see FIG. 11(B)) is set to known. On the other hand, if the explanation (output result from LLM500) differs from the expert's judgment, the LLM decision 305 (see FIG. 11(B)) is set to unknown, and the expert sets an appropriate explanation.

[0160] This kind of check can also be performed automatically. Below, we will show an example of automatically checking whether a target term is known or unknown in LLM500.

[0161] 12 and 13 are schematic diagrams showing an example in which the LLM 500 determines whether each term stored in the term 303 (see FIG. 11(B)) of the prior knowledge DB 300 is known or unknown. FIGS. 12 and 13 show an example in which the LLM 500 determines whether the term 303 "rabbit" is known or unknown. Here, an example is shown in which an explanation of the target prior knowledge (for example, the term 303 "rabbit") is input to the LLM 500 using explanation 304, and a question (answer 502) that can be answered with Yes / No is generated.

[0162] 12 shows an example in which a prompt 501 is input to the LLM 500, and the LLM 500 outputs a response 502 in response to this input. The prompt 501 includes "#instruction" and "#sentence". The "#instruction" includes an instruction to create a question that can be answered with Yes / No. Furthermore, the explanation 304 "Rabbit is a compact car produced and sold by A Motors" associated with the term 303 "Rabbit" in the prior knowledge DB 300 is inserted into the "#sentence".

[0163] In this manner, it is possible to obtain an answer 502 from the LLM 500 by inputting a prompt 501 into the LLM 500, the prompt 501 including an explanation 304 associated with a term 303 in the prior knowledge DB 300.

[0164] Figure 13 shows an example in which a prompt 511 including the answer 502 shown in Figure 12 is input to the LLM 500, and the LLM 500 outputs an answer 512 in response to this input. The prompt 511 includes a "#instruction" and a "#question". The "#instruction" includes an instruction to answer the question generated in Figure 12 (a question that can be answered with Yes / No) with Yes / No. Furthermore, the question generated in Figure 12 (answer 502) is inserted into the "#question".

[0165] In this way, by inputting a prompt 511 including an answer 502 obtained from the LLM 500 into the LLM 500, it is possible to obtain an answer 512 from the LLM 500. Based on this answer 512, the prior knowledge verification unit 80 (see FIG. 3) can set the LLM judgment 305 to either known or unknown. For example, if the answer obtained from the LLM 500 is an appropriate answer (e.g., if all questions are answered correctly), the LLM judgment 305 can be set to known. On the other hand, if the answer obtained from the LLM 500 is not an appropriate answer (e.g., if at least one question is answered incorrectly or the answer is unclear), the LLM judgment 305 can be set to unknown.

[0166] In the example shown in Fig. 13, the answer 512 from the LLM 500 includes "No, no, I don't have that information." In other words, two questions are incorrect and one question is an unknown answer. Therefore, in the example shown in Figs. 12 and 13, the LLM for the term 303 "rabbit" is determined to be unknown. In other words, "unknown" is stored in the LLM decision 305.

[0167] In this way, when prior knowledge such as an existing vocabulary list has been prepared and term explanations exist, it is possible to perform LLM judgment automatically. Note that, in cases where prior knowledge terms are listed but no term explanations exist, as will be described later, it is possible to generate term explanations using LLM500, and it is possible to perform LLM judgment automatically using those term explanations.

[0168] [Example of management server operation] Fig. 14 is a flow chart showing an example of a known unknown determination process in the management server 1. This known unknown determination process is executed by the control unit 11 (see Fig. 3) based on a program stored in the storage unit 18 (see Fig. 3). This known unknown determination process is executed, for example, at the timing when a start operation for starting the known unknown determination process is performed. This known unknown determination process will be described with appropriate reference to Figs. 1 to 13.

[0169] In step S601, the prior knowledge verification unit 80 acquires terms to be judged (target terms) from among the terms 303 stored in the prior knowledge DB 300. For example, the prior knowledge verification unit 80 acquires terms whose information is not stored in the LLM judgment 305 from among the terms 303 stored in the prior knowledge DB 300 in the order of the identification information 301.

[0170] In step S602, the prior knowledge confirmation unit 80 acquires the explanation 304 stored in the prior knowledge DB 300 in association with the target term acquired in step S601. For example, when "rabbit" is acquired as the target term, the explanation 304 stored in "rabbit" is acquired, which is "Rabbit is a compact car produced and sold by A Motors."

[0171] In step S603, the prior knowledge confirmation unit 80 inputs a prompt to the LLM 500 to generate multiple sets of questions that can be answered with Yes / No and their answers regarding the explanation acquired in step S602, and acquires the answers from the LLM 500. For example, the prior knowledge confirmation unit 80 inputs the prompt 501 shown in Fig. 12 to the LLM 500, and acquires the answers 502 from the LLM 500. Note that the answers 502 are held in memory because they are used in the determination process in step S605.

[0172] In step S604, the prior knowledge confirmation unit 80 inputs a prompt including an instruction to answer with Yes / No to the question (a question that can be answered with Yes / No) included in the answer from the LLM 500 acquired in step S603, to the LLM 500, and acquires the answer from the LLM 500. For example, the prior knowledge confirmation unit 80 inputs prompt 511 shown in FIG.

[0173] In step S605, the prior knowledge verification unit 80 judges whether or not all of the answers from the LLM 500 acquired in step S604 are correct. If all of the answers from the LLM 500 are correct, the process proceeds to step S606. On the other hand, if all of the answers from the LLM 500 are not correct (i.e., at least some of the answers are incorrect or unclear), the process proceeds to step S607. For example, the answer (Yes / No) included in the answer 502 from the LLM 500 (see FIG. 12) is "Yes, No, Yes." Also, the three answers included in the answer 512 from the LLM 500 (see FIG. 13) are "No, No, I don't have that information." Also, when these answers are compared, the answers that match are correct, and the answers that do not match are incorrect. In the example shown in FIG. 13, of the three answers "No, No, I don't have that information" included in answer 512, only the second one is correct ("No"), the first one is incorrect ("Yes" and "No"), and the third one is an unknown answer. In this case, it is determined that all of the answers from LLM500 are incorrect, and the process proceeds to step S607. Note that the determination criteria shown here are merely examples, and the determination process may be performed using other determination criteria. For example, if answers from LLM500 acquired in step S604 that are equal to or greater than a threshold are correct, it may be determined that the target term is known to LLM500.

[0174] In step S606, the prior knowledge checking unit 80 determines that the target term is known to the LLM 500.

[0175] In step S607, the prior knowledge checking unit 80 determines that the target term is unknown to the LLM 500.

[0176] In step S608, the prior knowledge checking unit 80 stores the known / unknown determination result of the LLM 500 for the target term in the LLM determination 305 of the prior knowledge DB 300. For example, when the target term “rabbit” is determined to be “unknown”, the prior knowledge checking unit 80 stores “unknown” in the LLM determination 305 associated with the term 303 “rabbit”.

[0177] [Example of building a knowledge graph] FIG. 5 shows an example in which each term related to prior knowledge is managed in a hierarchical structure. Specifically, an example is shown in which each term related to prior knowledge is organized into a network structure including relationships. In this case, relationships are maintained between higher-level, lower-level, and other entities (terms). Here, an entity means a unique expression such as a proper noun. FIG. 5 etc. also shows an example in which, for information that LLM500 does not understand, that information is used as a reference to trace back to the higher-level information, and each piece of information at the higher level is passed to LLM500.

[0178] Here, a description will be given with reference to FIG. 5. For example, assume that "RB3-A" 315 appears in a sentence to be input to the LLM 500. In this case, the terms (entities) higher than "RB3-A" 315 are traced. In the example shown in FIG. 5, "Rabbit (3rd generation)" 314, "Rabbit" 313, "Eco-Toku Car" 312, and "Compact Car" 311 exist as terms (entities) higher than "RB3-A" 315. As described above, regarding whether or not the LLM 500 recognizes each entity (term related to prior knowledge), a previously confirmed judgment result is stored in the LLM judgment 305 (see FIG. 11(B)). With reference to FIG. 11(B), the term 303 for which "known" is stored in the LLM judgment 305 is only "compact car" 311, and the others are "unknown." In this way, even if the term "RB3-A" 315 with an LLM judgment of 305 "unknown" appears in a sentence to be input to LLM 500, by tracing upward using that term as a reference, a term with an LLM judgment of 305 "known" will be reached. In other words, for each term related to prior knowledge, by tracing upward using the target term as a reference, a concept understood by LLM 500 may be reached.

[0179] In the example shown in FIG. 5, when the target term is "RB3-A" 315, an entity with LLM judgment 305 "known" such as "compact car" 311 is reached. Or, an entity with no information registered above it such as "eco-friendly car" 312 is reached. In this way, when an entity with LLM judgment 305 "known" is reached or an entity with no information registered above it is reached, each piece of information related to each entity ("Rabbit (3rd generation)" 314, "Rabbit" 313, "compact car" 311, or "eco-friendly car" 312) up to the target term "RB3-A" 315 as a reference ("compact car" 311, "eco-friendly car" 312) is passed to the LLM 500. By passing information (e.g., explanation 304) related to a term that the LLM 500 understands and a term at a lower level (target term) to the LLM 500, the LLM 500 can understand the term at the lower level. In this way, by passing only information (eg, description 304) related to the target term and higher-level terms related thereto to LLM 500, the amount of information passed to LLM 500 can be minimized.

[0180] [Example of automatic hierarchy of terms] As described above, if the task of structuring each term into a hierarchy is performed manually, the task becomes cumbersome. Therefore, the following shows an example of generating information for structuring each term into a hierarchical structure using LLM500.

[0181] For example, it is possible to create a knowledge graph using the LLM500. Here, a knowledge graph is a graph that represents various information using a graph structure. This knowledge graph is composed of nodes (e.g., entities) and edges (e.g., relationships between entities) that connect the nodes.

[0182] 15 and 16 are diagrams showing a schematic flow of creating a knowledge graph for each term included in glossary 531 using LLM 500.

[0183] FIG. 17 is a diagram showing a display example of a knowledge graph generated using the LLM 500.

[0184] 15 shows an example of input data 530 input to the LLM 500. The input data 530 includes a glossary 531 and a prompt 532.

[0185] Glossary 531 is an example of a document of business knowledge (such as a specification). For example, glossary 531 is an example of a glossary related to automobiles produced and sold by A Motors. Glossary 531 is stored in business knowledge DB 350 (see FIG. 11(A)). For example, glossary 531 including each term and its explanation can be used.

[0186] Prompt 532 is an example of a prompt for using LLM 500 to create a knowledge graph for each term included in glossary 531, and includes "#instruction" and "#output format." For example, "#instruction" includes an instruction to create a knowledge graph. Furthermore, "#output format" includes instructions to output in JSON (JavaScript (registered trademark) Object Notation), to give nodes id, name, and description attributes, to give edges src_id, dst_id, and relation attributes, and to make edges move from the concrete to the abstract of concepts.

[0187] JSON is a notation that allows data in which multiple elements are combined in a complex structure to be written concisely. Markdown notation, etc., may also be used in addition to JSON.

[0188] The id is identification information for identifying each node. For example, a serial number can be used as the id. The name is information indicating the entity corresponding to each node. For example, each term included in the glossary 531 is extracted as the name. The description is information indicating the explanation (term explanation) of the entity corresponding to each node. For example, the term 303 stored in the prior knowledge DB 300 (see FIG. 11(B)) corresponds to the name, and the explanation 304 corresponds to the description.

[0189] src_id means the source ID (ID of the source of the relationship). Also, dst_id means the destination ID (ID of the destination of the relationship). That is, the node (node ​​ID) pointed to by the edge can be identified by src_id and dst_id. Also, the hierarchical relationship of each node can be identified by src_id and dst_id. For example, src_id 306 stored in the prior knowledge DB 300 (see FIG. 11(B)) corresponds to src_id, and dst_id 307 corresponds to dst_id.

[0190] In addition, by using prompt 532 to draw an edge in a direction from the concrete (lower) to the abstract (higher) concept, an arrow is generated that points from the lower concept term to the higher concept term, as shown in FIG. 17.

[0191] In this manner, when input data 530 including glossary 531 and prompt 532 is input to LLM 500, output data 534 (see FIG. 16) for input data 530 is output from LLM 500.

[0192] Fig. 16 shows an example of output data 534 for input data 530. The output data 534 includes each term (name) extracted from glossary 531, its associated attribute information (description), and edge information (src_id, dst_id, relation) associated with them. Fig. 17 shows graph information displayed based on output data 534.

[0193] 17, the relationship between the terms 541 to 546 can be visually and easily understood, which makes it possible to easily check the terms included in the glossary 531.

[0194] In this way, it is possible to automate the process of hierarchically organizing the terms in the prior knowledge DB 300. It is also possible to automate the process of storing information in each of the terms 303 and descriptions 304 in the prior knowledge DB 300.

[0195] [Example of judgment of term description] 16 and the corresponding explanations can be stored in the terms 303 and explanations 304 of the prior knowledge DB 300. However, it is preferable that these terms and explanations are evaluated by an expert or using the LLM500 or the like, and are appropriately corrected based on the evaluation results. Therefore, an example of evaluating the terms, explanations, etc. included in the output data 534 will be shown below.

[0196] For example, a prompt to generate an explanation for each term (node) included in the output data 534 is input to the LLM 500, and the output result is obtained. For example, for the term "rabbit" included in the output data 534, a prompt including a question such as "Please give a complete explanation of the word rabbit in about 100 characters" is input to the LLM 500, and the output result is obtained. In this case, it is preferable not to provide background information such as the glossary 531 to the LLM 500, and to only ask about the knowledge of the LLM 500. In other words, it is preferable to have the LLM 500 generate a term explanation for each term without prior knowledge.

[0197] Next, the term and the corresponding explanation included in the output data 534 are compared with the output result (explanation of the term) in response to a prompt to generate an explanation for the term included in the output data 534, and it is possible to determine whether the term and the corresponding explanation included in the output data 534 are appropriate or not based on the degree of match, similarity, etc. If they match, or if the degree of similarity is equal to or greater than a reference value (i.e., the degree of similarity is high), it is possible to determine that the term and the corresponding explanation included in the output data 534 are appropriate. On the other hand, if the degree of similarity is less than the reference value (i.e., the degree of similarity is low), it is possible to determine that the term and the corresponding explanation included in the output data 534 are not appropriate and require correction. For this similarity determination process, a known determination process can be used.

[0198] Also, an expert may check the term "rabbit" and the output result from LLM 500 (explanation of rabbit) and make a judgment. For example, if the expert judges that the explanation of rabbit, which is the output result from LLM 500, is appropriate or requires only minor correction, that explanation is adopted. That is, the term "rabbit" is stored in terms 303 of prior knowledge DB 300, and the output result from LLM 500 (explanation of rabbit) is stored in explanation 304 corresponding to this "rabbit."

[0199] On the other hand, if the explanation of the rabbit, which is the output result from the LLM 500, differs from the expert's judgment, the expert will correct the explanation or create a new appropriate explanation to set a valid explanation for the rabbit. In this way, it is possible for the expert to extract explanations with different meanings as explanations for the rabbit and set a valid explanation. Also, if an explanation with a different meaning is extracted by the expert, it may be determined that the LLM 500 does not understand (learn) the term corresponding to that explanation, and "unknown" may be stored in the LLM judgment 305 for that term.

[0200] In the above, an example in which an expert checks a term (e.g., rabbit) and an output result from the LLM 500 (e.g., a description of rabbit) has been described, but the present invention is not limited to this. For example, if a database (e.g., including a term, its description, etc.) corresponding to the glossary 531 exists, the contents of the database (e.g., a description of rabbit) corresponding to the term (e.g., rabbit) may be matched with the output result from the LLM 500 (e.g., a description of rabbit) to determine the similarity between the two. In this case, a known character matching process may be adopted. Even if a database corresponding to the glossary 531 does not exist, the contents of the glossary 531, a term (e.g., rabbit), and an output result from the LLM 500 (e.g., a description of rabbit) may be matched with each other to determine the similarity between the two. In this case, for example, a matching process using an LLM, a known character matching process, etc. may be adopted.

[0201] 18 and 19 are diagrams that show a schematic flow of creating a knowledge graph for each term included in glossary 551 using LLM 500. Also, FIG. 20 is a diagram showing a display example of a knowledge graph created using LLM 500. Note that the examples shown in FIGS. 18 to 20 are examples in which glossary 551 is used instead of glossary 531 in FIGS. 15 to 17, and other parts correspond to FIGS. 15 to 17. For this reason, explanations regarding FIGS. 18 to 20 will be omitted here.

[0202] [Example of management server operation] Fig. 21 is a flowchart showing an example of prior knowledge generation processing in the management server 1. This prior knowledge generation is executed by the control unit 11 (see Fig. 3) based on a program stored in the storage unit 18 (see Fig. 3). This prior knowledge generation is executed, for example, at the timing when a start operation for starting the prior knowledge generation processing is performed. This prior knowledge generation processing will be described with appropriate reference to Figs. 1 to 20.

[0203] In step S611, the prior knowledge confirmation unit 80 acquires industry knowledge information. For example, the prior knowledge confirmation unit 80 acquires information (e.g., glossary 531, glossary 551) necessary for prior knowledge generation from among the various pieces of information stored in the business knowledge DB 350.

[0204] In step S612, the prior knowledge confirmation unit 80 inputs a prompt to the LLM 500 instructing the extraction of terms included in the industry knowledge information acquired in step S611 and the generation of graph information showing the relationships between the terms, and acquires a response from the LLM 500. For example, as shown in Figures 15 and 16, input data 530 including a glossary 531 is input to the LLM 500, and output data 534 is acquired as a response from the LLM 500.

[0205] In step S613, the prior knowledge confirmation unit 80 inputs a prompt to the LLM 500 to generate an explanation for the term included in the answer obtained in step S612, and obtains an answer from the LLM 500. For example, a prompt to generate an explanation for each term (node) included in the output data 534 shown in Fig. 16 is input to the LLM 500, and the output result is obtained. In this case, it is preferable not to provide background information such as a glossary to the LLM 500, and to only ask about the knowledge of the LLM 500.

[0206] In step S614, the prior knowledge verification unit 80 performs an evaluation process to compare the explanation of the term included in the answer from the LLM 500 acquired in step S612 with the explanation of the term included in the answer from the LLM 500 acquired in step S613, and evaluates the explanation for each term. In this case, as described above, it is possible to compare the explanations to determine the similarity between the two, and evaluate the explanation for each term based on the similarity. In addition, the contents of the database corresponding to each glossary, each term, and the explanation of each term may be matched to determine the similarity between the two, or a matching process using an LLM, a known character matching process, or the like may be adopted. In addition, at least a part of the content may be checked by an expert, and the confirmation result may be reflected.

[0207] In step S615, the prior knowledge verification unit 80 judges whether or not all of the explanations for each term are equal to or higher than the standard in the evaluation process in step S614. If all of the explanations for each term are equal to or higher than the standard, the process proceeds to step S616. On the other hand, if at least some of the explanations for each term are below the standard, the process proceeds to step S617.

[0208] In step S616, the prior knowledge verification unit 80 stores each term evaluated in the evaluation process in step S613 and the explanation for each term in the term 303 and explanation 304 in the prior knowledge DB 300, respectively.

[0209] In step S617, the prior knowledge confirmation unit 80 presents to the user (e.g., an expert) an explanation of the term determined to be below the standard in the evaluation process in step S613. Note that, in the evaluation process in step S613, each term determined to be above the standard and the explanation for each term may be stored in the term 303 and explanation 304 in the prior knowledge DB 300. Also, each term determined to be below the standard and presented to the user (e.g., an expert) and the explanation for each term can be stored in the term 303 and explanation 304 in the prior knowledge DB 300 after being confirmed, corrected, etc. by the user.

[0210] In this way, when storing information in each of the terms 303 and explanations 304 in the prior knowledge DB 300, the work can be automated.

[0211] [Example of LLM's response to user's question] FIG. 22 is a diagram showing an example of the case where the LLM 500 outputs a response 740 to a question 710 of a user U1. FIG. 22 shows an example of the case where the LLM 500 outputs a response 740 to a question 710 of a user U1, "Please tell me what options are available for Rabbit's audio." Here, the description will be made with reference to the sequence chart shown in FIG. 10. The information group 720 stored in the business knowledge DB 350 is, for example, the glossaries 531 and 551 and the specification (RB3-A) 351 shown in FIG. 11(A). FIG. 10 shows an example of obtaining a response to an inquiry or request from a user from the language model server 3, but here, as in the example shown in FIG. 22, an example of obtaining a question 710 from a user U1 from the LLM 500 (language model server 3) is shown. Note that the question includes a concept equivalent to the inquiry or request.

[0212] In step S21, the input information management unit 120 of the user terminal 2 accepts input information regarding a question 710 from the user U1, "Please tell me what options are available for Rabbit's audio." For example, the question 710 from the user U1 is accepted by inputting text information, voice information, or the like.

[0213] In step S22, the input information management unit 120 of the user terminal 2 transmits the received input information to the management server 1.

[0214] In step S41, the input information acquisition unit 60 of the management server 1 acquires the input information transmitted from the user terminal 2 via the communication unit 19.

[0215] In step S42, the prior knowledge narrowing unit 62 of the management server 1 uses the input information acquired by the input information acquiring unit 60 and the prior knowledge information stored in the prior knowledge DB 300 to determine provided word candidates to be provided to the language model server 3 as part of the prompt (provided information). For example, the term "rabbit" included in the question 710 "Please tell me what options are available for Rabbit's audio" is extracted as a provided word candidate from among the terms 303 stored in the prior knowledge DB 300.

[0216] In step S43, the prior knowledge narrowing unit 62 of the management server 1 adds words in a higher hierarchy than the words determined as the provided word candidates to the provided word candidates. For example, a higher-level term "Eco-Toku Car" linked to the term "Rabbit" extracted as the provided word candidate in step S42 is added to the provided word candidates. As described later, the term "RB3-A" related to the term "option" included in the question 710 "Please tell me what options can be selected for the audio of Rabbit," and higher-level terms "Rabbit (3rd generation)," "Rabbit," and "Eco-Toku Car" linked to the term "RB3-A" may be added to the provided word candidates.

[0217] In step S44, the information to be provided generation unit 63 of the management server 1 generates a prompt (information to be provided) to be provided by the management server 1 to the language model server 3 based on various information such as the result of narrowing down by the prior knowledge narrowing down unit 62 and the input information acquired by the input information acquisition unit 60, in order to improve the accuracy of the response to the inquiry or request.

[0218] Specifically, the information to be provided generating unit 63 determines a prompt 730 to be input to the LLM 500, using the result of the narrowing down by the prior knowledge narrowing down unit 62, the prior knowledge information stored in the prior knowledge DB 300, and the business knowledge information stored in the business knowledge DB 350. The prompt 730 shown in Fig. 22 corresponds to the prompt 730 shown in Fig. 7, and is shown with some of the information in the rectangle omitted.

[0219] As shown in FIG. 7, the prompt 730 includes "#work instructions" 731, "#prior knowledge" 732, "#reference information" 733, and "#inquiry content" 734. "#work instructions" 731 stores specific instructions for generating an answer to a question 710 from a user U1. For example, "#work instructions" 731 is a standard prompt for an inquiry response task. Note that the prompt shown in FIG. 7 is an example and is not limited to this. Other prompts capable of generating an answer to the question 710 from a user U1 may also be used.

[0220] "#inquiry" 734 stores a question 710 from user U1, "Please tell me what options are available for the Rabbit's audio." Note that the inquiry content is not limited to this. If car company A is an automobile company that sells a variety of car models, from sports cars to SUVs and commercial vans, it is possible to apply this to inquiries related to these. In addition, for example, it is possible to apply this to inquiries in various fields, not limited to the manufacturing industry, from the service industry to government. In other words, this embodiment is not limited to a specific industry.

[0221] The term 303 and the explanation 304 stored in the prior knowledge DB 300 are stored in "#prior knowledge" 732. As described above, the term "rabbit" included in the question 710 "Please tell me what options can be selected for the audio of Rabbit." is extracted from the terms 303 stored in the prior knowledge DB 300, and this term "rabbit" and the higher-level term "Eco-Toku Car" connected to the term "rabbit" are identified as candidates for provided words. Therefore, the term "rabbit" and the explanation of "rabbit" and the term "Eco-Toku Car" and the explanation of "Eco-Toku Car" are stored in "#prior knowledge" 732.

[0222] As described in FIG. 7, when the question 710 "Please tell me what options are available for the Rabbit audio system," includes a term related to information stored in the business knowledge DB 350, the term 303 related to the term may be extracted. For example, the 10th page (see FIG. 11(C)) of the specification (specification (RB3-A) 351) of RB3-A stored in the business knowledge DB 350 includes a description of the term "option" included in the question 710 "Please tell me what options are available for the Rabbit audio system." In this case, the term "RB3-A" related to the term "option" is extracted from the terms 303 stored in the prior knowledge DB 300, and the term "RB3-A" and the higher-level terms "Rabbit (3rd generation)," "Rabbit," and "Eco-friendly car" connected to the term "RB3-A," and their respective explanations are identified. Therefore, the terms “RB3-A”, “Rabbit (third generation)”, “Rabbit”, and “Eco-Toku vehicle”, along with their respective descriptions, are stored in “#prior knowledge” 732.

[0223] "#Reference information" 733 includes supplementary information (context) to be referred to in relation to the question 710 "Please tell me what options are available for Rabbit's audio." As described above, the specification (specification (RB3-A) 351) of RB3-A stored in the business knowledge DB 350 includes a description of the term "options" included in the question 710 "Please tell me what options are available for Rabbit's audio." In this case, the specification (RB3-A) 351 stored in the business knowledge DB 350 is stored in "#Reference information" 733. However, in this case, it is preferable to store only the relevant part of the specification (RB3-A) 351 (for example, the part that describes the options for audio). Note that, for the relevant part, it is possible to extract a sentence part that includes descriptions such as "audio" and "options" by using a known character recognition process or the like.

[0224] 3, the prior knowledge narrowing down unit 62 and the provided information generation unit 63 are configured as different processing units, and in FIG. 10, the prior knowledge narrowing down unit 62 determines a provided word candidate in step S42, adds a word in a higher hierarchy than the word determined as the provided word candidate by the prior knowledge narrowing down unit 62 to the provided word candidates in step S43, and the provided information generation unit 63 generates a prompt (provided information) in step S44, but this is not limiting. For example, the prior knowledge narrowing down unit 62 and the provided information generation unit 63 may be configured as a single processing unit (control unit), and each process of steps S42 to S44 may be executed as a series of processes by the control unit (prior knowledge narrowing down unit 62, provided information generation unit 63).

[0225] In step S45, the provision information generating unit 63 of the management server 1 transmits the generated prompt (provision information) to the language model server 3 (LLM 500).

[0226] In step S61, the provided information etc. acquiring unit 180 of the language model server 3 acquires the prompt (provided information) transmitted from the management server 1 via the communication unit 170.

[0227] In step S62, the answer information generating unit 181 of the language model server 3 generates answer information (answer 740) to the question 710 from the user U1, based on the prompt (provided information) acquired by the acquired unit 180 for provided information.

[0228] In step S63, the answer information generation unit 181 of the language model server 3 transmits the generated answer information (answer 740) to the management server 1.

[0229] In step S46, the answer information management unit 64 of the management server 1 acquires the answer information (answer 740) transmitted from the language model server 3 via the communication unit 19.

[0230] In step S47, the answer information management unit 64 of the management server 1 transmits the acquired answer information (answer 740) to the user terminal 2.

[0231] In step S23, the answer information management unit 121 of the user terminal 2 acquires the answer information (answer 740) transmitted from the management server 1 via the communication unit 110.

[0232] In step S24, the answer information management unit 121 of the user terminal 2 displays the acquired answer information (answer 740) on the output unit 130 of the user terminal 2. For example, the answer 740 is displayed on the output unit 130 as shown in FIG.

[0233] [Examples of Effects of This Embodiment] As mentioned above, if it is not clear that "Rabbit" is the name of a car model, the quality of the LLM500's response to the question "Please tell me what options are available for the Rabbit's audio system" may be significantly degraded. Therefore, it is possible to pass all prior knowledge about the many car models handled by A Motors to the LLM500 along with the question. However, since the amount of information that can be passed to the LLM500 is limited, it is difficult to pass all prior knowledge about the many car models to the LLM500.

[0234] Therefore, in this embodiment, by extracting prior knowledge within a range necessary for answering a question, it is possible to pass necessary prior knowledge even within a limited amount of information that can be passed to the LLM500, and the quality of the answer can be improved. For example, it is possible to confirm prior knowledge that the LLM500 does not grasp in advance to the LLM500. In other words, it is possible to confirm in advance whether the LLM500 can correctly answer each piece of prior knowledge. Then, it is possible to pass only prior knowledge that the LLM500 does not grasp to the LLM500, and it is possible to make prompt generation more efficient. In this case, it is possible to extract prior knowledge that the LLM500 does not grasp in a chain reaction and pass it to the LLM500 based on the hierarchical structure information. Also, for example, when the LLM500 is changed or the LLM500 is upgraded, it is possible to follow the evolution of the LLM500 by re-verifying prior knowledge that is not grasped against the LLM500.

[0235] Furthermore, LLM500 may require a long processing time and calculation cost depending on the amount of information in the prompt. For this reason, even if the amount of information that can be passed to LLM500 is large, for example, it is possible to improve the calculation efficiency of LLM500 and reduce the cost of calculations by extracting prior knowledge within the range necessary to answer the question.

[0236] In this embodiment, it is possible to automate the process of hierarchically classifying the terms in the prior knowledge DB 300. It is also possible to automate the process of storing information in each of the terms 303 and descriptions 304 in the prior knowledge DB 300.

[0237] [Example of executing processing on other devices or systems] In the above, an example was shown in which the extraction process, control process, etc. were executed in the management server 1 (an example of an information processing device), but all or part of each process may be executed in other devices. In this case, an information processing system is configured by each device that executes part of each process. For example, at least part of each process can be executed using various information processing devices such as a server, a device that can be used by a user (e.g., a smartphone, a tablet terminal, a personal computer), a server that can be connected via a predetermined network such as the Internet, and various electronic devices.

[0238] Furthermore, a part (or the whole) of the information processing system capable of executing the functions of the management server 1, etc. may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0239] [Configuration Example of This Embodiment and Its Effects] Here, the configurations of the management server 1, the user terminal 2, and the language model server 3 can be appropriately combined as necessary in addition to the combinations described above. Therefore, the configuration examples shown below may also be configured in consideration of various combinations.

[0240] The management server 1 (an example of an information processing device) is an information processing device capable of acquiring answer data (e.g., answer 740 shown in FIG. 8 ) to question data (e.g., “Please tell me what options are available for Rabbit’s audio?”) input to the language model server 3 (or LLM 500) (an example of a large-scale language model) from the language model server 3 (or LLM 500). The management server 1 includes a prior knowledge narrowing unit 62 that extracts a first character string “rabbit” associated with an LLM judgment 305 “unknown” (an example of judgment information that does not reach the criterion) from among the terms (e.g., rabbit, audio, option) (an example of a character string) included in the question data “Please tell me what options are available for Rabbit’s audio?” in a prior knowledge DB 300 (an example of a database) that associates and holds an LLM judgment 305 (an example of judgment information) indicating whether or not knowledge of a term 303 (an example of a predetermined character string) in the language model server 3 (or LLM 500) has reached a certain criterion, the term 303, and its explanation 304 (an example of explanation information). The information generating unit 63 and the answer information managing unit 64 (an example of a control unit) input a prompt 730 (an example of input data) including the first character string "rabbit" and the explanation 304 (an example of explanatory information) "Rabbit is a compact car produced and sold by A Motors" associated with the first character string "rabbit" in question data "Please tell me what options can be selected for audio equipment of Rabbit" to the language model server 3 (or LLM500) and acquire answer data (for example, answer 740 shown in FIG. 8) to the question data "Please tell me what options can be selected for audio equipment of Rabbit" from the language model server 3 (or LLM500). For example, as shown in FIG. 7 and FIG. 22, it is possible to input the prompt 730 to the language model server 3 (or LLM500) and acquire answer data (for example, answer 740 shown in FIG. 8) to the question data "Please tell me what options can be selected for audio equipment of Rabbit". The information processing method according to the present embodiment is an information processing method including each of these processes. Moreover, the program according to this embodiment is a program that causes a computer to execute each of these processes.In other words, the program according to this embodiment is a program that causes a computer to realize each function that the management server 1 can execute.

[0241] According to this configuration, by extracting prior knowledge within the range necessary to answer a question, it is possible to pass on the necessary prior knowledge even within the finite amount of information that can be passed on to the language model server 3 (or LLM 500), thereby improving the quality of the answer.

[0242] Furthermore, the prior knowledge DB300 (an example of a database) may hold hierarchical structure information (src_id306, dst_id307) showing the relationship between a plurality of terms 303 (an example of a character string) in a tree structure in association with each term 303 (an example of a character string). The prior knowledge narrowing unit 62 (an example of an extracting unit) extracts the term "Eco-Toku Car" (an example of one or more second character strings) associated with the LLM judgment 305 "Unknown" (an example of judgment information) in which knowledge of the term does not reach a certain standard from among the character strings (Eco-Toku Car, Compact Car) linked to the upper side of the term "Rabbit" based on the extracted term "Rabbit" (an example of a first character string) in the tree structure of the prior knowledge DB300. The provision information generating unit 63 (an example of a control unit) may include the term "Rabbit" and the term "Eco-Toku Car" and the explanation 304 (an example of explanation information) associated with each of the term "Rabbit" and the term "Eco-Toku Car" in the prompt 730 (an example of input data). For example, as shown in Figures 7 and 22, a prompt 730 can be input to the language model server 3 (or LLM 500) to obtain answer data (e.g., answer 740 shown in Figure 8) to the question data "Please tell me what options are available for Rabbit's audio."

[0243] According to this configuration, by extracting prior knowledge that LLM500 does not grasp based on hierarchical structure information, it is possible to appropriately pass necessary prior knowledge to the language model server 3 (or LLM500) even within the finite amount of information that can be passed to the language model server 3 (or LLM500), thereby improving the quality of answers.

[0244] The above-mentioned character strings may be terms used in a specific field (for example, an industry corresponding to a specific organization (such as a company) (for example, the automobile industry if the specific organization is an automobile manufacturing or sales company)). The tree structure of the prior knowledge DB300 may be a tree structure in which abstractions in the specific field are at the upper level and concreteness is at the lower level. The prior knowledge narrowing unit 62 (an example of an extraction unit) extracts the term "Eco-Toku Car" (an example of the second character string) from the character strings at the upper level to which the extracted term "Rabbit" (an example of the first character string) belongs in the tree structure of the prior knowledge DB300.

[0245] According to this configuration, by extracting terms in a specific field that LLM500 does not understand based on hierarchical structure information, it is possible to appropriately pass necessary terms even within the limited amount of information that can be passed to the language model server 3 (or LLM500), thereby improving the quality of answers.

[0246] The prior knowledge narrowing unit 62 (an example of an extraction unit) may extract specific information (e.g., page 10 of the specification of RB3-A in which the term "option" is described (see FIG. 11(C))) related to a term (an example of a character string) (e.g., Rabbit, Audio, Option) included in the question data "Please tell me what options can be selected for the audio of Rabbit" in the business knowledge DB350 (an example of a specific information database) that holds specific information used in a specific field. The provision information generating unit 63 (an example of a control unit) may include the first character string "Rabbit", the explanation 304 (an example of explanatory information) associated with the first character string "Rabbit" and the first character string "Rabbit is a compact car produced and sold by A Motors", and specific information related to the first character string "Rabbit" (e.g., page 10 of the specification of RB3-A) in the prompt 730 (an example of input data).

[0247] According to this configuration, by including in the prompt 730 specific information related to terms in a specific field that the language model server 3 (or LLM 500) does not understand, the language model server 3 (or LLM 500) can understand the terms in the specific field and specific information related to them, thereby improving the quality of the response from the language model server 3 (or LLM 500).

[0248] A prompt 501 (see FIG. 12) (an example of first question data) for creating a question for explanatory information of a term to be determined (an example of a character string term to be determined) (e.g., "rabbit"), which is a term to be determined (an example of a character string), is input to the large-scale language model, and an answer 502 (an example of answer data) for the prompt 501 is obtained from the language model server 3 (or LLM 500), and a response is made to the question included in the answer 502 (e.g., Is rabbit a car? Yes, ..., Is rabbit produced and sold by A Motors? Yes). The language model server 3 (or LLM 500) may further include a prior knowledge confirmation unit 80 (an example of a judgment information setting unit) that inputs a prompt 511 (see FIG. 13) (an example of second question data) to the language model server 3 (or LLM 500) to obtain an answer 512 (an example of answer data) to the prompt 511 from the language model server 3 (or LLM 500) and sets an LLM judgment 305 (an example of judgment information) of the judgment target term (e.g., "rabbit") based on the answer result (e.g., No, No, I don't have that information) included in the answer 512. In the example shown in FIGS. 12 and 13, "unknown" is set as the LLM judgment 305 of the judgment target term "rabbit."

[0249] According to this configuration, when setting the LLM judgment 305 that indicates whether or not knowledge of the term 303 (an example of a predetermined character string) reaches a certain standard in the language model server 3 (or the LLM 500), it is possible to automate the setting work. This makes it possible to set the LLM judgment 305 quickly and accurately.

[0250] The prior knowledge confirmation unit 80 (an example of a judgment information setting unit) inputs a prompt 501 (see FIG. 12 ) (an example of first question data) to the language model server 3 (or the LLM 500) to generate multiple questions that can be answered in the affirmative (YES) or negative (NO) for the explanatory information of a judgment target term (an example of a judgment target character string term) (e.g., “rabbit”), acquires an answer 502 (an example of answer data) to the prompt 501 from the language model server 3 (or the LLM 500), and generates multiple questions (e.g., “is rabbit a car?”, “Yes,” “…,” “Rabbit”) included in the answer 502. A prompt 511 (see FIG. 13) (an example of second question data) instructing a user to answer either affirmatively (YES) or negatively (NO) to the question "Is A Motors produced and sold by A Motors? Yes" may be input to the language model server 3 (or LLM 500), and an answer 512 (an example of answer data) to the prompt 511 may be obtained from the language model server 3 (or LLM 500), and an LLM decision 305 (an example of decision information) for the target term (e.g., "rabbit") may be set based on the answer result (e.g., "No, No, I don't have that information") included in the answer 512. In the example shown in FIGS. 12 and 13, the LLM decision 305 for the target term "rabbit" is set to "unknown."

[0251] According to this configuration, when setting the LLM judgment 305 in the language model server 3 (or the LLM 500), which indicates whether knowledge of the term 303 (an example of a predetermined character string) has reached a certain standard, it is possible to automatically set it using a question that can be answered with either affirmative (YES) or negative (NO). This makes it possible to set the LLM judgment 305 quickly and accurately.

[0252] The language model server 3 (or LLM 500) may further include a prior knowledge confirmation unit 80 (an example of a setting unit) that inputs input data 530 (see FIG. 15) (an example of third question data) instructing extraction of terms (entities) included in a glossary 531 (see FIG. 15) (an example of specific information) used in the above-mentioned specific field and generation of graph information showing relationships between the terms, obtains output data 534 (see FIG. 16) (an example of answer data) for the input data 530 from the language model server 3 (or LLM 500), inputs a prompt (shown in step S613 (see FIG. 21)) (an example of fourth question data) for generating an explanation for the term included in the output data 534 to the language model server 3 (or LLM 500), obtains answer data for the prompt (fourth question data) from the language model server 3 (or LLM 500), and sets explanation 340 (an example of explanation information) in prior knowledge DB 300 (an example of a database) using the explanation for the term included in the answer data.

[0253] According to this configuration, it is possible to automate the setting work when setting the explanation 340 in the prior knowledge DB 300. This makes it possible to set the explanation 340 quickly and accurately.

[0254] The system may further include a prior knowledge confirmation unit 80 (an example of a setting unit) that inputs input data 530 (see FIG. 15) (an example of third question data) that instructs extraction of terms (entities) included in a glossary 531 (see FIG. 15) (an example of specific information) used in the specific field described above and generation of graph information showing the relationships between the terms to the language model server 3 (or LLM 500), obtains output data 534 (see FIG. 16) (an example of answer data) for the input data 530 from the language model server 3 (or LLM 500), and sets a tree structure in a prior knowledge DB 300 (an example of a database) based on the terms and graph information included in the output data 534.

[0255] According to this configuration, it is possible to automate the task of hierarchizing each term in the prior knowledge DB 300. As a result, when hierarchizing each term, the hierarchizing process can be performed quickly and accurately.

[0256] The information processing method according to the present embodiment is an information processing method capable of acquiring answer data (e.g., answer 740 shown in FIG. 8 ) to question data (e.g., “Please tell me what options are available for Rabbit’s audio?”) input to the language model server 3 (or LLM 500) (an example of a large-scale language model) from the language model server 3 (or LLM 500). In the language model server 3 (or LLM 500), an LLM decision 305 (an example of decision information) indicating whether or not knowledge of a term 303 (an example of a predetermined character string) has reached a certain standard is stored in a prior knowledge DB 300 (an example of a database) associated with the term 303 and its explanation 304 (an example of explanation information). The information processing method performs an extraction process (step S100) to extract the first character string “Rabbit” associated with the LLM decision 305 “Unknown” (an example of decision information that has not reached the standard) from among the terms (e.g., rabbit, audio, option) (an example of a character string) included in the question data “Please tell me what options are available for Rabbit’s audio.” The program includes a control process (step S42) for inputting a prompt 730 (an example of input data) including the first character string "rabbit" and the explanation 304 (an example of explanatory information) "Rabbit is a compact car produced and sold by A Motors" associated with the first character string "rabbit" into the question data "Please tell me what options can be selected for the audio system of Rabbit." to the language model server 3 (or LLM 500), and acquiring answer data (for example, answer 740 shown in FIG. 8) to the question data "Please tell me what options can be selected for the audio system of Rabbit." from the language model server 3 (or LLM 500) (steps S42 to S46). The program according to this embodiment is a program that causes a computer to execute each of these processes. In other words, the program according to this embodiment is a program that causes a computer to realize each function that the management server 1 can execute.

[0257] According to this configuration, by extracting prior knowledge within the range necessary to answer a question, it is possible to pass on the necessary prior knowledge even within the finite amount of information that can be passed on to the language model server 3 (or LLM 500), thereby improving the quality of the answer.

[0258] Note that each processing procedure shown in this embodiment is an example for realizing this embodiment, and the order of some of the processing procedures may be changed within the scope that allows the realization of this embodiment, and some of the processing procedures may be omitted or other processing procedures may be added.

[0259] Each process shown in this embodiment is executed based on a program for causing a computer to execute each processing procedure. Therefore, this embodiment can also be understood as an embodiment of a program for realizing the function of executing each process, and a recording medium for storing the program. For example, the program can be stored in the storage device of the information processing device by an update process for adding a new function to the information processing device. This makes it possible to cause the updated information processing device to execute each process shown in this embodiment.

[0260] Although the embodiments of the present invention have been described above, the above-mentioned embodiments merely show some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above-mentioned embodiments. [Explanation of symbols]

[0261] 1... management server, 2... user terminal, 3... language model server, 11... control unit, 60... input information acquisition unit, 61... prior knowledge management unit, 62... prior knowledge narrowing unit, 63... provided information generation unit, 64... answer information management unit, 80... prior knowledge confirmation unit, 81... hierarchical management unit, 100... control unit, 120... input information management unit, 121... answer information management unit, 160... control unit, 180... provided information etc. acquisition unit, 181... answer information generation unit, 300... prior knowledge DB, 350... business knowledge DB

Claims

1. An information processing device capable of acquiring answer data to question data input to a large-scale language model from the large-scale language model, comprising: an extraction unit configured to extract, from among character strings included in the question data, a first character string associated with determination information indicating that knowledge of a character string in the large-scale language model does not reach a certain standard, in a database that stores the character string and explanatory information of the character string in association with the determination information; a control unit that inputs input data, in which the first character string and the explanatory information associated with the first character string are included in the question data, into the large-scale language model, and acquires answer data to the question data from the large-scale language model; An information processing device comprising:

2. the database holds hierarchical structure information that indicates relationships between a plurality of the character strings in a tree structure in association with each of the character strings; the extraction unit extracts, from among strings linked upward from the first string in the tree structure using the first string as a criterion, one or more second strings associated with determination information that does not reach the criterion; The control unit causes the input data to include the first character string, the second character string, and the explanation information associated with the first character string and the second character string, respectively. The information processing device according to claim 1 .

3. The character string is a term used in a particular field, the tree structure is a tree structure in which abstractions in the specific field are at an upper level and concreteness is at a lower level, The extraction unit extracts the second character string from character strings on an upper layer side of the tree structure to which the first character string belongs. The information processing device according to claim 2 .

4. the extraction unit extracts, from a specific information database that holds specific information used in the specific field, the specific information related to a character string included in the question data; The control unit causes the input data to include the first character string, the explanation information associated with the first character string, and the specific information related to the first character string. The information processing device according to claim 3 .

5. The present invention further includes a determination information setting unit that inputs first question data to the large-scale language model, the first question data being intended to create a question for explanatory information of a determination target string, which is the string to be determined, and acquires a question included in answer data to the first question data from the large-scale language model, inputs second question data to the large-scale language model, the second question data being intended to answer the question, and acquires answer data to the second question data from the large-scale language model, and sets determination information for the determination target string based on the answer result included in the answer data. The information processing device according to claim 1 .

6. The determination information setting unit inputs the first question data, which is intended to create a plurality of questions that can be answered in either affirmative or negative with respect to explanatory information of the determination target string, into the large-scale language model, and acquires answer data to the first question data from the large-scale language model, inputs the second question data, which is intended to answer in either affirmative or negative with respect to the plurality of questions included in the answer data, into the large-scale language model, and acquires answer data to the second question data from the large-scale language model, and sets determination information for the determination target string based on the answer results included in the answer data. The information processing device according to claim 5 .

7. a setting unit that inputs third question data instructing extraction of terms included in specific information used in the specific field and generation of graph information indicating relationships between the terms into the large-scale language model, and acquires answer data to the third question data from the large-scale language model; inputs fourth question data instructing generation of an explanation for the terms included in the answer data into the large-scale language model, and acquires the answer data to the fourth question data from the large-scale language model; and sets the explanation information in the database using an explanation for the terms included in the answer data to the fourth question data. The information processing device according to claim 3 .

8. a setting unit that inputs third question data, which instructs extraction of terms included in specific information used in the specific field and generation of graph information indicating relationships between the terms, into the large-scale language model, acquires answer data to the third question data from the large-scale language model, and sets the tree structure based on the terms and graph information included in the answer data. The information processing device according to claim 3 .

9. An information processing method capable of acquiring answer data to question data input to a large-scale language model from the large-scale language model, comprising: an extraction process in which a computer extracts, from among character strings included in the question data, a first character string associated with determination information indicating whether knowledge of a specific character string in the large-scale language model has reached a certain standard, the specific character string, and explanatory information of the specific character string, in a database in which the computer associates the specific character string with the determination information; a control process in which the computer inputs input data, in which the first character string and the explanatory information associated with the first character string are included in the question data, into the large-scale language model, and acquires answer data to the question data from the large-scale language model; An information processing method comprising:

10. A program for causing a computer to execute the program, capable of acquiring answer data to question data input to a large-scale language model from the large-scale language model, the program comprising: an extraction step of extracting, from among character strings included in the question data, a first character string associated with determination information indicating whether knowledge of a specific character string in the large-scale language model has reached a certain standard, the character string, and explanatory information of the character string, in a database that stores the character string and the determination information indicating whether knowledge of the specific character string in the large-scale language model has reached the certain standard; a control step of inputting input data, in which the first character string and the explanatory information associated with the first character string are included in the question data, into the large-scale language model, and acquiring answer data to the question data from the large-scale language model; A program that causes a computer to execute the following.

Images