Information processing device, information processing method, and information processing program

Abstract

To automatically generate an appropriate prompt with consideration for UX.SOLUTION: An information processing device comprises a generation unit and a modification unit. The generation unit generates a learning model corresponding to each LLM and obtained by learning combinations of input questions and response results output to the questions for each LLM on the basis of each of response results of each of a plurality of LLMs to the questions generated using a first prompt which instructs generation of a question which can evaluate response differences based on the response results of the plurality of LLMs for the same question. The modification unit modifies the first prompt so that the same response results can be obtained in the plurality of LLMs on the basis of the response results to the user's question output using the learning model generated by the generation unit.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, a technique for automatically generating a prompt for answer generation of a large language model (LLM) is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, for example, when an LLM failure occurs or during switching, it is not possible to generate an answer that does not impair the UX. Therefore, there is room for further improvement in automatically generating an appropriate prompt in consideration of the UX.

[0005] The present application has been made in view of the above, and an object thereof is to automatically generate an appropriate prompt in consideration of the UX.

Means for Solving the Problems

[0006] The information processing apparatus according to the present application includes a generation unit that generates a learning model corresponding to each LLM by learning a combination of an input question and an answer result output for each LLM for the question, based on each of the answer results of a plurality of LLMs for a question generated using a first prompt that instructs to generate a question capable of evaluating an answer difference based on the answer results of a plurality of LLMs for the same question, and a modification unit that modifies the first prompt so that the same answer result is obtained in the plurality of LLMs, based on the answer result for the user's question output using the learning model generated by the generation unit.

Effect of the Invention

[0007] According to one aspect of the embodiment, there is an effect that an appropriate prompt considering the UX can be automatically generated.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments for carrying out the information processing apparatus, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing apparatus, information processing method, and information processing program according to the present application are not limited by this embodiment. Also, in the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] (Embodiment) When a predetermined user asks questions to a plurality of LLMs such as GPT (Generative Pre-trained Transformer), the answer results obtained from each LLM may be different (at least different to the extent that a difference in the answer results can be recognized). For example, when the LLM is changed during an LLM failure or switching, the answer results for the same question may fluctuate (vary). The present application has been made in view of the above, and aims to automatically generate an appropriate prompt considering UX so that the answer results for the same question do not fluctuate even when the LLM is changed.

[0011] 〔1. Configuration of Information Processing System〕 The information processing system 1 shown in FIG. 1 will be described. As shown in FIG. 1, the information processing system 1 includes a terminal device 10 and an information processing apparatus 100. The terminal device 10 and the information processing apparatus 100 are communicably connected by wire or wirelessly via a predetermined communication network (network N). FIG. 1 is a diagram showing a configuration example of the information processing system 1 according to the embodiment.

[0012] The terminal device 10 is an information processing device used by a user who asks questions to an LLM such as GPT. The terminal device 10 is used, for example, by a user who performs a trend analysis of users who use an e-commerce street for marketing. The terminal device 10 may be any device as long as it can realize the processing in the embodiment. Also, the terminal device 10 may be a device such as a smartphone, a tablet terminal, a notebook PC, a desktop PC, a mobile phone, or a PDA. FIG. 2 shows the case where the terminal device 10 is a smartphone.

[0013] The terminal device 10 is a smart device such as a smartphone or a tablet, and is a portable terminal device that can communicate with an arbitrary server device via a wireless communication network such as 4G to 5G (Generation) or LTE (Long Term Evolution). Also, the terminal device 10 has a screen such as a liquid crystal display, and has a screen having a touch panel function, and may receive various operations on display data such as content, such as a tap operation, a slide operation, and a scroll operation, from the user using a finger or a stylus. In FIG. 2, the terminal device 10 is used by the user U1.

[0014] The information processing device 100 is an information processing device aimed at automatically generating an appropriate prompt in consideration of the UX, and may be any device as long as it can realize the processing in the embodiment. The information processing device 100 automatically generates a prompt that absorbs fluctuations so that, for example, even when the LLM is changed, the answer results for the same question do not fluctuate. Specifically, the information processing device 100 generates in advance a learning model corresponding to each LLM based on the answer results of a plurality of LLMs for a question generated using a predetermined prompt, and based on the answer result output using the learning model when the learning model is applied (when the user inputs), changes (or regenerates) a predetermined prompt so that the same answer result is obtained in a plurality of LLMs. The information processing device 100 is, for example, an information processing device that provides services such as a trend analysis of users who use an e-commerce street.

[0015] In FIG. 1, the case where the terminal device 10 and the information processing device 100 are separate devices is shown, but the terminal device 10 and the information processing device 100 may be integrated.

[0016] 〔2. An Example of Information Processing〕 FIG. 2 is a diagram showing an example of information processing of the information processing system 1 according to the embodiment. The information processing device 100 acquires (or generates) a predetermined prompt (hereinafter, appropriately referred to as the "first prompt") that instructs to generate a question such that the answer results of a plurality of LLMs for the same question are different to such an extent that a difference in the answer results can be recognized (step S101).

[0017] FIG. 3 is a diagram showing an example of the first prompt according to the embodiment. The information processing device 100 inputs an instruction sentence such as "Please generate a question that can confirm the response difference for the same query among a plurality of LLMs based on the following data.", an "element" specifying an element, a "question previously confirmed" specifying the content of a past question, "news content" specifying news content, a "question and answer between users" specifying a combination of question and answer on a predetermined service, a "review of a product" specifying a combination of a product name and a review, and a "conversation between users" specifying data of a conversation sequence of users using a predetermined service as the first prompt to a predetermined LLM. Note that the predetermined LLM generates a question based on this information.

[0018] The information processing apparatus 100 acquires the answer results of a plurality of LLMs for a question generated using the first prompt (step S102). Then, the information processing apparatus 100 generates a learning model corresponding to each LLM based on each of the answer results of the plurality of LLMs (step S103). Here, the learning model according to the embodiment will be described. The learning model according to the embodiment is a learning model that learns the combination of the input question (such as the question input by the user) and the answer result output for each LLM for the question. For example, it is a learning model generated by collecting the answer results for each LLM by inputting a question to the target LLM and learning using the answer results corresponding to each LLM. For example, LLM1 is a learning model that learns question A and answer result B output by inputting question A to LLM1, and LLM2 is a learning model that learns question A and answer result C output by inputting question A to LLM2.

[0019] In the learning model according to the embodiment, the input of the model is the name of the original LLM, the name of the converted LLM, and the query in the original LLM, and the output of the model is the query in the converted LLM. Taking an example of specific learning data, it is "LLM name A(1), query A(2), output(3); LLM name B(4), query B(5), output(6)". That is, the learning model according to the embodiment is a model that, when inputting (1), (2), and (4), finds (6) equivalent to (3) and infers (fills in the blank) (5) and outputs it.

[0020] The above steps S101 to S103 are the preprocessing of the information processing of the information processing system 1 according to the embodiment. Hereinafter, the case where the user U1 asks question A will be described as an example.

[0021] When the user U1 asks question A, the information processing apparatus 100 accepts question A (step S104). Then, the information processing apparatus 100 acquires the answer result for question A for each LLM by inputting question A to the learning model corresponding to each LLM (step S105).

[0022] The information processing apparatus 100 changes (or regenerates) the first prompt so that the same answer result can be obtained based on the answer results for each LLM for question A (step S106). At this time, the information processing apparatus 100 changes the first prompt using a predetermined prompt (hereinafter, appropriately referred to as the "second prompt") that instructs to change the prompt so that the same answer result can be obtained as the answer results of a plurality of LLMs for the user's question.

[0023] FIG. 4 is a diagram showing an example of the second prompt according to the embodiment. The information processing apparatus 100 inputs, as the second prompt, to each of a plurality of LLMs data including instruction texts such as "Please rewrite the prompt for <source LLM> input by the user into a prompt for <destination LLM>.", and "Please generate a prompt so that the user input results in the same response while taking into account the latest user data, content data, etc.", an "element" that designates an element, a "pair of queries to the LLM and its responses" that designates a combination of the source LLM name / destination LLM name, a query, and a response, a "news content" that designates a search keyword and the number of users, and a "user input" that designates the content input by the user. Each of the plurality of LLMs generates a query based on this information.

[0024] The information processing apparatus 100 provides the answer result of the LLM targeted by question A to the user U1. Specifically, the information processing apparatus 100 transmits information for displaying the answer result of the LLM targeted by question A to the terminal device 10 (step S107). When the terminal device 10 receives the information transmitted from the information processing apparatus 100, it displays the answer result for question A based on the received information.

[0025] (Variation 1 of Information Processing: Determination of the Same Answer Result) In step S106 according to the above embodiment, the information processing apparatus 100 may, for example, determine whether the same answer result is obtained in a plurality of target LLMs. Then, for example, when it is determined that the same answer result is not obtained, the information processing apparatus 100 may change the first prompt so that the same answer result is obtained. Further, for example, when it is determined that the same answer result is obtained, in step S107, the information processing apparatus 100 may provide the answer result to the question A to the user U1.

[0026] (Variation 2 of information processing: Narrow down to the latest data) In the above embodiment, the second prompt may be a prompt that instructs to change the prompt so that an answer result based on data satisfying a predetermined condition is obtained. For example, the second prompt may be a prompt that instructs to change the prompt so that an answer result based on the latest data (for example, the latest data such as Q&A log data or news data) or an answer result based on continuously changing data (at least an answer result that emphasizes and reflects the continuously changing data) is obtained. In this way, the second prompt may be a prompt for regenerating the first prompt by supplementing the latest data and the like. In step S106 according to the above embodiment, the information processing apparatus 100 may change the first prompt using the second prompt that instructs to change the prompt so that an answer result based on data satisfying a predetermined condition is obtained.

[0027] [3. Configuration of the terminal device] Next, the configuration of the terminal device 10 according to the embodiment will be described with reference to FIG. 5. FIG. 5 is a diagram showing a configuration example of the terminal device 10 according to the embodiment. As shown in FIG. 5, the terminal device 10 includes a communication unit 11, an input unit 12, an output unit 13, and a control unit 14.

[0028] (Communication unit 11) The communication unit 11 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 11 is connected to a predetermined network N by wire or wirelessly, and performs information transmission and reception with an information processing apparatus 100 or the like via the predetermined network N.

[0029] (Input unit 12) The input unit 12 receives various operations from the user. In FIG. 2, it receives various operations from the user U1. For example, the input unit 12 may receive various operations from the user via the display surface by means of a touch panel function. Further, the input unit 12 may receive various operations from buttons provided on the terminal device 10 or from a keyboard or mouse connected to the terminal device 10. For example, the input unit 12 receives an operation for asking a question.

[0030] (Output unit 13) The output unit 13 is a display screen such as a tablet terminal realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. For example, the output unit 13 displays the information transmitted from the information processing apparatus 100. For example, the output unit 13 displays the answer result to the user's question transmitted from the information processing apparatus 100.

[0031] (Control unit 14) The control unit 14 is, for example, a controller, and is realized by various programs stored in the storage device inside the terminal device 10 being executed with the RAM as the working area by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. For example, these various programs include the programs of the applications installed in the terminal device 10. For example, these various programs include the programs of the applications for displaying the information (such as the answer results to the user's questions) transmitted from the information processing device 100. Further, the control unit 14 is realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0032] As shown in FIG. 5, the control unit 14 has a receiving unit 141 and a transmitting unit 142, and realizes or executes the operations of information processing described below.

[0033] (Receiving Unit 141) The receiving unit 141 receives, for example, the information transmitted from the information processing device 100. For example, the receiving unit 141 receives the information for displaying the answer results to the user's questions transmitted from the information processing device 100.

[0034] (Transmitting Unit 142) The transmitting unit 142 transmits, for example, the operation information performed by the user. Further, the transmitting unit 142 transmits the information regarding the question input by the user (such as the information indicating the question content).

[0035] [4. Configuration of Information Processing Device] Next, with reference to FIG. 6, the configuration of the information processing apparatus 100 according to the embodiment will be described. FIG. 6 is a diagram showing a configuration example of the information processing apparatus 100 according to the embodiment. As shown in FIG. 6, the information processing apparatus 100 includes a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing apparatus 100 may include an input unit (e.g., a keyboard or a mouse) that receives various operations from the administrator of the information processing apparatus 100, and a display unit (e.g., a liquid crystal display) that displays various information.

[0036] (Communication Unit 110) The communication unit 110 is realized by, for example, a NIC or the like. Then, the communication unit 110 is connected to the network N by wire or wirelessly, and information is transmitted and received between the communication unit 110 and the terminal device 10 or the like via the network N.

[0037] (Storage Unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 6, the storage unit 120 includes a prompt information storage unit 121 and a learning model information storage unit 122.

[0038] The prompt information storage unit 121 stores information related to the prompt. Here, FIG. 7 shows an example of the prompt information storage unit 121 according to the embodiment. The information stored in the prompt information storage unit 121 is used, for example, for generating questions in the preprocessing according to the embodiment (generating questions for learning the learning model) and for changing the first prompt according to the embodiment. As shown in FIG. 7, the prompt information storage unit 121 has items such as "prompt ID" and "prompt information".

[0039] "Prompt ID" indicates identification information for identifying a prompt. "Prompt information" indicates information included in a prompt. In the example shown in FIG. 7, an example is shown in which conceptual information such as "Prompt information #1" and "Prompt information #2" is stored in "Prompt information". However, in reality, an instruction text such as "Please generate a query that can confirm the response difference for the same query among multiple LLMs based on the following data." or information indicating elements is stored.

[0040] The learning model information storage unit 122 stores information regarding the learning model. Here, FIG. 8 shows an example of the learning model information storage unit 122 according to the embodiment. The information stored in the learning model information storage unit 122 is used, for example, for learning the learning model. As shown in FIG. 8, the learning model information storage unit 122 has items such as "Learning model ID", "LLM", and "Learning model information".

[0041] "Learning model ID" indicates identification information for identifying a learning model. "LLM" indicates which LLM such as GPT it is. "Learning model information" indicates learning data for learning the learning model. In the example shown in FIG. 8, an example is shown in which conceptual information such as "Learning model information #1" and "Learning model information #2" is stored in "Learning model information". However, in reality, information indicating a combination of the input question and the answer result output for the question is stored.

[0042] (Control unit 130) The control unit 130 is a controller and is realized, for example, by various programs stored in the storage device inside the information processing device 100 being executed with the RAM as a work area by a CPU, MPU, etc. Further, the control unit 130 is realized by an integrated circuit such as an ASIC or FPGA.

[0043] As shown in FIG. 6, the control unit 130 includes an acquisition unit 131, a generation unit 132, a modification unit 133, and a provision unit 134, and realizes or executes the operations of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 6, and any other configuration may be used as long as it can perform the information processing described later.

[0044] (Acquisition Unit 131) The acquisition unit 131 acquires various types of information from the storage unit 120. Further, the acquisition unit 131 stores the acquired various types of information in the storage unit 120.

[0045] The acquisition unit 131 acquires various types of information from an external information processing device. The acquisition unit 131 acquires various types of information from other information processing devices such as the terminal device 10.

[0046] The acquisition unit 131 acquires, for example, a first prompt. Specifically, when the acquisition unit 131 asks the same question to a plurality of target LLMs, the acquisition unit 131 acquires a first prompt, which is a prompt that instructs to generate a question such that the answer results of the plurality of LLMs for the same question are different to such an extent that a difference can be recognized at least in the answer results.

[0047] The acquisition unit 131 acquires, for example, a question generated based on the first prompt. For example, the acquisition unit 131 acquires a question output by inputting the first prompt into a predetermined LLM. Further, the acquisition unit 131 acquires, for example, the answer results of a plurality of LLMs for a question generated using the first prompt. For example, the acquisition unit 131 acquires the respective answer results of a plurality of LLMs output by inputting a question generated using the first prompt to the plurality of target LLMs.

[0048] When the acquisition unit 131 acquires a question asked by a user, for example, the acquisition unit 131 inputs the acquired question into a learning model corresponding to each target LLM to acquire the answer result for the question for each LLM.

[0049] (Generation Unit 132) The generation unit 132 generates a learning model corresponding to each LLM, for example, based on each of the answer results of a plurality of LLMs acquired by the acquisition unit 131. Specifically, the generation unit 132 generates a learning model obtained by training the combination of the input question and the answer result output for each LLM using the first prompt for the question. In other words, the generation unit 132 inputs a question to the target LLM, collects the answer results for each LLM, and generates a learning model trained using the answer results corresponding to each LLM.

[0050] (Modification unit 133) The modification unit 133 modifies (or regenerates) the first prompt so that, for example, the same answer result is obtained in a plurality of LLMs. For example, the modification unit 133 modifies the first prompt based on the answer result for the user's question output using the learning model generated by the generation unit 132 so that the same answer result is obtained in a plurality of LLMs. At this time, the modification unit 133 determines whether the same answer result is obtained in a plurality of LLMs, and if it is determined that the same answer result is not obtained, the first prompt may be modified so that the same answer result is obtained in a plurality of LLMs.

[0051] The modification unit 133 modifies the first prompt using, for example, the second prompt. Specifically, the modification unit 133 modifies the first prompt using the second prompt, which is a prompt instructing to modify the prompt so that the same answer result is obtained as the answer results of a plurality of LLMs for the user's question. For example, the modification unit 133 modifies the first prompt using the second prompt instructing to modify the prompt so that an answer result based on data satisfying a predetermined condition, such as the latest data or continuously changing data, is obtained.

[0052] (Provision unit 134) The providing unit 134 provides, for example, the answer result to the user's question to the user. For example, the providing unit 134 provides information for displaying the answer result to the user's question. Specifically, the providing unit 134 provides information for displaying the answer result of the LLM targeted by the user's question among the plurality of LLMs.

[0053] [5. Information Processing Flow] Next, with reference to FIG. 9, the information processing procedure by the information processing system 1 according to the embodiment will be described. FIG. 9 is a flowchart showing the information processing procedure by the information processing system 1 according to the embodiment.

[0054] As shown in FIG. 9, the information processing apparatus 100 generates a learning model corresponding to each LLM based on the answer results of the plurality of LLMs for the question generated using the first prompt (step S201).

[0055] When the information processing apparatus 100 receives a question from the user, it changes the first prompt so that the same answer result can be obtained in the plurality of LLMs based on the answer result to the user's question output using the learning model (step S202).

[0056] The information processing apparatus 100 provides the user with the answer result of the LLM targeted by the user's question among the plurality of LLMs (step S203).

[0057] [6. Effects] As described above, the information processing apparatus 100 according to the embodiment includes a generation unit 132 and a modification unit 133. The generation unit 132 generates, based on each of the answer results of a plurality of LLMs for a question generated using a first prompt that instructs to generate a question that can evaluate the answer difference based on the answer results of the plurality of LLMs for the same question, a learning model corresponding to each LLM, which is a learning model obtained by learning the combination of the input question and the answer result output for the question for each LLM. The modification unit 133 modifies the first prompt so that the same answer result can be obtained in a plurality of LLMs based on the answer result for the user's question output using the learning model generated by the generation unit 132.

[0058] Thereby, the information processing apparatus 100 according to the embodiment can automatically generate a prompt that absorbs fluctuations so that, for example, even when the LLM is changed, the answer results for the same question do not fluctuate, enabling the automatic generation of an appropriate prompt considering the UX.

[0059] Further, the modification unit 133 determines whether the same answer result is obtained in a plurality of LLMs, and when it is determined that the same answer result is not obtained, modifies the first prompt so that the same answer result can be obtained.

[0060] Thereby, the information processing apparatus 100 according to the embodiment can, for example, change the prompt only when the same answer result is not obtained, enabling more effective automatic generation of the prompt.

[0061] Further, the modification unit 133 modifies the first prompt using a second prompt that instructs to modify the prompt so that the same answer result is obtained as the answer results of a plurality of LLMs for the user's question.

[0062] As a result, the information processing apparatus 100 according to the embodiment can enable more effective automatic generation of prompts by using other prompts that change the prompt so that, for example, the answer results for the same question do not fluctuate.

[0063] Further, the changing unit 133 changes the first prompt by using a second prompt that instructs to change the prompt so that an answer result based on data satisfying a predetermined condition can be obtained.

[0064] As a result, the information processing apparatus 100 according to the embodiment can enable more effective automatic generation of prompts by using other prompts that change the prompt so that, for example, an answer result based on appropriate data can be obtained.

[0065] Further, the information processing apparatus 100 according to the embodiment is characterized by further including a providing unit 134 that provides the user with the answer result of the LLM targeted by the user's question among a plurality of LLMs.

[0066] As a result, the information processing apparatus 100 according to the embodiment can enable, for example, the provision of an appropriate answer result in accordance with the user's question.

[0067] [7. Hardware Configuration] Further, the information processing apparatus 100 according to the above-described embodiment is realized, for example, by a computer 1000 having a configuration as shown in FIG. 10. FIG. 10 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing apparatus 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0068] The CPU 1100 operates based on programs stored in the ROM 1300 or the HDD 1400 and controls each part. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs dependent on the hardware of the computer 1000, and the like.

[0069] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, and the like. The communication interface 1500 acquires data from other devices via a predetermined communication network and sends it to the CPU 1100, and sends data generated by the CPU 1100 to other devices via a predetermined communication network.

[0070] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. Further, the CPU 1100 outputs the generated data to the output devices via the input / output interface 1600.

[0071] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads such a program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc), a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0072] For example, when computer 1000 functions as information processing apparatus 100 according to an embodiment, the CPU 1100 of computer 1000 realizes the functions of control unit 130 by executing a program loaded onto RAM 1200. The CPU 1100 of computer 1000 reads and executes these programs from recording medium 1800. As another example, these programs may be acquired from another device via a predetermined communication network.

[0073] [8. Others] Also, among the respective processes described in the above embodiment, all or part of the processes described as being automatically performed can be manually performed, or all or part of the processes described as being manually performed can be automatically performed by a known method. Additionally, regarding the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings, they can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

[0074] Moreover, each component of each illustrated device is conceptually functional and does not necessarily need to be physically configured as shown in the figure. That is, the specific form of the distribution and integration of each device is not limited to that shown, and all or part of it can be functionally or physically distributed and integrated in arbitrary units according to various loads, usage situations, etc.

[0075] Also, the above-described embodiments can be appropriately combined within a range that does not conflict with the processing content.

[0076] As described above, some of the embodiments of the present application have been described in detail with reference to the drawings. However, these are merely examples, and the present invention can be implemented in other forms with various modifications and improvements based on the knowledge of those skilled in the art, starting from the aspects described in the column of the disclosure of the invention.

[0077] Also, the "section (section, module, unit)" described above can be read as "means", "circuit", etc. For example, the acquisition section can be read as an acquisition means or an acquisition circuit.

Explanation of symbols

[0078] 1 Information processing system 10 Terminal device 11 Communication section 12 Input section 13 Output section 14 Control section 100 Information processing device 110 Communication section 120 Storage section 121 Prompt information storage section 122 Learning model information storage section 130 Control section 131 Acquisition section 132 Generation section 133 Modification section 134 Provision section 141 Reception section 142 Transmission section N Network

Claims

1. A generation unit that generates a learning model corresponding to each LLM, which learns a combination of an input question and an answer result output for each LLM for the question, based on each of the answer results of a plurality of LLMs for a question generated using a first prompt that instructs to generate a question that can evaluate an answer difference based on the answer results of a plurality of LLMs for the same question; A modification unit that modifies the first prompt so that the same answer result is obtained in the plurality of LLMs, based on the answer result for the user's question output using the learning model generated by the generation unit; An information processing apparatus comprising the above.

2. The modification unit: Determines whether the same answer result is obtained in the plurality of LLMs, and when it is determined that the same answer result is not obtained, modifies the first prompt so that the same answer result is obtained. The information processing apparatus according to Claim 1, characterized by the above.

3. The modification unit: Modifies the first prompt using a second prompt that instructs to modify the prompt so that the same answer result is obtained as the answer results of a plurality of LLMs for the user's question. The information processing apparatus according to Claim 1, characterized by the above.

4. The modification unit: Modifies the first prompt using the second prompt that instructs to modify the prompt so that the answer result based on data satisfying a predetermined condition is obtained. The information processing apparatus according to Claim 3, characterized by the above.

5. A provision unit that provides the user with the answer result of the LLM targeted by the user's question among the plurality of LLMs; The information processing apparatus according to Claim 1, further comprising the above.

6. An information processing method executed by a computer, comprising: A generation step of generating a learning model corresponding to each LLM, which learns a combination of an input question and an answer result output for each LLM for the question, based on each of the answer results of a plurality of LLMs for a question generated using a first prompt that instructs to generate a question that can evaluate an answer difference based on the answer results of a plurality of LLMs for the same question; A modification step of modifying the first prompt so that the same answer result can be obtained in the plurality of LLMs based on the answer result for the user's question output using the learning model generated in the generation step; An information processing method characterized by including the above. **Claim 7** A generation procedure for generating a learning model corresponding to each LLM, which is a learning model that has learned a combination of the input question and the answer result output for each LLM for the question, based on the answer results of the plurality of LLMs for the question generated using the first prompt that instructs to generate a question that can evaluate the answer difference based on the answer results of the plurality of LLMs for the same question; A modification procedure of modifying the first prompt so that the same answer result can be obtained in the plurality of LLMs based on the answer result for the user's question output using the learning model generated in the generation procedure; An information processing program characterized by causing a computer to execute the above.

Images