Information processing device, information processing method, and information processing program

Abstract

To automatically generate an appropriate prompt with consideration for UX.SOLUTION: An information processing device comprises a generation unit and a modification unit. The generation unit generates a learning model corresponding to each LLM and obtained by learning combinations of input questions and response results output to the questions for each LLM on the basis of each of response results of each of a plurality of LLMs to the questions generated using a first prompt which instructs generation of a question which can evaluate response differences based on the response results of the plurality of LLMs for the same question. The modification unit modifies the first prompt so that the same response results can be obtained in the plurality of LLMs on the basis of the response results to the user's question output using the learning model generated by the generation unit.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, a technique for automatically generating a prompt for answer generation of a large language model (LLM) has been known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, for example, when an LLM failure occurs or when switching, it is impossible to generate an answer that does not impair the UX. Therefore, there is room for further improvement in automatically generating an appropriate prompt in consideration of the UX.

[0005] The present application has been made in view of the above, and an object thereof is to automatically generate an appropriate prompt in consideration of the UX.

Means for Solving the Problems

[0006] The information processing device according to the present invention is characterized by comprising: a generation unit that generates a learning model corresponding to each LLM, which is trained on a combination of an input question and the answer result output by each LLM for the question, based on the answer results of multiple LLMs to a question generated using a first prompt that instructs the generation unit to generate a question capable of evaluating the difference in answers based on the answer results of multiple LLMs to the same question; and a modification unit that modifies the first prompt so that the same answer result is obtained in the multiple LLMs, based on the answer result to the user's question output using the learning model generated by the generation unit. [Effects of the Invention]

[0007] According to one embodiment, it is possible to automatically generate appropriate prompts that take UX into consideration. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of the configuration of an information processing system according to an embodiment. [Figure 2] Figure 2 shows an example of information processing according to the present invention. [Figure 3] Figure 3 shows an example of the first prompt according to this embodiment. [Figure 4] Figure 4 shows an example of a second prompt according to this embodiment. [Figure 5] Figure 5 shows an example of the configuration of a terminal device according to this embodiment. [Figure 6] Figure 6 shows an example of the configuration of an information processing device according to the embodiment. [Figure 7] Figure 7 shows an example of a prompt information storage unit according to the embodiment. [Figure 8] Figure 8 shows an example of a learning model information storage unit according to an embodiment. [Figure 9] Figure 9 is a flowchart showing an example of information processing according to the embodiment. [Figure 10]Figure 10 is a hardware configuration diagram showing an example of a computer that implements the functions of an information processing device. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing device, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing device, information processing method, and information processing program according to the present application. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted.

[0010] (Embodiment) When a user asks a question to multiple Generative Pre-trained Transformers (LLMs) such as GPT, the answers obtained from each LLM may differ (at least to a degree that the difference in answers is recognizable). For example, when an LLM fails or is switched, the answer to the same question may fluctuate. This invention was made in view of the above, and aims to automatically generate appropriate prompts that take UX into consideration, so that even when an LLM is changed, the answer to the same question does not fluctuate.

[0011] [1. Configuration of the Information Processing System] The information processing system 1 shown in Figure 1 will now be described. As shown in Figure 1, the information processing system 1 includes a terminal device 10 and an information processing device 100. The terminal device 10 and the information processing device 100 are connected to each other via a predetermined communication network (network N) by wired or wireless means. Figure 1 is a diagram showing an example configuration of the information processing system 1 according to an embodiment.

[0012] Terminal device 10 is an information processing device used by users who ask questions to an LLM such as GPT. Terminal device 10 is used, for example, by users who perform trend analysis of users who use an online shopping mall for marketing purposes. Terminal device 10 can be any device as long as it can realize the processing in the embodiment. Also, terminal device 10 may be a smartphone, tablet device, notebook PC, desktop PC, mobile phone, PDA, or other device. Figure 2 shows the case where terminal device 10 is a smartphone.

[0013] Terminal device 10 is, for example, a smart device such as a smartphone or tablet, and is a mobile terminal device that can communicate with any server device via a wireless communication network such as 4G-5G (Generation) or LTE (Long Term Evolution). Terminal device 10 also has a screen, such as an LCD display, which has touch panel functionality and may accept various operations on displayed data such as content from the user, such as tapping, sliding, and scrolling, using a finger or stylus. In Figure 2, terminal device 10 is used by user U1.

[0014] The information processing device 100 is an information processing device aimed at automatically generating appropriate prompts that take UX into consideration, and can be any device as long as it can realize the processing in the embodiment. For example, the information processing device 100 automatically generates prompts that absorb fluctuations so that even when the LLM is changed, there is no fluctuation in the answer result to the same question. Specifically, the information processing device 100 pre-generates a learning model corresponding to each LLM based on the answer results of multiple LLMs to a question generated using a predetermined prompt, and when the learning model is applied (when the user inputs), it changes (or regenerates) the predetermined prompts based on the answer result output using the learning model so that the same answer result is obtained in multiple LLMs. The information processing device 100 is an information processing device that provides services such as trend analysis of users using an online shopping mall.

[0015] In FIG. 1, the case where the terminal device 10 and the information processing device 100 are separate devices is shown, but the terminal device 10 and the information processing device 100 may be integrated.

[0016] 〔2. An Example of Information Processing〕 FIG. 2 is a diagram showing an example of information processing of the information processing system 1 according to the embodiment. The information processing device 100 acquires (or generates) a predetermined prompt (hereinafter, appropriately referred to as the "first prompt") that instructs to generate a question such that the answer results of a plurality of LLMs for the same question are different enough to be recognizable that there are differences in the answer results at least (step S101).

[0017] FIG. 3 is a diagram showing an example of the first prompt according to the embodiment. The information processing device 100 inputs an instruction sentence such as "Please generate a query that can confirm the response difference for the same query among multiple LLMs based on the following data.", an "element" specifying an element, a "query previously confirmed" specifying the past query content, a "news content" specifying news content, a "user-to-user Q&A" specifying a combination of Q&A on a predetermined service, a "review for a product" specifying a combination of a product name and a review, and a "user conversation" specifying data of the conversation series of users using a predetermined service as the first prompt to a predetermined LLM. Note that the predetermined LLM generates a question based on this information.

[0018] The information processing device 100 acquires the answer results of multiple LLMs to a question generated using the first prompt (step S102). Then, the information processing device 100 generates a learning model corresponding to each LLM based on each of the answer results of the multiple LLMs (step S103). Here, the learning model according to the embodiment will be described. The learning model according to the embodiment is a learning model that has learned combinations of input questions (such as questions entered by the user) and the answer results output for each LLM in response to the questions. For example, it is a learning model that is generated by collecting answer results for each LLM by inputting a question to the target LLM and then training the learning model using the answer results corresponding to each LLM. For example, LLM1 is a learning model that has learned question A and answer result B output when question A is input to LLM1, and LLM2 is a learning model that has learned question A and answer result C output when question A is input to LLM2.

[0019] In the learning model according to the embodiment, the input to the model is the name of the source LLM, the name of the converted LLM, and a question in the source LLM, and the output of the model is a question in the converted LLM. An example of specific training data is "LLM name A (1), question A (2), output (3); LLM name B (4), question B (5), output (6)". In other words, the learning model according to the embodiment, when given (1), (2), and (4) as input, finds (6) which is equivalent to (3), and infers (fills in) (5) and outputs it.

[0020] Steps S101 to S103 described above constitute the pre-processing of information processing in the information processing system 1 according to the embodiment. The following explanation will be given using the case where user U1 asks question A as an example.

[0021] When user U1 asks question A, the information processing device 100 accepts question A (step S104). Then, the information processing device 100 inputs question A to the learning model corresponding to each LLM and obtains the answer result to question A for each LLM (step S105).

[0022] The information processing device 100 modifies (or regenerates) the first prompt so that the same answer result is obtained based on the answer result of each LLM to question A (step S106). At this time, the information processing device 100 modifies the first prompt using a predetermined prompt (hereinafter referred to as the "second prompt") that instructs the prompt to be modified so that the same answer result is obtained as the answer result of multiple LLMs to the user's question.

[0023] Figure 4 shows an example of a second prompt according to the embodiment. The information processing device 100 inputs the following as a second prompt to each of the multiple LLMs: instructional statements such as "Please rewrite the prompt for <source LLM> entered by the user to a prompt for <destination LLM>" and "Please generate prompts so that the user input results in the same response, taking into account the latest user data, content data, etc.", an "element" specifying an element, a "pair of questions and answers to the LLM" specifying the source LLM name / destination LLM name and a combination of questions and answers, "news content" specifying data including search keywords and the number of users, and "user input" specifying the content input by the user. The multiple LLMs each generate questions based on this information.

[0024] The information processing device 100 provides user U1 with the answer result for the LLM that corresponds to question A. Specifically, the information processing device 100 transmits information to the terminal device 10 to display the answer result for the LLM that corresponds to question A (step S107). When the terminal device 10 receives the information transmitted from the information processing device 100, it displays the answer result for question A based on the received information.

[0025] (Information processing variation 1: Determination of identical response results) In step S106 of the above embodiment, the information processing device 100 may, for example, determine whether the same answer result was obtained in the multiple LLMs that were targeted. If the information processing device 100 determines, for example, that the same answer result was not obtained, it may change the first prompt so that the same answer result is obtained. Alternatively, if the information processing device 100 determines, for example, that the same answer result was obtained, it may provide the answer result for question A to user U1 in step S107.

[0026] (Information processing variation 2: Focusing on the latest data) In the above embodiment, the second prompt may be a prompt that instructs the system to change the prompt so that a response result based on data that satisfies predetermined conditions is obtained. For example, the second prompt may be a prompt that instructs the system to change the prompt so that a response result based on the latest data (e.g., the latest data such as question and answer log data or news data) is obtained (at least a response result that gives importance to and reflects the latest data), or a response result based on continuously changing data (at least a response result that gives importance to and reflects continuously changing data). Thus, the second prompt may be a prompt for regenerating the first prompt by supplementing it with the latest data, etc. In step S106 of the above embodiment, the information processing device 100 may change the first prompt using, for example, a second prompt that instructs the system to change the prompt so that a response result based on data that satisfies predetermined conditions is obtained.

[0027] [3. Configuration of the terminal device] Next, the configuration of the terminal device 10 according to the embodiment will be described using Figure 5. Figure 5 is a diagram showing an example of the configuration of the terminal device 10 according to the embodiment. As shown in Figure 5, the terminal device 10 has a communication unit 11, an input unit 12, an output unit 13, and a control unit 14.

[0028] (Communications Section 11) The communication unit 11 is implemented, for example, by a NIC (Network Interface Card). The communication unit 11 is connected to a predetermined network N by wire or wireless connection and sends and receives information to and from the information processing device 100 via the predetermined network N.

[0029] (Input section 12) The input unit 12 accepts various operations from the user. In Figure 2, it accepts various operations from user U1. For example, the input unit 12 may accept various operations from the user via the display surface using a touch panel function. Alternatively, the input unit 12 may accept various operations from buttons provided on the terminal device 10, or from a keyboard or mouse connected to the terminal device 10. For example, the input unit 12 accepts operations for asking questions.

[0030] (Output section 13) The output unit 13 is a display screen for a tablet terminal, for example, which is implemented using a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various types of information. For example, the output unit 13 displays information transmitted from the information processing device 100. For example, the output unit 13 displays the answer result to a user's question transmitted from the information processing device 100.

[0031] (Control Unit 14) The control unit 14 is, for example, a controller, and is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs stored in the memory device inside the terminal device 10 using RAM (Random Access Memory) as the working area. For example, these various programs include application programs installed on the terminal device 10. For example, these various programs include application programs that display information transmitted from the information processing device 100 (such as the answer result to a user's question). The control unit 14 is also implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0032] As shown in Figure 5, the control unit 14 has a receiving unit 141 and a transmitting unit 142, and realizes or executes the information processing operations described below.

[0033] (Receiver 141) The receiving unit 141 receives information transmitted from, for example, the information processing device 100. For example, the receiving unit 141 receives information transmitted from the information processing device 100 for displaying the answer result to a user's question.

[0034] (Transmitter 142) The transmitting unit 142 transmits, for example, information about operations performed by the user. The transmitting unit 142 also transmits information related to questions entered by the user (such as information indicating the content of the question).

[0035] [4. Configuration of Information Processing Device] Next, the configuration of the information processing device 100 according to the embodiment will be described using Figure 6. Figure 6 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Figure 6, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 may also have an input unit (for example, a keyboard or mouse) that receives various operations from the administrator of the information processing device 100, and a display unit (for example, a liquid crystal display) for displaying various information.

[0036] (Communications Department 110) The communication unit 110 is implemented, for example, by a NIC. The communication unit 110 is connected to the network N by wire or wireless connection and sends and receives information to and from terminal devices 10, etc., via the network N.

[0037] (Storage unit 120) The memory unit 120 is implemented by, for example, semiconductor memory elements such as RAM and flash memory, or storage devices such as hard disks and optical discs. As shown in Figure 6, the memory unit 120 has a prompt information storage unit 121 and a learning model information storage unit 122.

[0038] The prompt information storage unit 121 stores information related to prompts. Here, Figure 7 shows an example of the prompt information storage unit 121 according to the embodiment. The information stored in the prompt information storage unit 121 is used, for example, to generate questions in the preprocessing according to the embodiment (generating questions for training the learning model) and to modify the first prompt according to the embodiment. As shown in Figure 7, the prompt information storage unit 121 has items such as "prompt ID" and "prompt information".

[0039] The "Prompt ID" indicates identification information used to identify a prompt. The "Prompt Information" indicates the information contained in the prompt. In the example shown in Figure 7, conceptual information such as "Prompt Information #1" and "Prompt Information #2" is stored in "Prompt Information," but in reality, it stores instructional statements such as "Based on the following data, generate a question that allows you to check the differences in responses to the same question across multiple LLMs," as well as information indicating elements.

[0040] The learning model information storage unit 122 stores information about the learning model. Here, Figure 8 shows an example of the learning model information storage unit 122 according to this embodiment. The information stored in the learning model information storage unit 122 is used, for example, for training the learning model. As shown in Figure 8, the learning model information storage unit 122 has items such as "Learning Model ID", "LLM", and "Learning Model Information".

[0041] The "Learning Model ID" indicates identification information for identifying the learning model. "LLM" indicates what type of LLM it is, such as GPT. "Learning Model Information" indicates the training data for training the learning model. In the example shown in Figure 8, conceptual information such as "Learning Model Information #1" and "Learning Model Information #2" is stored in "Learning Model Information," but in reality, information such as the combination of the input question and the output answer result for that question is stored.

[0042] (Control unit 130) The control unit 130 is a controller, and is implemented, for example, by a CPU or MPU executing various programs stored in the memory device inside the information processing device 100 using RAM as the working area. Alternatively, the control unit 130 can be implemented by an integrated circuit such as an ASIC or FPGA.

[0043] As shown in Figure 6, the control unit 130 includes an acquisition unit 131, a generation unit 132, a modification unit 133, and a provision unit 134, and realizes or executes the information processing operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 6, and other configurations are also acceptable as long as they perform the information processing described later.

[0044] (Acquisition part 131) The acquisition unit 131 acquires various information from the storage unit 120. The acquisition unit 131 also stores the acquired information in the storage unit 120.

[0045] The acquisition unit 131 acquires various information from external information processing devices. The acquisition unit 131 also acquires various information from other information processing devices such as the terminal device 10.

[0046] The acquisition unit 131 acquires, for example, a first prompt. Specifically, the acquisition unit 131 acquires a first prompt, which is a prompt instructing the system to generate a question such that, when the same question is asked to multiple target LLMs, the answer results of the multiple LLMs to the same question are at least different enough to recognize that there is a difference in the answer results.

[0047] The acquisition unit 131 acquires, for example, a question generated based on a first prompt. For example, the acquisition unit 131 acquires a question output by inputting the first prompt to a predetermined LLM. The acquisition unit 131 also acquires, for example, the answer results of multiple LLMs to a question generated using the first prompt. For example, the acquisition unit 131 acquires the answer results of each of the multiple LLMs output by inputting a question generated using the first prompt to multiple target LLMs.

[0048] The acquisition unit 131, for example, acquires a question asked by the user, and then inputs the acquired question into a learning model corresponding to each LLM, thereby obtaining the answer result for each LLM.

[0049] (Generation unit 132) The generation unit 132 generates a learning model corresponding to each LLM based on the response results of multiple LLMs acquired by the acquisition unit 131, for example. Specifically, the generation unit 132 generates a learning model that has been trained on combinations of input questions and response results output for each LLM using a first prompt in response to the questions. In other words, the generation unit 132 collects response results for each LLM by inputting questions to the target LLMs and generates a learning model that has been trained using the response results corresponding to each LLM.

[0050] (Change 133) The modification unit 133 modifies (or regenerates) the first prompt, for example, so that the same answer result is obtained in multiple LLMs. For example, the modification unit 133 modifies the first prompt so that the same answer result is obtained in multiple LLMs based on the answer result to the user's question output using the learning model generated by the generation unit 132. In this case, the modification unit 133 may determine whether the same answer result was obtained in multiple LLMs, and if it determines that the same answer result was not obtained, it may modify the first prompt so that the same answer result is obtained in multiple LLMs.

[0051] The modification unit 133 modifies the first prompt using, for example, a second prompt. Specifically, the modification unit 133 modifies the first prompt using a second prompt, which is a prompt that instructs the modification unit to modify the prompt so that the same answer result is obtained as the result of multiple LLMs answering the user's question. For example, the modification unit 133 modifies the first prompt using a second prompt that instructs the modification unit to modify the prompt so that an answer result is obtained based on data that meets predetermined conditions, such as the latest data or continuously changing data.

[0052] (Provider 134) The providing unit 134 provides the user with the answer to the user's question, for example. For example, the providing unit 134 provides information for displaying the answer to the user's question. Specifically, the providing unit 134 provides information for displaying the answer to the LLM that is the subject of the user's question from among multiple LLMs.

[0053] [5. Information Processing Flow] Next, the information processing procedure by the information processing system 1 according to the embodiment will be explained using Figure 9. Figure 9 is a flowchart showing the information processing procedure by the information processing system 1 according to the embodiment.

[0054] As shown in Figure 9, the information processing device 100 generates a learning model corresponding to each LLM based on the answer results of multiple LLMs to the question generated using the first prompt (step S201).

[0055] When the information processing device 100 receives a question from the user, it modifies the first prompt based on the answer result to the user's question output using the learning model, so that the same answer result is obtained in multiple LLMs (step S202).

[0056] The information processing device 100 provides the user with the answer result of the LLM that is the subject of the user's question from among the multiple LLMs (step S203).

[0057] [6. Effects] As described above, the information processing device 100 according to the embodiment includes a generation unit 132 and a modification unit 133. The generation unit 132 generates a learning model corresponding to each LLM, which is trained on combinations of input questions and the answer results output by each LLM for the question, based on each of the answer results of the multiple LLMs for the question generated using a first prompt that instructs the generation unit 132 to generate a question that can evaluate the difference in answers based on the answer results of multiple LLMs for the same question. The modification unit 133 modifies the first prompt so that the same answer result can be obtained in the multiple LLMs, based on the answer result to the user's question output using the learning model generated by the generation unit 132.

[0058] As a result, the information processing device 100 according to the embodiment can automatically generate prompts that absorb fluctuations, so that even when the LLM is changed, the answer results to the same question do not fluctuate. This enables the automatic generation of appropriate prompts that take UX into consideration.

[0059] Furthermore, the modification unit 133 determines whether the same answer result was obtained in multiple LLMs, and if it determines that the same answer result was not obtained, it modifies the first prompt so that the same answer result can be obtained.

[0060] As a result, the information processing device 100 according to the embodiment can change the prompt only when the same answer result is not obtained, for example, thus enabling more effective automatic prompt generation.

[0061] Furthermore, the modification unit 133 modifies the first prompt using a second prompt that instructs the system to modify the prompt so that the same answer result is obtained as the answer result of multiple LLMs to the user's question.

[0062] As a result, the information processing device 100 according to the embodiment can, for example, automatically generate more effective prompts by using other prompts that change the prompt so that there is no variation in the answer results to the same question.

[0063] Furthermore, the modification unit 133 modifies the first prompt using a second prompt that instructs the system to modify the prompt so that a response result based on data that meets predetermined conditions can be obtained.

[0064] As a result, the information processing device 100 according to the embodiment can, for example, automatically generate more effective prompts by using other prompts that modify the prompt so that a response result based on appropriate data is obtained.

[0065] Furthermore, the information processing device 100 according to the embodiment is characterized by further having a providing unit 134 that provides the user with the answer result of the LLM that is the subject of the user's question from among a plurality of LLMs.

[0066] This enables the information processing device 100 according to the embodiment to provide, for example, appropriate answer results in line with the user's questions.

[0067] [7. Hardware Configuration] Furthermore, the information processing device 100 according to the above-described embodiment is realized by a computer 1000 having the configuration shown in Figure 10. Figure 10 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

[0068] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0069] The HDD1400 stores programs executed by the CPU1100, as well as data used by such programs. The communication interface1500 acquires data from other devices via a predetermined communication network and sends it to the CPU1100, and transmits data generated by the CPU1100 to other devices via the predetermined communication network.

[0070] The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input / output interface 1600. The CPU 1100 acquires data from input devices via the input / output interface 1600. The CPU 1100 also outputs the generated data to output devices via the input / output interface 1600.

[0071] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 can be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), tape media, magnetic recording medium, or semiconductor memory.

[0072] For example, when the computer 1000 functions as an information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded on the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from other devices via a predetermined communication network.

[0073] [8. Other] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0074] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0075] Furthermore, the embodiments described above can be combined as appropriate, as long as the processing content is not contradictory.

[0076] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention.

[0077] Furthermore, the terms "section, module, unit" mentioned above can be replaced with "means" or "circuit," etc. For example, the acquisition unit can be replaced with acquisition means or acquisition circuit. [Explanation of Symbols]

[0078] 1. Information Processing System 10 Terminal devices 11 Communications Department 12 Input section 13 Output section 14 Control Unit 100 Information Processing Devices 110 Communications Department 120 Storage section 121 Prompt Information Storage Unit 122 Learning Model Information Storage Unit 130 Control Unit 131 Acquisition Department 132 Generation part 133 Changes 134 Provision Department 141 Receiving Unit 142 Transmitter N Network

Claims

1. A generation unit generates a learning model corresponding to each LLM, which is trained on combinations of input questions and the output answers of each LLM for the question, based on the answer results of each of the multiple LLMs for the question generated using a first prompt that instructs the generation unit to generate a question that can evaluate the difference in answers based on the answer results of multiple LLMs for the same question. A modification unit modifies the first prompt based on the answer results to the user's questions output using the learning model generated by the generation unit, so that the same answer results are obtained in the multiple LLMs. An information processing device characterized by having the following features.

2. The aforementioned modified part is, The system determines whether the same answer result was obtained in the multiple LLMs, and if it determines that the same answer result was not obtained, it modifies the first prompt to obtain the same answer result. The information processing apparatus according to feature 1.

3. The aforementioned modified part is, Modify the first prompt using a second prompt that instructs the system to modify the prompt so that the same answer result is obtained as the result of multiple LLM responses to the user's question. The information processing apparatus according to feature 1.

4. The aforementioned modified part is, The first prompt is modified using the second prompt, which instructs the system to change the prompt so that the response result based on data that satisfies predetermined conditions is obtained. The information processing apparatus according to claim 3.

5. A providing unit that provides the user with the answer result of the LLM that is the subject of the user's question from among the multiple LLMs. The information processing apparatus according to claim 1, further comprising the above.

6. A method of information processing performed by a computer, A generation process involves generating a learning model that corresponds to each LLM, based on the answer results of multiple LLMs to the generated question, using a first prompt that instructs the system to generate a question that allows evaluation of the difference in answers based on the answer results of multiple LLMs to the same question, and learning models that have learned the combination of the input question and the answer result output by each LLM for that question. A modification step involves modifying the first prompt based on the answer results to the user's questions output using the learning model generated in the generation step, so that the same answer results are obtained in the multiple LLMs. An information processing method characterized by including

7. A generation procedure for generating a learning model that corresponds to each LLM, based on the answer results of multiple LLMs to the generated question, using a first prompt that instructs the system to generate a question that allows evaluation of the difference in answers based on the answer results of multiple LLMs to the same question, and which has learned the combination of the input question and the answer result output by each LLM for that question, and A modification procedure to modify the first prompt so that the same answer result is obtained in the multiple LLMs, based on the answer result to the user's question output using the learning model generated by the generation procedure, An information processing program characterized by causing a computer to execute it.

Images