Prompt adjustment program, prompt adjustment method, and information processing device

Abstract

The present invention makes it possible to achieve optimization of a prompt at a small calculation cost by configuring a prompt by: generating a plurality of sample prompts corresponding to a plurality of types of combinations on the basis of a plurality of items corresponding to a plurality of blocks constituting the prompt, and configuration information represented by a combination of possible values for the respective items; and determining, as an adjustment item for adjusting the prompt, an item included in the combination used for generating the sample prompt having the highest inference accuracy among results obtained by inputting the plurality of sample prompts to a language model.

Description

Prompt adjustment program, prompt adjustment method, and information processing device

[0001] The present invention relates to a prompt adjustment program, a prompt adjustment method, and an information processing device.

[0002] In recent years, language generation AI (Artificial Intelligence) has been attracting attention, and language generation AI is being used to perform tasks such as QA (Question and Answer) and information recommendation. In order to improve the accuracy of individual tasks in language generation AI, fine-tuning of pre-trained models has conventionally been performed, but fine-tuning requires enormous costs (computation time and computational resources).

[0003] Furthermore, changing the sentences (prompts) input to the pre-trained model can improve the accuracy of the output of the pre-trained model. For this reason, for example, a technique (prompting technique) for generating sentences to be input as prompts to large language models (LLMs) has been attracting attention (see, for example, Patent Literature 1). Conventionally, a technique for optimizing short prompts of a few words has also been known (see, for example, Non-Patent Literature 1).

[0004] Patent No. 7,325,152 U.S. Patent Application Publication No. 2023 / 0316001

[0005] Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, Zhiting Hu, "RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning", [online], May 25, 2022, CVPR2021, [Retrieved November 29, 2023], Internet <URL: / arxiv.org / abs / 2205.12548>

[0006] However, in such conventional prompt generation methods, the computational cost is calculated as the length of the prompt raised to the power of the number of vocabulary elements, so the computational cost becomes enormous when attempting to generate prompts with a large (long) vocabulary element.

[0007] In one aspect, the present invention aims to enable prompt optimization to be achieved with low computational cost.

[0008] Therefore, this prompt adjustment program causes a computer to execute a process of generating a plurality of sample prompts corresponding to a plurality of combinations based on configuration information that represents the configuration of a prompt as a combination of a plurality of items corresponding to a plurality of blocks that make up the prompt and the possible values ​​that each item can take, and determining, as an adjustment item for adjusting the prompt, an item included in the combination used to generate the sample prompt that has the highest inference accuracy in the results obtained by inputting the plurality of sample prompts into a language model.

[0009] According to one embodiment, prompt optimization can be achieved at low computational cost.

[0010] 10 is a block diagram showing an example of the hardware (HW) configuration of a computer 10 that realizes the functions of a prompt adjustment device as an example of a first embodiment. FIG. 11 is a diagram illustrating an example of the functional configuration of a prompt adjustment device as an example of a first embodiment. FIG. 12 is a diagram illustrating an example of a prompt configuration. FIG. 13 is a diagram illustrating an example of formulation information in a prompt adjustment device as an example of a first embodiment. FIG. 14 is a diagram illustrating an example of formulation information and an example of all combinations of possible attribute values ​​in a prompt adjustment device as an example of a first embodiment. FIG. 15 is a diagram illustrating an example of management information created by an optimization unit of a prompt adjustment device according to the first embodiment. FIG. 16 is a diagram illustrating an example of prompt design information generated by an optimization unit of a prompt adjustment device according to an example of a first embodiment. FIG. 17 is a flowchart illustrating the processing of the optimization unit of a prompt adjustment device according to the first embodiment. FIG. 18 is a flowchart illustrating the details of the processing of step S4 in the flowchart shown in FIG. 8. FIG. 19 is a diagram illustrating an example of the functional configuration of a prompt adjustment device as an example of a second embodiment. FIG. 19 is a diagram illustrating an example of data mining input information in a prompt adjustment device according to the second embodiment. FIG. 19 is a diagram illustrating an example of interactions generated by an interaction extraction unit of a prompt adjustment device according to the second embodiment. FIG. 19 is a diagram illustrating an example of a limited search range. 18 is a flowchart for explaining the processing of an optimization unit of the prompt adjustment device according to the second embodiment. FIG. 19 is a flowchart for explaining the processing of a pre-processing unit of the prompt adjustment device according to the second embodiment. FIG. 20 is a flowchart for explaining the processing of a post-processing unit of the prompt adjustment device according to the second embodiment. FIG. 21 is a flowchart for explaining the processing of an optimization processing unit of the prompt adjustment device according to the second embodiment. FIG. 22 is a flowchart for explaining a modification of the flowchart shown in FIG. 17. FIG. 23 is a flowchart for explaining a modification of the formulation information.

[0011] Hereinafter, embodiments of the prompt adjustment program, prompt adjustment method, and information processing device will be described with reference to the drawings. However, the embodiments shown below are merely examples, and are not intended to exclude various modifications or applications of techniques not explicitly stated in the embodiments. In other words, each embodiment can be implemented with various modifications (e.g., combinations of the embodiments and modifications) within the scope of its intent. Furthermore, each figure does not intend to include only the components shown in the figure, but may include other functions, etc.

[0012] (I) Description of the First Embodiment (A) Configuration (A-1) Hardware Configuration Fig. 1 is a block diagram showing an example of the hardware (HW) configuration of a computer 10 that realizes the functions of a prompt adjustment device 1a as an example of the first embodiment. When multiple computers are used as HW resources that realize the functions of the prompt adjustment device 1a, each computer may have the HW configuration exemplified in Fig. 1.

[0013] The prompt adjustment device 1a is an information processing device that adjusts prompts to be input to a language model so that the output from the language model is highly accurate. The language model may be a large-scale language model (LLM).

[0014] As shown in FIG. 1, the computer 10 may include, as its HW configuration, a processor 10a, a graphics processing unit 10b, a memory 10c, a storage unit 10d, an IF (Interface) unit 10e, an IO (Input / Output) unit 10f, and a reading unit 10g, for example.

[0015] The processor 10a is an arithmetic processing unit that performs various controls and calculations, and is an example of a control unit. The processor 10a may be connected to each block in the computer 10 via a bus 10j so that they can communicate with each other. The processor 10a may be a multiprocessor including multiple processors, a multi-core processor having multiple processor cores, or a configuration having multiple multi-core processors.

[0016] Examples of the processor 10a include integrated circuits (ICs) such as a CPU, MPU, APU, DSP, ASIC, and FPGA. Note that the processor 10a may be a combination of two or more of these integrated circuits. CPU is an abbreviation for Central Processing Unit, MPU is an abbreviation for Micro Processing Unit, APU is an abbreviation for Accelerated Processing Unit, DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific IC, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0017] The graphics processing device 10b controls screen display for an output device such as a monitor in the IO unit 10f. The graphics processing device 10b may also be configured as an accelerator that executes machine learning processing and inference processing using a machine learning model. The graphics processing device 10b may be any of a variety of arithmetic processing devices, such as a graphics processing unit (GPU), an APU, a DSP, an ASIC, an FPGA, or other integrated circuits (ICs).

[0018] The memory 10c is an example of HW that stores various types of data, programs, and other information. The memory 10c may be, for example, a volatile memory such as a dynamic random access memory (DRAM) or a non-volatile memory such as a persistent memory (PM), or both.

[0019] The storage unit 10d is an example of HW that stores various types of data, programs, and other information. Examples of the storage unit 10d include various storage devices such as a magnetic disk device such as a hard disk drive (HDD), a semiconductor drive device such as a solid state drive (SSD), and a nonvolatile memory. Examples of nonvolatile memory include a flash memory, a storage class memory (SCM), and a read-only memory (ROM).

[0020] The storage unit 10d may store a program 10h (prompt adjustment program) that realizes all or part of the various functions of the computer 10.

[0021] For example, the processor 10a of the prompt adjustment device 1a can implement a prompt adjustment function, which will be described later, by loading a program 10h stored in the storage unit 10d into the memory 10c and executing it.

[0022] The IF unit 10e is an example of a communication IF that controls the connection and communication between the computer 10 and other computers. For example, the IF unit 10e may include an adapter that complies with a LAN (Local Area Network) such as Ethernet (registered trademark) or optical communication such as FC (Fibre Channel). The adapter may support either or both wireless and wired communication methods.

[0023] For example, the prompt adjustment device 1a may be connected to other information processing devices, databases, etc. (not shown) via the IF unit 10e and a network so that they can communicate with each other. The program 10h may be downloaded to the computer 10 from the network via the communication IF and stored in the storage unit 10d.

[0024] The IO unit 10f may include one or both of an input device and an output device. Examples of input devices include a keyboard, a mouse, and a touch panel. Examples of output devices include a monitor, a projector, and a printer. The IO unit 10f may also include a touch panel that combines an input device and a display device. The output device may be connected to the graphics processing device 10b.

[0025] The reading unit 10g is an example of a reader that reads data and program information recorded on the recording medium 10i. The reading unit 10g may include a connection terminal or device to which the recording medium 10i can be connected or inserted. Examples of the reading unit 10g include an adapter compliant with USB (Universal Serial Bus) or the like, a drive device that accesses a recording disk, and a card reader that accesses a flash memory such as an SD card. Note that the recording medium 10i may store the program 10h, and the reading unit 10g may read the program 10h from the recording medium 10i and store it in the memory unit 10d.

[0026] Examples of the recording medium 10i include non-transitory computer-readable recording media such as magnetic / optical disks and flash memories. Examples of magnetic / optical disks include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs, and HVDs (Holographic Versatile Discs). Examples of flash memories include semiconductor memories such as USB memories and SD cards.

[0027] The above-described HW configuration of the computer 10 is an example. Therefore, the HW in the computer 10 may be increased or decreased (for example, adding or deleting any block), divided, integrated in any combination, or the HW may be added or deleted as needed.

[0028] (A-2) Example of Functional Configuration FIG. 2 is a diagram illustrating an example of the functional configuration of a prompt adjustment device 1a as an example of the first embodiment.

[0029] 2, the prompt adjustment device 1a may illustratively include functions as an input processing unit 11, an optimization unit 12a, and an output processing unit 17. The functions of the input processing unit 11, the optimization unit 12a, and the output processing unit 17 may be realized, for example, by the above-mentioned processor 10a executing a program 10h (prompt adjustment program).

[0030] The input processing unit 11 receives input of multiple (multiple types of) attributes included in the prompt and possible values ​​for each attribute by, for example, a user via a keyboard, mouse, network, or the like (not shown).

[0031] Input data is also input to the input processing unit 11. The input data includes a question (question sentence) and its correct answer. The input data including the question and the correct answer may be called sample data with correct answer. The input data may also be input by a user or the like via a keyboard, mouse, network, or the like.

[0032] The input data is changed as appropriate depending on the task. Here, for example, a movie recommendation task is taken as an example, in which a user inputs a list of movies they have watched and a list of movies they would like to recommend next, and a movie that the user is likely to watch next is selected from the list of movies and output.

[0033] The "questions" included in the input data correspond to the "list of movies the user has watched" and the "list of multiple movies the user would like to recommend next," which are input as described above. The "correct answers" correspond to the "correct movie candidates." Below are specific examples of questions and their correct answers.

[0034] Example question: History list "AAA, BBB, CCC" (AAA etc. are movie names), Recommended candidate list "DDD, EEE, FFF, GGG, HHH" (DDD etc. are movie names) Example correct answer: GGG (GGG is the name of a movie)

[0035] The input processing unit 11 passes each piece of input information to the optimization unit 12a. The input processing unit 11 may, for example, store each piece of input information in a predetermined storage area of ​​the memory 10c or the storage unit 10d. The optimization unit 12a may obtain the information by reading it from the storage area.

[0036] The optimization unit 12a generates information for adjusting (creating) a prompt based on the input data. Information for adjusting or creating a prompt can be referred to as prompt design information. By inputting a prompt created based on this prompt design information into the LLM, a highly accurate answer can be obtained that improves task performance using the LLM (e.g., question answering or information recommendation). In other words, the optimization unit 12a generates prompt design information for creating an optimized prompt.

[0037] As shown in FIG. 2, the optimization unit 12a functions as a formulation processing unit 13.

[0038] The formulation processor 13 formulates the configuration of the prompt. The formulation processor 13 formulates the prompt by combining multiple attributes included in the prompt and the values ​​that each attribute can take. Formulation can also be called parameterization or formatting.

[0039] The formulation processor 13 divides the prompt into a plurality of blocks. Each block is assigned a function in the prompt. A block is an element that defines the prompt. A block can also be called an attribute.

[0040] The formulation processing unit 13 may, for example, treat each of the multiple sentences included in the prompt as a block, or may determine the block based on specific words included in the sentence, and can be implemented with appropriate modifications.

[0041] 3 is a diagram showing an example of a prompt configuration, in which symbol A indicates an example of a prompt, and symbol B indicates a block configuration corresponding to the prompt shown by symbol A.

[0042] FIG. 3 illustrates an example prompt for an information recommendation task that allows the LLM to make movie recommendations.

[0043] A prompt is composed of a combination of multiple blocks. The prompt shown in Figure 3 has a task block, an input format block, an output format block, an output note block, an example block, and a question block. The task block, input format block, output format block, output note block, example block, and question block are each examples of blocked items.

[0044] These task blocks, input format blocks, output format blocks, output note blocks, example blocks, and question blocks correspond to the prompts P1, P2, P3, P4, P5, and P6, respectively, indicated by the symbol A in Figure 3. Note that the contents of the Inputs in P5 and P6 correspond to the "questions" in the input data described above.

[0045] As mentioned above, each block is assigned a function in the prompt: for example, the task block indicates the task, and the question block poses the actual question to the LLM.

[0046] In the prompt indicated by symbol A in Figure 3, the task block is at the beginning and the question block is at the end, but the accuracy of the LLM's response is likely to change if the task block is placed second, third, etc., at the end, or if the question block is placed first, second, third, etc. In particular, the attribute related to the order of "which block is placed in what order" in the prompt is considered to be important in terms of performance.

[0047] Hereinafter, the low / high accuracy of the results obtained by inputting a prompt into the LLM may be simply referred to as low / high accuracy.

[0048] The formulation processor 13 formulates a prompt by treating the elements that define the prompt as attributes and setting possible values ​​for each of the multiple attributes. This makes it possible to treat the prompt as a problem in which the optimal combination is automatically determined. Hereinafter, the information that associates each attribute with the possible values ​​for each attribute, created by the formulation processor 13 when formulating the prompt, may be referred to as formulation information.

[0049] The formulation information represents attribute settings associated with formulating a prompt. The formulation processing unit 13 may create the formulation information based on the input sample prompt. The sample prompt may be a prompt generated by applying a combination of parameters to each piece of sample data (for example, a history list including specific movie names and a recommendation candidate list) and outputting it.

[0050] FIG. 4 is a diagram showing an example of formulation information in the prompt adjustment device 1a as an example of the first embodiment.

[0051] In the formulation information shown in FIG. 4, for example, the order from the top of the prompt (1st to 6th) is used as an attribute, and possible values ​​include task block, input format block, output format block, output note block, example block, and question block. The order from the top of the prompt is an example of the position of a block in the prompt. Therefore, the attribute "order from the top of the prompt" is an example of an item that represents the position of a block in the prompt.

[0052] Thus, each of the first to sixth attributes is set to one of a task block, an input format block, an output format block, an output note block, an example block, and a question block.

[0053] In the example shown in Figure 4, in addition to the first to sixth attributes (in order from the beginning), the language used for instruction statements and the language used for output statements are also set as attributes. The possible values ​​for the language used for instruction statements and the language used for output statements are English and Japanese. The "language used for instruction statements" corresponds to the language of the task block, input / output format block, and output note block, and the "language used for output statements" corresponds to the language of the example block and question block.

[0054] Furthermore, the above-mentioned possible values ​​of the language used in the instruction sentence and the language used in the output sentence, namely, English and Japanese, are examples of variations in the expression of the block.

[0055] The optimization unit 12a may perform the following settings (prompt design), for example, by selecting and setting a value selected from the possible values ​​for each attribute.

[0056] 1st: Task block 2nd: Input format block 3rd: Output format block 4th: Output notes block 5th: Example block 6th: Question block Language used for instructions: English Language used for output statements: English

[0057] The optimization unit 12a generates multiple (multiple types of) prompt design information by switching the value selected from the possible values ​​for each attribute. The value selected from the possible values ​​set for each attribute can be called a parameter, input parameter, or hyperparameter.

[0058] The optimization unit 12a generates all combinations of parameters for all attributes by switching between possible values ​​(parameters) for each attribute.

[0059] FIG. 5 is a diagram showing an example of formulation information and examples of all combinations of possible attribute values ​​in the prompt adjustment device 1a as an example of the first embodiment.

[0060] 5, symbol A indicates another example of formulation information, and symbol B indicates all possible combinations of attribute values ​​corresponding to the formulation information indicated by symbol A. However, in this example, the question block is formulated as having only short.

[0061] In the formulation information illustrated by symbol A in FIG. 5, the order from the beginning of the prompt (first to third) is used as an attribute, and the possible values ​​are an instruction block, a question block, and a format block.

[0062] 5, in addition to the first to third blocks (in order from the beginning), the length of the description of the first block (1stlen), the length of the description of the second block (2ndlen), and the length of the description of the third block (3rdlen) are also set as attributes. The possible values ​​for the length of the description of the first block (1stlen), the length of the description of the second block (2ndlen), and the length of the description of the third block (3rdlen) are short and long. However, if the first block (1st) {or the second block (2nd), or the third block (3rd)}} is a question, the only possible value for 1stlen (or 2ndlen, 3rdlen) is short.

[0063] The aforementioned possible values ​​of "short" and "long" for the length of the explanation for the first block (1stlen), the length of the explanation for the second block (2ndlen), and the length of the explanation for the third block (3rdlen) are examples of variations in block representation. For example, the accuracy of prompts can change significantly depending on whether the prompt for a task block is short (brief and concise) or long (long and detailed).

[0064] In this example of attribute settings indicated by symbol A in Figure 5, the total number of possible combinations of values ​​for each attribute is 48 (= 6 x 2 x 2 x 2) ÷ 2 (because question can only be short) = 24, as indicated by symbol B. The number of possible combinations of values ​​for each attribute is represented as K. In the example indicated by symbol B in Figure 3, K = 24.

[0065] The formulation processing unit 13 stores the generated formulation information and all combinations of values ​​that each attribute can take in a predetermined storage area of ​​the memory 10c.

[0066] The formulation processing unit 13 creates formulation information (configuration information) that represents the configuration of a prompt as a combination of multiple (attributes: order, language; see FIG. 4) or multiple (attributes: order, length; see FIG. 5) corresponding to the multiple blocks that make up the prompt, and the values ​​that each item can take.

[0067] The optimization unit 12a generates a plurality (K) of prompts (sample prompts) by performing processing such as rearranging the sample data with correct answers to match each of all (K) combinations of possible values ​​of each attribute generated by the formulation processing unit 13. The optimization unit 12a generates K×N sample prompts by using a plurality (N) of sample data with correct answers to generate a plurality (K) of prompts (sample prompts).

[0068] The optimization processing unit 18 generates a plurality of sample prompts corresponding to a plurality of combinations based on the formulation information and a plurality of sample data with correct answers.

[0069] The optimization unit 12a inputs each of the generated K × N sample prompts into the LLM, compares the output of the LLM with the correct answer, and calculates the accuracy for each combination. Here, the accuracy for each combination may be calculated by, for example, calculating the average accuracy of the N sample data with correct answers. Note that the accuracy calculation method is not limited to this and can be modified as appropriate.

[0070] The optimization unit 12 may create management information for managing the accuracy of each combination. The management information includes combinations of multiple sample data with correct answers and combinations of values ​​that can be taken by multiple attributes in the formulation information.

[0071] FIG. 6 is a diagram illustrating management information created by the optimization unit 12a of the prompt adjustment device 1a according to the first embodiment.

[0072] FIG. 6 shows management information in the case where four (N=4) sample data with correct answers are used based on the formulation information exemplified in FIG.

[0073] In the management information exemplified in FIG. 6, a combination id, a sample_id, an attribute combination pattern, and whether or not the answer is correct are associated with each other.

[0074] The attribute combination pattern may be referred to as a parameter set. The combination id is identification information that identifies the attribute combination pattern. The sample_id is identification information that identifies the sample data with correct answers. Whether or not the answer is correct may indicate whether or not the result obtained by inputting a prompt generated based on the corresponding sample data with correct answers and the attribute combination pattern into the LLM is correct, or may be calculated using a commonly used accuracy index. Here, an example of simply whether or not the answer is correct is shown.

[0075] FIG. 7 is a diagram illustrating prompt design information generated by the optimization unit 12a of the prompt adjustment device 1a as an example of the first embodiment.

[0076] In the example shown in FIG. 7, symbol A indicates an example in which prompt design information is represented in a table format, and symbol B indicates an example in which prompt design information is represented in JSON (JavaScript Object Notation) format.

[0077] The optimization unit 12a passes the generated prompt design information to the output processing unit 17. For example, the optimization unit 12a may store the prompt design information in a predetermined storage area of ​​the memory 10c or the storage unit 10d. The output processing unit 17 may obtain the information by reading out the information stored in these storage areas.

[0078] The optimization unit 12a extracts (generates) the most accurate combination as prompt design information. The attributes and possible values ​​included in the combination included in this prompt design information are used as adjustment items for adjusting the prompt.

[0079] That is, the optimization unit 12a determines, as the adjustment item for adjusting the prompt, the item included in the combination used to generate the sample prompt that has the highest inference accuracy when multiple sample prompts are input into an LLM (language model).

[0080] The output processing unit 17 performs processing for outputting the prompt design information generated by the optimization unit 12a. For example, the output processing unit 17 may output the prompt design information to another system.

[0081] (B) Operation The processing of the optimization unit 12a of the prompt adjustment device 1a according to the first embodiment configured as described above will be described with reference to the flowchart (steps S1 to S6) shown in FIG.

[0082] In step S1, the optimization unit 12a extracts, from the input data stored by the input processing unit 11, a plurality of (N) sample data sets with correct answers to be used for optimizing the prompts.

[0083] In step S2, the formulation processing unit 13 parameterizes (formulates) the configuration of one or more sample prompts to create formulation information.

[0084] In step S3, the optimization unit 12a generates all (K) combinations of parameters for all attributes by switching the values ​​(parameters) that can be taken for each attribute in the formulation information. The optimization unit 12a also generates K × N sample prompts by using N pieces of sample data with correct answers to generate K sample prompts.

[0085] In step S4, the optimization unit 12a inputs each of the generated K × N sample prompts into the LLM and calculates the accuracy of each output LLM output. Here, for each sample prompt, it calculates whether each output LLM output is correct (0: incorrect, 1: correct).

[0086] In step S5, the optimization unit 12a calculates the accuracy for each combination by calculating the average accuracy of the N sample prompts, and in step S6, the optimization unit 12a extracts (generates) the combination with the highest accuracy as prompt design information, after which the process ends.

[0087] Next, the details of the process of step S4 in the flowchart shown in FIG. 8 will be described with reference to the flowchart (steps S41 to S48) shown in FIG.

[0088] In step S41, the optimization unit 12a creates a table with N x K rows, each of which has a column with a combination ID, a sample_id, an attribute combination pattern, and whether or not the combination is correct as terms. The "correct or not" column is left blank. In this table, rows are created in ascending order of combination ID, and for the same combination ID, rows are created in ascending order of sample_id. The attribute combination pattern is the value of each attribute corresponding to the combination ID.

[0089] In step S42, a loop process is started in which the control up to step S47 is repeatedly performed for all nodes present in the N samples. The optimization unit 12a extracts one sample from the N samples. This extracted sample is called sample n.

[0090] In step S43, a loop process is started in which the control up to step S46 is repeatedly performed for all K combinations. The optimization unit 12a extracts one combination from the K combinations. This extracted combination is referred to as combination k.

[0091] In step S44, the optimization unit 12a generates a prompt for the combination k based on the sample n.

[0092] In step S45, the optimization unit 12a inputs the generated prompt into the LLM and obtains the result (output).

[0093] In step S46, the optimization unit 12a registers the result of the LLM in the (N×k+n)th row of the management information created in step S41. For example, if the prompt is a classification question, the optimization unit 12a registers information indicating whether the result is the correct answer, and if the prompt is a ranking question, the optimization unit 12a registers information on the rank and calculation results such as a general accuracy index based on the rank (rank) of the correct answer.

[0094] In the management information illustrated in FIG. 6, when the first row is row 0, the combination id=k, sample_id=n is the N×k+nth row.

[0095] In step S47, loop end processing corresponding to step S43 is performed. When processing for all combinations is completed, the process proceeds to step S48.

[0096] In step S48, a loop end process corresponding to step S42 is performed. When the process for all samples is completed, the process ends.

[0097] (C) Effect As described above, according to the prompt adjustment device 1a of the first embodiment, the formulation processing unit 13 (optimization unit 12a) formulates the prompt by dividing it into multiple blocks, and creates formulation information that associates multiple attributes with the values ​​that each attribute can take.

[0098] Furthermore, the optimization unit 12a processes the sample data with correct answers in accordance with all possible combinations of values ​​for each attribute, thereby generating a plurality of sample prompts.

[0099] Then, the optimization unit 12a inputs each of the generated sample prompts into the LLM, compares the output of the LLM with the correct answer, calculates the accuracy for each combination, and extracts (generates) the combination with the highest accuracy as prompt design information.

[0100] This makes it easy to generate prompts that can get highly accurate results from the LLM.

[0101] (II) Description of the Second Embodiment (A) Configuration In the prompt adjustment device 1a of the first embodiment described above, the optimization unit 12a inputs each of the generated sample prompts into the LLM, compares the output of the LLM with the correct answer, and calculates the accuracy for each combination. Therefore, for example, if the formulation information has a large number of attributes, the number of possible combinations of values ​​for each attribute becomes enormous, which may increase the calculation cost.

[0102] Therefore, the prompt adjustment device 1b as an example of the second embodiment aims to easily generate prompts that can obtain highly accurate results from an LLM and to reduce calculation costs. The prompt adjustment device 1b is an information processing device that adjusts prompts to be input to a language model so that the output from the language model is highly accurate.

[0103] FIG. 10 is a diagram illustrating a functional configuration of a prompt adjustment device 1b as an example of the second embodiment.

[0104] 10, the prompt adjustment device 1b of the second embodiment includes an optimization unit 12b instead of the optimization unit 12a of the first embodiment, and other parts are configured in the same manner as the prompt adjustment device 1a of the first embodiment. Also, the prompt adjustment device 1b of the second embodiment has the same hardware configuration as the prompt adjustment device 1a of the first embodiment.

[0105] The optimization unit 12b includes an interaction extraction unit 14, a pre-processing unit 15, a post-processing unit 16, and an optimization unit 18 in addition to the formulation processing unit 13 similar to that of the first embodiment.

[0106] In the drawings, the same reference numerals as those already mentioned indicate the same parts, and therefore the description thereof will be omitted.

[0107] The interaction extraction unit 14 uses a known data mining technique to detect interactions between multiple attributes in the input data. An interaction is a synergistic effect that appears when multiple attributes are combined. An interaction is an example of a result of data mining.

[0108] The interaction extraction unit 14 receives, for example, CSV (Comma Separated Values) data in which the combination of hyperparameters is the attribute and whether the output is correct or not is the label, and extracts interactions having characteristics of correct answers (incorrect answers).

[0109] The interaction extraction unit 14 may receive data mining input information in CSV format, such as the example shown in FIG.

[0110] FIG. 11 is a diagram illustrating data mining input information in the prompt adjustment device 1b according to the second embodiment.

[0111] The optimization unit 12b may create data mining input information such as the example shown in Fig. 11. The data mining input information is information generated based on formulation information and sample data with correct answers, similar to the management information in the first embodiment, and is formed by associating each combination of attributes with information indicating whether the output from the LLM is correct or not.

[0112] The interaction extraction unit 14 performs data mining on the input data mining input information to calculate (extract, detect) interactions between multiple attributes. An interaction is a synergistic effect that appears when multiple attributes are combined.

[0113] The interaction extraction unit 14 extracts interactions between attributes and combinations of attributes through data mining. The interactions extracted by the interaction extraction unit 14 are input to calculation terms for Bayesian optimization by the optimization processing unit 18, which will be described later. Wide Learning (registered trademark), for example, may be used as a data mining method.

[0114] The interaction extraction unit 14 performs data mining using as input data mining input information that associates the results (whether correct or not) obtained by inputting multiple sample prompts into an LLM (language model) with the combinations used to generate the sample prompts.

[0115] 12 and 13 are diagrams illustrating interactions generated by the interaction extraction unit 14 of the prompt adjustment device 1b according to the second embodiment. Note that Fig. 12 illustrates an interaction with high accuracy, and Fig. 13 illustrates an interaction with low accuracy.

[0116] 12 and 13, labels correspond to chunks. Label=1 indicates good accuracy, and label=0 indicates poor accuracy. Chunks indicate interactions.

[0117] Also, "^" means "and" and corresponds to AND in a logical expression. =0 / =1 indicates that the value of the attribute is not applicable / is not relevant.

[0118] For example, the first line of Figure 12, "1stlen_short=0 ∧ 2nd_instruction=1 ∧ 2ndlen_short=0", indicates that the attribute "1stlen" is not "short" (=0), and the attribute "2nd" is "instruction" (=1), and the attribute "2ndlen" is not "short" (=0). In other words, a prompt where a long (non-short) instruction comes second and the first is long (non-short) indicates good precision.

[0119] 13, "1stlen_short=1 ∧ 2nd_instruction=1 ∧ 2ndlen_short=1" indicates that the attribute "1stlen" is "short" (=1), the attribute "2nd" is "instruction" (=1), and the attribute "2ndlen" is "short" (=1). In other words, a short instruction coming second and a short prompt coming first indicate poor accuracy.

[0120] The interaction extraction unit 14 passes the generated interactions to the post-processing unit 16 and the optimization unit 12. The interaction extraction unit 14 may, for example, store the generated interactions in a predetermined storage area of ​​the memory 10c or the storage unit 10d. The post-processing unit 16 and the optimization unit 12 may obtain the generated interactions by reading out the information stored in these storage areas.

[0121] The preprocessing unit 15 reduces the data to be processed (data mining) by the interaction extraction unit 14 .

[0122] The preprocessing unit 15 prepares a list of conditions that are qualitatively known to result in low accuracy (poor performance), such as a single attribute or a combination of attributes. This list may be called an exclusion list. The exclusion list is stored, for example, in a predetermined storage area of ​​the storage unit 12d.

[0123] The exclusion list may store, for example, the condition that the "question block" has the characteristic of completion (the sentence ends midway and the remaining part is left to be answered) and that the "question block" is placed at the beginning of the prompt. The conditions registered in the exclusion list may be called exclusion conditions.

[0124] In addition, the exclusion list may store a condition that it contains expressions that do not match the learning data (general sentences) of the LLM.

[0125] Furthermore, the exclusion list may store conditions where the language changes frequently during the course of a conversation.

[0126] In addition, the pre-processing unit 15 uses, for example, sample data with correct answers to create a sample prompt that matches a single attribute pattern or a combination of attributes that corresponds to the conditions of the exclusion list, and checks in advance whether the results obtained by inputting this sample prompt into the LLM are actually of poor accuracy.

[0127] For example, the preprocessing unit 15 uses sample data with correct answers to set the "question block" as the m-th block from the beginning, and while randomly changing the other blocks, creates prompts, inputs them into the LLM, and checks the accuracy of the output results. This trial is repeated while switching m=1, 2, 3, ..., and if the accuracy is significantly low (for example, the accuracy is below a threshold), the value of the "question block" is not entered in the m-th block.

[0128] That is, the preprocessing unit 15 generates sample prompts (second sample prompts) using sample data with correct answers under conditions (exclusion conditions) that are qualitatively known to result in low accuracy (poor performance) for single-attribute patterns or combinations of attributes, and then inputs these sample prompts into the LLM to verify in advance the accuracy of the results obtained. If the verification results show that the accuracy is significantly low, such as below a preset threshold, the preprocessing unit 15 stores the combination patterns that fall under these exclusion conditions as a skip list, and skips the processing (input to the LLM and processing by the interaction extraction unit 14 (data mining)) for combinations whose conditions are included in the list.

[0129] In this way, the preprocessing unit 15 reduces the variations in attribute combinations to be included in data mining. A skip list is a list of pairs of attributes and their values. A skip list can also be called an exclusion list. A skip list is a list of pairs of attributes and their values ​​that are determined by formulation.

[0130] If the experiment verifies that the accuracy is low, the preprocessing unit 15 excludes the combination patterns that meet the conditions in the exclusion list from the processing (data mining) of the interaction extraction unit 14 .

[0131] The preprocessing unit 15 compares the combinations of multiple attributes and their possible values ​​proposed by Bayesian optimization based on the formulation information created by the formulation processing unit 13 with the above-mentioned skip list, and if the proposed attribute pattern or attribute combination pattern is included in the exclusion conditions registered in the skip list, skips subsequent processing (input to the LLM and its accuracy evaluation, and input to the interaction extraction unit 14). Skipping subsequent processing (input to the LLM and its accuracy evaluation, and input to the interaction extraction unit 14) for attribute patterns or attribute combination patterns that meet the skip list conditions can be referred to as pruning.

[0132] The pre-processing unit 15 does not generate combination patterns in advance and then exclude them, but when a combination that matches the skip list is presented from Bayesian optimization, it proceeds to the next proposal loop without processing.

[0133] For example, if "1st: question" is in the skip list, and "1st: question" is included in the Bayesian optimization proposal, the process will be skipped and the next proposal will be advanced. As a result, the skipped combination will not be added to the input for data mining. In other words, the skipped combination will not be submitted to the LLM to calculate accuracy, and the result will not be added to the input for data mining.

[0134] The preprocessing unit 15 excludes configuration information that includes a combination that satisfies the exclusion condition from the target of data mining.

[0135] In addition, before excluding configuration information containing combinations that satisfy the exclusion conditions from the target of data mining, the preprocessing unit 15 generates a second sample prompt using sample data for the combination that satisfies the exclusion conditions, and if the accuracy of the result obtained by inputting this second sample prompt into the language model is below a threshold, excludes the combination that satisfies the exclusion conditions from the target of data mining.

[0136] The post-processing unit 16 uses statistical testing to reduce the interactions output from the interaction extraction unit 14 based on p-values, where p-values ​​represent statistical probabilities.

[0137] The post-processing unit 16 tests the combinations that appear in the interactions output from the interaction extraction unit 14 and calculates p-values. A known statistical testing method, such as the Breslow-Day test, may be used for the test. The post-processing unit 16 excludes interactions with p-values ​​equal to or greater than a threshold value (e.g., 0.05) from the targets of processing by the optimization processing unit 18, which will be described later. The exclusion of interactions from the targets of processing by the optimization processing unit 18 may be referred to as pruning.

[0138] The post-processing unit 16 may rank (sort) and cut off the interactions obtained as a result of data mining by the interaction extraction unit 14 by p-values ​​using a statistical test to determine whether the interactions are significant or not.

[0139] The post-processing unit 16 tests whether interactions between parameter combinations affect inference performance. The post-processing unit 16 excludes interactions whose p-values ​​are equal to or greater than a threshold value (e.g., 0.05), i.e., interactions that have no effect, from the processing targets (search candidates) of the optimization unit 18.

[0140] The post-processing unit 16 excludes interactions (data mining results) whose p-value (a value representing the impact of data mining results on inference performance) exceeds a threshold (in this second embodiment, is equal to or greater than the threshold) from the optimization target.

[0141] The post-processing unit 16 performs modeling for cutoff. The post-processing unit 16 performs logistic modeling using the effects of individual factors and interactions, and performs testing. Specifically, the post-processing unit 16 performs modeling using interactions whose coefficients calculated by logistic regression are not 0. The post-processing unit 16 calculates the effects of each factor and its interactions using logistic regression.

[0142] The optimization processing unit 18 uses the results of data mining by the interaction extraction unit 14 to reduce the number of attribute combinations by using a known optimization method, and determines combinations for generating prompts. As a known optimization method, for example, Bayesian optimization may be used.

[0143] The optimization processor 18 explicitly narrows down the search space of the Bayesian optimization acquisition function and selects parameters therefrom for generating prompt design information.

[0144] The optimization processing unit 18 searches for adjustment items for adjusting prompts through optimization based on the results (interactions) of the data mining.

[0145] The optimization processing unit 18 receives the remaining interactions generated by the interaction extraction unit 14 after the pruning by the post-processing unit 16 .

[0146] The optimization processing unit 18 processes only the interactions resulting from combinations of attributes, which are output from the interaction extraction unit 14 and are cut off by the post-processing unit 16, as calculation terms for Bayesian optimization, thereby reducing the time required for optimization.

[0147] Bayesian optimization is a method for arriving at an optimal solution with a small number of trials by sampling points with a high probability of being the optimal solution. Note that when performing Bayesian optimization, variables must be independent.

[0148] Here, the process of narrowing down the search range using interactions in Bayesian optimization in the optimization processing unit 18 will be described.

[0149] For example, in the case of all possible combinations of formulation information and attribute values ​​shown in Fig. 5, the original search range has three possibilities for "1st", "2nd", and "3rd", and two possibilities for "1stlen", "2ndlen", and "3rdlen". However, if 1st (2nd, 3rd) is "question", there is only one possibility for 1stlen (2ndlen, 3rdlen) (only "short").

[0150] Here, for example, it is assumed that the following two interactions are extracted from the multiple interactions illustrated in FIG.

[0151] Condition for good accuracy: 1stlen_short=0 ∧ 2nd_instruction=1 ∧ 2ndlen_short=0 Condition for good accuracy: 1st_instruction=0 ∧ 1stlen_short=0 ∧ 2ndlen_short=0 The optimization processing unit 18 calculates the AND condition of the above interactions. As a result, 1st_instruction=0 ∧ 1stlen_short=0 ∧ 2nd_instruction=1 ∧ 2ndlen_short=0 is obtained.

[0152] In this AND condition, the 1st search range is instruction=0, meaning anything other than instruction. 1stlen is short=0, meaning anything other than short. 2nd is instruction=1, meaning instruction. 2ndlen is short=0, meaning anything other than short.

[0153] Therefore, the search range is limited as shown in the limited range of FIG.

[0154] Fig. 14 is a diagram illustrating an example of a limited search range, in which the limited search range for Bayesian optimization is associated with each attribute.

[0155] For example, as described above, the search range for 1st is other than "instruction," so in Fig. 14, "format" and "question" are set as a limited range in association with the attribute 1st (first block). Also, the search range for 2nd is "instruction," so "instruction" is set in association with the attribute 2nd (second block). The search range for 3rd is other than 1st and 2nd, so it is inevitably determined once 1st is determined. A parameter set searched for with a limited range is expected to be highly accurate.

[0156] In the prompt adjustment device 1b of the second embodiment, the interaction extraction unit 14 calculates the interactions between attributes, and then the optimization processing unit 18 inputs the interaction terms into Bayesian optimization. The optimization processing unit 18 narrows down and determines the optimal combination through Bayesian optimization. That is, the results of data mining are used to explicitly narrow down the search space of the acquisition function for Bayesian optimization, and parameters are selected from there.

[0157] The optimization processing unit 18 determines the most accurate combination among the combinations proposed by Bayesian optimization as the adjustment item. For example, a trial number indicating how many times the proposal will be made may be set in advance by Bayesian optimization, and the proposal may be repeated until the trial number is reached.

[0158] This allows the optimization unit 12b to perform Bayesian optimization and optimize the prompt without trying all possible combinations of data.

[0159] Immediately after the start of processing by the optimization unit 12b, the number of data is small, making it difficult to extract useful interactions. However, as the processing progresses, the amount of data increases, making it easier to ensure statistical significance, and the optimization processing unit 18 can accelerate the narrowing down process.

[0160] (B) Operation The processing of the optimization unit 12b of the prompt adjustment device 1b according to the second embodiment configured as described above will be described with reference to the flowchart (steps S11 to S21) shown in FIG.

[0161] In step S11, the optimization unit 12a extracts, from the input data stored by the input processing unit 11, a plurality of (N) sample data sets with correct answers to be used for optimizing the prompts.

[0162] In step S12, the formulation processing unit 13 parameterizes (formulates) the configuration of one or more sample prompts to create formulation information.

[0163] In step S13, the preprocessing unit 15 creates a skip list from the qualitative exclusion list and stores the skip list in a predetermined storage area of ​​the memory 10c or the storage unit 10d. Details of the process of step S13 will be described later with reference to FIG. 16.

[0164] In step S14, the optimization unit 12a sets the number of loops to N (number of loops=N).

[0165] In step S15, the optimization processing unit 18 proposes parameters by Bayesian optimization based on the accuracy of the parameters from the first loop to the previous loop. The proposed parameters may be called proposed parameters.

[0166] In step S16, the preprocessing unit 15 checks whether the proposed parameter is included in the skip list. If the check result shows that the proposed parameter is included in the skip list (see the YES route in step S16), the process returns to step S15.

[0167] If the proposed parameter is not included in the skip list (see the NO route from step S16), the process proceeds to step S17.

[0168] In step S17, the optimization unit 12b generates a sample prompt using the proposed parameters, calculates whether the LLM is correct for this created sample prompt, and stores the result in the corresponding item indicating whether it is correct or not in the data mining input information.

[0169] In step S18, the interaction extraction unit 14 performs data mining using as input the parameters from the first to last loop in the data mining input information and a table indicating whether the answer is correct or not, to calculate interactions.

[0170] In step S19, the post-processing unit 16 performs testing, calculates the p-value of each interaction output from the interaction extraction unit 14, and performs pruning by excluding interactions whose p-values ​​are equal to or greater than a threshold value and leaving only interactions whose p-values ​​are less than the threshold value.

[0171] In step S20, the optimization processing unit 18 adds the remaining interactions to the Bayesian optimization term and selects the next parameter set.

[0172] In step S21, the optimization unit 12 checks whether a termination condition is satisfied. The termination condition may be, for example, detecting that the accuracy of the LLM output no longer improves, or reaching a predetermined number of loops, and may be implemented in an appropriate modified form.

[0173] If the termination condition is not satisfied (see the NO route in step S21), the process returns to step S15, whereas if the termination condition is satisfied (YES in step S21), the process ends.

[0174] Next, the processing of the preprocessing unit 15 of the prompt adjustment device 1b according to the second embodiment will be described with reference to the flowchart (steps S21 to S28) shown in Fig. 16. The flowchart shown in Fig. 16 shows details of the processing of step S13 in the flowchart shown in Fig. 15.

[0175] In step S21, the preprocessing unit 15 sets the skip list to empty. In step S22, the preprocessing unit 15 automatically compares the exclusion list with the attributes and possible values ​​formulated for the target data, thereby determining the items in the exclusion list to be verified. This allows for a correspondence to be achieved when the attribute names / values ​​written in the exclusion list and the attribute names / values ​​set in the formulation do not completely match, but are substantially the same. For example, the exclusion list may contain "QUESTION:1," but the attribute / possible value set in the formulation may be "1st: question." This corresponds to a case where the attribute and possible values ​​are swapped or the case is different, such as when the attribute and possible values ​​are swapped or when the uppercase / lowercase letters are different.

[0176] In step S23, a loop process is started in which the control up to step S26 is repeatedly performed for all items to be verified in the exclusion list.

[0177] In step S24, the pre-processing unit 15 performs a pre-check using the sample data with correct answers. The pre-processing unit 15 verifies the accuracy of the LLM results using the sample prompts created using the sample data with correct answers.

[0178] In step S25, the preprocessing unit 15 checks whether the accuracy of the LLM result is low by, for example, comparing the accuracy of the LLM result with a preset threshold value. If the accuracy of the LLM result is less than the threshold value, i.e., if the accuracy is low (see the YES route in step S25), the process proceeds to step S26.

[0179] In step S26, the preprocessing unit 15 adds the combination patterns that meet the corresponding conditions in the exclusion list to the skip list, and then proceeds to step S27.

[0180] Also, if the result of checking in step S25 shows that the accuracy of the LLM result is equal to or greater than the threshold, that is, if the accuracy is high (see the NO route in step S25), the process proceeds to step S27.

[0181] In step S27, loop end processing corresponding to step S23 is performed. Here, when processing for all items to be verified in the exclusion list is completed, the skip list is returned in step S28. Thereafter, processing ends. Note that in the optimization processing unit 18, if the parameter set proposed by Bayesian optimization matches the skip list, subsequent processing is not performed and the optimization processing unit 18 proceeds to the next Bayesian optimization proposal.

[0182] Next, the processing of the post-processing unit 16 of the prompt adjustment device 1b according to the second embodiment will be described with reference to the flowchart (steps S31 to S35) shown in FIG.

[0183] In step S31, a loop process is started in which the control up to step S33 is repeatedly performed for all interactions generated by the interaction extraction unit 14.

[0184] In step S32, the post-processing unit 16 calculates a p-value using statistical verification.

[0185] In step S33, a loop end process corresponding to step S31 is performed. When the process for all interactions is completed, the process proceeds to step S34.

[0186] In step S34, the post-processing unit 16 sorts all of the interactions based on the p-values. For example, the post-processing unit 16 sorts all of the interactions in ascending order based on the p-values.

[0187] In step S35, the post-processing unit 16 performs pruning on the sorted interaction column. For example, the post-processing unit 16 may select and retain only interactions whose p-values ​​are less than a preset threshold (p-value<threshold). Alternatively, the post-processing unit 16 may select (extract) and retain only a predetermined number (m) of interactions whose p-values ​​are less than the threshold and are ranked from the top of the p-values. Then, the processing ends.

[0188] Next, the Bayesian optimization process by the optimization processing unit 18 of the prompt adjustment device 1b according to the second embodiment will be described with reference to the flowchart (steps S71 to S75) shown in Fig. 18. Note that Fig. 18 illustrates a process for maximizing accuracy.

[0189] In step S71, the optimization processing unit 18 sets the loop count to N (loop count=N). N corresponds to the number of trials that represent how many times a proposal is made.

[0190] In step S72, the optimization processing unit 18 starts a loop process in which steps S73 to S74 are repeatedly performed for all loops represented by the total number of loops. The number of parameter proposal loops is represented by i. i is a natural number equal to or less than N, and its initial value is 1 (i=1).

[0191] In step S73, the optimization processing unit 18 proposes parameters (proposed parameters) by Bayesian optimization based on the accuracy of the parameters from loop count 1 to (i-1).

[0192] In step S74, the optimization processing unit 18 calculates the accuracy using the proposed parameters. In step S75, loop end processing corresponding to step S72 is performed. When the number of loops reaches N, this flow ends.

[0193] In this way, in the Bayesian optimization by the optimization processing unit 18, the next parameters are proposed using the results obtained up to that point.

[0194] (C) Effect As described above, according to the prompt adjustment device 1b of the second embodiment, the interaction extraction unit 14 detects interactions of multiple attributes using a data mining technique based on data mining input information generated based on formulation information and sample data with correct answers.

[0195] The optimization processor 18 can then improve the efficiency of the optimization technique by narrowing down the combinations of attributes using the results of data mining by the interaction extractor 14, and can determine combinations (prompt design information) for generating accurate prompts from a huge number of combinations with a limited number of Bayesian optimization loops. In other words, the optimization processor 18 explicitly narrows down the search space of the acquisition function for Bayesian optimization using the interactions obtained by data mining, and selects parameters from there.

[0196] This allows prompts that can provide highly accurate results from LLM to be generated in a short time with a small amount of calculation, which means that the calculation time and cost required to generate the prompts can be reduced.

[0197] The preprocessing unit 15 checks attribute combinations in advance based on an exclusion list that registers conditions that are qualitatively known to result in low accuracy (poor performance), generates a skip list, and performs pruning. In other words, if an attribute combination proposed by Bayesian optimization is included in the skip list, the process of generating a prompt using that combination and submitting it to the LLM to calculate accuracy, and the process of using the result in data mining, are skipped. This reduces the number of variations in attribute combinations to be included in data mining in the interaction extraction unit 14, thereby also reducing calculation time and calculation costs.

[0198] Furthermore, the preprocessing unit 15 uses sample data with correct answers to create sample prompts in accordance with combination patterns corresponding to the conditions in the exclusion list, and inputs these sample prompts into the LLM to check in advance whether the results obtained are truly poor in accuracy. This improves the reliability of pruning by the preprocessing unit 15.

[0199] The post-processing unit 16 tests the combinations that appear in the interactions output from the interaction extraction unit 14 and calculates p-values. The post-processing unit 16 then performs a cutoff to exclude interactions with p-values ​​equal to or greater than a threshold from the targets of processing by the optimization unit 18, which will be described later. This reduces the calculation time and cost required for Bayesian optimization by the optimization unit 18.

[0200] In addition, the post-processing unit 16 uses statistical testing to reduce the interactions output from the interaction extraction unit 14 based on p-values. This makes it possible to improve calculation efficiency by not considering interactions that are not statistically significant (that are unlikely to directly affect accuracy). Furthermore, although interactions of ordinal variables are not independent, performing testing also guarantees independence between attributes, making it possible to use Bayesian optimization.

[0201] The interaction extraction unit 14 calculates interactions between statistically significant attributes, thereby ensuring the independence between terms to be included in Bayesian optimization in the optimization processing unit 18. Furthermore, the post-processing unit 16 performs testing to calculate p-values, and by using these p-values ​​to limit the number of interaction terms and run Bayesian optimization, it is possible to optimize prompts while reducing calculation time and cost without trying every possible combination of data.

[0202] (III) Others The disclosed technology is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of each embodiment. The configurations and processes of each embodiment can be selected or combined as needed.

[0203] For example, in the prompt adjustment device 1b of the second embodiment described above, at least one of the pre-processing unit 15 and the post-processing unit 16 may be excluded from the configuration.

[0204] Furthermore, in the second embodiment described above, the post-processing unit 16 performs a statistical test to narrow down the interactions, but the present invention is not limited to this.

[0205] For example, the post-processing unit 16 may extract multiple (n) interactions using a specific method and include only the n interactions in the Bayesian optimization term.

[0206] As a specific method for extracting the n interactions, the post-processing unit 16 may, for example, randomly extract the n interactions.

[0207] Fig. 19 shows a modification of the flowchart shown in Fig. 17. The post-processing unit 16 may execute the process of the flowchart shown in Fig. 19 instead of the process shown in the flowchart of Fig. 17.

[0208] In the flowchart shown in FIG. 19, the post-processing unit 16 randomly extracts n interactions in step S51, and then ends the process.

[0209] The post-processing unit 16 may also calculate the weight of each interaction by logistic regression, extract m interactions in order of absolute values ​​of the coefficients, and randomly extract n interactions from these m interactions. Note that the m interactions extracted in order of absolute values ​​of the coefficients are important interactions for accuracy prediction.

[0210] Fig. 20 shows another modification of the flowchart shown in Fig. 17. The post-processing unit 16 may execute the process of the flowchart shown in Fig. 20 instead of the process shown in the flowchart of Fig. 17.

[0211] In the flowchart shown in FIG. 20, in step S61, the post-processing unit 16 calculates the weight of each interaction by logistic regression, and extracts m interactions in order of absolute values ​​of the coefficients.

[0212] Then, in step S62, the post-processing unit 16 randomly extracts n interactions from the m interactions, and then ends the process.

[0213] In each of the above-described embodiments, the formulation processing unit 13 may use a Lehmer code to represent the order of prompts from the beginning in the formulation information.

[0214] Fig. 21 is a diagram showing a modified example of the formulation information, in which the attribute of order is expressed using Lehmer code.

[0215] In the formulation information shown in FIG. 21, an instruction block, a question block, and a format block are used as attributes, and possible values ​​that can be taken as Lehmer codes are shown in association with each other.

[0216] This allows the possible values ​​for each attribute (order attribute) to be expressed in Lehmer code (see symbol P1). By expressing the order of attributes in Lehmer code, for example, even if the Instruction block is 0, the order of other blocks can also be 0, and the variables become independent. For example, Bayesian optimization requires that variables be independent, but applying Lehmer code ensures independence, making it applicable to optimization.

[0217] 21, in addition to the above-mentioned Instruction block, Format block, and Question block, the length of the explanation of the Instruction block, the length of the Format block, and the length of the Question block are also set as attributes. The length of the explanation of the Instruction block, the length of the Format block, and the length of the Question block can take the values ​​short and long.

[0218] In the second embodiment described above, the preprocessing unit 15 lists single attributes or combinations of attributes with low accuracy (poor performance), but this is not limiting. The preprocessing unit 15 may also list single attributes or combinations of attributes with high accuracy (good performance). The configuration of the prompt is not limited to the example shown in FIG. 3. Furthermore, the formulation information is not limited to those shown in FIGS. 4, 5, 21, etc.

[0219] Furthermore, the above disclosure enables those skilled in the art to implement and manufacture each embodiment.

[0220] 1a, 1b Prompt adjustment device 10 Computer 10a Processor 10b Graphics processing device 10c Memory 10d Storage unit 10e IF unit 10f IO unit 10g Reading unit 10h Program 10i Recording medium 10j Bus 11 Input processing unit 12a, 12b Optimization unit 13 Formulation processing unit 14 Interaction extraction unit 15 Preprocessing unit 16 Postprocessing unit 17 Output processing unit 18 Optimization processing unit

Claims

1. Based on configuration information representing the configuration of a prompt as a combination of a plurality of items corresponding to a plurality of blocks constituting the prompt and possible values for each item, generate a plurality of sample prompts corresponding to a plurality of types of the combinations, and determine, as adjustment items for adjusting the prompt, items included in the combination used for generating the sample prompt with the highest inference accuracy among the results obtained by inputting the plurality of sample prompts into a language model, and cause a computer to execute the process. A prompt adjustment program characterized by this.

2. Based on configuration information representing the configuration of a prompt as a combination of a plurality of items corresponding to a plurality of blocks constituting the prompt and possible values for each item, generate a plurality of sample prompts corresponding to a plurality of types of the combinations, perform data mining using, as input, information associating the results obtained by inputting the plurality of sample prompts into a language model with the combinations used for generating the sample prompts, and based on the result of the data mining, cause a computer to execute the process of searching for adjustment items for adjusting the prompt by optimization. A prompt adjustment program characterized by this.

3. For configuration information including combinations that meet an exclusion condition, cause the computer to execute the process of skipping the data mining. The prompt adjustment program according to claim 2, characterized by this.

4. Before skipping the data mining for configuration information including combinations that meet an exclusion condition, generate a second sample prompt using sample data with a combination that meets the exclusion condition, and if the accuracy of the result obtained by inputting the second sample prompt into the language model is equal to or lower than a threshold value, cause the computer to execute the process of skipping the data mining for the combination that meets the exclusion condition. The prompt adjustment program according to claim 3, characterized by this.

5. The computer is caused to execute a process of excluding, from the target of the optimization, the result of the data mining whose value representing the influence on the inference performance exceeds a threshold value. The prompt adjustment program according to any one of claims 2 to 4.

6. The process of searching for the adjustment item includes a process of explicitly narrowing down the search space of the acquisition function of the optimization using the result of the data mining, and selecting the adjustment item from within the narrowed-down range. The prompt adjustment program according to claim 2.

7. The prompt adjustment program according to claim 1 or 2, wherein the item represents the position of the block in the prompt.

8. The prompt adjustment program according to claim 1 or 2, wherein the item represents the expression variation of the block in the prompt.

9. Based on the configuration information representing the configuration of the prompt as a combination of a plurality of items corresponding to a plurality of blocks constituting the prompt and the possible values of each item, a plurality of sample prompts corresponding to a plurality of types of the combinations are generated, and the computer executes a process of determining, as an adjustment item for adjusting the prompt, an item included in the combination used for generating a sample prompt having the highest inference accuracy among the results obtained by inputting the plurality of sample prompts into a language model. A prompt adjustment method.

10. Based on the configuration information representing the configuration of the prompt as a combination of a plurality of items corresponding to a plurality of blocks constituting the prompt and the possible values of each item, a plurality of sample prompts corresponding to a plurality of types of the combinations are generated, data mining is performed using, as input, information associating the results obtained by inputting the plurality of sample prompts into a language model with the combinations used for generating the sample prompts, and the computer executes a process of searching for an adjustment item for adjusting the prompt by optimization based on the result of the data mining. A prompt adjustment method.

11. For the configuration information including combinations that meet the exclusion conditions, the computer executes a process of skipping the data mining, which is the prompt adjustment method according to claim 10.

12. Before skipping the data mining for the configuration information including combinations that meet the exclusion conditions, a second sample prompt is generated using sample data with the combinations that meet the exclusion conditions. When the accuracy of the result obtained by inputting the second sample prompt into the language model is below the threshold, the computer executes a process of skipping the data mining for the combinations that meet the exclusion conditions, which is the prompt adjustment method according to claim 11.

13. The computer executes a process of excluding the data mining results whose values representing the influence on the inference performance exceed the threshold from the optimization targets, which is the prompt adjustment method according to any one of claims 10 to 12.

14. The process of searching for the adjustment items includes a process of explicitly narrowing down the search space of the acquisition function of the optimization using the data mining results and selecting the adjustment items from the narrowed range, which is the prompt adjustment method according to claim 10.

15. The item represents the position of the block in the prompt, which is the prompt adjustment method according to claim 9 or 10.

16. The item represents the expression variation of the block in the prompt, which is the prompt adjustment method according to claim 9 or 10.

17. Based on configuration information representing the configuration of the prompt as a combination of a plurality of items corresponding to a plurality of blocks constituting the prompt and possible values for each item, generate a plurality of sample prompts corresponding to a plurality of types of the combinations, and determine, as adjustment items for adjusting the prompt, items included in the combination used for generating the sample prompt with the highest inference accuracy among the results obtained by inputting the plurality of sample prompts into a language model. An information processing apparatus characterized by comprising a control unit that executes a process.

18. Based on configuration information representing the configuration of the prompt as a combination of a plurality of items corresponding to a plurality of blocks constituting the prompt and possible values for each item, generate a plurality of sample prompts corresponding to a plurality of types of the combinations, perform data mining using, as input, information associating the results obtained by inputting the plurality of sample prompts into a language model with the combinations used for generating the sample prompts, and search for adjustment items for adjusting the prompt by optimization based on the results of the data mining. An information processing apparatus characterized by comprising a control unit that executes a process.

19. The information processing apparatus according to claim 18, wherein the control unit executes a process of skipping the data mining for configuration information including a combination in the configuration information that meets an exclusion condition.

20. Before the control unit skips the data mining for configuration information including a combination in the configuration information that meets an exclusion condition, generate a second sample prompt using sample data with the combination that meets the exclusion condition, and if the accuracy of the result obtained by inputting the second sample prompt into the language model is equal to or lower than a threshold value, execute a process of skipping the data mining for the combination that meets the exclusion condition. The information processing apparatus according to claim 19.

21. The information processing apparatus according to any one of claims 18 to 20, wherein the control unit executes a process of excluding from the optimization target the result of the data mining for which a value representing an influence of the result of the data mining on the inference performance exceeds a threshold value.

22. The information processing apparatus according to claim 18, wherein the process of searching for the adjustment item includes a process of explicitly narrowing down a search space of an acquisition function for the optimization using the result of the data mining and selecting the adjustment item from within the narrowed-down range.

23. The information processing apparatus according to claim 17 or 18, wherein the item represents a position of the block in the prompt.

24. The information processing apparatus according to claim 17 or 18, wherein the item represents a representation variation of the block in the prompt.

Images