Prompt generation device and prompt generation method

Abstract

To enable a prompt generation device for generating a prompt to be input to a large language model to easily generate a prompt that produces a more consistent response from the large language model, a prompt generation device according to the present invention includes: a creation unit that creates a prompt to be input to a large language model on the basis of a question or problem for a predetermined event; an acquisition unit that acquires a plurality of responses by inputting the prompt to the large language model a plurality of times; a calculation unit that calculates an evaluation index indicating consistency of the plurality of responses; and a correction unit that corrects the prompt so that a higher evaluation index can be obtained if the evaluation index is less than a predetermined threshold.

Description

Prompt generation device and prompt generation method

[0001] The present invention relates to a prompt generation device and a prompt generation method.

[0002] Large language models (LLMs) such as GPT-4 (Generative Pre-trained Transformer-4) are known (see, for example, Non-Patent Document 1).

[0003] Furthermore, prompt engineering is known as a technique for effectively formatting questions or tasks to obtain more desirable results from large-scale language models (see, for example, Non-Patent Document 2).

[0004] OpenAI,"GPT-4 Technical Report",arXiv preprint arXiv:2303.08774 (2023),<https: / / arxiv.org / pdf / 2303.08774.pdf> ,Internet, [Retrieved June 23, 2023] Yang, Xiaohan, et al., "What gpt knows about who is who", arXiv preprint arXiv:2205.07407 (2022), < https: / / arxiv.org / pdf / 2205.07407.pdf>, Internet, [Retrieved June 23, 2023]

[0005] There is a demand for applying large-scale language models as a recommendation system for network operations, and for the system to provide solutions to questions input by operators about how to deal with network operations, thereby making network operations more efficient and reducing the burden on operators.

[0006] However, with conventional large-scale language models, the answers given were inconsistent even when the same question was asked, making it difficult to provide a unified operational method for dealing with the same event.

[0007] An embodiment of the present invention has been made in consideration of the above-mentioned problems, and enables a prompt generation device that generates prompts to be input into a large-scale language model to easily generate prompts that have higher consistency in answers using the large-scale language model.

[0008] In order to solve the above problem, a prompt generation device according to an embodiment of the present invention includes a creation unit that creates a prompt to be input into a large-scale language model based on a question or problem regarding a predetermined event; an acquisition unit that inputs the prompt into the large-scale language model multiple times to acquire multiple answers; a calculation unit that calculates an evaluation index that indicates the consistency of the multiple answers; and a modification unit that, if the evaluation index is less than a predetermined threshold, modifies the prompt so as to obtain a higher evaluation index.

[0009] According to an embodiment of the present invention, a prompt generation device that generates prompts to be input into a large-scale language model can easily generate prompts that have higher consistency in answers using the large-scale language model.

[0010] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to the present embodiment; FIG. 2 is a flowchart illustrating an example of the processing of a prompt generation device according to the present embodiment; FIG. 3 is a diagram illustrating an example of an initial prompt according to the present embodiment; FIG. 4 is a diagram illustrating an example of a prompt for prompt correction according to the present embodiment; FIG. 5 is a diagram illustrating an example of a prompt after correction according to the present embodiment; and FIG. 6 is a diagram illustrating an example of the hardware configuration of a computer according to the present embodiment.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0012] 1 is a diagram showing an example of the configuration of an information processing system according to this embodiment. The information processing system 1 includes, for example, an operator terminal 10, an LLM (Large Language Model) server 20, a domain-specific knowledge DB (Database) 30, and a prompt generation device 100. The prompt generation device 100 is connected to the operator terminal 10, the LLM server 20, the instruction server 20, the domain-specific knowledge DB 30, etc. so as to be able to communicate with them.

[0013] The information processing system 1 is a system that uses a large-scale language model (hereinafter referred to as LLM) to provide an appropriate answer to a question or problem about a predetermined event input from, for example, an operator terminal 10 .

[0014] Here, the LLM is a natural language processing model trained using large amounts of text data, and is capable of returning appropriate answers to user questions. Applying this LLM to network (communication network) operations is expected to reduce the burden on operators. For example, in situations where network operation issues, such as responding to a failure, require an operator to input a question about how to resolve the issue into the LLM, and the LLM will then respond with the solution. By utilizing the LLM in this way, it becomes possible to use the LLM as a recommendation system for network operation.

[0015] Here, the following description will be given assuming that the information processing system 1 is a recommendation system that answers questions about how to deal with network operations. However, the information processing system 1 is not limited to this, and may be a system that answers questions about or how to deal with problems regarding various events other than network operations.

[0016] The operator terminal 10 is an information terminal used by an operator and has a computer configuration such as a personal computer (PC), a tablet terminal, or a smartphone. In this embodiment, the operator inputs a question or problem related to network operation into the operator terminal 10, and the operator terminal 10 transmits the input question or problem to the prompt generation device 100.

[0017] The prompt generation device (prompt generation server) 100 is, for example, an information processing device having a computer configuration or a system including multiple computers. The prompt generation device 100 receives a question or problem sent from the operator terminal 10 and generates an optimal prompt using a prompt generation algorithm. Here, a prompt is instruction information output to the LLM and includes, for example, information such as an instruction statement, a condition, or an output format specification. Note that a prompt is also called an AI (artificial intelligence) prompt.

[0018] The prompt generation device 100 is connected to the LLM server 20, and generates optimal prompts by repeatedly correcting prompts while utilizing the response content from the LLM server 20. The prompt generation device 100 is also connected to the domain-specific knowledge DB 30, and creates prompts for prompt correction according to prompt correction guidelines appropriate for each domain.

[0019] The LLM server 20 receives prompts sent from the prompt generation device 100 and generates answers to the operator's questions or problems using a large-scale language model such as GPT-4. The LLM server 20 is also connected to a domain knowledge DB 30, and when generating answers, creates appropriate answers based on domain knowledge (e.g., network configuration information, etc.). In the following description, the LLM server 20 may be simply referred to as "LLM."

[0020] In this embodiment, the domain-specific knowledge DB 30 is a database that stores domain-specific knowledge related to network operation. The domain-specific knowledge DB 30 provides the LLM server 20 with domain knowledge to improve the accuracy of answers. The domain-specific knowledge DB 30 also stores prompt revision guidelines, and the prompt generation device 100 creates prompts for prompt revision in accordance with these revision guidelines.

[0021] (Functional Configuration of Prompt Generation Device) The prompt generation device 100 realizes, for example, the functional configuration shown in Fig. 1 by executing a predetermined program on a computer included in the prompt generation device 100. In the example of Fig. 1, the prompt generation device 100 includes a control unit 101, a generation unit 102, an acquisition unit 103, a calculation unit 104, and a correction unit 105. Note that at least a portion of the above functional configurations may be realized by hardware.

[0022] The control unit 101 accepts questions or problems from the operator terminal 10, inputs prompts generated based on the accepted questions or problems to the LLM server 20, and provides answers obtained from the LLM server 20 to the operator terminal 10.

[0023] The LLM server 20 is an example of a large-scale language model (LLM) according to the present embodiment. The large-scale language model according to the present embodiment may be, for example, a large-scale language model provided by an external cloud service or the like.

[0024] The generator 102 executes a generation process to generate a prompt to be input to the large-scale language model based on a question or problem regarding a predetermined event. For example, the generator 102 generates an initial prompt to be input to the LLM server 20 based on a question or problem regarding network operation that the controller 101 receives from the operator terminal 10.

[0025] The acquisition unit 103 executes an acquisition process of inputting a prompt into the large-scale language model multiple times to acquire multiple answers. For example, the acquisition unit 103 inputs the prompt generated by the generation unit 102 or the same prompt corrected by the correction unit 105 into the LLM server 20 multiple times to acquire multiple answers.

[0026] The calculation unit 104 executes a calculation process to calculate an evaluation index indicating the consistency of a plurality of answers. For example, the calculation unit 104 calculates a consistency score, a similarity, or the like indicating the consistency of the plurality of answers acquired by the acquisition unit 103. Note that the consistency score or the similarity is an example of an evaluation index indicating the consistency of the plurality of answers.

[0027] When the evaluation index indicating the consistency of the multiple answers is less than a predetermined threshold, the correction unit 105 executes a correction process to correct the prompt so as to obtain a higher evaluation index. For example, when the consistency score (or similarity) calculated by the calculation unit 104 is less than a predetermined threshold, the correction unit 105 corrects the prompt using a large-scale language model such as the LLM server 20 so as to obtain a higher consistency score (or similarity).

[0028] Preferably, the correction unit 105 repeatedly corrects the prompts using, for example, a prompt correction guideline related to network operation contained in the domain-specific knowledge DB 30 until an evaluation index indicating the consistency of multiple answers becomes equal to or greater than a predetermined threshold.

[0029] (Problems with the Prior Art) Conventional LLMs provide inconsistent answers even when the same question is asked, making it difficult to apply them directly to network operations, for example. This is because multiple operators are involved in network operations, and if the LLM returns inconsistent answers, it becomes difficult for each operator to take a unified approach to the same event.

[0030] Although it is known that highly consistent (less inconsistent) answers can be expected depending on how the question is written, it is difficult for operators to create appropriate questions each time. Therefore, there is a need for technology that can automatically generate optimal prompts.

[0031] In traditional LLM, two main approaches are taken to improve the consistency of answers: parameter tuning and prompt engineering.

[0032] Parameter tuning allows for the adjustment of the diversity or consistency of generated text by adjusting parameters such as temperature or top_p to control the behavior of the LLM (see, for example, Non-Patent Document 1). However, this method makes it difficult to achieve proper parameter tuning, and incorrect parameter settings can significantly reduce the accuracy and diversity of answers. Furthermore, some parameters in this method are not externalized and cannot be changed by the user, which can make it difficult to perform appropriate tuning.

[0033] Prompt engineering is a technique for effectively formatting questions or tasks to obtain more desirable results from LLM (see, for example, Non-Patent Document 2). Prompt engineering typically involves techniques such as providing clear instructions, reformatting questions, adding context, and providing step-by-step instructions. Proper prompt engineering can improve the consistency and accuracy of answers. However, it is difficult for operators to manually perform prompt engineering each time they create a question.

[0034] As such, with conventional LLMs, even when the same question was asked, the answers were inconsistent, making it difficult to provide a unified operational method for dealing with the same event.

[0035] Therefore, the prompt generation device 100 according to this embodiment, with the functional configuration shown in FIG. 1, provides, for example, an automatic prompt optimization technology for supporting network operations. This technology allows operators to easily obtain consistent answers from the LLM, which is expected to enable quick and efficient resolution of network operation problems. This technology is also expected to reduce the burden on operators and improve the efficiency of network management.

[0036] Thus, according to this embodiment, a prompt generation device that generates prompts to be input into a large-scale language model can easily generate prompts that have higher consistency in answers using the large-scale language model.

[0037] <Processing Flow> Next, a specific flow of the prompt generation method according to this embodiment will be described. Fig. 2 is a flowchart showing an example of processing performed by the prompt generation device according to this embodiment. This processing represents an example of processing performed by, for example, the prompt generation device 100 having the functional configuration described in Fig. 1.

[0038] In step S201, the generator 102 executes a generation process to generate an initial prompt to be input to the large-scale language model based on a question or problem regarding a predetermined event. For example, the generator 102 generates an initial prompt to be input to the LLM server 20 based on a question or problem regarding network operation input by the control unit 101 from the operator terminal 10.

[0039] The initial prompt accurately and succinctly phrases the question or problem so that it can be answered appropriately by a target large-scale language model (e.g., GPT-4). The initial prompt may be based on user input, an existing template, or an automatically generated question.

[0040] 3 is a diagram showing an example of an initial prompt according to this embodiment. In the example of FIG. 3, the initial prompt 300 is configured to include "event information" and "instructions." This initial prompt 300 prompts the target LLM to appropriately extract information related to abnormal traffic in the network device and generate a response.

[0041] When creating initial prompts, it is important to select an appropriate question format or wording, taking into consideration the level of difficulty of the target question or problem, background knowledge, etc. This is expected to make it easier for the target LLM to understand the question or problem, and to improve the consistency and accuracy of their answers.

[0042] The creation of the initial prompt in step S201 is the starting point for the prompt optimization process in this embodiment.

[0043] In step S202, the acquisition unit 103 executes an acquisition process in which the prompt is input to the LLM multiple times to acquire multiple answers. For example, the acquisition unit 103 inputs the initial prompt generated by the generation unit 102 or the modified prompt modified by the modification unit 105 to the LLM server 20 multiple times to acquire multiple answers to the prompt. Here, the reason for acquiring answers multiple times is to evaluate the consistency of the multiple answers and improve the prompt.

[0044] In step S203, the calculation unit 104 calculates an evaluation index indicating the consistency of the multiple answers acquired by the acquisition unit 103. For example, the calculation unit 104 calculates the similarity between the answers using a known BERT score, and uses this as the evaluation index for the question. The BERT score is an evaluation index that uses a natural language processing model called a BERT (Bidirectional Encoder Representations from Transformers) model, and evaluates the similarity of sentences in the range of 0 to 1. A higher BERT score indicates higher consistency of the answers. (References: Zhang, Tianyi, et al., "Bertscore: Evaluating text generation with bert", arXiv preprint arXiv: 1904.09675 (2019))<https: / / arxiv.org / pdf / 1904.09675.pdf> )

[0045] When there are three or more answers, the calculation unit 104 determines the average value of the BERT scores (real numbers between 0 and 1, the larger the number, the higher the similarity) of all answer pairs as the consistency score of that answer.

[0046] For example, the calculation unit 104 calculates the BERT score in the following procedure: (1) The calculation unit 104 inputs each generated answer into a BERT model to obtain an embedding vector of the sentence. (2) The calculation unit 104 calculates the cosine similarity between each embedding vector for all answer pairs. (3) The calculation unit 104 scales the calculated cosine similarity to a range from 0 to 1 to obtain the BERT score for each answer pair. (4) The calculation unit 104 sets the average value of the BERT scores of all answer pairs as the consistency score of the multiple answers.

[0047] In step S204, the correction unit 105 determines whether the evaluation index indicating the consistency of the multiple answers calculated by the calculation unit 104 is equal to or greater than a predetermined threshold. Here, the predetermined threshold is a value set in advance to determine whether the consistency of the multiple answers is sufficient.

[0048] This threshold can be adjusted depending on the actual task or application scenario. For example, if the threshold is set low, inconsistent answers are tolerated, making it easier to obtain an answer with fewer iterations. On the other hand, if the threshold is set high, the opposite occurs, and more iterations are required to obtain a final answer.

[0049] If the evaluation index indicating the consistency of the multiple responses is less than a predetermined threshold, the correction unit 105 determines that the consistency of the multiple responses to the current prompt is low, and shifts the process to step S205. On the other hand, if the evaluation index indicating the consistency of the multiple responses is equal to or greater than a predetermined threshold, the correction unit 105 determines that the consistency of the multiple responses to the current prompt is sufficiently high, and shifts the process to step S206.

[0050] In step S205, the modifying unit 105 modifies the prompt so as to obtain a higher evaluation index, and the process returns to step S202. Preferably, the modifying unit 105 modifies the prompt using the LLM.

[0051] For example, the corrector 105 inputs the contents of the prompt and the correction guideline to the LLM server 20, and corrects the prompt to obtain a more consistent answer. For example, the corrector 105 creates a prompt for prompt correction as shown in FIG. 4 and inputs it to the LLM server 20.

[0052] FIG. 4 is a diagram illustrating an example of a prompt for prompt correction according to this embodiment. In the example of FIG. 4, the prompt 400 for prompt correction includes a correction guideline 401. The correction guideline 401 is created in advance based on the concept of prompt engineering and is suited to the task or usage scenario. In the example of FIG. 4, the correction guideline 401 includes guidelines such as "clarify the purpose of the question and the expected answer," "specify the content of the question," and "specify the information and format to be included in the answer." The correction unit 105 can correct the prompt so that more consistent answers can be obtained by having the LLM improve the prompt according to these guidelines.

[0053] As a modified example, the correction unit 105 may prepare multiple prompts for prompt correction and change the correction guidelines depending on the value of the evaluation index indicating consistency. For example, when the value of the evaluation index indicating consistency is small, the correction unit 105 may use a prompt that instructs a major correction of the prompt. Furthermore, when the degree of improvement of the evaluation index indicating consistency is small, the correction unit 105 may try a different prompt for prompt correction that has significantly different instructions from the prompt for prompt correction that has been used so far. Furthermore, the correction unit 105 may correct the prompt using an LLM other than the LLM server 20.

[0054] The correction unit 105 can repeat the prompt correction process based on the determination result of step S204, and can continue until an appropriate answer is obtained. However, to prevent infinite corrections, it is preferable that the correction unit 105 set an upper limit on the number of corrections and terminate the prompt correction process when the upper limit is reached. Furthermore, if the evaluation index indicating consistency does not increase even after multiple corrections, it is preferable that the correction unit 105 review the initial prompt or correction guidelines and re-execute the process. This realizes a more efficient prompt correction process.

[0055] 5 is a diagram showing an example of a prompt after correction according to this embodiment. In the example of FIG. 5, the prompt 500 after correction is configured to include "event information" and "instructions," similar to the initial prompt 300 described in FIG. 3. Furthermore, the "instructions" included in the prompt 500 after correction have been improved in accordance with the correction guideline 401 so as to obtain a more consistent response.

[0056] When the process proceeds to step S206, the control unit 101 provides the answer acquired using the prompt generated in the processes of steps S201 to S205. For example, the control unit 101 inputs the prompt generated in the processes of steps S201 to S205 to the LLM server 20 based on the question or problem received from the operator terminal 10. The control unit 101 also provides the answer acquired from the LLM server 20 to the operator terminal 10.

[0057] The process of FIG. 2 allows the prompt generation device 100, which generates prompts to be input to a large-scale language model, to easily generate prompts that have higher consistency in answers given by the large-scale language model.

[0058] <Example of Hardware Configuration> The prompt generation device 100 has, for example, the hardware configuration of a computer 600 as shown in Fig. 6. Alternatively, the prompt generation device 100 may be configured by a plurality of computers 600.

[0059] Fig. 6 is a diagram showing an example of the hardware configuration of a computer according to this embodiment. The prompt generation device 100 has, for example, the hardware configuration of a computer 600 as shown in Fig. 5. In the example of Fig. 6, the computer 600 has a processor 601, a memory 602, a storage device 603, a communication device 604, an input device 605, an output device 606, a bus B, etc.

[0060] The processor 601 is, for example, an arithmetic unit such as a CPU (Central Processing Unit) that executes predetermined programs to realize various functions. The memory 602 is a storage medium readable by the computer 600, and includes, for example, a RAM (Random Access Memory) and a ROM (Read Only Memory). The storage device 603 is a computer-readable storage medium, and may include, for example, a HDD (Hard Disk Drive), an SSD (Solid State Drive), various optical disks, and magneto-optical disks.

[0061] The communication device 604 includes one or more pieces of hardware (communication devices) for communicating with other devices via a wireless or wired network. The input device 605 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 606 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. Note that the input device 605 and the output device 606 may be integrated into one device (e.g., an input / output device such as a touch panel display).

[0062] The bus B is commonly connected to the above components and transmits, for example, address signals, data signals, and various control signals. The processor 601 is not limited to a CPU, and may be, for example, a DSP (Digital Signal Processor), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0063] (Supplementary Note) The prompt generation device 100 in this embodiment is not limited to being implemented by a dedicated device, but may also be implemented by a general-purpose computer. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be read into a computer system and executed. Note that the term "computer system" here includes hardware such as the OS and peripheral devices.

[0064] Furthermore, the term "computer-readable recording medium" includes various storage devices such as portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices 603 built into computer systems. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and devices that store a program for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases.

[0065] Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in a computer system, or may be one that is realized using hardware such as a PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

[0066] <Effects of the Embodiment> The effects of the present embodiment are as follows: (1) Improved consistency of answers using LLM (Large-Scale Language Model): Through the improved prompts of the present embodiment, the consistency of answers generated by the LLM is improved, reducing confusion for users. (2) Improved accuracy of answers from the LLM through clarification of prompts: Clarification of prompts makes it easier for the LLM to generate appropriate answers, improving answer accuracy. This improves the efficiency with which users access necessary information. (3) Facilitating acquisition of appropriate answers through efficient prompt modification: This embodiment enables efficient modification of prompts, increasing the probability of obtaining appropriate answers. This allows users to quickly solve problems. (4) Reduced burden on operators: The automatic prompt optimization technology of the present embodiment reduces the difficulty for operators in creating appropriate questions each time, thereby reducing the burden on operators. (5) Improved efficiency of network management: This embodiment enables operators to easily obtain consistent answers, which is expected to lead to quick and efficient problem resolution in network operation. This improves the efficiency of network management. (6) Expanding the scope of application of LLM: The improvement in quality of the information processing system 1 according to this embodiment has the potential to expand the scope of application of LLM. This is expected to promote the use of LLM in new application fields.

[0067] Due to the above effects, this embodiment can improve the quality of responses in the information processing system 1 or strengthen the ability to respond to users' information needs. Furthermore, this embodiment can quickly and efficiently solve problems in network operation, reducing the burden on operators. This is expected to improve the productivity of the entire organization and enable more efficient network management. Furthermore, this embodiment can be applied to other industries and application fields, which may produce even greater effects.

[0068] <Key Points of the Present Embodiment> The technical key points of the present embodiment are as follows. (1) Automatic prompt generation algorithm: In this embodiment, an algorithm is developed that automatically generates appropriate prompts based on collected and analyzed data. This algorithm proposes optimal prompts depending on the type of question or the purpose of the answer, reducing the operator's workload. (2) Prompt evaluation function: This embodiment includes a function for evaluating the effectiveness of generated prompts. This function continuously improves the quality of prompts, enabling operators to ask more effective questions. (3) Iterative optimization process: In this embodiment, the prompt generation algorithm is iteratively executed and optimized based on the generated prompts and their evaluation results. This ensures that optimal prompts are always provided, resulting in efficient network operation support. (4) Introduction of domain-specific knowledge: This embodiment can incorporate domain-specific knowledge related to network operation. This allows operators to obtain appropriate answers even when they ask specialized questions, resulting in more effective operation support.

[0069] Summary of Embodiments This specification discloses at least the prompt generation device and prompt generation method described in the following paragraphs. (Item 1) A prompt generation device comprising: a generation unit that generates a prompt to be input to a large-scale language model; an acquisition unit that inputs the prompt to the large-scale language model multiple times to acquire multiple answers; a calculation unit that calculates an evaluation index indicating the consistency of the multiple answers; and a correction unit that, if the evaluation index is less than a predetermined threshold, corrects the prompt so as to obtain a higher evaluation index. (Item 2) The prompt generation device described in paragraph 1, wherein the correction unit corrects the prompt using the large-scale language model. (Item 3) The prompt generation device described in paragraph 1 or 2, wherein the prompt includes a question or problem regarding an event in network operation, and the correction unit repeatedly corrects the prompt using a prompt correction guideline related to the network operation until the evaluation index becomes equal to or greater than the predetermined threshold. (4) A prompt generation method in which a computer executes the following processes: a process of generating a prompt to be input into a large-scale language model; a process of inputting the prompt into the large-scale language model multiple times to obtain multiple answers; a process of calculating an evaluation index indicating the consistency of the multiple answers; and a process of modifying the prompt so as to obtain a higher evaluation index if the evaluation index is less than a predetermined threshold.

[0070] (5) A storage medium storing a program that causes a computer to execute the following processes: a process of generating a prompt to be input into a large-scale language model; a process of inputting the prompt into the large-scale language model multiple times to obtain multiple answers; a process of calculating an evaluation index that indicates the consistency of the multiple answers; and a process of modifying the prompt so as to obtain a higher evaluation index if the evaluation index is less than a predetermined threshold.

[0071] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0072] REFERENCE SIGNS LIST 1 Information processing system 10 Operator terminal 20 LLM server 30 Domain specific knowledge DB 101 Control unit 102 Generation unit 103 Acquisition unit 104 Calculation unit 105 Correction unit

Claims

1. A prompt generation device comprising: a generation unit that generates a prompt to be input into a large-scale language model; an acquisition unit that inputs the prompt into the large-scale language model multiple times to acquire multiple answers; a calculation unit that calculates an evaluation index indicating the consistency of the multiple answers; and a modification unit that modifies the prompt so as to obtain a higher evaluation index if the evaluation index is less than a predetermined threshold.

2. The prompt generation device according to claim 1, wherein the correction unit corrects the prompt using the large-scale language model.

3. The prompt generating device according to claim 1 or 2, wherein the prompt includes a question or problem regarding an event in network operation, and the correction unit repeatedly corrects the prompt using a correction guideline for the prompt regarding the network operation until the evaluation index becomes equal to or greater than the predetermined threshold value.

4. A prompt generation method in which a computer executes the following processes: generating a prompt to be input into a large-scale language model; inputting the prompt into the large-scale language model multiple times to obtain multiple answers; calculating an evaluation index indicating the consistency of the multiple answers; and, if the evaluation index is less than a predetermined threshold, modifying the prompt so as to obtain a higher evaluation index.