Prompt creation assistance device and prompt creation assistance method

Abstract

A prompt creation assistance device according to an embodiment of the present invention includes an extraction unit, a label generation unit, a chain-of-thought generation unit, and a recording unit. The extraction unit extracts an operation that is executable by a system from document data including a description of an operation of the system and a command related to the operation, or from an operation history for the system. The label generation unit generates a label summarizing an explanation that corresponds to the executable operation included in the document data. The chain-of-thought generation unit generates a chain-of-thought in which an inference step for realizing the executable operation is described linguistically. The recording unit records the label and the chain-of-thought in association with each other in a database.

Description

Prompt creation support device and prompt creation support method

[0001] The present invention relates to a prompt creation support device and a prompt creation support method.

[0002] In recent years, systems for improving business processes have become in-house. For example, developers use large-scale language models (LLMs) as programming tools. Developers input their own prompts into the LLM, which generates a series of commands to instruct the system to perform the desired operations.

[0003] Ori Ram, et al., “In-Context Retrieval-Augmented Language Models. Transactions of the Association for Computational Linguistics”, 11:1316-1331. 2023. Maciej Besta, et al., “Graph of Thoughts: Solving Elaborate Problems with Large Language Models”, ArXiv, abs / 2308.09687. (2023).Junchi Yu, et al., “Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Models”, ArXiv, abs / 2310.03965. (2023).

[0004] However, if a developer does not have sufficient knowledge or skills about the system to be operated, he or she will be unable to create high-quality prompts to instruct the large-scale language model. For example, the developer may mistakenly create a prompt that causes the system to perform an operation that the developer does not want (or that the system cannot perform). Therefore, the developer is unable to create a prompt that causes the system to perform the operation that the developer wants. Also, even if the developer does not have the knowledge, the large-scale language model may be able to create the desired prompt if it does have the knowledge. However, some systems require confidential information and specialized knowledge to operate, making it difficult to train a large-scale language model to learn knowledge about all systems.

[0005] An object of the present invention is to improve user convenience.

[0006] A prompt creation assistance device according to an embodiment includes an extraction unit, a label generation unit, a thought chain generation unit, and a recording unit. The extraction unit extracts operations executable by the system from document data containing descriptions of system operations and commands related to the operations, or from an operation history of the system. The label generation unit generates labels summarizing explanatory portions corresponding to the executable operations contained in the document data. The thought chain generation unit generates chains of thought that linguistically describe inference steps that realize the executable operations. The recording unit associates the labels with the thought chains and records them in a database.

[0007] According to the present invention, it is possible to improve convenience for users.

[0008] FIG. 1 is a block diagram of a prompt creation support system. FIG. 2 is a block diagram of a prompt creation support device. FIG. 3 is a conceptual diagram of a target system assumed in this embodiment. FIG. 4 is a flowchart showing the LoT creation process of the prompt creation support device. FIG. 5 is a flowchart showing details of the CoT generation process by the CoT generation unit and the reference information assignment unit. FIG. 6 is a diagram showing an example of CoT generation by the CoT generation unit. FIG. 7 is a conceptual diagram of LoT stored in the LoT DB. FIG. 8 is a flowchart showing the prompt creation support process by the prompt creation support device. FIG. 9 is a diagram showing a specific example of search processing related to an initial prompt. FIG. 10 is a diagram showing the data flow of the prompt creation support system for an initial prompt. FIG. 11 is a block diagram showing an example of the hardware configuration of the prompt creation support device.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. In this embodiment, multiple parts with the same reference numerals operate in the same manner, and redundant description will be omitted as appropriate.

[0010] 1 is a block diagram of a prompt creation support system 100 including a prompt creation support device 1 according to this embodiment. The prompt creation support system 100 includes the prompt creation support device 1, a large-scale language model 2 (hereinafter referred to as LLM2), a target system 3, an external information database (DB) 4, and a Lake-of-Thought (LoT) database (DB) 5.

[0011] The prompt creation support device 1 generates multiple chains of thought (hereinafter referred to as CoT) to be stored in the LoT DB 5. The prompt creation support device 1 also receives prompt input from the user U and generates prompts that can be executed by the target system 3. CoT is a method of prompt engineering in LLM, and is a method for realizing complex reasoning using LLM by including intermediate reasoning steps, in other words, a series of procedures leading up to solving a task, as prompts.

[0012] The LLM 2 receives prompt input from the prompt creation support device 1 and outputs operation instructions and commands for the target system.

[0013] The target system 3 is a system to be operated by the user U. For example, it is software such as a simulator or robotic process automation (RPA), an application, or a network, a server, and software or an application that manages or controls them.

[0014] The external information DB 4 is a database of instructions, manuals, and operational rules relating to the operation and functions of the target system 3. The external information DB 4 may be information that can be collected by searching on the Internet.

[0015] The LoT DB 5 is a database that stores LoTs that associate a plurality of CoTs, and associates and stores a plurality of CoTs that can operate the target system 3. The LoT will be described later with reference to FIG.

[0016] In the prompt creation support system 100, the prompt creation support device 1, the LLM 2, the target system 3, the external information DB 4, and the LoT DB 5 may be connected via a wired network, a wireless network, or via the Internet.

[0017] Next, a prompt creation support device 1 according to this embodiment will be described with reference to the block diagram of Fig. 2. The prompt creation support device 1 includes a processing circuit 11, a storage unit 12, a display unit 13, an input unit 14, and a communication unit 15.

[0018] The processing circuit 11 includes an operation extraction unit 111, an explanation extraction unit 112, a label generation unit 113, a chain of thought (CoT) generation unit 114, a recording unit 115, a reference information assignment unit 116, a prompt acquisition unit 117, a search unit 118, a prompt generation unit 119, and a system control unit 120.

[0019] The operation extraction unit 111 refers to the external information DB 4 or the like, and extracts operations that can be executed by the target system 3 from document data that includes descriptions of operations of the target system 3 and commands related to the operations, or from the operation history for the target system 3. The explanation extraction unit 112 extracts explanation parts that correspond to the executable operations, included in the document data.

[0020] The label generation unit 113 generates labels that summarize the explanatory parts extracted by the explanation extraction unit 112. The CoT generation unit 114 generates a CoT that linguistically describes inference steps that realize executable operations. LLM2 may be used to generate the CoT.

[0021] The recording unit 115 records the LoT in which the label and the thought chain are associated with each other in the LoT DB 5. The reference information assigning unit 116 assigns reference information to each Thought in the CoT so as to associate the reference information with each Thought.

[0022] The prompt acquisition unit 117 acquires from the user U an initial prompt, which is a prompt for an operation or process desired by the user on the target system 3. The search unit 118 searches the LoT DB 5 using the initial prompt input by the user to identify a label that matches or is similar to the initial prompt. The search unit 118 also searches the LoT DB 5 using the initial prompt input by the user to identify a series of inference steps required to realize an operation related to the initial prompt across multiple CoTs.

[0023] The prompt generation unit 119 generates a CoT associated with the identified label as an execution prompt. The prompt generation unit 119 also generates a new CoT for the identified series of steps as an execution prompt. The execution prompt is input to the LLM2, which generates a series of processing steps that can be executed in the target system. The processing steps are executed sequentially in the target system, and operations that achieve the user's intended purpose are performed. The system control unit 120 controls the overall operation of the prompt creation assistance device 1. For example, it controls the display of data to the user and the maintenance of the LoT DB5.

[0024] The storage unit 12 records an initial prompt, an execution prompt, reference information, etc. Although not shown, the storage unit 12 may also store an LLM2.

[0025] The display unit 13 displays information related to the initial prompt and the execution prompt. The input unit 14 accepts input from the user U. The communication unit 15 performs data communication between the LLM 2, the target system 3, the external information DB 4, and the LoT DB 5.

[0026] Next, the target system 3 assumed in this embodiment will be described with reference to the conceptual diagram of Fig. 3. Fig. 3 shows an example of a stack configuration from hardware to software that realizes the execution environment of the target system 3. System 3A is, for example, an example of a platform that executes containerized services, and shows an example in which a virtual execution environment "VMware (registered trademark)", an OS (operating system) "Linux (registered trademark)", an orchestrator "Kubernetes (registered trademark)", and applications written in the programming language "Python (registered trademark)" are installed on the hardware.

[0027] System 3B is an example of a browser application, and shows an example in which the OS "Windows (registered trademark)", the browser "Microsoft Edge (registered trademark)", and the application are written in the programming language "JavaScript (registered trademark)" on hardware.

[0028] For example, when user U inputs a prompt to the LLM 2 to operate the target system 3, it may be necessary to describe specific processes, procedures, and commands for that operation for each stack in order to realize that operation. Taking system 3A as an example, the virtual execution environment "VMware," the OS (operating system) "Linux," and the orchestrator "Kubernetes" may each require specific settings, such as version confirmation and prior installation of required applications. In the prompt creation support device 1 according to this embodiment, the LoT DB 5 is created so that the desired operation can be performed even when user U generates a prompt without being aware of the stack configuration or without requiring knowledge of the stack.

[0029] Next, an example of LoT creation processing by the prompt creation assistance device 1 according to this embodiment will be described with reference to the flowchart of FIG.

[0030] In step S1, the operation extraction unit 111 extracts operations that can be executed by the system from the operation history. For example, any operation may be extracted from the operation history as the executable operation. In step S2, the explanation extraction unit 112 searches for an explanation portion that includes the executable operation extracted by the operation extraction unit 111 from external information stored in the external information DB4, such as manuals, operation instructions, and operation rules for the system components.

[0031] In step S3, the explanation extraction unit 112 determines whether an explanation portion has been extracted from the reference information (whether the search has yielded a hit). If the search has yielded a hit, the process proceeds to step S4A. If the search has not yielded a hit, i.e., if the relevant explanation portion has not been extracted, the process proceeds to step S4B.

[0032] In step S4A, the explanation extraction unit 112 extracts an explanation portion from external information. In step S4B, the explanation extraction unit 112 searches the Internet for information corresponding to the explanation portion. In step S4C, the explanation extraction unit 112 determines whether or not an explanation portion was extracted using the Internet (whether the search resulted in a hit). If the search resulted in a hit, the process proceeds to step S4A, where the corresponding explanation portion is extracted. If the search resulted in no hit, i.e., if the corresponding explanation portion could not be extracted, the process ends.

[0033] In step S5, the label generation unit 113 generates labels by summarizing the explanatory part. Specifically, the label generation unit 113 may use the heading of the explanatory part as the label, or may use a trained model trained to summarize sentences, input the explanatory part into the trained model, and use the summary output from the trained model as the label.

[0034] In step S6, the CoT generation unit 114 and the reference information assignment unit 116 generate a CoT. Specifically, the CoT generation unit 114 generates, for example, the execution order of commands included in the description portion as a CoT, and the reference information assignment unit 116 assigns a corresponding description to each Thought as reference information. Furthermore, if a similar CoT (hereinafter, similar CoT) is recorded in advance in the LoT DB 5, the similar CoT may be used.

[0035] In step S7, the recording unit 115 associates the CoT with the label and records them in the LoT DB 5. The label associated with the CoT is also called a prompt label.

[0036] Next, details of the CoT generation process performed by the CoT generation unit 114 and the reference information assignment unit 116 in step S6 will be described with reference to the flowchart of FIG.

[0037] In step S61, the CoT generation unit 114 uses a target label (for convenience of explanation, referred to as a target label) to be recorded in the LoT DB 5 to search for a CoT (hereinafter, a similar CoT) that is similar to the target label within the LoTs recorded in the LoT DB 5. Specifically, if the degree of match in wording or processing content between the target label and a label recorded in the LoT DB 5 is equal to or greater than a threshold, the target label is determined to be a similar CoT.

[0038] In step S62, the CoT generation unit 114 determines whether a similar CoT has been extracted (whether the search has yielded a hit). If the search has yielded a hit, the process proceeds to step S63A. If the search has not yielded a hit, i.e., if a similar CoT has not been extracted, the process proceeds to step S63B.

[0039] In step S63A, the CoT generation unit 114 evaluates whether the similar CoT realizes the operation of the target label. That is, the similar CoT is input to the LLM2 as a prompt, and it is determined whether the output obtained from the LLM2 can realize the desired operation on the target system 3.

[0040] In step S64A, if the similar CoT can perform label manipulation, the CoT generation unit 114 determines that the similar CoT can perform manipulation based on the target label, and therefore does not need to generate a new CoT. Therefore, the process ends. Note that the target label may be added as a label of the similar CoT. If the similar CoT cannot perform label manipulation, the process proceeds to step S63B.

[0041] In step S63B, the CoT generation unit 114 sets the target label as a prompt.

[0042] In step S64B, the CoT generation unit 114 generates a new CoT based on the LoT and external information. The CoT generation unit 114 may sequentially combine the Thoughts present in the LoT with the Thoughts extracted from the external information to generate a new CoT. At this time, the reference information assignment unit 116 assigns, to each Thought, a description portion related to the inference step corresponding to the Thought as reference information, and associates the Thoughts. Of course, the CoT generation unit 114 may sequentially combine the inference steps extracted only from the external information as Thoughts to generate a new CoT, and the reference information assignment unit 116 may assign reference information corresponding to each Thought. If necessary, the Thoughts included in the CoT may be further subdivided. In this way, each Thought in the CoT is linked to reference information that serves as the basis.

[0043] In step S65, the CoT generation unit 114 combines (integrates) common Thoughts among those included in the newly generated CoT.

[0044] Next, an example of CoT generation in the CoT generation unit 114 will be described with reference to FIG.

[0045] As an example, the CoT generation method described in the aforementioned Non-Patent Document 2 is adopted here. Specifically, using a technique described as GoT (Graph-of-Thought), the following processes are executed: Splitting process, in which LLM2 infers multiple patterns and generates multiple Thoughts, Refining process, in which LLM2 modifies the Thoughts to make them appropriate, Aggregating process, in which LLM2 integrates multiple Thoughts, Evaluation process, in which CoT is evaluated for its ability to realize prompts, and Backtracking process, in which if the CoT is rejected as a result of the Evaluation, the use of the Thought in other CoTs is also prohibited.

[0046] In the example of FIG. 6 , a prompt (Though) for the task "Build WebConsole" and a prompt for the splitting process for the LLM are input to the LLM, generating two branching Thoughts. Evaluation processing is then performed for each Thought to determine whether the task can be realized. In the upper branch, Thoughts are linked together, linking commands for realizing the "Build WebConsole" task with reference information for those commands. Meanwhile, in the lower branch, because no commands for realizing the task follow, a backtracking process is performed to prevent connection to other CoTs.

[0047] Next, the concept of the LoT stored in the LoT DB 5 will be described with reference to Fig. 7. Fig. 7 shows an example of a state in which two CoTs included in the LoT are combined.

[0048] CoT71 is a CoT generated by the method shown in Fig. 6, and CoT72 is a CoT generated by a different method. Reference information 74 is associated with though 73 of each of CoT71 and CoT72.

[0049] 7, the Thoughts of CoT71 and CoT72 are combined to form a new CoT. Specifically, Thought 73a of CoT72 may be the first initial Thought, and a new CoT may be formed in the order of Thought 73b of CoT71, Thought 73c of CoT72, and Thought 73d of CoT72. A new label may be associated with the new CoT and recorded in LoT DB5.

[0050] Next, the prompt creation support process performed by the prompt creation support device 1 will be described with reference to the flowchart of FIG.

[0051] In step S81, the prompt acquisition unit 117 acquires an initial prompt from the user. In step S82, the search unit 118 uses the acquired initial prompt to search the LoT DB 5 for a label that matches or is similar to the initial prompt.

[0052] In step S83, the search unit 118 determines whether a label that matches or is similar to the initial prompt exists in the LoT, that is, whether the search for the initial prompt by the search unit 118 has yielded a hit. If the search yields a hit, the process proceeds to step S84A. If the search yields no hit, that is, if a label that matches or is similar to the initial prompt cannot be extracted, the process proceeds to step S84B.

[0053] In step S84A, the search unit 118 identifies a label in the LoT DB 5 that matches or is similar to the prompt.

[0054] In step S85, the prompt generation unit 119 generates a CoT corresponding to the identified label as an execution prompt.

[0055] In step S86, the prompt generating unit 119 or the system control unit 120 presents the extracted execution prompt to the user.

[0056] Meanwhile, in step 84B, the search unit 118 searches for a Thought that matches or is similar to the initial prompt, and then, based on the corresponding Thought, traces the associated or adjacent Thoughts in the LoT Db5. In this case, the search unit 118 continues evaluating the adjacent Thoughts and searching for subsequent Thoughts to see if they can be a Thought that satisfies the task of the initial prompt. If a Thought that is evaluated as satisfying the task of the initial prompt is found, the search for Thoughts is terminated, and the multiple Thoughts obtained as a result of the search are extracted as a new CoT.

[0057] In step S87, the label generation unit 113 summarizes the initial prompt obtained in step S81 and generates a label. The recording unit 115 assigns the label generated from the initial prompt to the new CoT. The recording unit 115 records the labeled CoT in the LoT DB 5. Then, the process proceeds to step S86, where the new CoT is presented to the user as an execution prompt.

[0058] Next, a specific example of search processing related to the initial prompt will be described with reference to Fig. 9. Fig. 9 assumes that the CoT shown in Fig. 6 is recorded in the LoT DB 5, and when the LoT DB 5 is searched using the initial prompt 91, no similar label is found.

[0059] Assume that "Build and Install mobile network simulator" is entered as the initial prompt 91.

[0060] Here, when the search unit 118 searches the LoT DB 5 using the initial prompt 91, the Thought "Build WebConsole" in the CoT shown in FIG. 6 is found as a similar Thought. The prompt generation unit 119 then extracts the Thoughts that follow the similar Thought "Build WebConsole." Then, a new CoT that realizes the task of the initial prompt 91 can be generated as an execution prompt using the method of the flowchart shown in FIG. 8. In this way, the Thoughts included in the LoT DB 5 have a graph structure in which common or adjacent Thoughts are connected to each other, so that a new CoT can be created that conforms to the initial prompt.

[0061] Next, the overall data flow of prompt creation support system 100 for the initial prompt will be described with reference to Fig. 10. In the example of Fig. 10, prompt creation support system 100 includes a prompt creation support device 1, an LLM 2, a target system 3, an LoT DB 5, and a parser gateway 6.

[0062] The user U inputs an initial prompt to the prompt creation support device 1. The prompt creation support device 1 creates an execution prompt from the LoT DB 5 using a label or Thought that matches or is similar to the initial prompt, and presents the execution prompt to the user U. If the user U agrees with the execution prompt, the execution prompt is input to the LLM 2 via the prompt creation support device 1. Alternatively, the user U may input the execution prompt directly to the LLM 2.

[0063] The LLM 2 receives an execution prompt and outputs operations and commands for executing the task intended by the initial prompt. The parser gateway 6 accepts the operations and commands output from the LLM 2 in response to the execution prompt, formats the operations and commands into a format executable by the target system 3, and filters the operations and commands to prevent prohibited operations from being executed. The target system 3 executes the formatted operations and commands and outputs the results.

[0064] Next, an example of the hardware configuration of the prompt creation assistance device 1 according to this embodiment will be described with reference to the block diagram shown in Fig. 11. As shown in Fig. 11, the prompt creation assistance device 1 includes, for example, a central processing unit (CPU) 101, a read-only memory (ROM) 102, a random access memory (RAM) 103, a display 104, a communication interface 105, storage 106, and an input / output interface 107.

[0065] The CPU 101 is an integrated circuit capable of executing various programs and controls the overall operation of the prompt creation assistance device 1. The CPU 101 may also be called a "processor." The ROM 102 is, for example, a non-volatile semiconductor memory and stores programs and control data for controlling the prompt creation assistance device 1. The RAM 103 is, for example, a volatile semiconductor memory and is used as a work area for the CPU 101. The display 104 is, for example, a liquid crystal display or a touch panel display and displays information. The communication interface 105 is, for example, a unit for exchanging information with the outside, such as a USB, HDMI (registered trademark), or network interface. The storage 106 is a non-volatile storage device. The storage 106 stores system software and the like for the prompt creation assistance device 1. The input / output interface 107 is an interface that executes data input and output.

[0066] In the above embodiments, the CPU 101 of the prompt creation support device 1 may be another circuit (or processor). For example, instead of a CPU, the prompt creation support device 1 may include a GPU (Graphics Processing Unit), an NPU (Neural Network Processing Unit), an MPU (Micro Processing Unit), or the like. Each of the processes described in each embodiment may be realized by dedicated hardware. The processes of the prompt creation support device 1 may be a mixture of processes executed by software and processes executed by hardware, or may be only one of them.

[0067] According to the present embodiment described above, the operation extraction unit extracts operations that can be performed by the target system 3 from document data including descriptions of operations of the target system 3 and commands related to the operations, or from the operation history for the target system 3. The label generation unit generates labels that summarize explanatory portions corresponding to the executable operations included in the document data. The CoT generation unit generates a CoT, which is a series of inference steps including commands necessary to realize the executable operations. The recording unit records the CoT in the LoT DB, associating the labels with the CoT.

[0068] As a result, when a user inputs a new prompt, the LoT can be searched to extract similar labels or similar Thoughts. Furthermore, because each Thought in the LoT is associated with reference information, prompts can be created to generate commands that achieve desired operations and processes on the target system, even if the user and the LLM do not have knowledge of the target system. This improves user convenience.

[0069] Each embodiment of the present invention is presented as an example and does not limit the scope of the present invention. Each embodiment can be implemented in various forms without departing from the spirit of the present invention. Each embodiment may be combined with other embodiments, and in such cases, combined effects can be obtained. Each embodiment includes multiple components, and various combinations of these multiple components can result in various inventions. Each embodiment or combination of each component is included within the scope of the present invention.

[0070] 1...prompt creation support device 2...large-scale language model 3...target system 3A, 3B...system 4...external information database (DB) 6...parser gateway 11...processing circuit 12...storage unit 13...display unit 14...input unit 15...communication unit 73, 73a, 73b, 73c, 73d...though 74...reference information 91...initial prompt 100...prompt creation support system 101...CPU 102...ROM 103...RAM 104...display 105...communication interface 106...storage 107...input / output interface 111...operation extraction unit 112...explanation extraction unit 113...label generation unit 114...CoT generation unit 115...recording unit 116...reference information assignment unit 117...prompt acquisition unit 118...search unit 119...prompt generation unit 120...System control unit

Claims

1. A prompt creation support device comprising: an extraction unit that extracts operations that can be performed by a system from document data containing descriptions of system operations and commands related to the operations, or from a history of operations on the system; a label generation unit that generates labels that summarize explanatory parts corresponding to the executable operations contained in the document data; a thought chain generation unit that generates a chain of thought that linguistically describes inference steps that realize the executable operations; and a recording unit that associates the labels with the thought chains and records them in a database.

2. The prompt creation support device of claim 1 further comprising: a search unit that searches the database using an initial prompt input by a user to identify a label that matches or is similar to the initial prompt; and a prompt generation unit that generates the thought chain associated with the identified label as an execution prompt.

3. The prompt creation support device described in claim 1 further comprises: a thought chain generation unit that, for each of the inference steps included in the thought chain, extracts from the document data description portions related to the inference step and associates them as reference information, and forms a graph structure that combines the multiple thought chains by combining common or adjacent inference steps among the multiple thought chains; a search unit that searches the graph structure using an initial prompt input by the user to identify a series of processing steps including commands necessary to realize operations related to the initial prompt across the multiple thought chains; and a prompt generation unit that generates a new thought chain related to the identified series of processing steps as an execution prompt.

4. A prompt creation support method in which a computer extracts operations that can be performed by a system from document data containing descriptions of system operations and commands related to those operations, or from a history of operations on the system, generates labels that summarize the explanatory parts included in the document data that correspond to the executable operations, generates a chain of thought that linguistically describes the inference steps that realize the executable operations, and records the label and the chain of thought in a database in association with each other.

Images