Prompt input support system, method for supporting input of prompt, and prompt input support program
The prompt input support system addresses the inconsistency issue in large-scale language models by evaluating and selecting the most robust AI for each prompt, resulting in accurate and noise-resistant responses.
Patent Information
- Application Number
- JP2023178390
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-10-16
- Publication Date
- 2025-04-28
- Estimated Expiration
- 2043-10-16
AI Technical Summary
Large-scale language models generate different responses to the same prompt based on variations in notation, leading to inconsistent and inaccurate outputs.
A prompt input support system that calculates feature amounts for each prompt candidate, acquires responses from multiple generation system AIs, associates responses with feature amounts, evaluates response accuracy, and selects the most robust AI for new prompts based on evaluation results.
The system achieves responses that are less susceptible to noise, ensuring accuracy and consistency across varying prompt representations.
Smart Images
Figure 2025068452000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a prompt input support system, a prompt input support method, and a prompt input support program for utilizing generative AI. [Background technology]
[0002] Today, generative AI is being used in various fields (e.g., Non-Patent Document 1). This generative AI generates text, images, and other media as responses by responding to prompts, which are text used to give instructions or ask questions. This generative AI model uses a large-scale language model, which is a machine learning model trained on a huge dataset of text data. It learns the regularities and structures of the input training data and generates new data with similar characteristics in response to the input prompt. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] Mizuho Research & Technologies Co., Ltd., "GPTTM Utilization Support Service "ATHEUS for Generative AI"", [online], May 1, 2023, Mizuho Research & Technologies Co., Ltd., [Retrieved August 11, 2023], Internet <URL:https: / / www.mizuho-rt.co.jp / ai_powerhouse / technology / technology / technology-03 / gptatheus-for-generative-ai.html> Summary of the Invention [Problem to be solved by the invention]
[0004] However, large-scale language models differ depending on the dataset and training method used for training. Furthermore, even if the prompt has the same meaning, it may generate different responses if the spelling varies. If the response is affected by the spelling variation of the prompt, it may not be possible to obtain an accurate response. [Means for solving the problem]
[0005] A prompt input support system that solves the above problem includes a control unit connected to multiple generative AIs. The control unit calculates features for each prompt candidate, acquires responses to which the prompt candidates are input to the multiple generative AIs, associates the features with the responses, acquires evaluation results of the responses, and, when a new prompt is acquired, selects a generative AI from which to acquire the response based on the evaluation results. Effect of the Invention
[0006] According to the present invention, it is possible to obtain a response that is less susceptible to the effects of noise. [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 is an explanatory diagram of a prompt input support system according to an embodiment. [Diagram 2] FIG. 2 is an explanatory diagram of a hardware configuration of the embodiment. [Diagram 3] FIG. [Figure 4] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] An embodiment of a prompt input support system, a prompt input support method, and a prompt input support program will be described with reference to Figures 1 to 4. In this embodiment, it is assumed that the robustness of a generative AI is evaluated. As shown in FIG. 1, the prompt input support system of this embodiment uses a user terminal 10, a support server 20, and multiple AI systems 30, all of which are connected to one another via a network.
[0009] (Hardware configuration example) FIG. 2 shows an example of the hardware configuration of an information processing device H10 that functions as the user terminal 10, the support server 20, the AI system 30, and the like.
[0010] The information processing device H10 includes a communication device H11, an input device H12, a display device H13, a storage device H14, and a processor H 15. Note that this hardware configuration is an example, and the information processing device H10 may include other hardware.
[0011] The communication device H11 is an interface that establishes a communication path with another device and transmits and receives data, and is, for example, a network interface or a wireless interface.
[0012] The input device H12 is a device that accepts input from a user, etc., and is, for example, a mouse, a keyboard, etc. The display device H13 is, for example, a display or a touch panel that displays various information.
[0013] The storage device H14 is a storage device that stores data and various programs for executing various functions of the user terminal 10, the assistance server 20, and the AI system 30. Examples of the storage device H14 include a ROM, a RAM, and a hard disk.
[0014] The processor H15 uses programs and data stored in the storage device H14 to control each process in the user terminal 10, the support server 20, and the AI system 30. Examples of the processor H15 include a CPU and an MPU. The processor H15 expands a program stored in a ROM or the like into a RAM and executes various processes corresponding to various processes. For example, when an application program in the user terminal 10, the support server 20, or the AI system 30 is started, the processor H15 operates a process that executes each process described below.
[0015] The processor H15 is not limited to a processor that performs software processing for all the processes it executes. For example, the processor H15 may include a dedicated hardware circuit (e.g., an application specific integrated circuit (ASIC)) that performs hardware processing for at least a part of the processes it executes. That is, the processor H15 may be configured, for example, as follows:
[0016] [1] One or more processors that operate according to a computer program (software). [2] One or more dedicated hardware circuits that perform at least some of the various processes [3] Circuits that include combinations of these The processor includes a CPU and memory, such as RAM and ROM, that stores program codes or instructions configured to cause the CPU to execute processes. Memory, i.e., computer-readable media, includes any available media that can be accessed by a general-purpose or special-purpose computer.
[0017] (Functions of each information processing device) The functions of the user terminal 10, the support server 20, and the AI system 30 will be described with reference to FIG.
[0018] The user terminal 10 is a computer terminal used by a user who uses this system. The support server 20 is a computer system for evaluating the robustness of a generative AI. The support server 20 includes a control unit 21, a basic information storage unit 22, and an evaluation information storage unit .
[0019] The control unit 21 performs processes (including a management stage, an evaluation stage, a prediction stage, etc.) described below. By executing a prompt input support program for this purpose, the control unit 21 functions as a management unit 211, an evaluation unit 212, a prediction unit 213, etc.
[0020] The management unit 211 executes a process that uses a plurality of generative AIs. The evaluation unit 212 executes a process for evaluating the robustness of the generative AI. The prediction unit 213 executes a process of generating a response to the prompt using a highly robust best practice (AI system 30).
[0021] Basic information for evaluating robustness is recorded in the basic information storage unit 22. The basic information is registered before the preparation stage processing. This basic information includes prompts for evaluating robustness and a number of noises to be added to the prompts.
[0022] A prompt is a question to elicit a response from the AI system 30. Noise is information for changing the content of a prompt. For example, the addition or modification of a character string is used as noise. This noise does not change the content (purpose) of the prompt, but varies the notation. For example, noise may be the addition of a space or the placement of a mask symbol at a predetermined position in a prompt intended to fill in the gaps by predicting a word at a predetermined position in a sentence. Noise also includes naturally occurring variations and variations that are added artificially.
[0023] The evaluation information storage unit 23 records best practices used to obtain responses. These best practices are recorded when the preparatory stage processing is performed and evaluation results of each AI system 30 are obtained. The evaluation information storage unit 23 records best practices for the features of a prompt.
[0024] The features of a prompt are data that indicate the content of the prompt. For example, a distributed representation can be used. In best practice, information (e.g., AI identifier, address) for identifying an AI system 30 that can obtain a stable and robust response according to the features of the prompt is recorded.
[0025] The AI system 30 is a generative AI that generates responses to prompts. The AI system 30 is individually configured according to a computational model and parameters, which are elements that determine the structure of the generative AI.
[0026] (Preparatory stage processing) The preparation stage process will be described with reference to FIG. First, the control unit 21 of the support server 20 executes the following process for each prompt.
[0027] Here, the control unit 21 of the support server 20 executes a prompt acquisition process (step S11). Specifically, the management unit 211 of the control unit 21 acquires prompt candidates to be used for evaluation from the basic information storage unit 22. Then, the management unit 211 calculates feature quantities of the prompt candidates.
[0028] Next, the control unit 21 of the support server 20 executes the following process for each noise. Here, the control unit 21 of the support server 20 executes a noise acquisition process (step S12). Specifically, the management unit 211 of the control unit 21 acquires, from the basic information storage unit 22, noise to be added to the prompt candidate.
[0029] Then, the control unit 21 of the support server 20 executes a noise addition process (step S13). Specifically, the management unit 211 adds the noise acquired from the basic information storage unit 22 to the prompt candidate to be processed.
[0030] Next, the control unit 21 of the support server 20 executes a response acquisition process for each AI system (step S14). Specifically, the evaluation unit 212 of the control unit 21 sequentially identifies the AI systems 30 to be evaluated. Then, the evaluation unit 212 transmits the prompt candidates to which noise has been added to the AI systems 30 to be evaluated, acquires the responses, and temporarily stores them. The above process is then repeated until it is completed for all noise.
[0031] Next, the control unit 21 of the support server 20 executes a robust evaluation process (step S15). Specifically, the evaluation unit 212 of the control unit 21 compares the effects of noise on the responses from each AI system 30. Here, the evaluation unit 212 identifies differences in responses to prompt candidates that include different noises.
[0032] Next, the control unit 21 of the support server 20 executes a best practice registration process (step S16). Specifically, the evaluation unit 212 of the control unit 21 identifies an AI system 30 that can be evaluated as having minimal variation based on a certain evaluation criterion even when noise is included. Then, the evaluation unit 212 registers the identified AI system 30 as a best practice in the evaluation information storage unit 23 in association with the feature amount of the prompt candidate. The above process is then repeated until all prompt candidates have been processed.
[0033] (Prediction stage) Next, the prediction stage process will be described with reference to FIG.
[0034] Here, the control unit 21 of the support server 20 executes a prompt acquisition process (step S21). Specifically, the management unit 211 of the control unit 21 acquires a new prompt input by the user from the user terminal 10. Then, the management unit 211 calculates the feature amount of the acquired new prompt.
[0035] Next, the control unit 21 of the support server 20 executes a best practice selection process (step S22). Specifically, the management unit 211 of the control unit 21 acquires the best practice corresponding to the feature amount of the prompt from the evaluation information storage unit 23.
[0036] Next, the control unit 21 of the support server 20 executes an input process to the best practice (step S23). Specifically, the prediction unit 213 of the control unit 21 inputs the prompt acquired from the user terminal 10 to the AI system 30 of the best practice.
[0037] Next, the control unit 21 of the support server 20 executes a response acquisition process (step S24). Specifically, the prediction unit 213 of the control unit 21 acquires the response from the AI system 30 and outputs it to the user terminal 10.
[0038] (Operation of the embodiment) By using a best practice AI system 30 that is less affected by noise, highly robust responses can be obtained.
[0039] (Effects of the embodiment) (1) In this embodiment, the control unit 21 of the support server 20 executes a prompt acquisition process (step S11) and a noise acquisition process (step S12), thereby making it possible to generate a plurality of prompts with modified expressions. (2) In this embodiment, the control unit 21 of the support server 20 executes a response acquisition process for each AI system 30 (step S14). This makes it possible to acquire responses from different generative AIs for the same prompt content.
[0040] (3) In this embodiment, the control unit 21 of the assistance server 20 executes a robust evaluation process (step S15), which makes it possible to evaluate the effect of noise on the prompt.
[0041] (4) In this embodiment, the control unit 21 of the support server 20 executes a best practice registration process (step S16). Then, the control unit 21 of the support server 20 executes a best practice selection process (step S22). This makes it possible to select a generative AI that is less affected by noise in the prompt.
[0042] (5) In this embodiment, the control unit 21 of the support server 20 executes a response acquisition process (step S24), thereby making it possible to acquire a robust response.
[0043] This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction. In the above embodiment, a user terminal 10, a support server 20, and multiple AI systems 30 are used. The hardware configuration is not limited to this. For example, the AI system 30 may be provided in the support server 20.
[0044] In the above embodiment, a character string is added to the prompt as noise. The method of adding noise is not limited to this. For example, reference data may be searched for the prompt to extract and add necessary knowledge. In this case, noise may be added to a site published on a network or reference data obtained from a database.
[0045] In the above embodiment, the control unit 21 of the support server 20 executes a prompt acquisition process (step S21). In this case, the management unit 211 calculates the feature amount of the prompt. The feature amount of the prompt is not limited to the distributed representation. For example, the type or field of the prompt may be used.
[0046] In the above embodiment, the control unit 21 of the support server 20 executes a robust evaluation process (step S15). This evaluation may be recorded in the evaluation information storage unit 23 by acquiring an evaluation result from a user. In the above embodiment, the control unit 21 of the support server 20 executes a robustness evaluation process (step S15). Here, the evaluation unit 212 identifies differences in responses to prompt candidates including different noises. Alternatively, robustness may be evaluated using the difference between a response to a prompt that does not include noise as a standard and a response to a prompt candidate including different noises. In this case, a generative AI that outputs a response that is less different from the response to a prompt that does not include noise is determined to have high robustness. In the above embodiment, the control unit 21 of the support server 20 executes the preparatory stage process. This preparatory stage process may be performed when a new generative AI is detected. [Explanation of symbols]
[0047] 10...user terminal, 20...support server, 21...control unit, 211...management unit, 212...evaluation unit, 213...prediction unit, 22...basic information storage unit, 23...evaluation information storage unit, 30...AI system.
Claims
1. A prompt input support system including a control unit connected to a plurality of generative AIs, The control unit: Calculate features for each prompt candidate, The prompt candidates are input to a plurality of generative AIs to obtain responses; Acquire an evaluation result of the response in association with the feature amount; A prompt input support system characterized in that, when a new prompt is obtained, a generation AI that obtains the response is selected based on the evaluation result.
2. The control unit: adding noise to the candidate prompts; obtaining a response of the prompt candidate to which the noise has been added from the generation system AI; The prompt input support system according to claim 1 , wherein the evaluation result is obtained according to an effect of the noise on the response.
3. The control unit: adding noise to the reference data to be added to the candidate prompts; obtaining responses of the candidate prompts together with the noisy reference data from the generative AI; The prompt input support system according to claim 1 , wherein the evaluation result is obtained according to an effect of the noise on the response.
4. A prompt input support method for supporting input of a prompt to a plurality of generative AIs using a prompt input support system having a control unit connected to the generative AIs, comprising: The control unit: Calculate features for each prompt candidate, The prompt candidates are input to a plurality of generative AIs to obtain responses; Acquire an evaluation result of the response in association with the feature amount; A prompt input support method characterized in that, when a new prompt is obtained, a generation system AI that obtains the response is selected based on the evaluation result.
5. A prompt input support program for supporting input of a prompt to a plurality of generative AIs using a prompt input support system having a control unit connected to the generative AIs, The control unit, Calculate features for each prompt candidate, The prompt candidates are input to a plurality of generative AIs to obtain responses; Acquire an evaluation result of the response in association with the feature amount; A prompt input support program that functions as a means for selecting a generation system AI that obtains the response based on the evaluation result when a new prompt is obtained.