Program, method, information processing apparatus, and system

Abstract

To evaluate quality of a prompt obtained by applying a prompt generation condition to a predetermined query.SOLUTION: A program causes a computer to execute: a query acquisition step of acquiring a query related to a predetermined question; a condition acquisition step of acquiring a generation condition which is a condition related to generating a prompt which serves as a sentence to be input to a large-scale language model; a prompt generation step of generating a prompt by applying the generation condition acquired in the condition acquisition step to the query received in the query acquisition step; a prompt transmission step of transmitting the prompt generated in the prompt generation step to a large-scale language model system; an answer receiving step of receiving results of answers for each of the prompts transmitted in the prompt transmission step; and a prompt evaluation step of receiving values of evaluation for the results of answers received in the answer receiving step.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present disclosure relates to a program, a method, an information processing device, and a system. [Background technology]

[0002] Large-scale language models (LLMs) are becoming increasingly recognized as useful in various information processing tasks. Patent Document 1 discloses an FAQ registration support system that can automatically create a new FAQ and automatically determine whether to add it to existing FAQs. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-036210 A Summary of the Invention [Problem to be solved by the invention]

[0004] The problem is that it is not possible to evaluate the quality of the prompt obtained by applying one or more prompt generation conditions, such as a prompt syntax, to a given query. Therefore, the present disclosure has been made to solve the above problem, and its purpose is to provide a technique for evaluating the quality of a prompt obtained by applying prompt generation conditions, such as one or more prompt syntaxes, to a given query. [Means for solving the problem]

[0005] A program to be executed by a computer having a processor and a memory unit, the program comprising: a query acquisition step in which the processor acquires a query related to a specified question; a condition acquisition step in which one or more generation conditions are conditions for generating a prompt that is an input sentence for a large-scale language model; a prompt generation step in which one or more prompts are generated by applying the one or more generation conditions acquired in the condition acquisition step to the query received in the query acquisition step; a prompt sending step in which the one or more prompts generated in the prompt generation step are transmitted to a large-scale language model system; an answer receiving step in which one or more answer results are received in the prompt sending step; and a prompt evaluation step in which an evaluation value for the one or more answer results received in the answer receiving step is received. Effect of the Invention

[0006] In accordance with the present disclosure, a prompt resulting from applying one or more prompt generation conditions, such as a prompt syntax, to a given query can be evaluated for quality. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a block diagram showing the functional configuration of the system 1. [Diagram 2] FIG. 2 is a block diagram showing the functional configuration of the server 10. [Diagram 3] 2 is a block diagram showing the functional configuration of a user terminal 20. FIG. [Figure 4] FIG. 13 is a diagram showing the data structure of a user table 1012. [Diagram 5] FIG. 13 is a diagram showing the data structure of a query table 1013. [Figure 6] FIG. 10 illustrates the data structure of a prompt table 1014. [Figure 7] FIG. 13 is a diagram showing the data structure of a condition table 1021. [Figure 8] 13 is a diagram showing the data structure of a service table 1022. [Figure 9] 13 is a flowchart showing the operation of an evaluation process. [Figure 10] 13 is a flowchart showing the operation of a proposal process. [Figure 11] 11 is a screen example showing the operation of the evaluation process. [Figure 12] 11 is a screen example showing the operation of a proposal process. [Figure 13] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In all the drawings explaining the embodiment, the same reference numerals are given to common components, and repeated explanations are omitted. Note that the following embodiment does not unduly limit the contents of the present disclosure described in the claims. In addition, not all of the components shown in the embodiment are essential components of the present disclosure. In addition, each figure is a schematic diagram and is not necessarily illustrated strictly.

[0009] <System 1 Configuration> The system 1 in this disclosure is an information processing system that provides an information service for evaluating prompt syntax. The system 1 includes a server 10, a user terminal 20, and an information processing device of an LLM system 50, which are connected via a network N. FIG. 1 is a block diagram showing the functional configuration of the system 1. FIG. 2 is a block diagram showing the functional configuration of the server 10. As shown in FIG. FIG. 3 is a block diagram showing the functional configuration of the user terminal 20. As shown in FIG.

[0010] Each information processing device is configured by a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the server 10, the user terminal 20, and the LLM system 50, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0011] <Server 10 Configuration> The server 10 is an information processing device that provides an information service for evaluating prompt syntax. The server 10 includes a storage unit 101 and a control unit 104 .

[0012] <Configuration of the storage unit 101 of the server 10> The storage unit 101 of the server 10 includes an application program 1011 , a user table 1012 , a query table 1013 , a prompt table 1014 , a condition table 1021 , and a service table 1022 .

[0013] The application program 1011 is a program for causing the control unit 104 of the server 10 to function as each functional unit. Application programs 1011 include applications such as a web browser application.

[0014] User table 1012 is a table that stores and manages information about member users (hereinafter, users) who use the service. When a user registers to use the service, the user's information is stored in a new record in user table 1012. This allows the user to use the service according to the present disclosure. The user table 1012 is a table having a user ID as a primary key, and columns of user ID, user name, and user attribute. FIG. 4 is a diagram showing the data structure of the user table 1012. As shown in FIG.

[0015] The user ID is an item for storing user identification information for identifying a user. The user identification information is an item for which a unique value is set for each user. The user name is an item for storing the name of the user. The user name may be set to any character string such as a nickname instead of a name. User attributes are attribute information related to the attributes of a user. Specifically, in addition to the user's gender, age, and occupation, it includes the user's place of residence, educational level (e.g., highest level of education, field of study, etc.), family situation (e.g., whether single or married, whether they have children, etc.), income, hobbies and interests, language ability, health condition, religion, political views, etc.

[0016] The query table 1013 is a table for storing and managing information regarding queries related to questions (query information). The query table 1013 is a table having a query ID as a primary key, and columns of a query ID, a user ID, a query, and a date and time. FIG. 5 is a diagram showing the data structure of the query table 1013. As shown in FIG.

[0017] The query ID is an item for storing query identification information for identifying a query. The query identification information is an item in which a unique value is set for each piece of query information. The user ID is an item for storing user identification information for identifying a user. The query is an item for storing character string (text) information related to a question. Specifically, a query related to the user's physical condition, a query related to a device owned by the user, etc. are stored. For example, queries regarding the user's physical condition include queries such as "What kind of diet should I follow to lower my blood pressure?" and "What kind of exercise is suitable for lowering my blood pressure?". For example, queries regarding devices owned by users include "My PC won't turn on, what should I do?" and "My PC application won't start, what should I do?". The date and time is an item for storing the date and time when a record is newly stored or updated in the query table 1013.

[0018] The prompt table 1014 is a table for storing and managing information (prompt information) relating to a prompt that serves as an input sentence to the LLM system 50 . The prompt table 1014 is a table having a prompt ID as a primary key, and columns of a prompt ID, a user ID, a query ID, a condition ID, a prompt, an answer result, an evaluation value, and a date and time. FIG. 6 is a diagram showing the data structure of the prompt table 1014. As shown in FIG.

[0019] The prompt ID is an item for storing prompt identification information for identifying a prompt. The prompt identification information is an item for which a unique value is set for each piece of prompt information. The user ID is an item for storing user identification information for identifying a user. The query ID is an item for storing query identification information for identifying a query. The condition ID is an item for storing generation condition identification information for identifying the generation condition information. The prompt is an item for storing a prompt that is an input sentence to the LLM system 50. Specifically, character string (text) information related to the prompt is stored. The answer result is an item for storing the answer result to the prompt received from the LLM system 50. Specifically, character string (text) information related to the answer sentence to the prompt is stored. The evaluation value is an item for storing an evaluation value indicating an evaluation result for a prompt. Specifically, an evaluation value indicating the quality of an answer result output from the LLM system 50 based on the prompt is stored. For example, the evaluation value may be an evaluation value based on six levels from 0 to 5, or may be evaluated by a score from 0 to 100. The rating value may be stored in association with the user attributes of the user who has evaluated the prompt. Specifically, the rating value may include the user attributes of the user who has evaluated the prompt. The date and time is an item for storing the date and time when a record is newly stored or updated in the prompt table 1014.

[0020] The condition table 1021 is a table for storing and managing information (generation condition information) related to the generation conditions for generating a prompt. The generation condition information includes information (syntax information) related to the syntax of the prompt, which serves as an input sentence for the large-scale language model (LLM). The condition table 1021 is a table having a condition ID as a primary key, and columns of a condition ID, a generation condition, and a condition evaluation value. FIG. 7 is a diagram showing the data structure of the condition table 1021. As shown in FIG.

[0021] The condition ID is an item for storing generation condition identification information for identifying the generation condition information. The generation condition identification information is an item for which a unique value is set for each generation condition information. The generation condition is an item for storing a generation condition for generating a prompt. The generation condition includes information for specifying a process for generating a specific prompt based on an input specific query. For example, the generation condition includes the following processes. The generation condition includes character string (text) information related to the prompt syntax. A predetermined prompt can be generated by applying a predetermined prompt syntax to an input query. Generally, a prompt is composed of elements that make up the prompt, such as "commands," "context," "input data," and "output indicators." A prompt syntax is usually composed of a template that combines these elements. For example, each combination of the expressions and order of these elements is stored separately as a different prompt syntax. The generation conditions include a process of generating a prompt translated into English or another language for an input text such as a query, a process of generating a prompt translated into English or another language for a part of a word or term contained in the prompt, and a process of translating a prompt generated by applying a predetermined prompt syntax into English or another language. The generation conditions include a process of replacing a predetermined character string with a different predetermined character string for a prompt generated by applying a predetermined prompt syntax to an input text such as a query. The condition evaluation value is an item that stores an evaluation value (condition evaluation value) that indicates the evaluation result for the generation condition. Specifically, an evaluation value that indicates the quality of the answer result output from the LLM system 50 based on the prompt generated based on the prompt syntax is stored. Specifically, the condition evaluation value is calculated based on the evaluation value in the prompt table 1014 by the proposal process described later.

[0022] The service table 1022 is a table for storing and managing information (service information) related to web services. The service table 1022 is a table having the service ID as a primary key, and columns of the service ID, the service name, and the user attribute. FIG. 8 is a diagram showing the data structure of the service table 1022. As shown in FIG.

[0023] The service ID is an item for storing service identification information for identifying a web service. The service identification information is an item for which a unique value is set for each piece of service information. The service identification information may be configured to store the name, URL, domain name, etc. of a website related to the web service. The service name is an item for storing the name of the web service. Any character string can be set as the service name. The user attribute is an item for storing the user attribute of a user who uses a web service. The user attribute may be manually associated by the user with respect to the web service, or may be statistically set according to the user attribute of the user who uses the web service. For example, the user attribute of an average user who uses the web service may be set as the user attribute of the service table 1022.

[0024] <Configuration of the control unit 104 of the server 10> The control unit 104 of the server 10 includes a user registration control unit 1041, an evaluation unit 1042, and a proposal unit 1043. The control unit 104 executes an application program 1011 stored in the storage unit 101, thereby realizing each functional unit.

[0025] The user registration control unit 1041 performs processing to store, in the user table 1012, information on users who wish to use the service according to the present disclosure. The information stored in the user table 1012 is generated when a user opens a web page operated by a service provider from any information processing terminal, enters information into a specific input form, and transmits the information to the server 10. The user registration control unit 1041 stores the received information in a new record in the user table 1012, completing the user registration. This allows the user stored in the user table 1012 to use the service. Before the user registration control unit 1041 registers user information in the user table 1012, the service provider may carry out a predetermined examination to restrict whether or not the user is permitted to use the service. The user ID may be any character string or number that can identify the user, any character string or number desired by the user, or an arbitrary character string or number may be automatically set by the user registration control unit 1041.

[0026] The evaluation unit 1042 executes an evaluation process, the details of which will be described later.

[0027] The proposing unit 1043 executes a proposing process, the details of which will be described later.

[0028] <Configuration of User Terminal 20> The user terminal 20 is an information processing device operated by a user who uses a service. The user terminal 20 may be, for example, a mobile terminal such as a smartphone or a tablet, a stationary PC (Personal Computer), or a laptop PC. It may also be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch-type terminal. The user terminal 20 includes a storage unit 201 , a control unit 204 , an input device 206 , and an output device 208 .

[0029] <Configuration of the storage unit 201 of the user terminal 20> The storage unit 201 of the user terminal 20 includes a user ID 2011 and an application program 2012 .

[0030] The user ID 2011 is an account ID of the user. The user transmits the user ID 2011 from the user terminal 20 to the server 10. The server 10 identifies the user based on the user ID 2011 and provides the user with the service according to the present disclosure. The user ID 2011 includes information such as a session ID temporarily assigned by the server 10 when identifying the user who is using the user terminal 20.

[0031] The application program 2012 may be stored in advance in the storage unit 201, or may be configured to be downloaded from a web server operated by a service provider via a communication IF. The application programs 2012 include applications such as a web browser application. The application program 2012 includes an interpreted programming language such as JavaScript™ that runs on a web browser application stored on the user terminal 20 .

[0032] <Configuration of the control unit 204 of the user terminal 20> The control unit 204 of the user terminal 20 includes an input control unit 2041 and an output control unit 2042. The control unit 204 realizes each functional unit by executing the application program 2012 stored in the storage unit 201.

[0033] <Configuration of the input device 206 of the user terminal 20> The input device 206 of the user terminal 20 includes a camera 2061, a microphone 2062, a position information sensor 2063, a motion sensor 2064, and a touch device 2065.

[0034] <Configuration of the output device 208 of the user terminal 20> The output device 208 of the user terminal 20 includes a display 2081 and a speaker 2082.

[0035] <Configuration of the LLM system 50> The LLM (Large Language Model) system 50 refers to a large-scale artificial intelligence model used in the field of natural language processing (NLP). By learning a large amount of text data (web pages, books, articles, etc.), these models can understand the patterns of the language used by humans and effectively perform natural language generation (NLG) tasks. The LLM system 50 is used in many NLP tasks such as generating responses to specific questions, automatically generating articles, summarizing text, translation, sentiment analysis, etc. It can also be applied to various uses such as education, entertainment, customer service, and product development. The following exist in the LLM system 50. ·OpenAI ChatGPT ·Google Bard ·Stable Diffusion ·midjourney

[0036] <Operation of system 1> The following describes each process of system 1. Figure 9 is a flowchart showing the operation of the evaluation process. FIG. 10 is a flowchart showing the operation of the proposal process. FIG. 11 is a screen example showing the operation of the evaluation process. FIG. 12 is a screen example showing the operation of the proposal process.

[0037] <Evaluation process> The evaluation process is a process for performing an evaluation for a plurality of generation conditions. The evaluation process may be executed in response to an input operation from the user, or may be executed periodically in response to the contents of a pre-prepared query table 1013 and condition table 1021, and the prompt table 1014 may be updated periodically.

[0038] <Evaluation process overview> The evaluation process is a series of steps that include accepting a question (query) from a user, acquiring generation conditions, generating a prompt, sending the prompt to a large-scale language model (LLM) system, receiving an answer result from the LLM system, storing the answer result, presenting the answer result to the user, accepting an evaluation of the answer result from the user, and storing the evaluation.

[0039] <Evaluation process details> The evaluation process will be described in detail below.

[0040] In step S101, the evaluation unit 1042 of the server 10 executes a query acquisition step of acquiring a query related to a predetermined question. Specifically, the evaluation unit 1042 of the server 10 acquires one or more queries by referring to the query table 1013. The evaluation unit 1042 of the server 10 may present a predetermined query reception screen or the like to the user and acquire queries in response to an input operation by the user in a query input field or the like of the query reception screen. The query table 1013 may be configured to store queries such as questions input in advance by the user in a predetermined web service or the like. In this case, predetermined queries may be accepted not only from a specific user but also from a large number of users via the Internet. In the present disclosure, as an example, the evaluation unit 1042 of the server 10 acquires a query A and a query B.

[0041] [Query A] What kind of diet should I follow to lower my blood pressure?

[0042] [Query B] My computer won't turn on, what should I do?

[0043] In step S102, the evaluation unit 1042 of the server 10 executes a condition acquisition step of acquiring one or more generation conditions that are conditions related to the generation of a prompt that serves as an input sentence to the large-scale language model (LLM). Specifically, the evaluation unit 1042 of the server 10 acquires one or more generation conditions by referring to the condition table 1021. The evaluation unit 1042 of the server 10 does not need to acquire all the generation conditions stored in the condition table 1021, and may be configured to acquire one or more of the generation conditions. For example, the evaluation unit 1042 of the server 10 may acquire one or more generation conditions randomly or according to a predetermined order. In this disclosure, as an example, a case in which the following prompt syntax A and prompt syntax B are obtained will be described.

[0044] [Prompt Syntax A] # Order: You are a chatbot that outputs the "best output sentence" based on an input sentence. Depending on the content of the input sentence, output the most appropriate output sentence. # Input statement Query # Output statement:

[0045] [Prompt Syntax B] # Order: You are a chatbot that responds with appropriate output sentences based on input sentences. Please output simple, easy-to-understand output that even elementary school students can understand. # Input statement Query # Output statement:

[0046] In step S103, the evaluation unit 1042 of the server 10 executes a prompt generation step of generating one or more prompts by applying one or more generation conditions acquired in the condition acquisition step to the query received in the query acquisition step. Specifically, the evaluation unit 1042 of the server 10 generates one or more prompts by applying one or more queries acquired in step S101 to each of the one or more generation conditions acquired in step S102. In the present disclosure, as an example, by combining a prompt syntax A and a prompt syntax B with a query A and a query B, respectively, the following four types of prompts A to D are generated.

[0047] [Prompt A] (Prompt syntax A, Query A) # Order: You are a chatbot that outputs the "best output sentence" based on an input sentence. Depending on the content of the input sentence, output the most appropriate output sentence. # Input statement What kind of diet should I follow to lower my blood pressure? # Output statement:

[0048] [Prompt B] (Prompt syntax A, Query B) # Order: You are a chatbot that outputs the "best output sentence" based on an input sentence. Depending on the content of the input sentence, output the most appropriate output sentence. # Input statement My computer won't turn on, what should I do? # Output statement:

[0049] [Prompt C] (Prompt syntax B, Query A) # Order: You are a chatbot that responds with appropriate output sentences based on input sentences. Please output simple, easy-to-understand output that even elementary school students can understand. # Input statement What kind of diet should I follow to lower my blood pressure? # Output statement:

[0050] [Prompt D] (Prompt syntax B, Query B) # Order: You are a chatbot that responds with appropriate output sentences based on input sentences. Please output simple, easy-to-understand output that even elementary school students can understand. # Input statement My computer won't turn on, what should I do? # Output statement:

[0051] In the present disclosure, a case in which one or more prompt syntaxes are applied to one or more queries to generate one or more prompts has been disclosed as an example. Note that the generation of a prompt is not limited to only applying a query to a prompt syntax. For example, a prompt may be generated by translating the generated prompt into English or another language. A prompt may be generated by translating a word or term included in the prompt into English or another language. In this case, the evaluation unit 1042 of the server 10 also stores the generated prompt in the prompt item of the prompt table 1014. In this case, generation condition identification information for identifying generation conditions such as the prompt syntax used when generating the prompt, generation conditions such as translation conditions (prompt generation conditions), etc. is stored in association with the prompt in the condition ID item of the prompt table 1014. In other words, the prompt table 1014 is not limited to prompt syntax, and may be configured to store prompt generation conditions related to conditions for generating a prompt, prompts, answer results, and evaluation values ​​in association with each other. This makes it possible to evaluate the merits and demerits of suitable prompt generation conditions for various prompt generation conditions when generating a prompt. Furthermore, the evaluation unit 1042 of the server 10 does not necessarily need to generate a plurality of prompts. For example, when one generation condition is acquired in step S102, the evaluation unit 1042 of the server 10 generates one prompt.

[0052] In step S104, the evaluation unit 1042 of the server 10 executes a prompt sending step of sending one or more prompts generated in the prompt generating step to a large scale language model (LLM) system. The prompt sending step executes a step of sending the prompt generated in the prompt generating step to an external large scale language model (LLM) system operated by a business operator different from the computer. Specifically, the evaluation unit 1042 of the server 10 transmits a request including one or more prompts A to D generated in step S103 to an API endpoint provided by the LLM system 50.

[0053] In step S105, the evaluation unit 1042 of the server 10 executes an answer receiving step of receiving one or more answer results to each of the one or more prompts transmitted in the prompt transmitting step. Specifically, the LLM system 50 is configured to be able to output a character string (text) in response to a prompt received at an API endpoint. The LLM system 50 transmits a response including one or more answer results A to D output based on the received one or more prompts A to D to the server 10. The evaluation unit 1042 of the server 10 receives and accepts the answer results A to D to the prompts A to D included in the response from the LLM system 50.

[0054] In step S106, the evaluation unit 1042 of the server 10 associates the prompt generated in the prompt generating step with the answer result received in the answer receiving step and stores them. Specifically, the evaluation unit 1042 of the server 10 associates the answer results A to D received in step S105 with each of the prompts A to D generated in step S103, and stores them in the prompt and answer result items of the prompt table 1014. At this time, the query ID (query A, query B) of the query used when generating the prompt and the condition ID (prompt syntax A, prompt syntax B) of the generation condition are stored as the query ID and condition ID in the prompt table 1014. As a result, the query identification information, condition identification information, prompt, and answer result are stored in association with each other in the prompt table 1014. Furthermore, when a query is received from a user in step S101, the user identification information of the user may be stored in the user ID field of the prompt table 1014.

[0055] In step S107, the evaluation unit 1042 of the server 10 executes a presenting step of presenting the answer result received in the answer receiving step to the user. Specifically, the evaluation unit 1042 of the server 10 transmits the answer results A to D stored in step S106 to the user terminal 20. The control unit 204 of the user terminal 20 displays the received answer results A to D on the display 2081 of the user terminal 20, thereby presenting them to the user. 11 is an example of an evaluation screen in the evaluation process. Evaluation page D1 is displayed on the display 2081 of the user terminal 20. Evaluation page D1 includes answer areas D101, D102, D103, and D104 corresponding to answer results A to D. Evaluation page D1 includes evaluation input fields D111 to D114 corresponding to the answer areas D101 to D104, respectively. Evaluation page D1 includes a send button D120. It is preferable that the evaluation page D1 displays answer results for the same query under different generation conditions so that they can be compared. For example, the evaluation page D1 displays answer areas D101 and D103 corresponding to prompt A, answer result A to prompt C, and answer result C side by side. Similarly, the evaluation page D1 displays answer areas D102 and D104 corresponding to prompt B, answer result B to prompt D, and answer result D side by side. This allows the user to compare answer result A and answer result C, and input the evaluation results in evaluation input fields D111 and D113 by operating the input device 206 of the user terminal 20. The user can compare answer result B and answer result D, and input the evaluation results in evaluation input fields D112 and D114 by operating the input device 206 of the user terminal 20. When the control unit 204 of the user terminal 20 receives the first answer result in step S105, the control unit 204 presents the first answer result to the user.

[0056] In step S108, the evaluation unit 1042 of the server 10 executes a prompt evaluation step of receiving evaluation values ​​for one or more answer results received in the answer receiving step. The prompt evaluation step executes a step of receiving evaluation values ​​for one or more answer results from the user. Specifically, the user operates the input device 206 of the user terminal 20 to input evaluation values ​​of the answer results A to D displayed in the answer areas D101 to D104 into the evaluation input fields D111 to D114. Specifically, the user operates the input device 206 of the user terminal 20 to input evaluation values ​​on a six-level scale from 0 to 5 indicating the goodness or badness of the answer results A to D into the evaluation input fields D111 to D114. The evaluation input fields D111 to D114 may be configured to allow input of a plurality of different evaluation values ​​for the answer results, such as "readability" and "accuracy." The user selects the send button D120 by operating the input device 206 of the user terminal 20. The control unit 204 of the user terminal 20 transmits to the server 10 the user ID 2011, the evaluation values ​​entered in the evaluation input fields D111-D114, and the prompt IDs associated with the answer results A-D corresponding to the evaluation input fields D111-D114, respectively.

[0057] In step S108, the prompt evaluation step executes a step of receiving an evaluation value obtained by having a large-scale language model (LLM) system evaluate one or more answer results. In the present disclosure, a configuration for accepting an input of an evaluation value from a user has been disclosed as an example, but the present invention is not limited thereto. For example, the evaluation unit 1042 of the server 10 may transmit the answer results A to D to the LLM system 50 and use the answer results as the evaluation value. For example, by using a prompt generated by applying answer results A to D to the following prompt syntax, the evaluation results for answer results A to D can be acquired from the LLM system 50.

[0058] [Prompt to get rating value] # Order: You are a chatbot that rates the quality of answers on a 6-point scale from 0 to 5. Depending on the content of the answer, please rate the quality of the answer in terms of "readability" and "accuracy." # Answer text Answer result # Evaluation result:

[0059] In step S109, the evaluation unit 1042 of the server 10 executes a prompt evaluation storage step of storing the user attributes related to the attributes of the user and the evaluation value received from the user in the prompt evaluation step in association with each other. Specifically, the evaluation unit 1042 of the server 10 stores the evaluation values ​​entered in the evaluation input fields D111 to D114 received from the user terminal 20 in the evaluation value field of a record identified in the prompt table 1014 based on the prompt ID associated with the answer results A to D corresponding to each of the evaluation input fields D111 to D114. The evaluation unit 1042 of the server 10 may store the user ID 2011 of the user who made the evaluation in association with the evaluation value in the evaluation value item of the prompt table 1014. The evaluation unit 1042 of the server 10 may also store the user attributes of the user who made the evaluation in association with the evaluation value in the evaluation value item of the prompt table 1014. As a result, the prompt table 1014 can not only store the evaluation for each prompt, but also store in association which user, what user attributes, and what evaluation value was input by which user. For example, the evaluation results for answer results A to D may differ depending on the user attributes, but it is possible to specify which of prompts A to D is suitable for a given user attribute based on the evaluation values ​​for answer results A to D according to the user attributes.

[0060] <Proposal processing> The proposal process is a process for proposing suitable generation conditions for each web service.

[0061] <Proposal process overview> The proposal process is a series of processes that accept a service selection from a user, identify the service attributes of the selected service, identify a user associated with the service attributes, obtain an evaluation value of a prompt by the user, calculate an evaluation value of the generation conditions, and propose suitable generation conditions to the user based on the evaluation value.

[0062] <Details of the proposal process> The proposal process will be described in detail below.

[0063] In step S301, the proposing unit 1043 of the server 10 executes a type input step of accepting input of the type of a web service from the user. The user operates the input device 206 of the user terminal 20, executes a browser application or the like, and opens the proposal page D3 by inputting the URL of a web page (proposal page) for executing the proposal process. The control unit 204 of the user terminal 20 transmits a request including a user ID 2011 for opening the proposal page to the server 10.

[0064] When the server 10 receives the request, it generates a proposal page and transmits it to the user terminal 20. The control unit 204 of the user terminal 20 displays the proposal page on the display 2081 of the user terminal 20 and presents it. 12 is an example of a proposal screen in the proposal process. Proposal page D3 is displayed on display 2081 of user terminal 20. Proposal page D3 includes a type input field D301 capable of receiving input of the type and genre of a web service. Note that type input field D301 may be an input field capable of receiving information for identifying a web service, such as the service name and service identification information of the web service, instead of the type of web service. The proposal page D3 includes generating conditions D311 to D314 and condition evaluation values ​​D321 to D324 for the generating conditions D311 to D314.

[0065] The user operates the input device 206 of the user terminal 20 to input the type of web service in the type input field D301. Specifically, the user may input information such as the main functions provided by the web service, the purpose, the user demographic, the platform (desktop, mobile, both, etc.), whether it is paid or free, whether account registration is required, etc., in the type input field D301. The user may also input information regarding the genre of the website, such as an EC site, a news site, an education site, an entertainment site, an accounting site, or a time attendance site, in the type input field D301. The control unit 204 of the user terminal 20 transmits information about the type of web service input in the type input field D301 to the server 10. The proposal unit 1043 of the server 10 receives and accepts the information about the type of web service.

[0066] In step S302, the proposing unit 1043 of the server 10 executes a service attribute specifying step of specifying a service attribute of the web service according to the type of the web service input in the type input step. Specifically, the suggestion unit 1043 of the server 10 searches the service table 1022 based on the information on the type of web service received in step S301, and acquires user attributes. For example, the suggestion unit 1043 of the server 10 may acquire user attributes of a plurality of web services according to the type of web service, and identify the service attributes of the web service based on the plurality of user attributes. In addition, when the proposal unit 1043 of the server 10 receives the service name, service identification information, etc. of a web service in step S301, it may search for the service name and service ID in the service table 1022 according to the service name and service identification information, obtain user attributes, and identify the user attributes as service attributes.

[0067] The proposing unit 1043 of the server 10 executes a user specifying step of specifying a user included in the service attribute specified in the service attribute specifying step. Specifically, the suggestion unit 1043 of the server 10 searches the user attribute item of the user table 1012 based on the service attribute identified in step S302, and obtains and identifies the user ID of a user who has the user attribute included in the service attribute. As a result, the proposal unit 1043 of the server 10 can identify the user ID of a user who is expected to be a user according to the service attributes of the web service, in accordance with the type of the web service input in step S301. For example, when the genre is "EC site", the user ID of a user having user attributes such as "female" and "30s" is identified.

[0068] In step S304, the suggestion unit 1043 of the server 10 executes an evaluation value acquisition step of acquiring an evaluation value of the prompt by the user identified in the user identification step. Specifically, based on the user ID identified in step S303, the suggestion unit 1043 of the server 10 searches the evaluation value item of the prompt table 1014 (the item of the user ID of the user who evaluated the stored evaluation value associated with the evaluation value included in the evaluation value) and obtains the prompt ID, evaluation value, and condition ID evaluated by the user. This makes it possible to obtain an evaluation value for past prompts by a user who is assumed to be a user according to the service attributes of the web service.

[0069] In step S305, the suggestion unit 1043 of the server 10 executes a condition evaluation step of calculating a condition evaluation value of one or more generating conditions acquired in the condition acquisition step, based on the evaluation value received in the prompt evaluation step. Specifically, the proposal unit 1043 of the server 10 calculates statistical values ​​such as the average, median, mode, maximum, minimum, etc. for each condition ID for the multiple evaluation values ​​obtained in step S304, and calculates them as evaluation values ​​(condition evaluation values) for the generation conditions identified by the condition ID. This makes it possible to calculate a condition evaluation value for each of one or more generation conditions based on evaluation values ​​of past prompts by users who are assumed to be users according to the service attributes of the web service.

[0070] In step S306, the proposing unit 1043 of the server 10 executes a condition proposing step of proposing one or more generation condition candidates to the user based on the condition evaluation value calculated in the condition evaluating step. Specifically, the suggestion unit 1043 of the server 10 searches the condition table 1021 for the condition ID of the generation condition whose condition evaluation value was calculated in step S304, and acquires the item of the generation condition. The suggestion unit 1043 of the server 10 transmits one or more generation conditions to the user terminal 20 in association with the condition evaluation value. The control unit 204 of the user terminal 20 displays the generating conditions D311 to D314 and the condition evaluation values ​​D321 to D324 included in the received generating condition information on the display 2081 of the user terminal 20, and presents them to the user.

[0071] In step S306, a condition proposing step executes a step of presenting one or more generation condition candidates to the user in order of the condition evaluation value. Specifically, the suggestion unit 1043 of the server 10 may transmit a predetermined number of pieces of generation condition information to the user terminal 20 in order according to the superiority or inferiority of the condition evaluation value. In addition, the suggestion unit 1043 of the server 10 may transmit to the user terminal 20 generation condition information having a condition evaluation value equal to or greater than a predetermined value, excluding generation condition information having a condition evaluation value less than a predetermined value. The control unit 204 of the user terminal 20 displays a predetermined number of generation conditions D311-D314 and condition evaluation values ​​D321-D324 on the display 2081 of the user terminal 20 in order according to the superiority or inferiority of the received condition evaluation values, and presents them to the user. The control unit 204 of the user terminal 20 displays the generation conditions D311-D314 and the condition evaluation values ​​D321-D324 having condition evaluation values ​​equal to or greater than a predetermined value on the display 2081 of the user terminal 20, and presents them to the user.

[0072] In step S306, a condition proposing step executes a step of proposing one or more generation condition candidates to the user according to the type of the web service accepted in the type input step. Specifically, the user can refer to the generation conditions D311-D314 and the condition evaluation values ​​D321-D324 displayed on the display 2081 of the user terminal 20 to select suitable generation conditions for developing a web service, etc., depending on the type of the web service, etc., and the attributes of the user.

[0073] In step S306, the condition proposal step calculates a condition evaluation value based on the user attributes of the user of the web service identified based on the type of web service accepted in the type input step, and on the evaluation value accepted from the user included in the user attributes, and executes a step of proposing one or more generation condition candidates to the user. In the present disclosure, service attributes are identified in step S302 according to the service type accepted in step S301, a user having user attributes included in the service attributes is identified in step S303, a prompt evaluation value by the user is obtained in step S304, and an evaluation value for each generation condition is calculated according to the evaluation value in step S305. This allows the user to select suitable generation conditions for developing a web service from the proposed generation condition candidates according to the type of the web service and the attributes of the user. It is possible to select more suitable generation conditions.

[0074] In the present disclosure, the proposal process is configured to input the service type in step S301, but steps S301 to S303 may be omitted. In this case, all evaluation values ​​included in the prompt table 1014 may be obtained in step S304, a condition evaluation value for the generation condition may be calculated based on all the evaluation values ​​in step S305, and generation condition candidates may be proposed in step S306. In this case, the user can receive a proposal of suitable prompt candidates with excellent condition evaluation values, regardless of the service type.

[0075] <Basic computer hardware configuration> 13 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication IF 991 (interface). These are electrically connected to each other by a communication bus 921.

[0076] The processor 901 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, and the like.

[0077] The main memory device 902 is for temporarily storing programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0078] The auxiliary storage device 903 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0079] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard. The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks that can connect to the Internet via a specified access point (e.g., Wi-Fi (registered trademark)), etc. In the case of wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. In the case of wired connection, the network also includes a network that is directly connected by a USB (Universal Serial Bus) cable or the like.

[0080] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration among multiple computers 90 and connecting them together via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0081] <Basic functional configuration of computer 90> A description will now be given of the functional configuration of a computer realized by the basic hardware configuration (FIG. 13) of a computer 90. The computer comprises at least the functional units of a control unit, a storage unit, and a communication unit.

[0082] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 connected to each other via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0083] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903, expanding the programs in the main storage device 902, and executing processes according to the programs. The control unit can realize functional units that perform various information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0084] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. Furthermore, the processor 901 can secure a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with a program. Furthermore, the control unit can cause the processor 901 to execute processes of adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0085] The term database refers to a relational database, which is used to manage sets of data called masters and tables in a tabular format structurally defined by rows and columns, by associating them with each other. In a database, a table is called a table or master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated. Usually, a column that serves as a primary key for uniquely identifying a record is set in each table and each master, but setting a primary key in a column is not essential. The control unit can cause the processor 901 to add, delete, or update records in a specific table or master stored in the storage unit according to various programs. Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0086] In addition, the database and master in this disclosure may include any data structure (such as a list, a dictionary, an associative array, or an object) in which information is structurally defined. The data structure also includes data that can be considered as a data structure by combining data with a function, class, method, or the like written in any programming language.

[0087] The communication unit is realized by the communication IF 991. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0088] <Additional Notes> The matters described in the above embodiments will be supplemented below.

[0089] (Appendix 1) A program to be executed by a computer having a processor and a memory unit, the program comprising: a query acquisition step (S101) in which the processor acquires a query related to a specified question; a condition acquisition step (S102) in which one or more generation conditions are acquired which are conditions for generating a prompt which is an input sentence to a large scale language model (LLM); a prompt generation step (S103) in which one or more prompts are generated by applying the one or more generation conditions acquired in the condition acquisition step to the query received in the query acquisition step; a prompt sending step (S104) in which one or more prompts generated in the prompt generation step are transmitted to a large scale language model (LLM) system; an answer receiving step (S105) in which one or more answer results are received in the prompt sending step; and a prompt evaluation step (S108) in which an evaluation value for the one or more answer results received in the answer receiving step is received. This allows the prompts obtained by applying one or more generation conditions to a given query to be evaluated for quality.

[0090] (Appendix 2) The program according to claim 1, further comprising: a condition evaluation step (S305) in which a processor calculates a condition evaluation value of one or more generation conditions acquired in the condition acquisition step based on the evaluation value received in the prompt evaluation step. This makes it possible to evaluate the quality of one or more production conditions.

[0091] (Appendix 3) The program described in Appendix 2, wherein the condition acquisition step (S102) includes a step of acquiring one or more prompt syntaxes related to the syntax of a prompt that is an input sentence to the large-scale language model (LLM), the prompt generation step (S103) includes a step of generating one or more prompts by applying the one or more prompt syntaxes acquired in the condition acquisition step to the query received in the query acquisition step, and the condition evaluation step (S305) includes a step of calculating a syntax evaluation value of the one or more prompt syntaxes acquired in the syntax acquisition step based on the evaluation value received in the prompt evaluation step. This allows one or more prompt constructs to be rated for their merits or demerits.

[0092] (Appendix 4) The program according to any one of appendixes 1 to 3, wherein the prompt evaluation step (S108) is a step of accepting an evaluation value for one or more answer results from the user. This allows the prompt to be evaluated as to whether it is good or bad in accordance with the evaluation value received from the user.

[0093] (Appendix 5) 5. The program according to claim 4, wherein the processor executes a prompt evaluation storage step (S109) of associating and storing user attributes related to the attributes of the user with the evaluation value received from the user in the prompt evaluation step. This allows the prompt to be evaluated in terms of its quality in relation to user attributes such as the user's gender, age, occupation, income, place of residence, and educational background.

[0094] (Appendix 6) The program according to any one of appendices 1 to 3, wherein the prompt evaluation step (S108) is a step of accepting an evaluation value obtained by having a large-scale language model (LLM) system evaluate one or more answer results. This allows the prompt to be evaluated as good or bad without the user having to perform any evaluation work.

[0095] (Appendix 7) The program according to claim 2 or 3, wherein the processor executes a condition proposing step (S306) of proposing one or more generation condition candidates to the user based on the condition evaluation value calculated in the condition evaluation step. This allows the user to select suitable generation conditions from the proposed generation condition candidates when developing a web service or the like.

[0096] (Appendix 8) The program according to claim 7, wherein the condition proposing step (S306) is a step of proposing one or more generation condition candidates to the user in order of condition evaluation value. This allows the user to select suitable generation conditions from the proposed generation condition candidates when developing a web service or the like.

[0097] (Appendix 9) A program as described in Appendix 7 or 8, wherein a processor executes a type input step (S301) of receiving input of a type of web service from a user, and a condition proposing step (S306) of proposing one or more candidate generation conditions to the user depending on the type of web service received in the type input step. This allows the user to select suitable generation conditions when developing a web service or the like, depending on the type of the web service or the attributes of the user.

[0098] (Appendix 10) The program described in Appendix 9, wherein the condition proposal step (S306) is a step of calculating a condition evaluation value based on the evaluation value received from the user included in the user attributes of the user of the web service identified based on the type of web service accepted in the type input step, and proposing one or more generation condition candidates to the proposed user. This allows the user to select suitable generation conditions for developing a web service from the proposed generation condition candidates according to the type of the web service and the attributes of the user. It is possible to select more suitable generation conditions.

[0099] (Appendix 11) A program described in any one of Appendices 1 to 10, wherein the prompt sending step (S104) is a step of sending the prompt generated in the prompt generation step to an external large-scale language model (LLM) system operated by an operator different from the computer. This allows the prompts obtained by applying one or more generation conditions to a given query to be evaluated by each external LLM system for quality.

[0100] (Appendix 12) A method implemented on a computer having a processor and a memory, the processor performing all of the steps performed in the invention according to any one of claims 1 to 11. This allows the prompts obtained by applying one or more generation conditions to a given query to be evaluated for quality.

[0101] (Appendix 13) An information processing device comprising a control unit and a memory unit, wherein the control unit executes all of the steps executed in the invention according to any one of Supplementary Note 1 to Supplementary Note 11. This allows the prompts obtained by applying one or more generation conditions to a given query to be evaluated for quality.

[0102] (Appendix 14) A system comprising means for performing all the steps performed in any one of the inventions according to appendix 1 to appendix 11. This allows the prompts obtained by applying one or more generation conditions to a given query to be evaluated for quality. [Explanation of symbols]

[0103] 1 system, 10 server, 101 memory unit, 104 control unit, 106 input device, 108 output device, 20 user terminal, 201 memory unit, 204 control unit, 206 input device, 208 output device, 50 LLM system, 501 memory unit, 504 control unit, 506 input device, 508 output device

Claims

1. A program to be executed by a computer having a processor and a storage unit, The processor, A query acquisition step of acquiring a query related to a predetermined question; a condition acquisition step of acquiring one or more generation conditions which are conditions for generating a prompt that is an input sentence to a large-scale language model (LLM); a prompt generating step of generating one or more prompts by applying the one or more generation conditions acquired in the condition acquiring step to the query received in the query acquiring step; a prompt sending step of sending the one or more prompts generated in the prompt generating step to a large language model (LLM) system; a response receiving step of receiving one or more response results to each of the one or more prompts transmitted in the prompt transmitting step; a prompt evaluation step of receiving an evaluation value for the one or more answer results received in the answer receiving step; A program that executes.

2. The processor, a condition evaluation step of calculating a condition evaluation value of the one or more generation conditions acquired in the condition acquisition step based on the evaluation value received in the prompt evaluation step; Execute The program according to claim 1.

3. The step of obtaining the condition includes obtaining one or more prompt syntaxes related to a syntax of a prompt that is an input sentence to a large-scale language model (LLM); the prompt generating step includes a step of generating the one or more prompts by applying the one or more prompt syntaxes acquired in the condition acquiring step to the query accepted in the query acquiring step, The condition evaluation step includes a step of calculating a syntax evaluation value of the one or more prompt syntaxes acquired in the condition acquisition step based on the evaluation value received in the prompt evaluation step. The program according to claim 2.

4. The prompt evaluation step is a step of receiving an evaluation value for the one or more answer results from a user. The program according to claim 1.

5. The processor, a prompt evaluation storage step of storing a user attribute related to the attribute of the user and the evaluation value received from the user in the prompt evaluation step in association with each other; Execute The program according to claim 4.

6. The prompt evaluation step is a step of receiving an evaluation value obtained by having the large-scale language model (LLM) system evaluate the one or more answer results. The program according to claim 1.

7. The processor, a condition suggestion step of suggesting one or more generation condition candidates to a user based on the condition evaluation value calculated in the condition evaluation step; Execute The program according to claim 2.

8. The condition proposing step is a step of proposing the one or more generation condition candidates to the user in order of the condition evaluation value. The program according to claim 7.

9. The processor, a type input step of receiving an input of a type of the web service from the user; Run The condition proposing step is a step of proposing the one or more generation condition candidates to the user according to the type of the web service accepted in the type input step. The program according to claim 7.

10. The condition suggestion step is a step of calculating the condition evaluation value based on the evaluation value received from the user included in the user attributes of the user of the web service specified based on the type of the web service received in the type input step, and proposing the one or more generation condition candidates to the user. The program according to claim 9.

11. The prompt transmitting step is a step of transmitting the prompt generated in the prompt generating step to an external large scale language model (LLM) system operated by a business operator different from that of the computer. The program according to claim 1.

12. A method implemented on a computer having a processor and a memory, the processor performing all of the steps performed in any of the inventions according to claims 1 to 11.

13. 12. An information processing apparatus comprising a control unit and a storage unit, the control unit executing all steps executed in the invention according to any one of claims 1 to 11.

14. A system comprising means for performing all the steps performed in any one of claims 1 to 11.

Images