Program, method, information processing apparatus, and system

Abstract

To evaluate quality of a prompt obtained by applying a prompt generation condition to a predetermined query.SOLUTION: In a system equipped with a network-connected server, a user terminal, and a large-scale language model system, the server executes: a query acquisition step S101 of acquiring a query related to a predetermined question; a condition acquisition step S102 of acquiring one or more generation conditions which are conditions related to generating a prompt which serves as a sentence to be input to a large-scale language model; a prompt generation step S103 of generating one or more prompts; a prompt transmission step S104 of transmitting the one or more prompts generated in the prompt generation step; an answer receiving step S105 of receiving one or more answer results for each of the one or more prompts; and a prompt evaluation step S108 of receiving values of evaluation for the one or more answer results received in the answer receiving step.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to a program, a method, an information processing device, and a system. [Background technology]

[0002] The usefulness of large-scale language models (LLMs) is becoming increasingly recognized in various information processing tasks. Patent Document 1 discloses an FAQ registration support system that can automatically create a new FAQ and automatically determine whether it should be added to existing FAQs. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-036210 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a problem in that it is not possible to evaluate the quality of the prompt obtained by applying prompt generation conditions such as one or more prompt syntaxes to a given query. Therefore, the present disclosure has been made to solve the above problem, and its purpose is to provide a technology for evaluating the quality of a prompt obtained by applying prompt generation conditions, such as one or more prompt syntaxes, to a specified query. [Means for solving the problem]

[0005] a condition acquisition step of acquiring one or more generation conditions that are conditions for generating a prompt that will be an input sentence to a large-scale language model; a prompt generation step of generating one or more prompts by applying the one or more generation conditions acquired in the condition acquisition step to the query received in the query acquisition step; a prompt sending step of transmitting the one or more prompts generated in the prompt generation step to a large-scale language model system; an answer receiving step of receiving one or more answer results for each of the one or more prompts sent in the prompt sending step; and a prompt evaluation step of receiving evaluation values ​​for the one or more answer results received in the answer receiving step. [Effects of the Invention]

[0006] According to the present disclosure, the prompts obtained by applying prompt generation conditions, such as one or more prompt syntaxes, to a given query can be evaluated for quality. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 2 is a block diagram showing the functional configuration of the system 1. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of the server 10. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of a user terminal 20. [Figure 4] FIG. 10 is a diagram showing the data structure of a user table 1012. [Figure 5] FIG. 10 is a diagram showing the data structure of a query table 1013. [Figure 6] FIG. 10 is a diagram showing the data structure of a prompt table 1014. [Figure 7] FIG. 10 is a diagram showing the data structure of a condition table 1021. [Figure 8] FIG. 10 is a diagram showing the data structure of a service table 1022. [Figure 9] 10 is a flowchart showing the operation of an evaluation process. [Figure 10] 10 is a flowchart showing the operation of a proposal process. [Figure 11] 10 is a screen example showing the operation of the evaluation process. [Figure 12] 10 is a screen example showing the operation of the proposal process. [Figure 13] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all drawings describing the embodiments, common components are designated by the same reference numerals, and repeated description will be omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Furthermore, not all components shown in the embodiments are necessarily essential components of the present disclosure. Furthermore, each drawing is a schematic diagram and is not necessarily a precise illustration.

[0009] <System 1 Configuration> The system 1 in this disclosure is an information processing system that provides an information service for evaluating prompt syntax. The system 1 includes a server 10, a user terminal 20, and an information processing device of an LLM system 50, which are connected via a network N. FIG. 1 is a block diagram showing the functional configuration of the system 1. As shown in FIG. FIG. 2 is a block diagram showing the functional configuration of the server 10. As shown in FIG. FIG. 3 is a block diagram showing the functional configuration of the user terminal 20. As shown in FIG.

[0010] Each information processing device is configured by a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by this hardware configuration will be described later. For each of the server 10, the user terminal 20, and the LLM system 50, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer will be omitted.

[0011] <Server 10 configuration> The server 10 is an information processing device that provides an information service for evaluating prompt syntax. The server 10 includes a storage unit 101 and a control unit 104 .

[0012] <Configuration of the storage unit 101 of the server 10> The storage unit 101 of the server 10 includes an application program 1011 , a user table 1012 , a query table 1013 , a prompt table 1014 , a condition table 1021 , and a service table 1022 .

[0013] The application program 1011 is a program for causing the control unit 104 of the server 10 to function as each functional unit. Application programs 1011 include applications such as a web browser application.

[0014] User table 1012 is a table that stores and manages information about member users (hereinafter, "users") who use the service. When a user registers to use the service, the user's information is stored in a new record in user table 1012. This allows the user to use the service according to the present disclosure. The user table 1012 is a table having a user ID as a primary key and columns of user ID, user name, and user attribute. FIG. 4 is a diagram showing the data structure of the user table 1012. As shown in FIG.

[0015] The user ID is an item that stores user identification information for identifying a user. The user identification information is an item that is set with a unique value for each user. The user name is an item for storing the name of the user. The user name may be set to any character string such as a nickname instead of a name. User attributes are attribute information related to a user's characteristics. Specifically, in addition to the user's gender, age, and occupation, it includes place of residence, educational level (e.g., highest level of education, field of study, etc.), family situation (e.g., whether single or married, whether they have children, etc.), income, hobbies and interests, language ability, health status, religion, and political views.

[0016] The query table 1013 is a table for storing and managing information about queries related to questions (query information). The query table 1013 is a table having a query ID as a primary key and columns of a query ID, a user ID, a query, and a date and time. FIG. 5 is a diagram showing the data structure of the query table 1013.

[0017] The query ID is an item that stores query identification information for identifying a query. The query identification information is an item that has a unique value set for each piece of query information. The user ID is an item for storing user identification information for identifying a user. The query is an item that stores character string (text) information related to a question. Specifically, a query related to the user's physical condition, a query related to a device owned by the user, etc. are stored. For example, queries regarding the user's physical condition include queries such as "What kind of diet should I follow to lower my blood pressure?" and "What kind of exercise is suitable to lower my blood pressure?". For example, queries regarding devices owned by users include "My PC won't turn on, what should I do?" and "My PC application won't start, what should I do?". The date and time is an item for storing the date and time when a record is newly stored or updated in the query table 1013.

[0018] The prompt table 1014 is a table for storing and managing information (prompt information) relating to prompts that are input statements to the LLM system 50. The prompt table 1014 is a table having a prompt ID as a primary key and columns of prompt ID, user ID, query ID, condition ID, prompt, answer result, evaluation value, and date and time. FIG. 6 is a diagram showing the data structure of the prompt table 1014.

[0019] The prompt ID is an item that stores prompt identification information for identifying a prompt. The prompt identification information is an item that has a unique value set for each piece of prompt information. The user ID is an item for storing user identification information for identifying a user. The query ID is an item for storing query identification information for identifying a query. The condition ID is an item for storing generation condition identification information for identifying the generation condition information. The prompt is an item that stores a prompt that serves as an input sentence to the LLM system 50. Specifically, character string (text) information related to the prompt is stored. The answer result is an item for storing the answer result to the prompt received from the LLM system 50. Specifically, character string (text) information related to the answer sentence to the prompt is stored. The evaluation value is an item that stores an evaluation value that indicates the evaluation result for the prompt. Specifically, an evaluation value that indicates the quality of the answer result output from the LLM system 50 based on the prompt is stored. For example, the evaluation value may be an evaluation value based on six levels from 0 to 5, or may be evaluated using a score from 0 to 100. The rating value may be stored in association with the user attributes of the user who has rated the prompt. Specifically, the rating value may include the user attributes of the user who has rated the prompt. The date and time is an item for storing the date and time when a new record is stored or updated in the prompt table 1014.

[0020] Condition table 1021 is a table for storing and managing information (generation condition information) related to generation conditions for generating a prompt. The generation condition information includes information (syntax information) related to the syntax of the prompt, which serves as an input sentence for the large-scale language model (LLM). The condition table 1021 is a table having a condition ID as a primary key, and columns of a condition ID, a generation condition, and a condition evaluation value. FIG. 7 is a diagram showing the data structure of the condition table 1021. As shown in FIG.

[0021] The condition ID is an item for storing generation condition identification information for identifying the generation condition information, and a unique value is set for each generation condition information. The generation condition is an item that stores the generation condition for generating a prompt. The generation condition includes information for specifying the process for generating a specific prompt based on an input specific query. For example, the generation condition includes the following process: The generation conditions include character string (text) information related to the prompt syntax. A predetermined prompt can be generated by applying a predetermined prompt syntax to an input query. Generally, a prompt is composed of elements such as "commands," "context," "input data," and "output indicators." A prompt syntax is usually composed of a template that combines these elements. For example, each combination of the expression and order of these elements is stored separately as a different prompt syntax. The generation conditions include generating a prompt translated into English or another language for input text such as a query, generating a prompt translated into English or another language for a portion of a word or term included in the prompt, and translating a prompt generated by applying a predetermined prompt syntax into English or another language. The generation conditions include a process of replacing a predetermined character string with a different predetermined character string for a prompt generated by applying a predetermined prompt syntax to input text such as a query. The condition evaluation value is an item that stores an evaluation value (condition evaluation value) that indicates the evaluation result for the generation condition. Specifically, an evaluation value that indicates the quality of the answer result output from the LLM system 50 based on the prompt generated based on the prompt syntax is stored. Specifically, the condition evaluation value is calculated based on the evaluation value in the prompt table 1014 by the proposal process described below.

[0022] The service table 1022 is a table for storing and managing information (service information) related to web services. The service table 1022 is a table having the service ID as a primary key and columns of the service ID, service name, and user attribute. FIG. 8 is a diagram showing the data structure of the service table 1022. As shown in FIG.

[0023] The service ID is an item that stores service identification information for identifying a web service. The service identification information is an item that has a unique value set for each piece of service information. The service identification information may be configured to store the name, URL, domain name, etc. of the website related to the web service. The service name is an item that stores the name of the web service. Any character string can be set as the service name. The user attributes are items that store the user attributes of users who use the web service. The user attributes may be manually associated by the user depending on the web service, or may be statistically set depending on the user attributes of users who use the web service. For example, the user attributes of an average user who uses the web service may be set as the user attributes in the service table 1022.

[0024] <Configuration of the control unit 104 of the server 10> The control unit 104 of the server 10 includes a user registration control unit 1041, an evaluation unit 1042, and a proposal unit 1043. The control unit 104 executes an application program 1011 stored in the storage unit 101, thereby realizing each functional unit.

[0025] The user registration control unit 1041 performs processing to store information about users who wish to use the service according to the present disclosure in the user table 1012. The information stored in the user table 1012 is generated when a user opens a web page operated by a service provider from any information processing terminal, enters information into a predetermined input form, and transmits the information to the server 10. The user registration control unit 1041 stores the received information in a new record in the user table 1012, completing the user registration. This allows the user stored in the user table 1012 to use the service. Before the user registration control unit 1041 registers the user information in the user table 1012, the service provider may conduct a predetermined examination to restrict whether or not the user is permitted to use the service. The user ID may be any character string or number that can identify the user, any character string or number desired by the user, or may be automatically set by the user registration control unit 1041.

[0026] The evaluation unit 1042 executes an evaluation process, the details of which will be described later.

[0027] The proposing unit 1043 executes a proposing process, the details of which will be described later.

[0028] <Configuration of User Terminal 20> The user terminal 20 is an information processing device operated by a user who uses a service. The user terminal 20 may be, for example, a mobile terminal such as a smartphone or tablet, a stationary personal computer (PC) or a laptop PC, or a wearable terminal such as a head mounted display (HMD) or a wristwatch terminal. The user terminal 20 includes a storage unit 201 , a control unit 204 , an input device 206 , and an output device 208 .

[0029] <Configuration of the storage unit 201 of the user terminal 20> The storage unit 201 of the user terminal 20 includes a user ID 2011 and an application program 2012 .

[0030] The user ID 2011 is the user's account ID. The user transmits the user ID 2011 from the user terminal 20 to the server 10. The server 10 identifies the user based on the user ID 2011 and provides the user with the service according to the present disclosure. The user ID 2011 includes information such as a session ID temporarily assigned by the server 10 to identify the user using the user terminal 20.

[0031] The application program 2012 may be stored in advance in the storage unit 201, or may be configured to be downloaded from a web server or the like operated by a service provider via a communication IF. The application programs 2012 include applications such as a web browser application. The application program 2012 includes an interpreted programming language such as JavaScript (registered trademark) that runs on a web browser application stored on the user terminal 20.

[0032] <Configuration of the control unit 204 of the user terminal 20> The control unit 204 of the user terminal 20 includes an input control unit 2041 and an output control unit 2042. The control unit 204 realizes each functional unit by executing the application program 2012 stored in the storage unit 201.

[0033] <Configuration of the input device 206 of the user terminal 20> The input device 206 of the user terminal 20 includes a camera 2061, a microphone 2062, a position information sensor 2063, a motion sensor 2064, and a touch device 2065.

[0034] <Configuration of the output device 208 of the user terminal 20> The output device 208 of the user terminal 20 includes a display 2081 and a speaker 2082. [[ID=​​​​​​​​​​​​​​​​​​​​​​​​​​FIG. 10 is a flowchart showing the operation of the proposal process. FIG. 11 is an example of a screen showing the operation of the evaluation process. FIG. 12 is an example of a screen showing the operation of the proposal process.

[0037] <Evaluation process> The evaluation process is a process for evaluating a plurality of generation conditions. The evaluation process may be performed in response to input operations from the user, or may be performed periodically in accordance with the contents of the pre-prepared query table 1013 and condition table 1021, and the prompt table 1014 may be periodically updated.

[0038] <Evaluation process overview> The evaluation process is a series of steps that include accepting a question (query) from a user, obtaining generation conditions, generating a prompt, sending the prompt to a large-scale language model (LLM) system, receiving an answer result from the LLM system, storing the answer result, presenting the answer result to the user, accepting an evaluation of the answer result from the user, and storing the evaluation.

[0039] <Evaluation process details> The evaluation process will be described in detail below.

[0040] In step S101, the evaluation unit 1042 of the server 10 executes a query acquisition step of acquiring a query related to a predetermined question. Specifically, the evaluation unit 1042 of the server 10 acquires one or more queries by referring to the query table 1013. The evaluation unit 1042 of the server 10 may present a predetermined query reception screen or the like to the user and acquire queries in response to the user's input operation into a query input field or the like on the query reception screen. The query table 1013 may also be configured to store queries such as questions that the user has input in advance to a predetermined web service or the like. In this case, predetermined queries may be accepted not only from a specific user but also from a large number of users via the Internet. In the present disclosure, as an example, the evaluation unit 1042 of the server 10 acquires a query A and a query B.

[0041] [Query A] What kind of diet should I follow to lower my blood pressure?

[0042] [Query B] My computer won't turn on, what should I do?

[0043] In step S102, the evaluation unit 1042 of the server 10 executes a condition acquisition step of acquiring one or more generation conditions that are conditions related to the generation of a prompt that serves as an input sentence to the large-scale language model (LLM). Specifically, the evaluation unit 1042 of the server 10 acquires one or more generation conditions by referring to the condition table 1021. Note that the evaluation unit 1042 of the server 10 does not need to acquire all of the generation conditions stored in the condition table 1021, and may be configured to acquire one or more of the generation conditions. For example, the evaluation unit 1042 of the server 10 may acquire one or more generation conditions randomly or in a predetermined order. In this disclosure, as an example, a case will be described in which the following prompt syntax A and prompt syntax B are acquired.

[0044] [Prompt Syntax A] # Order: You are a chatbot that outputs the "best output sentence" based on the input sentence. Depending on the input sentence, output the most appropriate output sentence. # Input statement Query # Output statement:

[0045] [Prompt Syntax B] # Order: You are a chatbot that responds to input sentences with appropriate output sentences. Please output concise and easy-to-understand output that even elementary school students can understand. # Input statement Query # Output statement:

[0046] In step S103, the evaluation unit 1042 of the server 10 executes a prompt generation step in which one or more prompts are generated by applying one or more generation conditions acquired in the condition acquisition step to the query received in the query acquisition step. Specifically, the evaluation unit 1042 of the server 10 generates one or more prompts by applying one or more queries acquired in step S101 to each of one or more generation conditions acquired in step S102. In the present disclosure, as an example, by combining query A and query B with prompt syntax A and prompt syntax B, respectively, the following four types of prompts A to D are generated.

[0047] [Prompt A] (Prompt syntax A, Query A) # Order: You are a chatbot that outputs the "best output sentence" based on the input sentence. Depending on the input sentence, output the most appropriate output sentence. # Input statement What kind of diet should I follow to lower my blood pressure? # Output statement:

[0048] [Prompt B] (Prompt syntax A, Query B) # Order: You are a chatbot that outputs the "best output sentence" based on the input sentence. Depending on the input sentence, output the most appropriate output sentence. # Input statement My computer won't turn on, what should I do? # Output statement:

[0049] [Prompt C] (Prompt syntax B, Query A) # Order: You are a chatbot that responds to input sentences with appropriate output sentences. Please output concise and easy-to-understand output that even elementary school students can understand. # Input statement What kind of diet should I follow to lower my blood pressure? # Output statement:

[0050] [Prompt D] (Prompt Syntax B, Query B) # Order: You are a chatbot that responds to input sentences with appropriate output sentences. Please output concise and easy-to-understand output that even elementary school students can understand. # Input statement My computer won't turn on, what should I do? # Output statement:

[0051] In the present disclosure, an example has been disclosed in which one or more prompt syntaxes are applied to one or more queries to generate one or more prompts. Note that the generation of a prompt is not limited to applying a query to a prompt syntax. For example, a prompt may be generated by translating a generated prompt into English or another language. A prompt may also be generated by translating words or terms included in the prompt into English or another language. In this case, the evaluation unit 1042 of the server 10 stores the generated prompt in the prompt field of the prompt table 1014. In this case, generation condition identification information for identifying generation conditions (prompt generation conditions), such as the prompt syntax used to generate the prompt and the translation conditions, is stored in the condition ID field of the prompt table 1014 in association with the prompt. In other words, the prompt table 1014 is not limited to prompt syntax, and may be configured to store prompt generation conditions related to the conditions for generating a prompt, prompts, answer results, and evaluation values ​​in association with each other. This makes it possible to evaluate the merits and demerits of suitable prompt generation conditions for various prompt generation conditions when generating a prompt. Furthermore, the evaluation unit 1042 of the server 10 does not necessarily have to generate a plurality of prompts. For example, when one generation condition is acquired in step S102, the evaluation unit 1042 of the server 10 generates one prompt.

[0052] In step S104, the evaluation unit 1042 of the server 10 executes a prompt sending step of sending one or more prompts generated in the prompt generating step to a large scale language model (LLM) system. The prompt sending step executes a step of sending the prompts generated in the prompt generating step to an external large scale language model (LLM) system operated by a business operator different from the computer. Specifically, the evaluation unit 1042 of the server 10 transmits a request including one or more prompts A to D generated in step S103 to an API endpoint provided by the LLM system 50.

[0053] In step S105, the evaluation unit 1042 of the server 10 executes an answer receiving step of receiving one or more answer results for each of the one or more prompts transmitted in the prompt transmitting step. Specifically, the LLM system 50 is configured to be able to output a character string (text) in response to a prompt received at an API endpoint. The LLM system 50 transmits a response including one or more answer results A to D output based on one or more received prompts A to D to the server 10. The evaluation unit 1042 of the server 10 receives and accepts the answer results A to D for the prompts A to D included in the response from the LLM system 50.

[0054] In step S106, the evaluation unit 1042 of the server 10 associates the prompt generated in the prompt generating step with the answer result received in the answer receiving step and stores them. Specifically, the evaluation unit 1042 of the server 10 associates the answer results A to D received in step S105 with each of the prompts A to D generated in step S103, and stores them in the prompt and answer result fields of the prompt table 1014. At this time, the query ID and condition ID of the prompt used to generate the prompt (query A, query B) are stored as query IDs and condition IDs of the generation conditions (prompt syntax A, prompt syntax B). As a result, the query identification information, condition identification information, prompts, and answer results are stored in association with each other in the prompt table 1014. Furthermore, when a query is received from a user in step S101, the user identification information of the user may be stored in the user ID field of the prompt table 1014.

[0055] In step S107, the evaluation unit 1042 of the server 10 executes a presentation step of presenting the answer result received in the answer reception step to the user. Specifically, the evaluation unit 1042 of the server 10 transmits the answer results A to D stored in step S106 to the user terminal 20. The control unit 204 of the user terminal 20 displays the received answer results A to D on the display 2081 of the user terminal 20, thereby presenting them to the user. 11 is an example of an evaluation screen in the evaluation process. Evaluation page D1 is displayed on display 2081 of user terminal 20. Evaluation page D1 includes answer areas D101, D102, D103, and D104 corresponding to answer results A to D. Evaluation page D1 includes evaluation input fields D111 to D114 corresponding to answer areas D101 to D104, respectively. Evaluation page D1 also includes a send button D120. It is preferable that the evaluation page D1 displays answer results for different generation conditions for the same query so that they can be compared. For example, the evaluation page D1 displays answer areas D101 and D103 corresponding to answer result A and answer result C for prompt A and prompt C side by side. Similarly, the evaluation page D1 displays answer areas D102 and D104 corresponding to answer result B and answer result D for prompt B and prompt D side by side. This allows the user to compare answer result A and answer result C and input the evaluation results in evaluation input fields D111 and D113 by operating the input device 206 of the user terminal 20. The user can compare answer result B and answer result D and input the evaluation results in evaluation input fields D112 and D114 by operating the input device 206 of the user terminal 20. If the control unit 204 of the user terminal 20 receives the first answer result in step S105, it presents the first answer result to the user.

[0056] In step S108, the evaluation unit 1042 of the server 10 executes a prompt evaluation step of receiving evaluation values ​​for one or more answer results received in the answer receiving step. The prompt evaluation step executes a step of receiving evaluation values ​​for one or more answer results from the user. Specifically, the user inputs evaluation values ​​of the answer results A to D displayed in the answer areas D101 to D104 into the evaluation input fields D111 to D114 by operating the input device 206 of the user terminal 20. Specifically, the user inputs evaluation values ​​on a six-level scale from 0 to 5 indicating the goodness or badness of the answer results A to D into the evaluation input fields D111 to D114 by operating the input device 206 of the user terminal 20. The evaluation input fields D111 to D114 may be configured to allow input of a plurality of different evaluation values ​​for the answer results, such as "readability" and "accuracy." The user selects the send button D120 by operating the input device 206 of the user terminal 20. The control unit 204 of the user terminal 20 transmits to the server 10 the user ID 2011, the evaluation values ​​entered in the evaluation input fields D111 to D114, and the prompt IDs associated with the answer results A to D corresponding to the evaluation input fields D111 to D114, respectively.

[0057] In step S108, a prompt evaluation step executes a step of receiving an evaluation value obtained by having a large-scale language model (LLM) system evaluate one or more answer results. Although the present disclosure has disclosed an example of a configuration in which an evaluation value is input from a user, the present disclosure is not limited to this. For example, the evaluation unit 1042 of the server 10 may transmit the answer results A to D to the LLM system 50 and use the answer results as the evaluation value. For example, by using a prompt generated by applying answer results A to D to the following prompt syntax, the evaluation results for answer results A to D can be obtained from the LLM system 50.

[0058] [Prompt to get rating] # Order: You are a chatbot that rates the quality of answers on a 6-point scale from 0 to 5. Depending on the content of the answer, please rate the quality of the answer in terms of "readability" and "accuracy." # Answer text Answer result # Evaluation result:

[0059] In step S109, the evaluation unit 1042 of the server 10 executes a prompt evaluation storage step of storing the user attributes related to the attributes of the user and the evaluation value received from the user in the prompt evaluation step in association with each other. Specifically, the evaluation unit 1042 of the server 10 stores the evaluation values ​​entered in the evaluation input fields D111 to D114 received from the user terminal 20 in the evaluation value field of the record identified in the prompt table 1014 based on the prompt ID associated with the answer results A to D corresponding to each of the evaluation input fields D111 to D114. The evaluation unit 1042 of the server 10 may store the user ID 2011 of the user who made the evaluation in association with the evaluation value in the evaluation value field of the prompt table 1014. The evaluation unit 1042 of the server 10 may also store the user attributes of the user who made the evaluation in association with the evaluation value in the evaluation value field of the prompt table 1014. As a result, the prompt table 1014 can not only store the evaluation for each prompt, but also store the evaluation value entered by which user and with what user attributes. For example, the evaluation results for answer results A to D may differ depending on the user attributes, but it is possible to identify which prompt A to D is suitable for a given user attribute based on the evaluation value for answer results A to D according to the user attributes.

[0060] <Proposal processing> The proposal process is a process of proposing suitable generation conditions for each web service.

[0061] <Proposal process overview> The proposal process is a series of steps that accepts a service selection from a user, identifies the service attributes of the selected service, identifies a user associated with the service attributes, obtains an evaluation value of a prompt from the user, calculates an evaluation value of the generation conditions, and proposes suitable generation conditions to the user based on the evaluation value.

[0062] <Details of proposal process> The proposal process will be described in detail below.

[0063] In step S301, the proposing unit 1043 of the server 10 executes a type input step of accepting input of the type of web service from the user. The user operates the input device 206 of the user terminal 20 to execute a browser application or the like, and opens the proposal page D3 by inputting the URL of a web page (proposal page) for executing the proposal process. The control unit 204 of the user terminal 20 transmits a request including a user ID 2011 for opening the proposal page to the server 10.

[0064] Upon receiving the request, the server 10 generates a proposal page and transmits it to the user terminal 20. The control unit 204 of the user terminal 20 displays the proposal page on the display 2081 of the user terminal 20 and presents it. 12 is an example of a proposal screen in the proposal process. Proposal page D3 is displayed on display 2081 of user terminal 20. Proposal page D3 includes a type input field D301 that can accept input of the type and genre of a web service. Note that type input field D301 may also be an input field that can accept information for identifying a web service, such as the service name and service identification information of the web service, rather than the type of web service. The proposal page D3 includes creation conditions D311 to D314 and condition evaluation values ​​D321 to D324 for the creation conditions D311 to D314.

[0065] The user operates the input device 206 of the user terminal 20 to input the type of web service in the type input field D301. Specifically, the user may input information such as the main functions provided by the web service, purpose, user demographic, platform (desktop, mobile, both, etc.), whether it is paid or free, and whether account registration is required, in the type input field D301. The user may also input information about the genre of the website, such as an e-commerce site, news site, education site, entertainment site, accounting site, or time attendance site, in the type input field D301. The control unit 204 of the user terminal 20 transmits information about the type of web service entered in the type entry field D301 to the server 10. The proposal unit 1043 of the server 10 receives and accepts the information about the type of web service.

[0066] In step S302, the proposing unit 1043 of the server 10 executes a service attribute specifying step of specifying a service attribute of a web service according to the type of web service input in the type input step. Specifically, the suggestion unit 1043 of the server 10 searches the service table 1022 and acquires user attributes based on the information about the type of web service received in step S301. For example, the suggestion unit 1043 of the server 10 may acquire user attributes of multiple web services according to the type of web service, and identify the service attributes of the web service based on the multiple user attributes. In addition, when the proposal unit 1043 of the server 10 receives the service name, service identification information, etc. of a web service in step S301, it may search the service name and service ID in the service table 1022 according to the service name and service identification information, obtain user attributes, and identify the user attributes as service attributes.

[0067] The proposing unit 1043 of the server 10 executes a user specifying step of specifying users included in the service attributes specified in the service attribute specifying step. Specifically, the suggestion unit 1043 of the server 10 searches the user attribute item in the user table 1012 based on the service attribute identified in step S302, and obtains and identifies the user ID of the user who has the user attribute included in the service attribute. As a result, the proposing unit 1043 of the server 10 can identify the user ID of a user who is expected to be a user according to the service attributes of the web service, in accordance with the type of web service input in step S301. For example, if the genre is "EC site," the user ID of a user having user attributes such as "female" and "in their 30s" is identified.

[0068] In step S304, the proposing unit 1043 of the server 10 executes an evaluation value obtaining step of obtaining an evaluation value of the prompt by the user identified in the user identifying step. Specifically, based on the user ID identified in step S303, the suggestion unit 1043 of the server 10 searches the evaluation value item of the prompt table 1014 (the item of the user ID of the user who evaluated the stored evaluation value associated with the evaluation value included in the evaluation value) and obtains the prompt ID, evaluation value, and condition ID evaluated by the user. This makes it possible to obtain evaluation values ​​for past prompts by users who are assumed to be users according to the service attributes of the web service.

[0069] In step S305, the proposing unit 1043 of the server 10 executes a condition evaluation step of calculating a condition evaluation value of one or more generating conditions acquired in the condition acquisition step, based on the evaluation value received in the prompt evaluation step. Specifically, the proposal unit 1043 of the server 10 calculates statistical values ​​such as the average, median, mode, maximum, and minimum values ​​for each condition ID for the multiple evaluation values ​​obtained in step S304, and calculates them as evaluation values ​​(condition evaluation values) for the generation conditions identified by the condition ID. This makes it possible to calculate a condition evaluation value for each of one or more generation conditions based on evaluation values ​​for past prompts by users who are assumed to be users according to the service attributes of the web service.

[0070] In step S306, the proposing unit 1043 of the server 10 executes a condition proposing step of proposing one or more generation condition candidates to the user based on the condition evaluation value calculated in the condition evaluating step. Specifically, the proposal unit 1043 of the server 10 searches the condition table 1021 for the condition ID of the generation condition whose condition evaluation value was calculated in step S304, and acquires the item of the generation condition. The proposal unit 1043 of the server 10 transmits one or more generation conditions to the user terminal 20 in association with the condition evaluation value. The control unit 204 of the user terminal 20 displays the generation conditions D311 to D314 and the condition evaluation values ​​D321 to D324 included in the received generation condition information on the display 2081 of the user terminal 20, and presents them to the user.

[0071] In step S306, a condition proposing step is executed to present one or more generation condition candidates to the user in order of the condition evaluation value. Specifically, the proposing unit 1043 of the server 10 may transmit a predetermined number of pieces of generation condition information to the user terminal 20 in order of merit of the condition evaluation value. Furthermore, the proposing unit 1043 of the server 10 may exclude generation condition information having a condition evaluation value less than a predetermined value and transmit generation condition information having a condition evaluation value equal to or greater than a predetermined value to the user terminal 20. The control unit 204 of the user terminal 20 displays a predetermined number of generation conditions D311 to D314 and condition evaluation values ​​D321 to D324 in order according to the received condition evaluation values ​​on the display 2081 of the user terminal 20, and presents them to the user. The control unit 204 of the user terminal 20 displays the generation conditions D311 to D314 and condition evaluation values ​​D321 to D324 having condition evaluation values ​​equal to or greater than a predetermined value on the display 2081 of the user terminal 20, and presents them to the user.

[0072] In step S306, a condition proposing step is executed to propose one or more creation condition candidates to the user according to the type of web service accepted in the type input step. Specifically, the user can refer to the generation conditions D311 to D314 and the condition evaluation values ​​D321 to D324 displayed on the display 2081 of the user terminal 20 to select the generation conditions that are suitable for developing a web service, etc., depending on the type of the web service, etc. and the attributes of the user.

[0073] In step S306, the condition proposal step calculates a condition evaluation value based on the evaluation value received from the user included in the user attributes of the user of the web service identified based on the type of web service received in the type input step, and executes a step of proposing one or more generation condition candidates to the user. In the present disclosure, in step S302, service attributes are identified according to the service type accepted in step S301, in step S303, a user having a user attribute included in the service attributes is identified, in step S304, a prompt evaluation value by the user is obtained, and in step S305, an evaluation value for each generation condition is calculated according to the evaluation value. This allows users to select suitable creation conditions for developing web services from candidate creation conditions that are proposed according to the type of web service and the attributes of the user, allowing users to select more suitable creation conditions.

[0074] In the present disclosure, the proposal process is configured to input the service type in step S301, but steps S301 to S303 may be omitted. In this case, all evaluation values ​​contained in prompt table 1014 may be obtained in step S304, a condition evaluation value for the generation condition may be calculated based on all the evaluation values ​​in step S305, and generation condition candidates may be proposed in step S306. In this case, the user can receive suitable prompt candidates with excellent condition evaluation values, regardless of the service type.

[0075] <Basic computer hardware configuration> 13 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main memory device 902, an auxiliary memory device 903, and a communication IF 991 (interface), which are electrically connected to one another by a communication bus 921.

[0076] The processor 901 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0077] The main memory device 902 is used to temporarily store programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0078] The auxiliary storage device 903 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0079] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards. A network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, networks include 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can connect to the Internet via a predetermined access point. In the case of a wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), and Bluetooth (registered trademark). In the case of a wired connection, networks also include those that are directly connected using a USB (Universal Serial Bus) cable, etc.

[0080] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration across multiple computers 90 and interconnecting them via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0081] <Basic functional configuration of computer 90> The following describes the functional configuration of a computer realized by the basic hardware configuration (FIG. 13) of the computer 90. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0082] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0083] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903, expanding them in the main storage device 902, and executing processing in accordance with the programs. The control unit can realize functional units that perform various types of information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0084] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. Furthermore, the processor 901 can allocate a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with the programs. Furthermore, the control unit can cause the processor 901 to execute processes for adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0085] A database refers to a relational database, which manages data sets called masters and tables in a tabular format structurally defined by rows and columns, by relating them to each other. In a database, a table is called a table, a master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated. Typically, each table and each master has a column set as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit can cause the processor 901 to add, delete, or update records in specific tables and masters stored in the storage unit according to various programs. Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0086] Note that the databases and masters in this disclosure may include any data structure in which information is structurally defined (such as a list, dictionary, associative array, or object). The data structure also includes data that can be considered as a data structure by combining data with functions, classes, methods, etc. written in any programming language.

[0087] The communication unit is realized by the communication IF 991. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0088] <Additional Notes> The matters described in the above embodiments will be supplemented below.

[0089] (Appendix 1) A program to be executed by a computer having a processor and a memory unit, the program comprising: a query acquisition step (S101) in which the processor acquires a query related to a predetermined question; a condition acquisition step (S102) in which one or more generation conditions are acquired that are conditions for generating a prompt that will be an input sentence to a large-scale language model (LLM); a prompt generation step (S103) in which one or more prompts are generated by applying the one or more generation conditions acquired in the condition acquisition step to the query received in the query acquisition step; a prompt sending step (S104) in which one or more prompts generated in the prompt generation step are sent to a large-scale language model (LLM) system; an answer receiving step (S105) in which one or more answer results for each of the one or more prompts sent in the prompt sending step; and a prompt evaluation step (S108) in which evaluation values ​​for the one or more answer results received in the answer receiving step are received. This allows the quality of the prompts obtained by applying one or more generation conditions to a given query to be evaluated.

[0090] (Appendix 2) The program according to Appendix 1, wherein the processor executes a condition evaluation step (S305) of calculating a condition evaluation value of one or more generation conditions acquired in the condition acquisition step based on the evaluation value received in the prompt evaluation step. This allows the quality of one or more production conditions to be evaluated.

[0091] (Appendix 3) A program described in Appendix 2, wherein the condition acquisition step (S102) includes a step of acquiring one or more prompt syntaxes related to the syntax of a prompt that will be an input sentence to the large-scale language model (LLM), the prompt generation step (S103) includes a step of generating one or more prompts by applying the one or more prompt syntaxes acquired in the condition acquisition step to the query received in the query acquisition step, and the condition evaluation step (S305) includes a step of calculating a syntax evaluation value of the one or more prompt syntaxes acquired in the syntax acquisition step based on the evaluation value received in the prompt evaluation step. This allows you to evaluate the merits and demerits of one or more prompt constructs.

[0092] (Appendix 4) The program according to any one of appendices 1 to 3, wherein the prompt evaluation step (S108) is a step of accepting an evaluation value for one or more answer results from the user. This allows the prompt to be evaluated as to whether it is good or bad in accordance with the evaluation value received from the user.

[0093] (Appendix 5) 5. The program according to claim 4, wherein the processor executes a prompt evaluation storage step (S109) of associating and storing user attributes relating to the attributes of the user with the evaluation value received from the user in the prompt evaluation step. This allows the quality of the prompt to be evaluated in relation to user attributes such as the user's gender, age, occupation, income, place of residence, and educational background.

[0094] (Appendix 6) The program according to any one of appendices 1 to 3, wherein the prompt evaluation step (S108) is a step of accepting an evaluation value obtained by having a large-scale language model (LLM) system evaluate one or more answer results. This allows the prompt to be evaluated for quality without requiring the user to perform any evaluation work.

[0095] (Appendix 7) The program according to claim 2 or 3, wherein the processor executes a condition proposing step (S306) of proposing one or more generation condition candidates to the user based on the condition evaluation value calculated in the condition evaluation step. This allows the user to select suitable creation conditions from the proposed creation condition candidates when developing a web service or the like.

[0096] (Appendix 8) The program according to appendix 7, wherein the condition proposing step (S306) is a step of proposing one or more generation condition candidates to the user in order of condition evaluation value. This allows the user to select suitable creation conditions from the proposed creation condition candidates when developing a web service or the like.

[0097] (Appendix 9) A program as described in Appendix 7 or 8, wherein a processor executes a type input step (S301) of receiving input of the type of web service from a user, and a condition proposing step (S306) of proposing one or more candidate generation conditions to the user depending on the type of web service received in the type input step. This allows the user to select suitable generation conditions when developing a web service or the like, depending on the type of the web service or the attributes of the user.

[0098] (Appendix 10) The program described in Appendix 9, wherein the condition proposal step (S306) is a step of calculating a condition evaluation value based on the evaluation value received from the user included in the user attributes of the web service user identified based on the type of web service received in the type input step, and proposing one or more generation condition candidates to the proposed user. This allows users to select suitable creation conditions for developing web services from candidate creation conditions that are proposed according to the type of web service and the attributes of the user, allowing users to select more suitable creation conditions.

[0099] (Appendix 11) A program described in any one of Appendices 1 to 10, wherein the prompt sending step (S104) is a step of sending the prompt generated in the prompt generation step to an external large-scale language model (LLM) system operated by a different operator than the computer. This allows the quality of the prompts obtained by applying one or more generation conditions to a given query to be evaluated by each external LLM system.

[0100] (Appendix 12) A computer-implemented method comprising a processor and a memory, wherein the processor performs all of the steps performed in the invention according to any one of appendixes 1 to 11. This allows the quality of the prompts obtained by applying one or more generation conditions to a given query to be evaluated.

[0101] (Appendix 13) An information processing device comprising a control unit and a memory unit, wherein the control unit executes all of the steps executed in the invention according to any one of Supplementary Note 1 to Supplementary Note 11. This allows the quality of the prompts obtained by applying one or more generation conditions to a given query to be evaluated.

[0102] (Appendix 14) A system comprising means for performing all steps performed in any of the inventions according to any one of appendixes 1 to 11. This allows the quality of the prompts obtained by applying one or more generation conditions to a given query to be evaluated. [Explanation of symbols]

[0103] 1 System, 10 Server, 101 Memory Unit, 104 Control Unit, 106 Input Device, 108 Output Device, 20 User Terminal, 201 Memory Unit, 204 Control Unit, 206 Input Device, 208 Output Device, 50 LLM System, 501 Memory Unit, 504 Control Unit, 506 Input Device, 508 Output Device

Claims

1. A program to be executed by a computer having a processor and a storage unit, the processor: a query acquisition step of acquiring a query related to a predetermined question; a condition acquisition step of acquiring one or more generation conditions that are conditions for generating a prompt that is an input sentence to a large-scale language model (LLM); a prompt generating step of generating one or more prompts by applying the one or more generation conditions acquired in the condition acquiring step to the query received in the query acquiring step; a prompt sending step of sending the one or more prompts generated in the prompt generating step to a large scale language model (LLM) system; an answer receiving step of receiving one or more answer results to each of the one or more prompts sent in the prompt sending step; a prompt evaluation step of receiving an evaluation value for the one or more answer results received in the answer receiving step; A program that executes.

2. the processor: a condition evaluation step of calculating a condition evaluation value of the one or more generation conditions acquired in the condition acquisition step based on the evaluation value received in the prompt evaluation step; To execute The program according to claim 1.

3. The step of obtaining the condition includes obtaining one or more prompt syntaxes related to the syntax of a prompt that is an input sentence to a large-scale language model (LLM); the prompt generating step includes a step of generating the one or more prompts by applying the one or more prompt syntaxes acquired in the condition acquiring step to the query accepted in the query acquiring step, the condition evaluation step includes a step of calculating a syntax evaluation value of the one or more prompt syntaxes acquired in the condition acquisition step based on the evaluation value received in the prompt evaluation step; The program according to claim 2.

4. The prompt evaluation step is a step of receiving an evaluation value for the one or more answer results from a user. The program according to claim 1.

5. the processor: a prompt evaluation storage step of storing user attributes related to the attributes of the user and the evaluation value received from the user in the prompt evaluation step in association with each other; To execute The program according to claim 4.

6. The prompt evaluation step is a step of receiving an evaluation value obtained by having the large-scale language model (LLM) system evaluate the one or more answer results. The program according to claim 1.

7. the processor: a condition proposing step of proposing one or more generation condition candidates to a user based on the condition evaluation value calculated in the condition evaluating step; To execute The program according to claim 2.

8. the condition proposing step is a step of proposing the one or more generation condition candidates to the user in order of the condition evaluation value; The program according to claim 7.

9. the processor: a type input step of receiving an input of a type of the web service from the user; Run the condition proposing step is a step of proposing the one or more generation condition candidates to the user according to the type of the web service accepted in the type input step; The program according to claim 7.

10. The condition proposing step is a step of calculating the condition evaluation value based on the evaluation value received from the user included in the user attributes of the user of the web service identified based on the type of the web service received in the type input step, and proposing the one or more generation condition candidates to the user. The program according to claim 9.

11. the prompt sending step is a step of sending the prompt generated in the prompt generating step to an external large-scale language model (LLM) system operated by a different operator from the computer. The program according to claim 1.

12. A method implemented on a computer having a processor and a memory, wherein the processor performs all of the steps performed in the invention according to any one of claims 1 to 11.

13. 12. An information processing device comprising a control unit and a storage unit, wherein the control unit executes all of the steps executed in the invention according to any one of claims 1 to 11.

14. A system comprising means for executing all steps performed in any one of the inventions according to claims 1 to 11.

Images