Information processing unit, information processing method and program

Abstract

To provide a technique for optimizing input information to a generative model.SOLUTION: An information processing unit is configured to: acquire a generation result obtained by generating input information to a generative model and inputting the input information to the generation model based upon instruction information, a search result for reference information based upon the instruction information, and a template, acquire a predetermined evaluation result by evaluating the generation result; and optimize at least one of a search method for the reference information or the template based upon the predetermined evaluation result.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Generative models such as large-scale language models (LLMs) are known. A generative model is a machine learning model that executes a given task according to input information called prompts and outputs the resulting generated data. Techniques for optimizing prompts have been proposed to obtain good generation results from generative models.

[0003] To obtain good results from generative models, a technique is known in which prompts contain reference information obtained by searching external data sources. This type of technique is called Retrieval Augmented Generation (RAG). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, Douwe Kiela, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", [online], [Searched on December 20, 2020], Internet<URL: https: / / arxiv.org / pdf / 2005.11401v4.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] The objective of the present disclosure is to provide a technology for optimizing input information to a generative model. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure includes at least one memory and at least one processor, and the at least one processor generates input information for a generative model based on instruction information, search results for reference information based on the instruction information, and a template, obtains a generation result obtained by inputting the input information into the generative model, evaluates the generation result to obtain a predetermined evaluation result, and optimizes at least one of a search method for reference information or a template based on the predetermined evaluation result. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram illustrating an example of the overall configuration of an information processing system. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of an optimization device. [Figure 3] FIG. 10 is a diagram illustrating an example of a feedback information input field. [Figure 4] FIG. 10 is a diagram illustrating an example of an evaluation value input field. [Figure 5] 10 is a flowchart illustrating an example of an optimization process according to an embodiment. [Figure 6] 10 is a flowchart illustrating an example of an evaluation data generation process. [Figure 7] 10 is a flowchart illustrating an example of a task execution process. [Figure 8] 10 is a flowchart illustrating an example of an evaluation value calculation process. [Figure 9] 10 is a flowchart illustrating an example of optimization processing in Modification 1. [Figure 10] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0009] [First embodiment] A first embodiment of the present disclosure is an information processing system that generates various data based on a generative model. The information processing system in this embodiment has a function of optimizing input information to be input to the generative model. In this embodiment, the generative model may be, for example, a large language model (LLM). The generative model is not limited to a large language model, and may be a model that generates other data such as audio or images, or may be a base model. The generative model may be, for example, a neural network.

[0010] The information processing system in this embodiment executes a predetermined generation task according to a search expansion generation framework. In this embodiment, the generation task may be, for example, a question answering task. A question answering task is a task in which data indicating a question is input and data indicating an answer to the question is generated. However, the generation task is not limited to a question answering task, and may be any task that generates data based on a generative model. For example, the generation task may be document summarization, translation, image generation, musical sound generation, etc.

[0011] Search expansion generation is a technique used to obtain good generation results from a generative model. In search expansion generation, reference information is searched for from a specified data source based on instruction information that instructs the execution of a task, and a prompt containing the instruction information and the search results of the reference information is input to a generative model. The generative model performs the task while taking into account reference information that is not included in the training data, so it can generate data appropriate for the instruction information.

[0012] Search expansion generation makes it easy to add new knowledge to the generative model. However, factors other than the generative model have a large impact on the generated results. One example of factors other than the generative model is the accuracy of data search. Furthermore, search expansion generation tends to increase the amount of data in prompts, which can easily increase the amount of calculation required for inference. In search expansion generation, optimizing the search method for reference information improves the accuracy of the reference information included in the prompt, thereby enabling prompt optimization.

[0013] A template engine is used to generate prompts to be input into a generative model. A template engine is a program that generates text data based on input data using predefined templates. Template engines are sometimes used to generate, for example, web pages or source code. Using a template engine, a user can efficiently generate complex prompts.

[0014] It is known that the results generated by a generative model vary depending on the content of the prompt. It is also known that the prompts that produce good results from a generative model vary depending on the type of generative model or the task that the generative model is made to execute. Techniques for optimizing prompts to obtain good results from a generative model are known. This type of technique is called prompt engineering, for example. When a prompt is generated using a template, the prompt can be optimized by optimizing the template.

[0015] This embodiment provides a technology for optimizing input information to a generative model within the framework of search expansion generation. In this embodiment, the optimization of input information to the generative model is achieved by optimizing at least one of a reference information search method and a template based on a predetermined index. In one aspect, according to this embodiment, by inputting optimized input information to the generative model, good generation results can be obtained from the generative model.

[0016] <Overall configuration of information processing system> The overall configuration of an information processing system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the overall configuration of an information processing system.

[0017] 1, the information processing system 1000 includes an optimization device 10, a search device 20, a generation device 30, and a terminal device 40. The optimization device 10, the search device 20, the generation device 30, and the terminal device 40 may be connected to each other via a communication network such as a LAN (Local Area Network) or the Internet so as to be able to communicate data with each other.

[0018] The optimization device 10 is an example of an information processing device such as a personal computer, a workstation, or a server that instructs the generation device 30 to execute a predetermined task in response to a request from the terminal device 40. The optimization device 10 may obtain search results for reference information from the retrieval device 20 based on instruction information input at the terminal device 40. The optimization device 10 may generate retrieval information based on the instruction information and transmit it to the retrieval device 20.

[0019] The instruction information is information that instructs the execution of a predetermined task. For example, the instruction information may include an execution instruction indicating the task to be executed, input data for the task, etc. For example, if the instruction information instructs the execution of a question-answering task, the instruction information may include text data indicating a question, etc.

[0020] The search information is information indicating search conditions for reference information. For example, the search information may include a query indicating the search conditions, a search module to be used, internal parameters of the search module, the number of reference information items to be included in the search results, etc.

[0021] A search module is software that performs a search. Examples of the search module include a search engine, a search algorithm, a search service, etc. Examples of the search module include VectorDB, BM25, Elasticsearch, etc.

[0022] The optimization device 10 may generate input information for the generative model 35 based on the search results of the reference information acquired from the search device 20. The optimization device 10 may generate the input information for the generative model 35 using a template. The optimization device 10 may acquire, from the generation device 30, a generation result obtained by inputting the generated input information into the generative model 35. The optimization device 10 may generate response information based on the generation result and transmit the response information to the terminal device 40.

[0023] The input information may include text data, image data, or audio data. The text data may be, for example, a natural language sentence called a prompt. The image data may be, for example, a still image or a video. The image data may include, for example, an image of the user. The audio data may include, for example, audio uttered by the user. The input information may include a result of recognizing the image data or audio data.

[0024] The response information is information indicating a response to the instruction information. For example, the response information may include information indicating whether the task was successfully executed, a generation result by a generative model, etc. For example, if the response information is in response to instruction information instructing the execution of a question-answering task, the response information may include text data indicating an answer to the question.

[0025] In this embodiment, the optimization device 10 optimizes setting information for executing a predetermined task. Hereinafter, the setting information for executing a predetermined task will be referred to as "hyperparameters." In this embodiment, the hyperparameters include at least either a search method for reference information or a template used to generate input information. The optimization device 10 may optimize at least one of the search method for reference information or the template among the hyperparameters.

[0026] The search device 20 is an example of an information processing device such as a personal computer, workstation, or server that searches for reference information in response to a request from the optimization device 10. The search device 20 may include a data source 25 that stores various pieces of reference information. As an example, the data source 25 may be a storage device or a database in which the reference information is stored. The data source 25 may be provided by an external device or system. Note that "external" means that it is not included in the information processing system 1000.

[0027] The search device 20 may be implemented as one or more devices or systems that perform searches using different search modules. The search device 20 may also be an external device or system. The search device 20 may retrieve reference information from external data sources 25.

[0028] The generating device 30 is an example of an information processing device such as a personal computer, a workstation, or a server that executes a predetermined task using a generative model. The generating device 30 may include a generative model 35. The generative model 35 is a generative model for executing the predetermined task. The generative model 35 may be, for example, a large-scale language model.

[0029] The generative model 35 may be realized by a single generative model, or by multiple generative models working together. The generative model 35 may be configured from multiple generative models according to the tasks to be performed.

[0030] The terminal device 40 is an example of an information processing terminal, such as a personal computer, a smartphone, or a tablet terminal, operated by a user of the information processing system 1000. The terminal device 40 may request the optimization device 10 to execute a predetermined task. The terminal device 40 may transmit instruction information input by the user to the optimization device 10. The terminal device 40 may receive response information from the optimization device 10 and present it to the user.

[0031] For example, the terminal device 40 may display the response information on a display device of the terminal device 40. For example, the terminal device 40 may output a voice synthesized from the response information from a speaker of the terminal device 40.

[0032] The overall configuration of the information processing system 1000 shown in FIG. 1 is an example, and various system configuration examples are possible depending on the application and purpose. The information processing system 1000 may be configured with one or more devices. Each device included in the information processing system 1000 may be a system configured with multiple devices. Each function included in the information processing system 1000 may be realized by any device that constitutes the system. Each component included in the information processing system 1000 may be included in any device that constitutes the system.

[0033] The data source 25 may be built in the optimization device 10 or the generation device 30. The data source 25 may be built in the terminal device 40. The data source 25 may be held in an external information processing device. The data source 25 may be held in a distributed manner in an external information processing system consisting of multiple devices. In this case, the information processing system 1000 may not include the search device 20.

[0034] The generative model 35 may be built into the optimization device 10 or the search device 20. The generative model 35 may be distributed and held in an external information processing system consisting of multiple devices. In this case, the information processing system 1000 may not include the generating device 30.

[0035] The information processing system 1000 may include a plurality of one or more of the optimization device 10, the search device 20, the generation device 30, and the terminal device 40. The optimization device 10, the search device 20, or the generation device 30 may be realized by a plurality of computers, or may be realized as a cloud computing service. Two or more of the optimization device 10, the search device 20, and the generation device 30 may be realized by a standalone computer. The classification of devices such as the optimization device 10, the search device 20, the generation device 30, and the terminal device 40 shown in FIG. 1 is an example.

[0036] <Functional configuration of the optimization device> The functional configuration of the optimization device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of the optimization device.

[0037] 2, the optimization device 10 includes an evaluation data storage unit 101, a data generation unit 102, an instruction receiving unit 103, a search unit 104, a generation unit 105, a response unit 106, a feedback unit 107, an evaluation unit 108, an evaluation history storage unit 109, and an optimization unit 110. The optimization device 10 functions as the evaluation data storage unit 101, the data generation unit 102, the instruction receiving unit 103, the search unit 104, the generation unit 105, the response unit 106, the feedback unit 107, the evaluation unit 108, the evaluation history storage unit 109, and the optimization unit 110 by executing an optimization program installed in advance.

[0038] The evaluation data storage unit 101 stores evaluation data. The evaluation data is used to calculate an evaluation value that serves as an index for optimization. The evaluation value may include a task evaluation value, a search evaluation value, and a feedback evaluation value. Note that the task evaluation value is an example of a first evaluation value, the search evaluation value is an example of a second evaluation value, and the feedback evaluation value is an example of a third evaluation value.

[0039] The task evaluation value is an evaluation value that evaluates the validity of the generated result relative to the correct answer information. The correct answer information may be predetermined for the instruction information. The evaluation data may include a dataset (hereinafter also referred to as a "correct answer dataset") for calculating the task evaluation value. The correct answer dataset may be a dataset that associates the instruction information with the correct answer information. The correct answer dataset is an example of a first dataset.

[0040] For example, when performing a question-answering task, the correct answer dataset may be question-answering data. The question-answering data may be data in which a question sentence, answer options, and correct answer options are associated with each other. The question-answering data may be data in which a question sentence and a model answer are associated with each other.

[0041] As another example, the correct answer dataset may be natural language inference data or calculation data. The natural language inference data may be data indicating a relationship between two or more sentences (e.g., an implication relationship). The calculation data may be data indicating a calculation formula and a correct answer value (e.g., 1 + 2 = 3).

[0042] The search evaluation value is an evaluation value obtained by evaluating the relevance between the instruction information and the search results. The evaluation data may include a dataset for calculating the search evaluation value (hereinafter also referred to as a "search dataset"). The search dataset may be a dataset that associates search information (for example, a query indicating search conditions) with reference information. The search dataset may further include information indicating the degree of relevance between the search information and the reference information. The search dataset is an example of a second dataset. The evaluation value obtained by evaluating the relevance between the instruction information and the search results may include an evaluation value obtained by evaluating the relevance between the search information generated based on the instruction information and the search results.

[0043] The feedback evaluation value is an evaluation value obtained by evaluating the validity of the generated result in response to the instruction information. The evaluation data may include feedback information for calculating the feedback evaluation value. The feedback information may be collected by the feedback unit 107 from the user to whom the response information is presented. The feedback information is an example of a third data set.

[0044] The evaluation data may be stored in a database in which items and values are associated with each other. As a method for associating items and values, for example, the items and values may be stored as a set, information that allows one piece of information to be obtained from the other piece of information may be stored, or one piece of information and identification information for the other piece of information may be stored as a set.

[0045] The data generation unit 102 generates evaluation data. Specifically, first, the data generation unit 102 acquires document data for generating the evaluation data. The data generation unit 102 may acquire document data input by a user. The data generation unit 102 may acquire document data from a data source 25. The data generation unit 102 may generate document data based on a generative model 35. The data generation unit 102 may generate document data based on another generative model different from the generative model 35. The data generation unit 102 may acquire multiple pieces of document data.

[0046] Next, the data generation unit 102 generates a supervised answer data set and a search data set based on the document data. The data generation unit 102 stores evaluation data including the generated supervised answer data set and search data set in the evaluation data storage unit 101.

[0047] <Generating the correct answer dataset> For example, the data generating unit 102 may generate a supervised answer dataset as follows: First, the data generating unit 102 generates input information for instructing the generation of instruction information and supervised answer information based on document data. When a question-answering task is performed, for example, the input information may be a prompt such as "Generate a question sentence from the following document and create a possible answer."

[0048] The data generation unit 102 may generate input information based on a portion of the document data. The portion of the document data may be, for example, data obtained by dividing the document data into a plurality of pieces based on the document structure. The document structure may be, for example, a chapter, a section, a paragraph, a sentence, or the like. The portion of the document data may be, for example, data obtained by dividing the document data into a plurality of pieces with a predetermined data amount.

[0049] Next, the data generation unit 102 transmits the generated input information to the generation device 30. The generation device 30 inputs the input information received from the optimization device 10 to the generation model 35. The generation model 35 generates one or more combinations of instruction information and correct answer information based on the input information, and outputs a generation result including the generated instruction information and correct answer information. The generation device 30 transmits the generation result output from the generation model 35 to the optimization device 10.

[0050] In the optimization device 10, the data generation unit 102 receives the generation results of the generative model 35 from the generation device 30. The data generation unit 102 generates a supervised answer data set that associates the instruction information and the supervised answer information included in the received generation results. The data generation unit 102 stores the generated supervised answer data set in the evaluation data storage unit 101.

[0051] The data generation unit 102 may generate a supervised answer dataset based on another generative model different from the generative model 35. The other generative model is an example of a second generative model. The supervised answer dataset generated by the data generation unit 102 may be verified and selected by a user. The supervised answer dataset may be a combination of instruction information and supervised answer information prepared in advance. In this case, the data generation unit 102 does not need to generate a supervised answer dataset.

[0052] <<Generating a search dataset>> For example, the data generation unit 102 may generate a search dataset as follows. First, the data generation unit 102 generates input information for generating search information based on document data. The data generation unit 102 may generate the input information based on a portion of the document data. For example, the input information may be a prompt such as "Please output a query linked to the following document."

[0053] Next, the data generation unit 102 transmits the generated input information to the generation device 30. The generation device 30 inputs the input information received from the optimization device 10 to the generative model 35. The generative model 35 generates one or more pieces of search information based on the input information and outputs a generation result including the generated search information. The generation device 30 transmits the generation result output from the generative model 35 to the optimization device 10.

[0054] In the optimization device 10, the data generation unit 102 receives the generation results generated by the generative model 35 from the generation device 30. The data generation unit 102 generates a search dataset that associates search information included in the received generation results with document data as reference information. The data generation unit 102 stores the generated search dataset in the evaluation data storage unit 101.

[0055] The data generation unit 102 may cause the generative model 35 to generate a relevance between the search information and the reference information along with the search information, and generate a search data set that further includes the relevance. The relevance may be an index indicating the degree of semantic match. The input information to the generative model 35 when generating the relevance may include a definition and an evaluation method for the relevance. As an example, the input information to the generative model 35 may include instruction information instructing the model 35 to generate a relevance between a value of 1 when the search information and the reference information are related and a value of 0 when the search information and the reference information are not related.

[0056] The data generation unit 102 may generate a search dataset based on another generative model different from the generative model 35. The other generative model is an example of a third generative model. The search dataset generated by the data generation unit 102 may be verified and selected by a user. The search dataset may be a combination of search information and reference information prepared in advance. In this case, the data generation unit 102 does not need to generate a search dataset.

[0057] The instruction receiving unit 103 receives input of instruction information. The instruction receiving unit 103 may receive the input of instruction information by receiving instruction information input via the terminal device 40. The instruction receiving unit 103 may receive the input of instruction information via an input device connected to the optimization device 10.

[0058] The search unit 104 searches for reference information based on the instruction information received by the instruction receiving unit 103. For example, the search unit 104 may generate search information based on the instruction information and transmit a search request including the search information to the search device 20. For example, the search unit 104 may cause the generative model 35 to generate the search information based on the instruction information. For example, the method for generating the search information may be a method based on ReAct (Reason + Act). The search unit 104 may receive search results transmitted by the search device 20 in response to the search request. The search results may include reference information that satisfies the search conditions indicated in the search information.

[0059] The search unit 104 may perform query expansion as preprocessing before performing a search. For example, the search unit 104 may perform query expansion by generating a list of keywords related to the query and a priority of each keyword based on a generative model. The generative model used to generate the keywords may be the generative model 35, or may be a generative model different from the generative model 35. The search unit 104 may perform query expansion by extracting a list of related keywords from dictionary data prepared in advance. Note that the dictionary data may be stored in advance in a storage unit such as a memory of the optimization device 10.

[0060] The search unit 104 may perform searches using multiple search modules and integrate the search results. The integration of search results may be performed, for example, by using Reciprocal Rank Fusion (RRF). In RRF, search results from each search module may be weighted and integrated into a single search result.

[0061] The generation unit 105 acquires a generation result by the generative model 35 based on the instruction information accepted by the instruction accepting unit 103 and the search results acquired by the search unit 104. For example, the generation unit 105 may generate input information for the generative model 35 based on the instruction information and the search results, and transmit the generated information to the generation device 30. The generation unit 105 may generate the input information for the generative model 35 by embedding the instruction information and the search results in a template. The generation unit 105 may generate the input information by processing the template, the instruction information, and the search results of the reference information based on a predetermined rule. The generation unit 105 may generate the input information by inputting the template, the instruction information, and the search results of the reference information into another generative model. The generation unit 105 may receive, from the generation device 30, a generation result generated by the generation device 30 inputting the input information into the generative model 35.

[0062] The response unit 106 generates response information in response to the instruction information accepted by the instruction accepting unit 103, based on the generation result acquired by the generation unit 105. The response unit 106 may generate response information including the generation result acquired by the generation unit 105, and transmit the response information to the terminal device 40. The response unit 106 may include a part of the generation result acquired by the generation unit 105 in the response information. The response unit 106 may transmit the generation result acquired by the generation unit 105 itself as response information to the terminal device 40. The response unit 106 may generate response information by processing the generation result acquired by the generation unit 105, and transmit the response information to the terminal device 40.

[0063] The feedback unit 107 acquires feedback information from a user who has been presented with the response information. The feedback unit 107 may acquire feedback information input to the terminal device 40. The feedback information is a result of the user who has been presented with the response information evaluating the validity of the generation result in response to the instruction information.

[0064] The feedback information may be collected using, for example, a black-box optimization tool. The black-box optimization tool may be, for example, Optuna (registered trademark). The black-box optimization tool may be pre-installed in the terminal device 40. The black-box optimization tool may be installed in the optimization device 10 and configured to acquire feedback information input by a user to the terminal device 40.

[0065] The evaluation unit 108 calculates an evaluation value that serves as an index used for optimization. The evaluation unit 108 may evaluate the generation result acquired by the generation unit 105. The evaluation unit 108 may evaluate the response information generated by the response unit 106. The evaluation result by the evaluation unit 108 may include a result of evaluation based on a predetermined index and an evaluation value calculated by the evaluation. The evaluation unit 108 may calculate at least one of a task evaluation value, a search evaluation value, and a feedback evaluation value.

[0066] <<Task Evaluation Value Calculation>> For example, the evaluation unit 108 may calculate the task evaluation value as follows. First, the evaluation unit 108 acquires correct answer information associated with the instruction information accepted by the instruction accepting unit 103 from the correct answer dataset stored in the evaluation data storage unit 101. Next, the evaluation unit 108 calculates the task evaluation value based on a comparison result between the generation result acquired by the generation unit 105 and the correct answer information. For example, when a question-answering task is executed, the evaluation unit 108 may calculate an accuracy rate indicating the rate at which the generation result matches the correct answer information.

[0067] As an example, the determination of whether the generated result matches the correct answer information may be performed on a rule-based basis. The rule-based determination may be a complete match or a partial match. Note that a complete match is a rule that treats the generated result as correct when it completely matches the correct answer information. A partial match is a rule that treats the generated result as correct when it contains part of the correct answer information.

[0068] As another example, the determination of whether the generated result matches the correct answer information may be performed based on a generative model. The generative model used for the determination may be the generative model 35, or may be a generative model different from the generative model 35. In the determination based on the generative model, the task may be to generate a truth value (e.g., True or False) for the combination of the generated result and the correct answer information, or may be to generate a discrete evaluation value (e.g., a score on a five-point scale).

[0069] <<Calculation of search rating>> For example, the evaluation unit 108 may calculate the search evaluation value as follows. First, the evaluation unit 108 acquires reference information associated with the search information generated by the search unit 104 from the search dataset stored in the evaluation data storage unit 101. Next, the evaluation unit 108 calculates the search evaluation value based on a comparison result between the search results acquired by the search unit 104 and the reference information acquired from the search dataset. For example, the evaluation unit 108 may determine whether the reference information acquired from the search dataset is included in the search results.

[0070] The evaluation unit 108 may calculate a search evaluation value based on the relevance included in the search data set. For example, the evaluation unit 108 may calculate a search evaluation value that is higher as reference information with a higher relevance is included in the higher search results.

[0071] The evaluation unit 108 may generate a relevance between the search information and the search results based on a generative model. The generative model that generates the relevance may be the generative model 35, or may be a generative model different from the generative model 35. The generation of the relevance by the generative model may be a task of generating a truth value (e.g., True or False) for a combination of the search information and the search results, or may be a task of generating a discrete evaluation value (e.g., a score on a five-point scale).

[0072] <Feedback evaluation value calculation> For example, the evaluation unit 108 may calculate the feedback evaluation value as follows: First, the evaluation unit 108 reads out the feedback information stored in the evaluation data storage unit 101. Next, the evaluation unit 108 calculates the feedback evaluation value based on the read feedback information.

[0073] For example, the feedback evaluation value may be calculated based on a trained evaluation model. The evaluation model may be a machine learning model (for example, a neural network) that uses the generated results as explanatory variables and the evaluation value as a target variable. The evaluation value may be a truth value (for example, True or False) or a discrete evaluation value (for example, a 5-point score).

[0074] The evaluation model may be trained using feedback information acquired by the feedback unit 107 as training data. The evaluation model may also be a generative model that has been additionally trained using the feedback information as training data. As an example, the additional training method may be fine tuning. The generative model that generates the evaluation value may be the generative model 35, or may be another generative model different from the generative model 35.

[0075] The evaluation history storage unit 109 stores history information of the evaluation values calculated by the evaluation unit 108. The history information of the evaluation values may be information that accumulates the evaluation values calculated by the evaluation unit 108. The number of pieces of history information stored in the evaluation history storage unit 109 is not limited. The evaluation history storage unit 109 may store a predetermined number of evaluation values that were generated most recently, or may store multiple evaluation values that were calculated within a predetermined most recent period.

[0076] The history information may include search information generated by the search unit 104, search results obtained by the search unit 104, input information generated by the generation unit 105, generation results obtained by the generation unit 105, and evaluation values calculated by the evaluation unit 108.

[0077] The optimization unit 110 optimizes the hyperparameters based on the evaluation values calculated by the evaluation unit 108. For example, the optimization unit 110 may optimize the hyperparameters based on any one of a task evaluation value, a search evaluation value, or a feedback evaluation value. The optimization unit 110 may optimize the hyperparameters based on history information stored in the evaluation history storage unit 109. The optimization unit 110 may optimize the hyperparameters by using black-box optimization to select one or more optimal options from multiple options for the hyperparameters. The optimization unit 110 may optimize at least one of a search method for reference information or a template among the hyperparameters.

[0078] The search method to be optimized may include, for example, at least one of the type of search module, the internal parameters of the search module, and the number of search results. The number of search results may be the number of reference information items to be included in the prompt or the number of search results output by the search module. When multiple search results are integrated, the type of search module may include multiple search modules. In this case, the search method to be optimized may include, for example, a method for integrating multiple search results, the importance of each search module, and the internal parameters of each search module. For example, the method for integrating search results may be reverse rank fusion. For example, the importance of a search module may be the weight assigned to each search result when integrating search results using reverse rank fusion.

[0079] The search method to be optimized may include a method for sorting search results. Examples of the method for sorting search results include the output order of the search module, the reverse order, and an order based on relevance. For example, the order based on relevance may be ascending or descending order. Another example of the order based on relevance may be an order in which highly relevant results are placed at the beginning or end and less relevant results are placed in the middle.

[0080] The internal parameters of the search module may include a chunk size. The chunk size is the amount of data used to divide large amounts of input data into the search module. As an example, if the search module is VectorDB, the internal parameters may include the chunk size. VectorDB is a method of generating embedding vectors from document data to be searched and extracting search results based on the similarity between the embedding vectors and the embedding vectors generated from the query. The chunk size is the amount of data used to generate embedding vectors from the document data and the query.

[0081] When the search module is VectorDB, the internal parameters of the search module may include parameters related to embedding. For example, the parameters related to embedding may include the type of language model used for embedding and the number of dimensions when reducing the dimensions of the embedding vector.

[0082] For example, the search method to be optimized may further include at least one of information about query expansion and information about reranking. For example, the information about query expansion may include whether to perform query expansion, the number of queries to expand, a method for integrating search results based on each query, internal parameters of each integration method, a prompt to be used for query expansion, etc. For example, the information about reranking may include whether to perform reranking, the reranking method, internal parameters of the reranking method, etc.

[0083] The templates to be optimized may include, for example, templates for generating input information for the generative model 35. When performing query expansion, the templates to be optimized may include, for example, templates for generating input information to be used in query expansion.

[0084] For example, the optimization unit 110 may optimize the hyperparameters according to a predetermined optimization algorithm. For example, the optimization algorithm may use a black-box optimization method or a gray-box optimization method. For example, the optimization unit 110 may use a black-box optimization tool. For example, the black-box optimization tool may be Optuna (registered trademark).

[0085] Examples of black-box optimization include multi-objective optimization, genetic algorithms, Bayesian optimization, evolutionary computing, and reinforcement learning. Examples of gray-box optimization include multi-fidelity optimization. Specifically, black-box optimization may use methods such as TPE (Tree-Structured Parzen Estimator), GP-BO (Gaussian Process - Bayesian Optimization), CMA-ES (Covariance Matrix Adaptation Evolution Strategy), and NSGA-II (Non-Dominated Sorting Genetic Algorithm-II).

[0086] In a genetic algorithm, new hyperparameters may be selected using tournament selection, for example. When tournament selection is used, a predetermined number of hyperparameters are randomly selected from multiple hyperparameters, and the hyperparameter with the highest evaluation value is selected. Alternatively, roulette selection may be used.

[0087] For example, the optimization unit 110 may optimize the hyperparameters by multi-objective optimization based on at least two of the task evaluation value, the search evaluation value, and the feedback evaluation value. As another example, the optimization unit 110 may optimize the hyperparameters based on an average value of at least two of the task evaluation value, the search evaluation value, and the feedback evaluation value.

[0088] The multi-objective optimization may be a method that uses a ranking-style evaluation value based on domination. An example of a multi-objective optimization method is NSGA-II. The multi-objective optimization may be constrained optimization. In constrained optimization, if one or more evaluation values are below a predetermined lower limit, the method may not select a new hyperparameter.

[0089] In template optimization, a new template may be generated based on a generative model. The generative model used to generate the template may be the generative model 35 or a generative model different from the generative model 35.

[0090] Specifically, the optimization unit 110 determines one or more templates to be rewritten (hereinafter also referred to as "parent templates") based on the evaluation value calculated by the evaluation unit 108 or the evaluation value stored in the evaluation history storage unit 109. A parent template is a template for which the evaluation value has been calculated by the evaluation unit 108. Next, the optimization unit 110 generates input information for the generative model. Then, the optimization unit 110 acquires a template (hereinafter also referred to as "child template") generated by inputting the input information into the generative model.

[0091] The input information to the generative model may include template information about one or more parent templates and information specifying the structure of the template.

[0092] The input information to the generative model may include an evaluation value for the parent template, and / or a search method used to evaluate the parent template and / or search results obtained by the search method.

[0093] The template information may include at least one of the template itself and input information generated using the template. The template structure may include, for example, a placeholder indicating a position where a predetermined item is to be embedded. Other examples of the template structure may include the type of markup language used to describe the template, the type of symbols used in the template, the order of content included in the template, or the repetition of items included in the template.

[0094] For example, the symbol may be a bullet point in a list, parentheses in a heading, etc. For example, the content may be information indicating instructions, context, conditions, etc., or information that subdivides these. The information specifying the repetition of an item may be whether or not a specific item is to be repeated, or the number of times it is to be repeated.

[0095] <User Interface> The user interface of the information processing system 1000 will be described with reference to Figs. 3 and 4. The user interface of the information processing system 1000 may include a feedback screen. The feedback screen is a screen for receiving feedback information from a user. The feedback screen may be displayed on a display device of the terminal device 40 or on a display device of the optimization device 10.

[0096] The feedback screen may include an area for displaying the generation results of the generative model 35. The feedback screen may include a feedback information input field for inputting feedback information from the user. The feedback information may include, for example, an evaluation value indicating the evaluation result by the user. The feedback information may include, for example, free-form text.

[0097] 3 is a diagram showing an example of a feedback information input field 600. As shown in FIG. 3, the feedback information input field 600 may have an evaluation value input field 601, a text input field 602, a submit button 603, and a cancel button 604.

[0098] The evaluation value input field 601 may have evaluation options. Each option may be associated with a predetermined evaluation value. The evaluation options may be, for example, "Good," "So-so," or "Bad." The evaluation options may be, for example, numbers, letters, or symbols indicating the evaluation value. The evaluation options may be, for example, on a scale of 3 to 5.

[0099] For example, the evaluation value input field 601 may accept input of an evaluation value by operating a radio button. For example, the evaluation value input field 601 may accept input of an evaluation value by operating a slider. The evaluation value input field 601 may accept input of a continuous value or input of a discrete value.

[0100] FIG. 4 is a diagram showing an example of an evaluation value input field. As shown in FIG. 4, the evaluation value input field 601 may be any one of an input form 611 for selecting an evaluation value by operating a radio button, an input form 612 for adjusting an evaluation value by operating a slider, or an input form 613 for performing multi-objective optimization. The multi-objective optimization may be performed by combining a quantitative evaluation value (such as AUC (Area Under the Curve)) and a subjective evaluation value (human). Note that a user (e.g., a programmer or a general user) may select which of the input forms 611 to 613 shown in FIG. 4 to use.

[0101] The user may refer to the generation results displayed on the feedback screen and input feedback information for the generation results in the feedback information input field 600. The feedback information may be acquired by recognizing the user's voice. The feedback information may be acquired by having the user select one or more generation results that the user feels are good or one or more generation results that the user feels are not good from multiple generation results generated using multiple hyperparameters.

[0102] The user may input the evaluation value through a user interface for comparative evaluation that simultaneously displays multiple generation results and allows the user to select the generation result that is most highly rated by the user. In the user interface, the evaluation value of a selected generation result may be higher than the evaluation values of non-selected generation results. In the user interface, a value indicating how superior a certain generation result is to other generation results may be input using a slider or the like. In addition, in the user interface, the generation results of the generative model 35 may be processed by a program to perform comparative evaluation.

[0103] The user may input an evaluation value through a user interface for ranking evaluation that simultaneously displays multiple generation results and allows the user to rank them. Alternatively, the user may input an evaluation value by changing the evaluation value calculated by the terminal device 40 or the optimization device 10. Furthermore, the user may input an evaluation value by combining the evaluation value input by the user with the evaluation value calculated by the terminal device 40 or the optimization device 10.

[0104] After the user inputs the feedback information, when the user presses the post button 603, the terminal device 40 acquires the feedback information input in the feedback information input field 600 and transmits it to the optimization device 10. In the optimization device 10, the feedback unit 107 receives the feedback information from the terminal device 40. The feedback unit 107 stores the received feedback information in the evaluation data storage unit 101.

[0105] When the user presses the stop button 604 without inputting feedback information, the terminal device 40 transmits a signal indicating that the evaluation is to be stopped to the optimization device 10. In the optimization device 10, the feedback unit 107 receives the signal indicating that the evaluation is to be stopped from the terminal device 40. Upon receiving the signal indicating that the evaluation is to be stopped, the feedback unit 107 ends the processing without storing the feedback information.

[0106] <Optimization process flow> The optimization process executed by the information processing system 1000 will be described with reference to Fig. 5 to Fig. 8. Fig. 5 is a flowchart showing an example of the optimization process in the embodiment. The optimization process is a process for optimizing hyperparameters.

[0107] In step S1, the optimization device 10 generates evaluation data. Specifically, the data generation unit 102 of the optimization device 10 generates a superset data set and a search data set. The optimization device 10 stores the evaluation data including the superset data set and the search data set in the evaluation data storage unit 101.

[0108] In step S2, the optimization device 10 executes a predetermined task based on the evaluation data generated in step S1. Specifically, first, the instruction receiving unit 103 of the optimization device 10 receives input of instruction information included in the supervised data set. Next, the search unit 104 of the optimization device 10 searches for reference information based on the instruction information. Subsequently, the generation unit 105 of the optimization device 10 acquires a generation result by the generative model 35 based on the search results of the instruction information and the reference information. Then, the response unit 106 of the optimization device 10 outputs response information based on the generation result.

[0109] In step S3, the optimization device 10 calculates evaluation values based on the evaluation data generated in step S1. Specifically, the evaluation unit 108 of the optimization device 10 calculates a task evaluation value, a search evaluation value, and a feedback evaluation value. The optimization device 10 stores history information of the evaluation values in the evaluation history storage unit 109.

[0110] In step S4, the optimization device 10 updates the hyperparameters based on the evaluation value calculated by the evaluation unit 108. The optimization device 10 may update the hyperparameters based on history information of the evaluation values stored in the evaluation history storage unit 109. Specifically, the optimization unit 110 of the optimization device 10 updates at least one of the search method or template for reference information based on the evaluation value.

[0111] In step S5, the optimization device 10 determines whether or not a predetermined convergence condition is satisfied. If it is determined that the predetermined convergence condition is satisfied (YES), the optimization device 10 ends the optimization process. On the other hand, if it is determined that the predetermined convergence condition is not satisfied (NO), the optimization device 10 returns the process to step S2.

[0112] The predetermined convergence condition is a condition that must be satisfied to terminate the hyperparameter optimization. For example, the predetermined convergence condition may be that the hyperparameter has been updated a predetermined number of times, that the maximum evaluation value has not been updated a predetermined number of times, or that the difference in evaluation value before and after the update is within a predetermined threshold (in other words, that the predetermined number of times no longer increases significantly due to the update), etc.

[0113] After returning to step S2, the optimization device 10 executes steps S2 to S5 again based on the hyperparameters updated in step S4. In this way, the optimization device 10 repeatedly calculates the evaluation value and updates the hyperparameters until a predetermined convergence condition is satisfied.

[0114] The tasks executed in step S2 may be the same task or different tasks in the iterative process including the calculation of the evaluation value and the update of the hyperparameters. The optimization device 10 may execute the optimization based on the task evaluation value, the optimization based on the search evaluation value, and the optimization based on the feedback evaluation value in different iterative processes. The optimization device 10 may execute the optimization based on the task evaluation value, the optimization based on the search evaluation value, and the optimization based on the feedback evaluation value simultaneously in the same iterative process.

[0115] As described above, in the optimization process, the optimization device 10 starts with default hyperparameters and obtains a generation result based on instruction information obtained from the correct answer dataset. Next, the optimization device 10 obtains an evaluation value by evaluating the generation result based on the evaluation data. Subsequently, the optimization device 10 samples the next hyperparameter based on the evaluation value. The optimization device 10 repeatedly executes the above process until the maximum evaluation value is obtained.

[0116] <Evaluation data generation process> The evaluation data generation process (step S1 in FIG. 5) in this embodiment will be described in more detail with reference to FIG. 6. FIG. 6 is a flowchart showing an example of the evaluation data generation process.

[0117] In step S11, the data generation unit 102 of the optimization device 10 acquires document data. The data generation unit 102 may acquire multiple pieces of document data.

[0118] In step S12, the data generation unit 102 of the optimization device 10 generates a supervised answer dataset based on the document data acquired in step S1. Specifically, the data generation unit 102 generates input information for instructing the generation of instruction information and supervised answer information based on the document data. Next, the data generation unit 102 transmits the generated input information to the generation device 30. Subsequently, the data generation unit 102 receives a generation result by the generative model 35 from the generation device 30. Then, the data generation unit 102 generates a supervised answer dataset that associates the instruction information and supervised answer information included in the generation result.

[0119] In step S13, the data generation unit 102 of the optimization device 10 generates a search dataset based on the document data acquired in step S1. Specifically, the data generation unit 102 generates input information for generating search information based on the document data. Next, the data generation unit 102 transmits the generated input information to the generation device 30. Subsequently, the data generation unit 102 receives the generation result by the generative model 35 from the generation device 30. Then, the data generation unit 102 generates a search dataset that associates the search information included in the generation result with document data as reference information.

[0120] In step S14, the data generation unit 102 of the optimization device 10 stores the evaluation data in the evaluation data storage unit 101. The evaluation data includes the supervised answer dataset generated in step S12 and the search dataset generated in step S13.

[0121] <<Task execution process>> The task execution process (step S2 in FIG. 5) in this embodiment will be described in more detail with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the task execution process.

[0122] In step S21, the terminal device 40 reads out the supervised answer data set stored in the evaluation data storage unit 101 of the optimization device 10. Next, the terminal device 40 acquires instruction information included in the supervised answer data set. The terminal device 40 transmits the acquired instruction information to the optimization device 10.

[0123] In the optimization device 10, the instruction receiving unit 103 receives instruction information from the terminal device 40. Next, the instruction receiving unit 103 receives input of the received instruction information. The instruction receiving unit 103 sends the received instruction information to the search unit 104 and the generation unit 105.

[0124] In step S22, the search unit 104 of the optimization device 10 receives instruction information from the instruction receiving unit 103. Next, the search unit 104 generates search information based on the received instruction information. At this time, the search unit 104 generates the search information using a search method for reference information included in the hyperparameter. The search unit 104 transmits the generated search information to the search device 20.

[0125] The search device 20 receives search information from the optimization device 10. Next, the search device 20 searches for reference information from the data source 25 based on the received search information. The search device 20 transmits the search results for the reference information to the optimization device 10.

[0126] In the optimization device 10, the search unit 104 receives the search results from the search device 20. The search unit 104 sends the received search results to the generation unit 105 and the evaluation unit .

[0127] In step S23, the generation unit 105 of the optimization device 10 receives instruction information from the instruction receiving unit 103. The generation unit 105 also receives search results from the search unit 104. Next, the generation unit 105 acquires reference information from the search results. Subsequently, the generation unit 105 generates input information for the generative model 35 based on the instruction information and the reference information. The generation unit 105 transmits the generated input information to the generation device 30.

[0128] In step S24, the generation device 30 receives input information from the optimization device 10. Next, the generation device 30 inputs the received input information to the generative model 35. The generative model 35 executes a predetermined task based on the input information and outputs the resulting generation result. The generation device 30 transmits the generation result output from the generative model 35 to the optimization device 10.

[0129] In the optimization device 10, the generation unit 105 receives the generation result from the generation device 30. The generation unit 105 sends the received generation result to the response unit 106 and the evaluation unit 108.

[0130] In step S25, the response unit 106 of the optimization device 10 receives the generation result by the generative model 35 from the generation unit 105. Next, the response unit 106 generates response information based on the received generation result. The response unit 106 transmits the generated response information to the terminal device 40.

[0131] In step S26, the terminal device 40 receives response information from the optimization device 10. Next, the terminal device 40 presents the received response information to the user. Subsequently, the terminal device 40 acquires feedback information from the user. The terminal device 40 transmits the acquired feedback information to the optimization device 10.

[0132] In the optimization device 10, the feedback unit 107 receives feedback information from the terminal device 40. Next, the feedback unit 107 stores the received feedback information in the evaluation data storage unit 101.

[0133] <Evaluation value calculation process> The evaluation value calculation process (step S3 in FIG. 5) in this embodiment will be described in more detail with reference to FIG. 8. FIG. 8 is a flowchart showing an example of the evaluation value calculation process.

[0134] In step S31, the evaluation unit 108 of the optimization device 10 receives the generation result by the generative model 35 from the generation unit 105. Next, the evaluation unit 108 acquires supervised answer information associated with the instruction information accepted by the instruction acceptance unit 103 from the supervised answer dataset stored in the evaluation data storage unit 101. Subsequently, the evaluation unit 108 compares the generation result with the supervised answer information. The evaluation unit 108 calculates a task evaluation value based on the comparison result between the generation result and the supervised answer information. The evaluation unit 108 sends the calculated task evaluation value to the optimization unit 110.

[0135] In step S32, the evaluation unit 108 of the optimization device 10 receives the search results for the reference information from the search unit 104. Next, the evaluation unit 108 acquires reference information associated with the search information generated by the search unit 104 from the search dataset stored in the evaluation data storage unit 101. Subsequently, the evaluation unit 108 compares the search results with the reference information. The evaluation unit 108 calculates a search evaluation value based on the comparison result between the search results and the reference information. The evaluation unit 108 sends the calculated search evaluation value to the optimization unit 110.

[0136] In step S33, the evaluation unit 108 of the optimization device 10 extracts feedback information related to the generation result obtained by the generation unit 105 from the feedback information stored in the evaluation data storage unit 101. Next, the evaluation unit 108 calculates a feedback evaluation value based on the extracted feedback information. The evaluation unit 108 sends the calculated feedback evaluation value to the optimization unit 110.

[0137] In step S34, the evaluation unit 108 of the optimization device 10 stores history information of the evaluation values in the evaluation history storage unit 109. The history information includes the task evaluation value calculated in step S31, the search evaluation value calculated in step S32, and the feedback evaluation value calculated in step S33. The history information may include the search information generated by the search unit 104, the search results obtained by the search unit 104, the input information generated by the generation unit 105, the generation results obtained by the generation unit 105, and the evaluation values calculated by the evaluation unit 108.

[0138] The information processing system 1000 may execute a new task using hyperparameters for which at least one of the reference information search method and the template has been optimized by the optimization device 10. In this case, the information processing system 1000 does not need to re-optimize the hyperparameters.

[0139] The hyperparameters for which at least one of the reference information search method and template has been optimized by the optimization device 10 may be used by another device or system other than the information processing system 1000. In this case, the other device or system does not need to re-optimize the hyperparameters.

[0140] [Variation 1] In the above embodiment, a configuration has been described in which at least one of a reference information search method and a template is optimized simultaneously. The reference information search method and the template may be optimized separately. In this modified example, a configuration will be described in which the reference information search method and the template are optimized separately.

[0141] The optimization process in this modification will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the optimization process in the first modification.

[0142] In step S1, the optimization device 10 generates evaluation data. The method for generating evaluation data is the same as the evaluation data generation process in the embodiment (see FIG. 6).

[0143] In step S2A, the optimization device 10 executes a predetermined task based on the evaluation data generated in step S1. The method of executing the task is the same as the task execution process in the embodiment (see FIG. 7).

[0144] In step S3A, the optimization device 10 calculates a search evaluation value based on the evaluation data generated in step S1. The method for calculating the search evaluation value is the same as the evaluation value calculation process in the embodiment (see step S32 in FIG. 8). The optimization device 10 stores history information of the search evaluation value in the evaluation history storage unit 109.

[0145] In step S4A, the optimization device 10 updates the search method for reference information based on the search evaluation value calculated in step S3 A. At this time, the optimization device 10 fixes the template and updates the search method.

[0146] In step S5A, the optimization device 10 determines whether a first convergence condition is satisfied. For example, the first convergence condition may be that the search method has been updated a predetermined number of times, or that the difference in search evaluation value before and after the update is within a predetermined threshold. If it is determined that the first convergence condition is satisfied (YES), the optimization device 10 proceeds to step S2B. On the other hand, if it is determined that the first convergence condition is not satisfied (NO), the optimization device 10 returns to step S2A.

[0147] In step S2B, the optimization device 10 executes a predetermined task based on the evaluation data generated in step S1. At this time, the optimization device 10 executes the predetermined task using the search method updated in step S3A.

[0148] In step S3B, the optimization device 10 calculates a task evaluation value and a feedback evaluation value based on the evaluation data generated in step S1. The calculation methods of the task evaluation value and the feedback evaluation value are the same as those in the evaluation value calculation process in the embodiment (see steps S31 and S33 in FIG. 8). The optimization device 10 stores history information of the task evaluation value and the feedback evaluation value in the evaluation history storage unit 109.

[0149] In step S4B, the optimization device 10 updates the template based on the task evaluation value and the feedback evaluation value calculated in step S3B. At this time, the optimization device 10 updates the template while fixing the search method.

[0150] In step S5B, the optimization device 10 determines whether a second convergence condition is satisfied. For example, the second convergence condition may be that the template has been updated a predetermined number of times, or that the differences in the task evaluation value and the feedback evaluation value before and after the update are within a predetermined threshold. If it is determined that the second convergence condition is satisfied (YES), the optimization device 10 terminates the optimization process. On the other hand, if it is determined that the second convergence condition is not satisfied (NO), the optimization device 10 returns the process to step S2B.

[0151] 9, the optimization device 10 optimizes the search method and then optimizes the template, but the order of optimization may be reversed. That is, the optimization device 10 may first optimize the template while keeping the search method for reference information fixed, and then optimize the search method for reference information by fixing the optimized template.

[0152] 9, the optimization of the search method and the optimization of the template are each performed once, but each optimization may be repeated multiple times, i.e., the optimization of the search method and the optimization of the template may be performed alternately and repeatedly.

[0153] Simultaneous optimization of hyperparameters, including the reference information search method and the template, can result in a large amount of computation. By optimizing either the reference information search method or the template, and then optimizing the other while keeping the other fixed, the amount of computation required for optimization can be expected to be reduced.

[0154] [Variation 2] In the above embodiment, a configuration has been described in which an evaluation value is calculated using all evaluation data. If an evaluation value based on a portion of the evaluation data is lower than the evaluation value for an evaluated hyperparameter, even if an evaluation value is calculated based on the remaining evaluation data, the hyperparameter is often not selected as an optimized hyperparameter. In this case, omitting the process of calculating an evaluation value based on the remaining evaluation data does not affect the optimization result.

[0155] In this modification, if the evaluation value of a certain hyperparameter calculated based on a portion of the evaluation data is less than a predetermined reference value, the evaluation of that hyperparameter is stopped. Hereinafter, the evaluation value based on a portion of the evaluation data is referred to as the "intermediate evaluation value."

[0156] As an example, a case will be described in which the evaluation unit 108 calculates the accuracy rate of a question answering task as the task evaluation value. First, the evaluation unit 108 acquires a predetermined number of correct answer datasets from the evaluation data storage unit 101. The predetermined number of correct answer datasets may be randomly extracted correct answer datasets, or may include correct answer datasets predetermined by a user. The correct answer datasets predetermined by a user may, for example, include questions that are considered difficult to answer, or may include questions that the user wants to be answered correctly without fail.

[0157] Next, the evaluation unit 108 acquires the generation results by the generative model 35 based on the instruction information included in the acquired predetermined number of correct answer datasets. The evaluation unit 108 calculates an intermediate evaluation value by comparing the generation results by the generative model 35 with the correct answer information included in the correct answer datasets.

[0158] The evaluation unit 108 compares the intermediate evaluation value with a predetermined reference value. The predetermined reference value may be, for example, the average value of past evaluation values, the best value of past evaluation values, or a fixed value specified by the user. The average value of past evaluation values may be calculated each time the evaluation unit 108 calculates an evaluation value, or may be calculated based on history information stored in the evaluation history storage unit 109.

[0159] If the intermediate evaluation value is less than a predetermined reference value, the evaluation unit 108 stops calculating the task evaluation value. On the other hand, if the intermediate evaluation value is equal to or greater than the predetermined reference value, the evaluation unit 108 acquires the remaining correct answer data set from the evaluation data storage unit 101 and continues calculating the task evaluation value based on the remaining correct answer data set.

[0160] In other words, in this modification, the evaluation unit 108 calculates an evaluation value using a portion of the evaluation data, and controls optimization of at least one of the reference information search method and the template based on the evaluation value. As an example, if the evaluation value is lower than a predetermined reference value, the evaluation unit 108 stops evaluating the generation results and evaluates the generation results generated by changing at least one of the reference information search method and the template.

[0161] In this modification, for hyperparameters for which the evaluation value calculated based on a portion of the evaluation data is not expected to exceed the past evaluation value, evaluation based on the remaining evaluation data is stopped. This significantly reduces the cost required to calculate the evaluation value until optimization is complete. Therefore, this modification allows for efficient hyperparameter optimization.

[0162] This modification can be applied regardless of the framework of search expansion generation. For example, this modification can be applied even when optimizing a template for generating input information for a generative model.

[0163] [Application example] In each embodiment of the present disclosure, an example of using a large-scale language model that generates text data has been described as an example of a generative model, but the uses of a generative model are not limited to this. Examples of uses of a generative model include image generation, code review, material generation, finance, etc.

[0164] Image generation may be the generation of characters such as anime characters. Material generation may be the generation of materials from raw materials. Code review may select items such as tests, data structures, and documentation through black-box optimization. Variations of items may include items from qualitative perspectives such as consistency, maintainability, and errors. Furthermore, variations of items may include items related to context (code differences). Finance may be the creation of individual stock investment strategies.

[0165] <Summary> As is clear from the above explanation, the optimization device 10 according to one embodiment of the present disclosure searches for reference information based on input instruction information, generates input information for a generative model based on the instruction information, the search results for the reference information, and a template, obtains a generation result obtained by inputting the input information into the generative model, evaluates the generation result to obtain a predetermined evaluation result, and optimizes at least one of the search method for reference information or the template based on the predetermined evaluation result.

[0166] The predetermined evaluation result may include a first evaluation value that evaluates the validity of the generated result with respect to correct answer information determined for the instruction information. The predetermined evaluation result may include a second evaluation value that evaluates the relevance of the instruction information to the search result. The predetermined evaluation result may include a third evaluation value that evaluates the validity of the generated result with respect to the instruction information. The third evaluation value may be a value based on feedback from a user regarding the generated result.

[0167] The optimization device 10 may calculate a first evaluation value based on a first dataset generated based on a second generative model. The optimization device 10 may calculate a second evaluation value based on a second dataset generated based on a third generative model. The optimization device 10 may acquire a third evaluation value by inputting the generation results to a trained model generated based on a third dataset including the generation results and feedback information.

[0168] The search method may include information about at least one of a search module, parameters of the search module, the number of search results, and query expansion.

[0169] The optimization device 10 may optimize the template using information about the search method. The optimization device 10 may optimize the search method and optimize the template based on the optimized search method. The optimization device 10 may optimize the template and optimize the search method based on the optimized template. The optimization device 10 may iteratively perform optimizing the search method and optimizing the template.

[0170] The predetermined evaluation result may include an evaluation value calculated using a portion of the dataset. The optimization device 10 may control optimization of at least one of a reference information search method or a template based on the evaluation value. If the evaluation value is lower than a predetermined reference value, the optimization device 10 may stop evaluating the generated results and evaluate the generated results generated by changing at least one of a reference information search method or a template.

[0171] The optimization device 10 may optimize at least one of the reference information retrieval method and the template by performing black-box optimization based on a predetermined evaluation result. The optimization device 10 may optimize at least one of the reference information retrieval method and the template by sampling hyperparameters based on the predetermined evaluation result.

[0172] As a result, according to one embodiment of the present disclosure, a technique for optimizing input information to a generative model can be provided. In one aspect, according to one embodiment, good generation results can be obtained from a generative model without changing the semantic content of the input information.

[0173] [Hardware configuration of information processing device] Some or all of the devices (optimization device 10, search device 20, generation device 30, and terminal device 40) in the above-described embodiments may be configured as hardware, or may be configured as software (program) information processing executed by a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or the like. In the case of software information processing, software that realizes at least some of the functions of each device in the above-described embodiments may be stored on a non-transitory storage medium (non-transitory computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or a USB (Universal Serial Bus) memory, and the software information processing may be executed by loading the software into a computer. The software may also be downloaded via a communication network. Furthermore, all or part of the software processing may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), thereby executing the software information processing by hardware.

[0174] The storage medium that stores the software may be a removable medium such as an optical disk, or a fixed medium such as a hard disk, memory, etc. The storage medium may be provided inside the computer (main storage device, auxiliary storage device, etc.) or outside the computer.

[0175] 10 is a block diagram showing an example of the hardware configuration of each device (optimization device 10, search device 20, generation device 30, and terminal device 40) in the above-described embodiment. Each device may be realized as a computer 7 including, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, which are connected via a bus 76.

[0176] Although the computer 7 in FIG. 10 includes one of each component, it may also include multiple of the same component. Also, while FIG. 10 shows one computer 7, the software may be installed on multiple computers, and each of the multiple computers may execute the same or different parts of the software. In this case, a distributed computing configuration may be used in which each computer communicates with the other computers via a network interface 74 or the like to execute the processing. That is, each device in the above-described embodiment (the optimization device 10, the search device 20, the generation device 30, and the terminal device 40) may be configured as a system in which one or more computers execute instructions stored in one or more storage devices to realize its functions. Furthermore, the system may be configured such that information sent from a terminal is processed by one or more computers provided on a cloud, and the processing results are sent to the terminal.

[0177] The various calculations of each device in the above-described embodiments (optimization device 10, search device 20, generation device 30, and terminal device 40) may be executed in parallel using one or more processors, or using multiple computers via a network. Furthermore, the various calculations may be distributed to multiple processing cores within a processor and executed in parallel. Furthermore, some or all of the processes, means, etc. of the present disclosure may be realized by at least one of a processor and a storage device provided on a cloud that can communicate with computer 7 via a network. In this way, each device in the above-described embodiments may be implemented in the form of parallel computing using one or more computers.

[0178] The processor 71 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that at least controls a computer or performs calculations. The processor 71 may be a general-purpose processor, a dedicated processing circuit designed to perform a specific calculation, or a semiconductor device that includes both a general-purpose processor and a dedicated processing circuit. The processor 71 may also include an optical circuit or a calculation function based on quantum computing.

[0179] The processor 71 may perform arithmetic processing based on data or software input from each device or the like configured inside the computer 7, and may output the calculation results or control signals to each device or the like. The processor 71 may control each component constituting the computer 7 by executing the OS (Operating System) of the computer 7, applications, etc.

[0180] Each device in the above-described embodiment (optimization device 10, search device 20, generation device 30, and terminal device 40) may be realized by one or more processors 71. Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other via wire or wirelessly.

[0181] The main memory device 72 may store instructions executed by the processor 71, various data, etc., and information stored in the main memory device 72 may be read by the processor 71. The auxiliary memory device 73 is a memory device other than the main memory device 72. Note that these memory devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory devices for saving various data, etc. in each device (optimization device 10, search device 20, generation device 30, and terminal device 40) in the above-described embodiments may be realized by the main memory device 72 or the auxiliary memory device 73, or may be realized by an internal memory built into the processor 71. For example, each memory unit in the above-described embodiments may be realized by the main memory device 72 or the auxiliary memory device 73.

[0182] When each device in the above-described embodiment (optimization device 10, search device 20, generation device 30, and terminal device 40) is configured with at least one storage device (memory) and at least one processor connected (coupled) to this at least one storage device, at least one processor may be connected to one storage device. Also, at least one storage device may be connected to one processor. Also, a configuration in which at least one processor out of multiple processors is connected to at least one storage device out of multiple storage devices may be included. Also, this configuration may be realized by storage devices and processors included in multiple computers. Furthermore, a configuration in which a storage device is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

[0183] The network interface 74 is an interface for connecting to the communication network 8 wirelessly or via a wire. The network interface 74 may be an appropriate interface, such as one that conforms to an existing communication standard. Information may be exchanged with an external device 9A connected via the communication network 8 via the network interface 74. The communication network 8 may be any one of a WAN (Wide Area Network), a LAN (Local Area Network), a PAN (Personal Area Network), etc., or a combination thereof, as long as information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet, an example of a LAN is IEEE802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication), etc.

[0184] The device interface 75 is an interface such as a USB that directly connects to the external device 9B.

[0185] The external device 9A is a device connected to the computer 7 via a network. The external device 9B is a device connected directly to the computer 7.

[0186] For example, the external device 9A or the external device 9B may be an input device. The input device may be a device such as a camera, a microphone, a motion capture device, various sensors, a keyboard, a mouse, or a touch panel, and provides acquired information to the computer 7. Alternatively, the external device 9A or the external device 9B may be a device equipped with an input unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0187] Furthermore, the external device 9A or the external device 9B may be, for example, an output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or a speaker that outputs sound or the like. Alternatively, the output device may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0188] Furthermore, the external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage such as an HDD.

[0189] Furthermore, the external device 9A or the external device 9B may be a device having some of the functions of the components of each device (the optimization device 10, the search device 20, the generation device 30, and the terminal device 40) in the above-described embodiments. That is, the computer 7 may transmit some or all of the processing results to the external device 9A or the external device 9B, or may receive some or all of the processing results from the external device 9A or the external device 9B.

[0190] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it also includes the addition of elements other than the enumerated elements (a, b, and c), such as having d, as in abcd.

[0191] In this specification (including claims), when expressions such as "using data as input / based on / according to / in response to data" (including similar expressions) are used, unless otherwise specified, this includes cases where the data itself is used, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of data, etc.) is used. Furthermore, when a statement is made that a result is obtained "using data as input / based on / according to / in response to data" (including similar expressions), this includes cases where the result is obtained based solely on the data, or where the result is influenced by other data, factors, conditions, and / or states other than the data itself, unless otherwise specified. Furthermore, when a statement is made that "data is output" (including similar expressions), this includes cases where the data itself is used as output, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of various data, etc.) is used as output, unless otherwise specified.

[0192] When the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that encompass any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, and physically connection / coupling. These terms should be interpreted appropriately according to the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted as being included in these terms without limitation.

[0193] In this specification (including the claims), the expression "A configured to B" may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting / configuration of element A being configured / set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, if element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure, etc. of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0194] Whenever words implying containing or possessing (e.g., "comprising / including," "having," etc.) are used in this specification (including the claims), they are intended to be open-ended terms that include the inclusion or possession of things other than the object designated by the object of the term. When the object of such words implying containing or possessing does not specify a quantity or suggests a singular number (e.g., expressions using the articles "a" or "an"), the expression should be construed as not being limited to a specific number.

[0195] In this specification (including the claims), even if expressions such as "one or more" and "at least one" are used in some places and expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

[0196] In this specification, when a particular advantage / result is described as being obtained with respect to a particular configuration of an embodiment, it should be understood that the same advantage / result can also be obtained with one or more other embodiments having the same configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or circumstances, and that the effect is not necessarily obtained with the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or circumstances are satisfied, and the effect does not necessarily occur in a claimed invention that defines the same or a similar configuration.

[0197] In this specification (including claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may cooperate to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" (including similar expressions) is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that hardware may include electronic circuits, devices including electronic circuits, etc.

[0198] In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data, or may store the entire data. Also, a configuration in which only some of the multiple storage devices store data may be included.

[0199] In this specification (including the claims), terms such as "first," "second," etc. are used merely as a way of distinguishing between two or more elements, and are not necessarily intended to impose technical meanings such as temporal aspect, spatial aspect, sequence, quantity, etc. Thus, for example, a reference to a first element and a second element does not necessarily mean that only two elements can be employed therein, that the first element must precede the second element, that the first element must be present in order for the second element to be present, etc.

[0200] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, partial deletions, etc. are possible within the scope of the conceptual idea and spirit of the present invention, which is derived from the content defined in the claims and their equivalents. For example, when numerical values or formulas are used in the above-described embodiments, they are shown for illustrative purposes and do not limit the scope of the present disclosure. Furthermore, the order of each operation shown in the embodiments is also illustrative and does not limit the scope of the present disclosure.

[0201] The disclosed technology may take the following forms as described below.

[0202] (Appendix 1) At least one memory; at least one processor; The at least one processor: generating input information for a generative model based on the instruction information, search results for reference information based on the instruction information, and a template; acquiring a generation result obtained by inputting the input information into the generation model; Evaluating the generated result to obtain a predetermined evaluation result; optimizing at least one of the search method for the reference information and the template based on the predetermined evaluation result; Information processing device.

[0203] (Appendix 2) the predetermined evaluation result includes a first evaluation value obtained by evaluating the validity of the generated result with respect to correct answer information determined for the instruction information; 10. The information processing device according to claim 1.

[0204] (Appendix 3) the predetermined evaluation result includes a second evaluation value that evaluates the relevance between the instruction information and the search result. 3. The information processing device according to claim 1 or 2.

[0205] (Appendix 4) the predetermined evaluation result includes a third evaluation value that evaluates the validity of the generation result with respect to the instruction information. 4. An information processing device according to any one of appendices 1 to 3.

[0206] (Appendix 5) the third evaluation value is a value based on feedback from a user regarding the generation result. 5. The information processing device according to claim 4.

[0207] (Appendix 6) The at least one processor: calculating the first evaluation value based on a first dataset generated based on a second generative model; 3. The information processing device according to claim 2.

[0208] (Appendix 7) The at least one processor: calculating the second evaluation value based on a second dataset generated based on a third generative model; 4. The information processing device according to claim 3.

[0209] (Appendix 8) The at least one processor: acquiring the third evaluation value by inputting the acquired generation result into a trained model generated based on a third dataset including the generation result and feedback information; 5. The information processing device according to claim 4.

[0210] (Appendix 9) The search method includes at least information about a search module, parameters of the search module, the number of search results, query expansion, and a method for sorting the search results. 9. An information processing device according to any one of appendices 1 to 8.

[0211] (Appendix 10) The at least one processor: optimizing the template using information about the search method; 10. An information processing device according to any one of appendices 1 to 9.

[0212] (Appendix 11) The at least one processor: optimizing the search method; optimizing the template based on the optimized search method; 11. An information processing device according to any one of appendices 1 to 10.

[0213] (Appendix 12) The at least one processor: optimizing the template; optimizing the search method based on the optimized template; 12. An information processing device according to any one of appendices 1 to 11.

[0214] (Appendix 13) The at least one processor: optimizing the search method; optimizing the template; Repeatedly execute 13. An information processing device according to any one of appendices 1 to 12.

[0215] (Appendix 14) the predetermined evaluation result includes an evaluation value calculated using a portion of the data set, The at least one processor: optimizing at least one of the search method for the reference information and the template based on the evaluation value; 14. An information processing device according to any one of appendices 1 to 13.

[0216] (Appendix 15) The at least one processor: If the evaluation value is lower than a predetermined reference value, the evaluation of the generated result is stopped, and the generated result is evaluated after changing at least one of the search method for the reference information or the template. 15. The information processing device according to claim 14.

[0217] (Appendix 16) The at least one processor: optimizing at least one of the reference information search method and the template by performing black-box optimization based on the predetermined evaluation result; 16. An information processing device according to any one of appendices 1 to 15.

[0218] (Appendix 17) The at least one processor: optimizing at least one of the reference information search method and the template by sampling hyperparameters based on the predetermined evaluation result; 17. An information processing device according to claim 1.

[0219] (Appendix 18) At least one processor generating input information for a generative model based on the instruction information, search results for reference information based on the instruction information, and a template; acquiring a generation result obtained by inputting the input information into the generation model; Evaluating the generated result to obtain a predetermined evaluation result; optimizing at least one of the search method for the reference information and the template based on the predetermined evaluation result; An information processing method that performs processing.

[0220] (Appendix 19) At least one processor has generating input information for a generative model based on the instruction information, search results for reference information based on the instruction information, and a template; acquiring a generation result obtained by inputting the input information into the generation model; Evaluating the generated result to obtain a predetermined evaluation result; optimizing at least one of the search method for the reference information and the template based on the predetermined evaluation result; A program for executing a process. [Explanation of symbols]

[0221] 10: Optimization device 20: Search device 25: Data Source 30:Generation device 35: Generative model 40: Terminal device 101: Evaluation data storage unit 102: Data generation unit 103: Instruction reception department 104: Search section 105: Generation part 106: Response section 107: Feedback section 108: Evaluation Department 109: Evaluation history memory unit 110: Optimization section 1000: Information Processing Systems

Claims

1. at least one memory; at least one processor; The at least one processor generating input information for a generative model based on the instruction information, search results for reference information based on the instruction information, and a template; acquiring a generation result obtained by inputting the input information into the generation model; Evaluating the generated result to obtain a predetermined evaluation result; optimizing at least one of the search method for the reference information and the template based on the predetermined evaluation result; Information processing device.

2. the predetermined evaluation result includes a first evaluation value obtained by evaluating the validity of the generated result with respect to correct answer information determined for the instruction information; The information processing device according to claim 1 .

3. the predetermined evaluation result includes a second evaluation value that evaluates the relevance between the instruction information and the search result; The information processing device according to claim 1 .

4. the predetermined evaluation result includes a third evaluation value that evaluates the validity of the generation result with respect to the instruction information. The information processing device according to claim 1 .

5. the third evaluation value is a value based on feedback from a user regarding the generation result. The information processing device according to claim 4 .

6. The at least one processor calculating the first evaluation value based on a first dataset generated based on a second generative model; The information processing device according to claim 2 .

7. The at least one processor calculating the second evaluation value based on a second dataset generated based on a third generative model; The information processing device according to claim 3 .

8. The at least one processor The third evaluation value is obtained by inputting the acquired generation result into a trained model generated based on a third dataset including the generation result and feedback information. The information processing device according to claim 4 .

9. The search method includes at least information about a search module, parameters of the search module, the number of search results, query expansion, and a method for sorting the search results. The information processing device according to claim 1 .

10. The at least one processor optimizing the template using information about the search method; 10. The information processing device according to claim 1.

11. The at least one processor optimizing the search method; optimizing the template based on the optimized search method; 10. The information processing device according to claim 1.

12. The at least one processor optimizing the template; optimizing the search method based on the optimized template; 10. The information processing device according to claim 1.

13. The at least one processor optimizing the search method; optimizing the template; Repeatedly execute 10. The information processing device according to claim 1.

14. the predetermined evaluation result includes an evaluation value calculated using a portion of the data set, The at least one processor optimizing at least one of the search method for the reference information and the template based on the evaluation value; 10. The information processing device according to claim 1.

15. The at least one processor If the evaluation value is lower than a predetermined reference value, the evaluation of the generated result is stopped, and the generated result is evaluated after changing at least one of the search method for the reference information or the template. The information processing device according to claim 14.

16. The at least one processor optimizing at least one of the reference information search method and the template by performing black-box optimization based on the predetermined evaluation result; 10. The information processing device according to claim 1.

17. The at least one processor optimizing at least one of the reference information search method and the template by sampling hyperparameters based on the predetermined evaluation result; 10. The information processing device according to claim 1.

18. At least one processor generating input information for a generative model based on the instruction information, search results for reference information based on the instruction information, and a template; acquiring a generation result obtained by inputting the input information into the generation model; Evaluating the generated result to obtain a predetermined evaluation result; optimizing at least one of the search method for the reference information and the template based on the predetermined evaluation result; An information processing method that performs processing.

19. At least one processor generating input information for a generative model based on the instruction information, search results for reference information based on the instruction information, and a template; acquiring a generation result obtained by inputting the input information into the generation model; Evaluating the generated result to obtain a predetermined evaluation result; optimizing at least one of the search method for the reference information and the template based on the predetermined evaluation result; A program for executing a process.

Images