Computer system, and parameter search support method
The system efficiently identifies optimal parameter sets for generative AI tasks by correlating free models with paid models, addressing high costs and inefficiencies in existing methods, enabling cost-effective execution of complex tasks.
Patent Information
- Application Number
- JP2024016269
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-02-06
- Publication Date
- 2025-08-19
AI Technical Summary
Existing methods for using generative AI to perform complex tasks, such as summarizing large documents, are hindered by the high cost and inefficiency of determining optimal parameter combinations, especially when using paid models, necessitating numerous trials to find the best settings.
A computer system that manages correlations between paid and free natural language processing models, using correlation information to identify free models correlated with paid models, and evaluates parameter sets based on evaluation indices to determine optimal parameter combinations for tasks involving multiple processes.
This approach allows for the efficient and cost-effective search for parameter sets, reducing the need for costly trials with paid models and optimizing task execution using free models with similar performance.
Smart Images
Figure 2025121072000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a technique for searching for parameters that control each process of a task that is composed of multiple processes, including the use of a generation AI. [Background technology]
[0002] In recent years, generative AI has emerged that receives instructions from users and generates text (see, for example, Non-Patent Document 1 and Non-Patent Document 2). In this invention, generative AI refers to a large-scale language model (LLM). A large-scale language model is a type of natural language processing model that can perform various language processing tasks.
[0003] There are paid and free generation AIs. Paid generation AIs incur a fee depending on the number of tokens in the input text. For example, if you input 1,000 tokens of input text into GPT-4, a paid generation AI, it will cost about 10 yen. [Prior art documents] [Patent documents]
[0004] [Non-Patent Document 1] OpenAI, R. "GPT-4 technical report." arXiv (2023): 2303-08774. [Non-patent document 2] Mosaic ML, "Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs", [online], [Retrieved December 25, 2023], Internet<URL: https: / / www.mosaicml.com / blog / mpt-7b> . [Non-patent document 3] LangChain, [online], [Retrieved December 25, 2023], Internet<URL:https: / / www.langchain.com / > Summary of the Invention [Problem to be solved by the invention]
[0005] Consider a task of summarizing 10,000 documents, each consisting of 100 tokens of text, using a paid generative AI to extract and analyze the key points of the documents. In this case, 1 million tokens of text are input to the generative AI. Since there is a limit to the number of tokens that can be input to the generative AI at one time, it is difficult to have the generative AI perform the above task as is.
[0006] In response to this, a technology is known that connects inputs and outputs to a generative AI to achieve complex tasks (see Non-Patent Document 3).
[0007] Using the technology described in Non-Patent Document 3, the above-mentioned task can be realized as a task that links the following steps: segmenting input text, inputting the segmented text to a generation AI, aggregating multiple texts output by the generation AI, and inputting the aggregated text back to the generation AI. In this case, it is necessary to set parameters that control the input to the generation AI, segmentation, and aggregation via prompts, which are texts that contain instructions for the generation AI.
[0008] To efficiently execute the above tasks, it is necessary to find the optimal combination of parameters (parameter set) in advance. If there are 10 patterns for each of the splitting and aggregation parameters, it will be necessary to try the task 100 times to find the optimal parameter combination. If a paid generation AI is used, it will cost approximately 1 million yen.
[0009] When determining the parameters to control each process of a task consisting of multiple processes, including the use of generative AI, it is effective to narrow down the combinations of parameters to be verified using free generative AI. [Means for solving the problem]
[0010] A representative example of the invention disclosed in the present application is as follows: That is, a computer system includes a computer having a processor, a storage device connected to the processor, and a network interface connected to the processor, and is accessible to a text generation system that receives a prompt, which is text including an instruction, and generates output text using a natural language processing model, and the system is configured to receive correlation information for managing a correlation between the natural language processing model of a first category and the natural language processing model of a second category, the correlation information being classified based on costs incurred in using the natural language processing model, and the plurality of processes of a target task including one or more text generation processes using a target natural language processing model that is the natural language processing model of the first category. and first parameter set information for managing first parameter sets, which are combinations of parameters for controlling each of the above, wherein the cost of the natural language processing model of the first classification is greater than the cost of the natural language processing model of the second classification, and the computer system refers to the correlation information to identify a verification natural language processing model from the natural language processing models of the second classification that is correlated with the target natural language processing model, calculates evaluation indices for evaluating the target task using the verification natural language processing model and the execution results of the target task while changing the first parameter set, and determines candidate first parameter sets based on the evaluation indices. [Effects of the Invention]
[0011] According to the present invention, it is possible to search for parameters for each process of a task consisting of multiple processes, including the use of a natural language processing model (generative AI), at low cost. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiment. [Brief explanation of the drawings]
[0012] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of the parameter search system according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer that configures the parameter search system of the first embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of a data structure of generated AI information according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a data structure of correlation information according to the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a data structure of search text information according to the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of a data structure of search parameter set information according to the first embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a data structure of evaluation information according to the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a data structure of verification result information according to the first embodiment. [Figure 9] 1 is a flowchart illustrating an outline of a process executed by the parameter search system according to the first embodiment. [Figure 10] FIG. 2 is a diagram showing an example of a screen displayed by the parameter search system of the first embodiment. [Figure 11] 10 is a flowchart illustrating an example of a search text generation process executed by the parameter search system of the first embodiment. [Figure 12] 10 is a flowchart illustrating an example of a parameter set generation process executed by the parameter search system according to the first embodiment. [Figure 13] 10 is a flowchart illustrating an example of a search process executed by the parameter search system according to the first embodiment. [Figure 14] 10 is a flowchart illustrating an example of a verification process executed by the parameter search system according to the first embodiment. [Figure 15] FIG. 2 is a diagram showing an example of a screen displayed by the parameter search system of the first embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of a functional configuration of a parameter search system according to a second embodiment. [Figure 17] FIG. 10 is a diagram illustrating an example of the data structure of analytical text information according to the second embodiment. [Figure 18] FIG. 10 is a diagram showing an example of the data structure of analysis parameter set information according to the second embodiment. [Figure 19] 10 is a flowchart illustrating an outline of a process executed by a parameter search system according to a second embodiment. [Figure 20] 10 is a flowchart illustrating an example of a generation AI classification process according to the second embodiment. [Figure 21] 10 is a flowchart illustrating an example of an analytical text generation process executed by the parameter search system according to the second embodiment. [Figure 22] 10 is a flowchart illustrating an example of a parameter set generation process executed by the parameter search system according to the second embodiment. [Figure 23A] 10 is a flowchart illustrating an example of a correlation analysis process executed by the parameter search system according to the second embodiment. [Figure 23B] 10 is a flowchart illustrating an example of a correlation analysis process executed by the parameter search system according to the second embodiment. [Figure 23C] 10 is a flowchart illustrating an example of a correlation analysis process executed by the parameter search system according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will readily understand that the specific configuration can be changed without departing from the spirit or intent of the present invention.
[0014] In the configuration of the invention described below, the same or similar configurations or functions are denoted by the same reference numerals, and redundant explanations will be omitted.
[0015] In this specification, the terms "first," "second," "third," etc. are used to identify components and do not necessarily limit the number or order. [Example]
[0016] Fig. 1 is a diagram illustrating an example of a functional configuration of a parameter search system according to the first embodiment. Fig. 2 is a diagram illustrating an example of a hardware configuration of a computer that constitutes the parameter search system according to the first embodiment.
[0017] The parameter search system 100 is connected to a text generation system 101 and a terminal 102 via a network 103 such as a LAN (Local Area Network).
[0018] The terminal 102 is a terminal operated by a user who instructs the execution of a task. The text generation system 101 provides an API for using the generation AI (LLM). The generation AI receives a prompt, which is text containing instructions, and generates output text. Note that the present invention is not limited to the number of terminals 102 and text generation systems 101. Note that the parameter search system 100 may have the generation AI.
[0019] The parameter search system 100 receives a parameter search request for a task including multiple processes from a terminal 102, and determines a combination of parameters (parameter set) for controlling each process of the task. The task in the first embodiment includes the use of one or more generation AIs.
[0020] In this example, we consider a task of creating a summary of a document. The task involves dividing the document, generating summaries for the divided documents, aggregating the summaries, and generating a final summary from the aggregated summaries. In this case, it is necessary to determine parameters that define each of the steps of dividing the document, generating the summary, and aggregating the summary.
[0021] The parameter search system 100 includes a computer 200 as shown in Fig. 2. The computer 200 includes a CPU 210, a memory 220, a hard disk drive (HDD) 230, an input device 240, an output device 250, and a network device 260. The hardware elements are connected to each other via a bus 270.
[0022] The CPU 210 functions as a module, which will be described later, by executing a program stored in the memory 220. In the following description, when a process is described using a module as the subject, it means that the CPU 210 is executing a program that realizes the module.
[0023] The parameter search system 100 holds generated AI information 120, correlation information 121, search text information 122, search parameter set information 123, evaluation information 124, and verification result information 125. The parameter search system 100 also has a data generation unit 110, a parameter set generation unit 111, a search unit 112, a verification unit 113, and a display unit 114.
[0024] The generated AI information 120 is information for managing usable generated AIs. The data structure of the generated AI information 120 will be explained using FIG.
[0025] The correlation information 121 is information for managing the correlation between generated AIs. The data structure of the correlation information 121 will be explained with reference to FIG.
[0026] The search text information 122 is information for managing text used to search for parameter sets. The data structure of the search text information 122 will be explained using FIG.
[0027] Search parameter set information 123 is information for managing candidate parameter sets. The data structure of search parameter set information 123 will be explained using FIG.
[0028] The evaluation information 1242 is information for managing the evaluation results of the usefulness of the parameter set. The data structure of the evaluation information 1242 will be explained using FIG.
[0029] The verification result information 125 is information for managing the verification results of the parameter set using a paid generation AI. The data structure of the verification result information 125 will be explained using FIG.
[0030] FIG. 3 is a diagram illustrating an example of the data structure of the generated AI information 120 according to the first embodiment.
[0031] The generated AI information 120 stores entries including an ID 301, a name 302, and a group type 303. There is one entry for one generated AI.
[0032] ID 301 is a field that stores the ID of the entry. Name 302 is a field that stores the name of the generated AI. Group type 303 is a field that stores the identification information of the group to which the generated AI belongs. In this embodiment, generated AIs are classified into paid generated AIs and free generated AIs. It should be noted that generated AIs can also be classified based on costs other than monetary amounts. For example, processing time, the amount of resources required for processing, etc. can be considered. It should be noted that generated AIs can be classified by comparing the cost of using the generated AI with a threshold value.
[0033] FIG. 4 is a diagram illustrating an example of the data structure of the correlation information 121 according to the first embodiment.
[0034] The correlation information 121 stores a correlation table 400 for each type of summary target. The correlation table 400 is information in a matrix format, with paid generated AIs in the rows and free generated AIs in the columns. The correlation degree, which indicates the strength of the correlation, is stored in the cells. The closer the correlation degree is to 1, the more correlated the generated AIs are.
[0035] FIG. 5 is a diagram illustrating an example of the data structure of the search text information 122 according to the first embodiment.
[0036] The search text information 122 stores entries each including an ID 501, an input text 502, an output text 503, and a target type 504. There is one entry for one input text.
[0037] ID 501 is a field that stores the ID of the entry. Input text 502 is a field that stores the text that is input to the generation AI. Output text 503 is a field that stores the text that is generated from the input text. The output text 503 stores the correct answer or ideal text. Target type 504 is a field that stores the type of the target to be summarized.
[0038] FIG. 6 is a diagram illustrating an example of the data structure of the search parameter set information 123 according to the first embodiment.
[0039] The search parameter set information 123 stores entries including an ID 601, a target type 602, a first parameter 603, a second parameter 604, and a third parameter 605. There is one entry for one parameter set.
[0040] The ID 601 is a field for storing the ID of the entry. The object type 602 is a field for storing the type of the abstract object.
[0041] The first parameter 603 is a field for storing parameters that control the division of a document. For example, a prompt containing the number of characters to be divided is conceivable. The second parameter 604 is a field for storing parameters that control the summarization of the divided documents. For example, a prompt containing the number of characters in the summary text of the divided documents is conceivable. The third parameter 605 is a field for storing parameters that control the aggregation of the summary text. For example, a prompt containing the number of characters in the aggregated text is conceivable.
[0042] In this embodiment, the processes of document division, document summarization, and document aggregation are each performed using a generation AI. Therefore, the generation AI is controlled using prompts including each parameter. For example, in the case of document division, a possible prompt would be "Please summarize to 500 characters." In the case of document summarization, a possible prompt would be "Please summarize to 100 characters." In the case of document aggregation, a possible prompt would be "Please summarize to 500 characters."
[0043] The document division may be performed by a dedicated application, in which case the parameters controlling the application are searched for.
[0044] FIG. 7 is a diagram illustrating an example of the data structure of the evaluation information 124 according to the first embodiment.
[0045] The evaluation information 124 stores entries including an ID 701, a text ID 702, a generated AI name 703, a parameter set ID 704, a target type 705, and an evaluation score 706. One entry exists for one evaluation result.
[0046] ID 701 is a field that stores the ID of the entry. Text ID 702 is a field that stores the ID of the search text used. Generation AI name 703 is a field that stores the name of the generation AI that was used. Parameter set ID 704 is a field that stores the ID of the parameter set that was used. Target type 705 is a field that stores the type of summary target. Evaluation score 706 is a field that stores the evaluation score that evaluates the results of the task.
[0047] FIG. 8 is a diagram illustrating an example of the data structure of the verification result information 125 according to the first embodiment.
[0048] The verification result information 125 stores entries including an ID 801, a generated AI name 802, a parameter set ID 803, input text 804, output text 805, and a selection flag 806. There is one entry for each combination of generated AI and parameter set.
[0049] ID 801 is a field that stores the ID of the entry. Creation AI name 802 is a field that stores the name of a candidate creation AI that requires payment. Parameter set ID 803 is a field that stores the ID of a candidate parameter set.
[0050] Input text 804 is a field for storing text input for a task using a generation AI corresponding to generation AI name 802, to which a parameter set corresponding to parameter set ID 803 is set. Output text 805 is a field for storing text generated by the task. Selection flag 806 is a field for storing a flag indicating whether or not the settings of the entry are to be used. If the settings are to be used, "1" is set in selection flag 806, and if the settings are not to be used, "0" is set in selection flag 806.
[0051] Next, the processing executed by the parameter search system 100 will be described.
[0052] Fig. 9 is a flowchart illustrating an outline of the processing executed by the parameter search system 100 of the first embodiment. Fig. 10 is a diagram illustrating an example of a screen displayed by the parameter search system 100 of the first embodiment.
[0053] When the display unit 114 of the parameter search system 100 receives access from the terminal 102, it presents a screen 1000 such as that shown in FIG.
[0054] The screen 1000 includes an input field 1001, an input field 1002, and an operation button 1003. The input field 1001 is a field for the user to input the paid generation AI that they wish to use. The input field 1002 is a field for inputting the type of abstraction target. The operation button 1003 is an operation button for instructing the parameter search system 100 to execute processing. When the operation button 1003 is operated, the parameter search system 100 starts processing.
[0055] First, the parameter search system 100 selects a generation AI to be used for searching for a parameter set from among free generation AIs (step S101).
[0056] Specifically, the parameter search system 100 identifies the correlation table 400 corresponding to the type of summary target specified by the user from among the correlation tables 400 stored in the correlation information 121. The parameter search system 100 refers to the correlation table 400 and selects the free generation AI that has the highest correlation with the paid generation AI specified by the user. Note that if multiple paid generation AIs are specified, a free generation AI is selected for each paid generation AI.
[0057] Next, the parameter search system 100 executes a search text generation process (step S102). The search text generation process will be described in detail later.
[0058] Next, the parameter search system 100 executes a parameter set generation process (step S103). The parameter set generation process will be described in detail later.
[0059] Next, the parameter search system 100 executes a search process (step S104). The details of the search process will be described later.
[0060] After the search process is completed, the parameter search system 100 determines whether or not multiple candidate parameter sets have been identified by the search process (step S105).
[0061] If the number of candidate parameter sets identified by the search process is one, the parameter search system 100 displays the candidate parameter set (step S107) and ends the process.
[0062] If the search process identifies multiple candidate parameter sets, the parameter search system 100 executes a verification process (step S106) and then terminates the process. The verification process will be described in detail later.
[0063] FIG. 11 is a flowchart illustrating an example of a search text generation process executed by the parameter search system 100 according to the first embodiment.
[0064] The data generating unit 110 acquires an input text corresponding to the type of the summary target (step S201).
[0065] For example, if the type of the object to be summarized is "news," the data generation unit 110 acquires news articles from a news site or the like. Alternatively, the input text may be acquired from a publicly available database. The data generation unit 110 acquires multiple input texts.
[0066] The data generator 110 generates output texts that are generated by tasks using input texts (step S202). An output text is generated for each input text.
[0067] For example, an administrator of the parameter search system 100 generates and registers output text from input text.
[0068] The data generating unit 110 registers the input text and the output text in the search text information 122 (step S203).
[0069] Specifically, the data generation unit 110 adds entries to the search text information 122 for the number of input texts, sets an ID in ID 501 of each entry, and sets the type of abstract target specified in target type 504 of each entry. The data generation unit 110 also sets a pair of input text and output text in each entry.
[0070] The search text generation process can be omitted if an input text of the specified type to be summarized is registered in the search text information 122. The search text generation process may also be performed in advance.
[0071] FIG. 12 is a flowchart illustrating an example of a parameter set generation process executed by the parameter search system 100 according to the first embodiment.
[0072] The parameter set generation unit 111 generates a parameter set (step S301).
[0073] For example, a method of generating a parameter set based on a rule, a method of generating a parameter set randomly, and a method of generating a parameter set manually are conceivable. A plurality of generation methods may also be combined.
[0074] The parameter set generation unit 111 registers the parameter set in the search parameter set information 123 (step S302).
[0075] Specifically, the parameter set generation unit 111 adds entries to the search parameter set information 123 for the number of generated parameter sets, sets an ID in ID 601 of each entry, and sets the specified type of abstract target in target type 602 of each entry. The parameter set generation unit 111 also sets a parameter set in each added entry.
[0076] The parameter set generation process can be omitted if a parameter set for the specified type of abstract target is registered in the search parameter set information 123. The parameter set generation process may also be executed in advance.
[0077] 13 is a flowchart illustrating an example of a search process executed by the parameter search system 100 of the embodiment 1. The search unit 112 executes the process described below for each specified paid generation AI.
[0078] The search unit 112 refers to the search parameter set information 123 and selects a parameter set to be used (step S401).
[0079] The search unit 112 refers to the search text information 122 and selects the input text to be used (step S402).
[0080] The search unit 112 executes the task using the search generation AI selected in step S101, the parameter set selected in step S401, and the input text selected in step S402 (step S403).
[0081] The search unit 112 calculates an evaluation score by comparing the text generated by the task with the output text corresponding to the selected input text (step S404). At this time, the search unit 112 adds an entry to the evaluation information 124 and sets various information.
[0082] For example, the search unit 112 calculates the similarity between two texts using the distributed representations of the texts, and sets this as the evaluation score.
[0083] The search unit 112 determines whether or not the process has been completed for all input texts (step S405).
[0084] If processing has not been completed for all input texts, search unit 112 returns to step S402.
[0085] When the process is completed for all input texts, the search unit 112 calculates the overall evaluation score (step S406).
[0086] Specifically, the search unit 112 refers to the evaluation information 124 and searches for an entry in which the name of the search generation AI selected in step S401 is set in the generation AI name 802 and the ID of the parameter set selected in step S402 is set in the parameter set ID 803. The search unit 112 calculates a comprehensive evaluation score using the evaluation scores 706 of the searched entries. For example, the sum or average of the evaluation scores is calculated as the comprehensive evaluation score. The search unit 112 stores first evaluation data that associates pairs of generation AIs and parameter sets with comprehensive evaluation scores.
[0087] The search unit 112 determines whether or not the process has been completed for all parameter sets (step S407).
[0088] If the process has not been completed for all parameter sets, the search unit 112 returns to step S401.
[0089] When the process has been completed for all parameter sets, the search unit 112 determines candidate parameter sets based on the overall evaluation scores (step S408).
[0090] For example, the search unit 112 determines a parameter set whose overall evaluation score is greater than a threshold as a candidate parameter set. At this time, the search unit 112 adds an entry to the verification result information 125, sets an ID to ID 801 of the added entry, and sets "0" to selection flag 806 of the added entry. The search unit 112 sets the name of a paid generation AI to generation AI name 802 of the added entry, and sets a candidate parameter set to parameter set ID 803.
[0091] Fig. 14 is a flowchart illustrating an example of a verification process executed by the parameter search system 100 according to the first embodiment. Fig. 15 is a diagram illustrating an example of a screen displayed by the parameter search system 100 according to the first embodiment.
[0092] The verification unit 113 receives an input text for verification (step S501).
[0093] 15 on the terminal 102. The screen 1500 includes a display field 1501, an input field 1502, an operation button 1503, and a display field 1504.
[0094] Display field 1501 is a field that displays paid generation AIs and candidate parameter sets. Display field 1501 includes check boxes for selecting paid generation AIs and candidate parameter sets to use. Display field 1501 also includes an operation button for displaying details of the parameter sets.
[0095] Input field 1502 is a field for inputting input text. For example, the user inputs a file path into input field 1502. Operation button 1503 is an operation button for issuing an instruction to execute verification. Display field 1504 is a field for displaying text generated by executing a task using the input text.
[0096] The user inputs input text into the input field 1502 and operates the operation button 1503 .
[0097] The verification unit 113 selects a combination of a paid generated AI and a candidate parameter set (step S502).
[0098] Specifically, the verification unit 113 selects one entry from the verification result information 125 .
[0099] The verification unit 113 executes a task for the input text using the paid generated AI and candidate parameters (step S503).
[0100] At this time, the verification unit 113 sets the input text in the input text 804 of the corresponding entry in the verification result information 125, and sets the text generated by the task in the output text 805.
[0101] The verification unit 113 determines whether or not the process has been completed for all pairs of paid generation AIs and candidate parameter sets (step S504).
[0102] If the processing has not been completed for all pairs of paid generated AIs and candidate parameter sets, the verification unit 113 returns to step S502.
[0103] When the process has been completed for all pairs of paid generation AIs and candidate parameter sets, the verification unit 113 displays the verification results via the display unit 114 (step S505), and ends the verification process.
[0104] For example, the generated text is displayed in a display field 1504 on a screen 1500. The user refers to the text and selects a combination of a paid generation AI and a parameter set to be used.
[0105] According to the first embodiment, a parameter set for controlling each process of a task can be searched for using a free generation AI that is correlated with a paid generation AI, thereby reducing the cost required for determining the parameter set.
[0106] In this embodiment, the execution of a task is triggered by inputting text, but this is not limiting. If information other than text is used as the execution trigger for a task, search input information may be prepared instead of the search text information 122. [Example]
[0107] In the second embodiment, the parameter search system 100 analyzes the correlation of the generated AI. The second embodiment will be described below, focusing on the differences from the first embodiment.
[0108] FIG. 16 is a diagram illustrating an example of the functional configuration of the parameter search system 100 according to the second embodiment.
[0109] The parameter search system 100 of the second embodiment differs from the parameter search system 100 of the first embodiment in that it holds analytical text information 126 and analytical parameter set information 127. The parameter search system 100 of the second embodiment also differs from the parameter search system 100 of the first embodiment in that it has a generation AI classification unit 115 and a correlation analysis unit 116.
[0110] The analytical text information 126 is information for managing text used to analyze the correlation between generation AIs. The data structure of the analytical text information 126 will be explained using FIG.
[0111] The analytical parameter set information 127 is information for managing parameter sets used to analyze the correlation between generated AIs. The data structure of the analytical parameter set information 127 will be explained using FIG.
[0112] FIG. 17 is a diagram illustrating an example of the data structure of the analytical text information 126 according to the second embodiment.
[0113] The analytical text information 126 stores entries each including an ID 1701, an input text 1702, an output text 1703, and a target type 1704. There is one entry for each input text.
[0114] ID 1701 is a field that stores the ID of the entry. Input text 1702 is a field that stores the text that is input to the generation AI. Output text 1703 is a field that stores the text that is generated from the input text. The output text 1703 stores the correct answer or ideal text. Target type 1704 is a field that stores the type of the target to be summarized.
[0115] FIG. 18 is a diagram illustrating an example of the data structure of the analysis parameter set information 127 according to the second embodiment.
[0116] The analytical parameter set information 127 stores entries including an ID 1801, a target type 1802, a first parameter 1803, a second parameter 1804, and a third parameter 1805. There is one entry for one parameter set.
[0117] The ID 1801 is a field for storing the ID of the entry. The object type 1802 is a field for storing the type of the abstract object.
[0118] The first parameter 1803 is a field for storing parameters for controlling the division of a document. For example, a prompt including the number of characters to be divided is conceivable. The second parameter 1804 is a field for storing parameters for controlling the summarization of the divided documents. For example, a prompt including the number of characters in the summary text of the divided documents is conceivable. The third parameter 1805 is a field for storing parameters for controlling the aggregation of the summary text. For example, a prompt including the number of characters in the aggregated text is conceivable.
[0119] FIG. 19 is a flowchart illustrating an outline of the process executed by the parameter search system 100 according to the second embodiment.
[0120] When the parameter search system 100 receives an execution instruction from the system administrator, it executes the process described below.
[0121] The parameter search system 100 executes a generation AI classification process (step S601). The generation AI classification process will be described in detail later.
[0122] Next, the parameter search system 100 executes an analytical text generation process (step S602). The analytical text generation process will be described in detail later.
[0123] Next, the parameter search system 100 executes a parameter set generation process (step S603). The parameter set generation process will be described in detail later.
[0124] Next, the parameter search system 100 executes a correlation analysis process (step S604). The correlation analysis process will be described in detail later.
[0125] FIG. 20 is a flowchart illustrating an example of the generation AI classification process according to the second embodiment.
[0126] The generation AI classifier 115 acquires cost information (step S701).
[0127] For example, the generation AI classification unit 115 accesses the web page of the service that provides the generation AI and acquires the cost information. Note that the cost information may be input by the administrator of the parameter search system 100.
[0128] The generated AI classification unit 115 classifies the generated AI based on the cost information (step S702).
[0129] Specifically, the generated AI classification unit 115 classifies the generated AI based on the comparison result between the cost and the threshold. In this embodiment, the threshold is set to 0, and the generated AI classification unit 115 classifies the generated AI into paid generated AI and free generated AI.
[0130] The generated AI classification unit 115 reflects the classification result in the generated AI information 120 (step S703).
[0131] Specifically, the generated AI classification unit 115 adds an entry to the generated AI information 120, sets an ID in the ID 301 of each entry, and sets the name of the generated AI in the name 302. The generated AI classification unit 115 also sets information about the group to which the generated AI belongs in the group type 303 of each entry.
[0132] The AI generation classification process may be performed in advance.
[0133] FIG. 21 is a flowchart illustrating an example of an analytical text generation process executed by the parameter search system 100 according to the second embodiment.
[0134] The data generating unit 110 selects a type of abstract object (step S801). It is assumed that the type of abstract object is set in advance.
[0135] The data generating unit 110 acquires an input text corresponding to the type of the summary target (step S802). The process of step S802 is the same as the process of step S201.
[0136] The data generator 110 generates an output text that is generated by a task using the input text (step S803). The process of step S803 is the same as the process of step S202.
[0137] The data generator 110 registers the input text and the output text in the analytical text information 126 (step S804).
[0138] Specifically, the data generation unit 110 adds entries to the analytical text information 126 for the number of input texts, sets an ID in ID 1701 of each entry, and sets the specified type of summary target in target type 1704 of each entry. The data generation unit 110 also sets a pair of input text and output text in each entry.
[0139] The data generating unit 110 determines whether or not the processing has been completed for all types of abstracts to be summarized (step S805).
[0140] If the processing has not been completed for all types of abstracts, the data generating unit 110 returns to step S801.
[0141] When processing has been completed for all types of summary targets, the data generation unit 110 ends the analytical text generation process.
[0142] FIG. 22 is a flowchart illustrating an example of a parameter set generation process executed by the parameter search system 100 according to the second embodiment.
[0143] The parameter set generating unit 111 selects a type of abstraction target (step S901). It is assumed that the type of abstraction target is set in advance.
[0144] The parameter set generating unit 111 generates a parameter set (step S902). The process of step S902 is the same as the process of step S301.
[0145] Parameter set generation unit 111 registers the parameter set in analytical parameter set information 127 (step S903). The process of step S903 is the same as the process of step S302.
[0146] The parameter set generation unit 111 determines whether or not the process has been completed for all types of abstraction targets (step S904).
[0147] If the processing has not been completed for all types of abstraction targets, the parameter set generation unit 111 returns to step S901.
[0148] When the processing is completed for all types of abstract targets, the parameter set generation unit 111 ends the parameter set generation processing.
[0149] 23A, 23B, and 23C are flowcharts illustrating an example of correlation analysis processing executed by the parameter search system 100 of Example 2. The correlation analysis unit 116 executes the processing described below for each type of summary target.
[0150] The correlation analysis unit 116 sets a task for analysis (step S1001). It is assumed that information on the task for analysis has been input in advance.
[0151] The correlation analysis unit 116 refers to the generated AI information 120 and selects a paid generated AI (step S1002).
[0152] Correlation analysis unit 116 refers to analysis parameter set information 127 and selects a parameter set for the type of the object to be summarized (step S1003).
[0153] The correlation analysis unit 116 refers to the analytical text information 126 and selects the input text of the type to be summarized (step S1004).
[0154] The correlation analysis unit 116 executes the task using the selected paid generation AI and the selected input text (step S1005).
[0155] The correlation analysis unit 116 calculates an evaluation score by comparing the text generated by the task with the output text corresponding to the selected input text (step S1006). At this time, the correlation analysis unit 116 adds an entry to the evaluation information 124 and sets various information. The processing in step S1006 is the same as the processing in step S404.
[0156] The correlation analysis unit 116 determines whether or not the processing has been completed for all input texts (step S1007).
[0157] If processing has not been completed for all input texts, the correlation analysis unit 116 returns to step S1004.
[0158] When the process is completed for all input texts, the correlation analysis unit 116 calculates a comprehensive evaluation score (step S1008).
[0159] Specifically, the correlation analysis unit 116 calculates the total score as the sum or average of the evaluation scores of the tasks using each input text. At this time, the correlation analysis unit 116 stores second evaluation data in the work area, which associates the name of the paid generation AI, the parameter set, and the total evaluation score.
[0160] The correlation analysis unit 116 determines whether or not the process has been completed for all parameter sets (step S1009).
[0161] If the processing has not been completed for all parameter sets, the correlation analysis unit 116 returns to step S1003.
[0162] When the processing has been completed for all parameter sets, the correlation analysis unit 116 determines whether the processing has been completed for all paid generation AIs (step S1010).
[0163] If the processing has not been completed for all paid generation AIs, the correlation analysis unit 116 returns to step S1002.
[0164] When the processing for all the paid generated AIs has been completed, the correlation analysis unit 116 starts processing for the free generated AIs. First, the correlation analysis unit 116 refers to the generated AI information 120 and selects a free generated AI (step S1011).
[0165] The correlation analysis unit 116 refers to the analytical parameter set information 127 and selects a parameter set for the type of the object to be summarized (step S1012).
[0166] The correlation analysis unit 116 refers to the analytical text information 126 and selects the input text of the type to be summarized (step S1013).
[0167] The correlation analysis unit 116 executes the task using the selected free generation AI and the selected input text (step S1014).
[0168] The correlation analysis unit 116 calculates an evaluation score by comparing the text generated by the task with the output text corresponding to the selected input text (step S1015). The process of step S1015 is the same as the process of step S404.
[0169] The correlation analysis unit 116 determines whether or not the processing has been completed for all input texts (step S1016).
[0170] If the processing has not been completed for all input texts, the correlation analysis unit 116 returns to step S1013.
[0171] When the process has been completed for all input texts, the correlation analysis unit 116 calculates a comprehensive evaluation score (step S1017). The process of step S1017 is the same as the process of step S1008.
[0172] The correlation analysis unit 116 determines whether or not the process has been completed for all parameter sets (step S1018).
[0173] If the processing has not been completed for all parameter sets, the correlation analysis unit 116 returns to step S1012.
[0174] When the process has been completed for all parameter sets, the correlation analysis unit 116 determines whether the process has been completed for all free generated AIs (step S1019).
[0175] If the processing has not been completed for all paid generation AIs, the correlation analysis unit 116 returns to step S1011.
[0176] When the processing for all the paid generated AIs has been completed, the correlation analysis unit 116 references the generated AI information 120 to generate the correlation table 400 (step S1020).
[0177] The correlation analysis unit 116 refers to the generated AI information 120 and selects a paid generated AI (step S1021).
[0178] The correlation analysis unit 116 refers to the generated AI information 120 and selects a free generated AI (step S1022).
[0179] The correlation analysis unit 116 calculates the degree of correlation between the selected paid generation AI and the selected free generation AI (step S1023). Specifically, the following process is executed.
[0180] (S1023-1) The correlation analysis unit 116 acquires the second evaluation data of the paid generation AI and sorts it in descending order of overall evaluation score. The correlation analysis unit 116 generates a list of the parameter sets of the top five second evaluation data. Note that the list may be generated in step S1008.
[0181] (S1023-2) The correlation analysis unit 116 acquires the second evaluation data of the free generation AI and sorts it in descending order of overall evaluation score. The correlation analysis unit 116 generates a list of the top five parameter sets of the second evaluation data. Note that the list may be generated in step S1017.
[0182] (S1023-3) The correlation analysis unit 116 compares the two lists and calculates the degree of correlation. For example, if the lists contain the same parameter sets and the rankings match, the correlation score is set to "2." If the lists contain the same parameter sets but the rankings do not match, the correlation score is set to "0.5." The correlation analysis unit 116 calculates the degree of correlation using the following formula (1):
[0183]
number
[0184] Generative AIs with similar parameter sets that have high task evaluation results are thought to be correlated.
[0185] (S1023-4) The correlation analysis unit 116 sets the correlation degree in the corresponding cell of the correlation table 400.
[0186] This concludes the description of the process in step S1023.
[0187] Next, the correlation analysis unit 116 determines whether or not the processing has been completed for all free generated AIs (step S1024).
[0188] If the processing has not been completed for all free generated AIs, the correlation analysis unit 116 returns to step S1022.
[0189] When the processing has been completed for all the free generated AIs, the correlation analysis unit 116 determines whether the processing has been completed for all the paid generated AIs (step S1025).
[0190] If the processing has not been completed for all paid generation AIs, the correlation analysis unit 116 returns to step S1021.
[0191] When the processing is completed for all paid generation AIs, the correlation analysis unit 116 ends the correlation analysis processing.
[0192] The correlation analysis of the generation AI is a process for generating correlation information 121. The cost required for this process is borne by the operator of the parameter search system 100, and is not borne by the user of the parameter search system 100. Therefore, it does not affect the cost required for parameter search.
[0193] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments are provided to explain the present invention in detail, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.
[0194] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, solid-state drives (SSDs), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.
[0195] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, and Java (registered trademark).
[0196] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or storage medium.
[0197] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines are necessarily shown in the product. All components may be interconnected. [Explanation of symbols]
[0198] 100 Parameter Search System 102 terminals 103 Network 110 Data Generation Unit 111 Parameter set generation unit 112 Search Department 113 Verification Department 114 Display section 115 Generative AI classification section 116 Correlation Analysis Department 120 Generated AI information 121 Correlation Information 122 Search text information 123 Search parameter set information 124 Rating Information 125 Verification result information 126 Text information for analysis 127 Analysis parameter set information 200 computers 210 CPU 220 memory 230 Hard Disk Drive (HDD) 240 input devices 250 output devices 260 Network Devices 270 Bus 400 Correlation Table 1000, 1500 screens
Claims
1. 1. A computer system comprising: a computer having a processor, a storage device connected to the processor, and a network interface connected to the processor; accessibly connected to a text generation system that accepts prompts, which are text including instructions, and generates output text using a natural language processing model; The system holds correlation information for managing a correlation between the natural language processing model of a first category and the natural language processing model of a second category, the correlation information being classified based on costs incurred in using the natural language processing model, and first parameter set information for managing a first parameter set, which is a combination of parameters for controlling each of a plurality of processes of a target task consisting of a plurality of processes including one or more text generation processes using a target natural language processing model that is the natural language processing model of the first category, the cost of the natural language processing model of the first classification is greater than the cost of the natural language processing model of the second classification; The computer system identifying a verification natural language processing model correlated with the target natural language processing model from among the natural language processing models of the second classification with reference to the correlation information; executing the target task using the natural language processing model for verification and calculating a first evaluation index for evaluating an execution result of the target task while changing the first parameter set; A computer system that determines the first parameter set as candidates based on the first evaluation index.
2. 2. The computer system of claim 1, Identifying the first parameter set as a candidate based on a comparison result between the first evaluation index and a threshold value; If there is only one candidate first parameter set, generating display information for displaying the candidate first parameter set; A computer system characterized in that, when there are multiple candidate first parameter sets, the target task using the target natural language processing model is executed while changing the first parameter set, and display information is generated to display the first parameter set and the execution results of the target task.
3. 3. The computer system according to claim 2, retaining second parameter set information for managing second parameter sets, which are combinations of parameters for controlling each of a plurality of processes of an analysis task that is composed of a plurality of processes including one or more text generation processes using the natural language processing model; for each of the natural language processing models of the first classification, while changing the second parameter set, performing the analysis task using the natural language processing model of the first classification and calculating a second evaluation index for evaluating an execution result of the analysis task; generating a first list by extracting, for each of the natural language processing models of the first classification, the second parameter sets that provide good results in executing the analysis task based on the second evaluation index; For each of the natural language processing models of the second classification, performing the analysis task using the natural language processing model of the second classification and calculating the second evaluation index while changing the second parameter set; generating a second list by extracting, for each of the natural language processing models of the second classification, the second parameter sets that provide good execution results for the analysis task based on the second evaluation index; calculating a degree of agreement of the second parameter set as a correlation index based on the first list and the second list for each combination of the natural language processing model of the first classification and the natural language processing model of the second classification; A computer system characterized by generating the correlation information by associating a combination of the natural language processing model of the first classification and the natural language processing model of the second classification, and the correlation index.
4. 4. The computer system according to claim 3, A computer system characterized in that the parameter for controlling the text generation process is the content of the instruction in the prompt.
5. A parameter search support method executed by a computer system, comprising: The computer system a computer having a processor, a storage device connected to the processor, and a network interface connected to the processor; accessibly connected to a text generation system that accepts prompts, which are text including instructions, and generates output text using a natural language processing model; The system holds correlation information for managing a correlation between the natural language processing model of a first category and the natural language processing model of a second category, the correlation information being classified based on costs incurred in using the natural language processing model, and first parameter set information for managing a first parameter set, which is a combination of parameters for controlling each of a plurality of processes of a target task consisting of a plurality of processes including one or more text generation processes using a target natural language processing model that is the natural language processing model of the first category, the cost of the natural language processing model of the first classification is greater than the cost of the natural language processing model of the second classification; The parameter search support method includes: a first step in which the computer system refers to the correlation information and identifies a verification natural language processing model that is correlated with the target natural language processing model from among the natural language processing models of the second classification; a second step in which the computer system executes the target task using the natural language processing model for verification while changing the first parameter set, and calculates a first evaluation index for evaluating an execution result of the target task; a third step in which the computer system determines the first parameter set as candidates based on the first evaluation index.
6. 6. The parameter search support method according to claim 5, The third step includes: the computer system identifying the first parameter set as a candidate based on a comparison result between the first evaluation index and a threshold; generating, by the computer system, display information for displaying the first parameter set when there is only one candidate first parameter set; a step in which, when there are multiple candidate first parameter sets, the computer system executes the target task using the target natural language processing model while changing the first parameter set, and generates display information for displaying the first parameter set and the execution results of the target task.
7. 7. The parameter search support method according to claim 6, the computer system holds second parameter set information for managing second parameter sets, which are combinations of parameters for controlling each of a plurality of processes of an analytical task that is composed of a plurality of processes including one or more text generation processes using the natural language processing model; The parameter search support method includes: a step in which the computer system executes the analytical task using the natural language processing model of the first classification and calculates a second evaluation index for evaluating an execution result of the analytical task while changing the second parameter set for each of the natural language processing models of the first classification; generating a first list by extracting, for each of the natural language processing models of the first classification, the second parameter set that provides good results in executing the analysis task based on the second evaluation index; a step in which the computer system executes the analysis task using the natural language processing model of the second classification and calculates the second evaluation index while changing the second parameter set for each of the natural language processing models of the second classification; generating a second list by extracting, for each of the natural language processing models of the second classification, the second parameter sets that provide good execution results for the analysis task based on the second evaluation index; a step in which the computer system calculates, for each combination of the natural language processing model of the first classification and the natural language processing model of the second classification, a degree of agreement of the second parameter set as a correlation index based on the first list and the second list; a step in which the computer system generates the correlation information by associating a combination of the natural language processing model of the first classification and the natural language processing model of the second classification, and the correlation index.
8. 8. The parameter search support method according to claim 7, A parameter search support method, characterized in that the parameter for controlling the text generation process is the content of the instruction of the prompt.