Evaluation methods for computer systems and machine learning models

Abstract

This evaluates the accuracy of the generative AI (machine learning model) responses to various usage scenarios. [Solution] The computer system communicates with a machine learning model and receives evaluation instructions including information specifying the usage scenario of the machine learning model, input prompts, and expected answers. It generates multiple conversion prompts by changing the expression pattern of the input prompts, inputs each of the input prompts and multiple conversion prompts into the machine learning model to obtain multiple generated answers, compares each of the multiple generated answers with the expected answer to calculate a similarity score representing the similarity between the expected answer and the generated answer, and performs statistical processing using the similarity scores of the multiple generated answers to calculate an index for evaluating the accuracy of the machine learning model's answers in the usage scenario.

Description

Technical Field

[0001] The present invention relates to an evaluation technique for generative AI.

Background Art

[0002] The utilization of generative AI is progressing for various domains and various purposes. The accuracy of the answers (outputs) of generative AI for the usage scenarios such as the domains and purposes where generative AI is introduced is important.

[0003] As a technique for evaluating the accuracy of a machine learning model, the technique described in Patent Document 1 is known. Patent Document 1 describes "an abnormality degree calculation unit that calculates the abnormality degree of the partial time-series data based on the feature amount of the partial time-series data that is at least a part of the time-series data, and when the abnormality degree of the partial time-series data exceeds a threshold value, an abnormality sign determination unit that determines the partial time-series data as abnormality sign data, and calculates the reliability of the abnormality sign data based on the feature data corresponding to the feature amount of the abnormality sign data, and when the reliability is unacceptable, a normal determination unit that determines the abnormality sign data as normal data. The normal determination unit stores the feature data corresponding to the feature amount of the abnormality sign data in a feature data storage unit, and the normal determination unit stores the feature data corresponding to the abnormality sign data in the feature data layer, which is a layer corresponding to the range of the value of the feature amount of the abnormality sign data, in the feature data storage unit in association with each other. For each feature data layer, the cumulative number of abnormality sign data of the abnormality sign data corresponding to the feature data layer is stored as the feature data. The normal determination unit calculates the reliability of the abnormality sign data based on the ratio of the cumulative number of abnormality sign data corresponding to the abnormality sign data to the total number of the cumulative number of abnormality sign data."

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] The technology described in Patent Document 1 assumes that the training data is known. However, in Large Language Models (LLMs) used to implement generative AI, the training data is often unknown.

[0006] The present invention aims to realize a technology for evaluating the accuracy of responses generated by AI in various usage scenarios. [Means for solving the problem]

[0007] A typical example of the invention disclosed in this application is as follows: A computer system comprising a processor, a storage device connected to the processor, and a network interface connected to the processor, which is communicatively connected to a machine learning model that accepts prompts including the content of a task as input, executes the task defined by the prompt, and generates an answer; the processor accepts evaluation instructions including information specifying the usage scenario of the machine learning model, an input prompt, and an expected answer; generates a plurality of conversion prompts by changing the expression pattern of the input prompt; inputs each of the input prompts and the plurality of conversion prompts to the machine learning model to obtain a plurality of generated answers; compares each of the plurality of generated answers with the expected answer to calculate a similarity score representing the similarity between the expected answer and the generated answer; calculates an index for evaluating the accuracy of the machine learning model's answer in the usage scenario by performing statistical processing using the similarity scores of the plurality of generated answers; and records a first calculation result including the index associated with the information specifying the usage scenario in the storage device. [Effects of the Invention]

[0008] According to the present invention, the accuracy of the responses generated by the AI ​​in the usage scenario can be evaluated. Other issues, configurations, and effects will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the functional configuration of the evaluation system in Example 1. [Figure 2] This figure shows an example of the hardware configuration of the computer that constitutes the evaluation system of Example 1. [Figure 3] This figure shows an example of a screen displayed by the evaluation system in Example 1. [Figure 4A] This figure shows an example of the information managed by the prompt / answer storage unit of Example 1. [Figure 4B] This figure shows an example of the information managed by the prompt / answer storage unit of Example 1. [Figure 5] This figure shows an example of the information managed by the comparison result storage unit in Example 1. [Figure 6] This figure shows an example of the information managed by the proficiency memory unit of Example 1. [Figure 7] This is a flowchart illustrating the overview of the processes performed by the evaluation system in Example 1. [Figure 8] This flowchart illustrates an example of the prompt conversion process performed by the evaluation system in Example 1. [Figure 9A] This flowchart illustrates an example of the response comparison process performed by the evaluation system in Example 1. [Figure 9B] This flowchart illustrates an example of the response comparison process performed by the evaluation system in Example 1. [Figure 10] This figure shows an example of a method for calculating similarity in the response comparison process of Example 1. [Figure 11] This flowchart illustrates an example of the proficiency determination process performed by the evaluation system in Example 1. [Figure 12] This figure shows an example of a screen displayed by the evaluation system in Example 1. [Figure 13] It is a diagram showing an example of the functional configuration of the evaluation system of Example 2. [Figure 14] It is a diagram showing an example of the screen displayed by the evaluation system of Example 2. [Figure 15A] It is a diagram showing an example of the information managed by the prompt / answer storage unit of Example 2. [Figure 15B] It is a diagram showing an example of the information managed by the prompt / answer storage unit of Example 2. [Figure 16] It is a diagram showing an example of the information managed by the comparison result storage unit of Example 2. [Figure 17] It is a diagram showing an example of the information managed by the proficiency storage unit of Example 2. [Figure 18] It is a diagram showing an example of the information managed by the parameter storage unit of Example 2. [Figure 19] It is a diagram showing an example of the information managed by the fitness storage unit of Example 2. [Figure 20] It is a flowchart explaining the outline of the process executed by the evaluation system of Example 2. [Figure 21A] It is a flowchart explaining an example of the answer comparison process executed by the evaluation system of Example 2. [Figure 21B] It is a flowchart explaining an example of the answer comparison process executed by the evaluation system of Example 2. [Figure 22] It is a diagram showing an example of the method for calculating the similarity score in the answer comparison process of Example 2. [Figure 23A] It is a flowchart explaining an example of the proficiency determination process executed by the evaluation system of Example 2. [Figure 23B] It is a flowchart explaining an example of the proficiency determination process executed by the evaluation system of Example 2. [Figure 24] It is a flowchart explaining an example of the fitness determination process executed by the evaluation system of Example 2. [Figure 25] It is a diagram showing an example of the screen displayed by the evaluation system of Example 2. [Figure 26]This figure shows an example of a screen displayed by the evaluation system in Example 2. [Modes for carrying out the invention]

[0010] The embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not to be construed as being limited to the embodiments described below. It will be readily apparent to those skilled in the art that the specific configuration can be modified without departing from the spirit or intent of the present invention.

[0011] In the configuration of the invention described below, identical or similar components or functions are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] The designations "First," "Second," "Third," etc., used in this specification are for the purpose of identifying constituent elements and do not necessarily limit their number or order. [Examples]

[0013] Figure 1 shows an example of the functional configuration of the evaluation system 100 of Example 1. Figure 2 shows an example of the hardware configuration of the computer that constitutes the evaluation system 100 of Example 1.

[0014] The evaluation system 100 evaluates the accuracy of the generative AI's responses in a given usage scenario. Here, the usage scenario is a concept that includes the domain in which the generative AI is used and the purpose for which the generative AI is used. The evaluation system 100 connects to the natural language processing service 101 via a network (not shown).

[0015] The natural language processing service 101 is a service provided by a system that implements generative AI. This system provides the service using an LLM. The evaluation system 100 may maintain multiple LLMs. The LLM receives prompts containing task content, such as questions written in natural language, understands the task content, and generates and outputs text that serves as the answer by executing the task. The accuracy of the generative AI's answer is equivalent to the accuracy of the LLM's answer.

[0016] The evaluation system 100 is a system implemented using a computer 200 as shown in Figure 2. The computer 200 has a processor 201, main memory 202, secondary memory 203, and a network interface 204. Each hardware element is connected via a bus 205.

[0017] The processor 201 executes the program stored in the main memory 202. By executing processing according to the program, the processor 201 operates as a functional unit (module) that realizes a specific function. In the following description, when the processing is described with a functional unit as the subject, it indicates that the processor 201 is executing the program that realizes that functional unit.

[0018] Main memory 202 is memory, etc., and stores programs executed by the processor 201 and information used by those programs. Main memory 202 is also used as a work area. Secondary memory 203 is a large-capacity storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive). Programs and information stored in main memory 202 may also be stored in secondary memory 203. In this case, the processor 201 reads the programs and information from secondary memory 203 and loads them into main memory 202. Network interface 204 is an interface for connecting to a network.

[0019] The evaluation system 100 includes an input unit 110, a prompt conversion unit 111, an answer comparison unit 112, an answer generation unit 113, a proficiency determination unit 114, a display unit 115, a prompt / answer storage unit 120, a comparison result storage unit 121, and a proficiency storage unit 122.

[0020] The prompt / answer storage unit 120 stores combinations of prompts and expected answers. The expected answers are ideal answers output from the LLM. The comparison result storage unit 121 stores the comparison result between the expected answers and the answers output by the LLM (generated answers). The proficiency storage unit 122 stores the proficiency level, which represents the accuracy of the LLM's answers in the usage scenario. If there is a large difference between the expected answers and the generated answers, it indicates that the LLM has not learned enough for the usage scenario and that its proficiency level is low.

[0021] The input unit 110 accepts input of various types of information. The display unit 115 displays various types of information. The display unit 115 also provides various interfaces.

[0022] The prompt conversion unit 111 generates a prompt by changing the expression pattern of the input prompt. Here, the expression pattern is a concept that includes language and grammar (style, word order, etc.). The answer comparison unit 112 compares the expected answer with the generated answer. The answer generation unit 113 sends the prompt to the natural language processing service 101 and obtains the answer. The proficiency determination unit 114 determines the proficiency of the LLM based on the comparison results.

[0023] Furthermore, regarding the functional units of the evaluation system 100, multiple functional units may be combined into a single functional unit, or a single functional unit may be divided into multiple functional units according to its function.

[0024] Figure 3 shows an example of a screen displayed by the evaluation system 100 of Example 1.

[0025] The display unit 115 of the evaluation system 100 displays screen 300 on a terminal or the like operated by the user. Screen 300 is an interface for setting prompts and expected answers, and includes input fields 301, selection fields 302, input fields 303, and operation buttons 304.

[0026] Input field 301 is for entering the domain in which the generation AI will be introduced, i.e., the usage scenario for the generation AI. Selection field 302 is for selecting the LLM to be evaluated. Input field 303 is for entering prompts and expected answers. Multiple combinations of prompts and expected answers can be entered in input field 303. Operation button 304 is an operation button for outputting evaluation instructions.

[0027] When the user operates the operation button 304, an evaluation instruction including the domain, a list of LLMs, and a combination of prompts and expected answers is output to the evaluation system 100.

[0028] Figures 4A and 4B show examples of information managed by the prompt / answer storage unit 120 of Embodiment 1.

[0029] The prompt / answer storage unit 120 manages, for example, tables 400 and 410.

[0030] Table 400 is a table for managing metadata of the input prompt and expected response combinations. Table 400 stores entries that include problem ID 401 and domain 402.

[0031] Problem ID 401 is a field that stores an identifier for identifying the combination of the input prompt and the expected answer. Domain 402 is a field that stores the domain in which the generation AI will be implemented.

[0032] Table 410 is a table for managing combinations of prompts and expected answers. Table 410 stores entries that include the problem ID 411, the expression pattern 412, the prompt 413, and the expected answer 414.

[0033] Problem ID 411 is the same field as Problem ID 401. Expression Pattern 412 is a field that stores the expression pattern of the prompt. Prompt 413 is a field that stores the prompt. Expected Answer 414 is a field that stores the expected answer.

[0034] The generation AI will produce different answers depending on the prompt's expression pattern, even for the same task. Therefore, in this embodiment, multiple prompts with different expression patterns are generated from the input prompt, and the accuracy of the generation AI's answers is evaluated while taking into account the variability in responses.

[0035] Tables 400 and 410 may be combined into a single table. Furthermore, the prompt / answer storage unit 120 may manage various types of information in a data format other than a table format.

[0036] Figure 5 shows an example of the information managed by the comparison result storage unit 121 in Example 1.

[0037] The comparison result storage unit 121 manages, for example, a table 500. The table 500 stores entries including ID 501, problem ID 502, representation pattern 503, model 504, generated response 505, similarity 506, and date and time 507.

[0038] ID501 is a field that stores an identifier to identify the comparison result. Problem ID502 is the same field as Problem ID401. Representation pattern 503 is the same field as representation pattern 412.

[0039] Model 504 is a field that stores the LLM's identification information. Generated response 505 is a field that stores the generated response output by the LLM. Similarity 506 is a field that stores the similarity score, which represents the similarity between the expected response and the generated response. Date and time 507 is a field that stores the date and time the comparison was performed.

[0040] The comparison result storage unit 121 may also manage various types of information in a data format other than a table format.

[0041] Figure 6 shows an example of the information managed by the proficiency memory unit 122 of Embodiment 1.

[0042] The proficiency memory unit 122 manages, for example, table 600. Table 600 stores entries including ID 601, model 602, domain 603, similarity score 604, variation score 605, and date and time 606.

[0043] ID601 is a field that stores identification information for the LLM proficiency assessment result. Model602 is a field that stores LLM identification information. Domain603 is a field that stores the domain.

[0044] The similarity score 604 is a field that stores the similarity score representing the similarity between the expected answer and the generated answer in the usage scenario. The variability score 605 is a field that stores the variability score representing the degree of variation in the generated answer in the usage scenario. The similarity score and variability score are examples of indicators that represent proficiency. The date and time 606 is a field that stores the date and time when the LLM proficiency level was determined.

[0045] The proficiency level storage unit 122 may also manage various types of information in a data format other than a table format.

[0046] Figure 7 is a flowchart illustrating the overview of the processes performed by the evaluation system 100 in Example 1.

[0047] When the evaluation system 100 receives an evaluation request from a user, it displays screen 300 and accepts the evaluation instructions (step S101).

[0048] The display unit 115 displays screen 300 on a terminal operated by the user, for example. The input unit 110 receives evaluation instructions via screen 300. The input unit 110 outputs the domains, LLM list, and combinations of prompts and expected answers included in the evaluation instructions to the prompt conversion unit 111.

[0049] The prompt conversion unit 111 adds entries to the table 400 equal to the number of prompt and expected answer combinations, and sets identification information in the problem ID 401 of the added entries. The prompt conversion unit 111 also sets the domain 402 of the added entries to the domain included in the evaluation instruction.

[0050] The prompt conversion unit 111 adds entries to the table 410 equal to the number of prompt and expected answer combinations, and sets identification information in the problem ID 411 of the added entries. The prompt conversion unit 111 also sets the prompt and expected answer in the prompt 413 and expected answer 414 of the added entries.

[0051] The prompt conversion unit 111 identifies the expression pattern of each prompt and sets the identified expression pattern to the expression pattern 412 of the corresponding entry. Possible methods for identifying the prompt expression pattern include using pre-configured rules and using LLM. A field for inputting the expression pattern may also be provided on screen 300.

[0052] The evaluation system 100 performs a prompt conversion process (step S102). Details of the prompt conversion process will be described later.

[0053] The evaluation system 100 performs a response comparison process (step S103). Details of the response comparison process will be described later.

[0054] The evaluation system 100 performs a proficiency assessment process (step S104). Details of the proficiency assessment process will be described later.

[0055] Figure 8 is a flowchart illustrating an example of the prompt conversion process performed by the evaluation system 100 of Example 1.

[0056] The prompt conversion unit 111 refers to table 400 and selects a problem ID (step S201).

[0057] The prompt conversion unit 111 obtains a combination of prompt and expected answer corresponding to the selected problem ID from the table 410 (step S202).

[0058] The prompt conversion unit 111 generates a list of expression patterns for the problem ID (step S203).

[0059] In this embodiment, it is assumed that a definition list containing predefined expression patterns is set in advance. The prompt conversion unit 111 obtains an expression pattern from the expression pattern 412 of the entry corresponding to the selected problem ID in table 410. The prompt conversion unit 111 generates an expression pattern list by excluding the expression pattern obtained from the definition list.

[0060] The prompt conversion unit 111 selects an expression pattern from the expression pattern list (step S204).

[0061] The prompt conversion unit 111 converts the acquired prompt into a selected expression pattern (step S205). In this embodiment, in the case of grammatical conversion, only the prompt is changed, while in the case of language conversion, both the prompt and the expected answer are converted.

[0062] The prompt conversion unit 111 registers the prompt whose expression pattern has been converted in the table 410 (step S206).

[0063] Specifically, the prompt conversion unit 111 adds an entry to the table 410, sets the selected problem ID in the entry's problem ID 411, and sets the selected expression pattern in the expression pattern 412. The prompt conversion unit 111 sets the prompt 413 of the added entry to the prompt with the converted expression pattern. The prompt conversion unit 111 sets the expected answer 414 of the added entry to the expected answer. In the case of grammar conversion, the obtained expected answer is set as is in expected answer 414, and in the case of language conversion, the expected answer 414 is set to the expected answer with the language converted.

[0064] The prompt conversion unit 111 determines whether processing has been completed for all expression patterns in the expression pattern list (step S207).

[0065] If processing has not been completed for all expression patterns in the expression pattern list, the prompt conversion unit 111 returns to step S204.

[0066] When processing is complete for all expression patterns in the expression pattern list, the prompt conversion unit 111 determines whether processing has been completed for all problem IDs (step S208).

[0067] If processing is not complete for all problem IDs, the prompt conversion unit 111 returns to step S201.

[0068] Once processing is complete for all problem IDs, the prompt conversion unit 111 terminates the prompt conversion process.

[0069] Figures 9A and 9B are flowcharts illustrating an example of the response comparison process performed by the evaluation system 100 in Example 1. Figure 10 shows an example of the method for calculating similarity in the response comparison process of Example 1.

[0070] The response comparison unit 112 selects an LLM from the list of LLMs included in the evaluation instructions (step S301).

[0071] The answer comparison unit 112 refers to table 400 and selects a problem ID (step S302).

[0072] The answer comparison unit 112 refers to table 410 and selects one combination from the prompt and expected answer combinations corresponding to the selected problem ID (step S303). At this time, the answer comparison unit 112 sets the evaluation count to 1.

[0073] The response comparison unit 112 obtains a generated response by sending the selected LLM information and the selected combination prompt to the response generation unit 113 (step S304).

[0074] The response generation unit 113 inputs a prompt to the natural language processing service 101, which provides a service using the selected LLM, and obtains a generated response. The response generation unit 113 then sends the generated response to the response comparison unit 112.

[0075] The response comparison unit 112 identifies the response items of the expected response and the response items of the generated response for the selected combination (step S305).

[0076] In this embodiment, the answer includes multiple answer items. For example, if the expected answer includes answer items in CSV format, the answer comparison unit 112 obtains the comma-separated answer items. Alternatively, answer items can be obtained from the generated answer by using a prompt that instructs the generation of the generated answer in CSV format. Furthermore, LLM may be used to obtain the answer items of both the expected answer and the generated answer.

[0077] The answer comparison unit 112 generates pairs of answer items for the expected answers and answer items for the generated answers (step S306).

[0078] The response comparison unit 112 selects a pair (step S307) and calculates the similarity (individual similarity) between the response items that make up the pair (step S308). Since the response items are strings, the similarity between strings can be calculated as the similarity between response items using known techniques such as Word2Vec. The response comparison unit 112 stores the pairs and their similarities in association.

[0079] The answer comparison unit 112 determines whether processing has been completed for all pairs (step S309).

[0080] If processing is not complete for all pairs, the answer comparison unit 112 returns to step S307.

[0081] Once processing is complete for all pairs, the response comparison unit 112 identifies the maximum individual similarity for each response item of the expected response based on the individual similarity of all pairs (step S310).

[0082] Here, the process of step S310 will be explained using Figure 10. In the matrix 1000 shown in Figure 10, the columns correspond to the answer items of the expected answers, and the rows correspond to the answer items of the generated answers. The cells store the individual similarity score corresponding to the pair of answer items. In step S310, the answer comparison unit 112 identifies the maximum value of the individual similarity score for each column.

[0083] The response comparison unit 112 calculates a similarity score representing the similarity between the expected response and the generated response using the maximum value of the individual similarity score for each response item of the expected response (step S311).

[0084] Specifically, the response comparison unit 112 calculates the similarity score as the average of the maximum individual similarity scores for each response item of the expected responses. Note that the calculation method described above is just one example and is not limited to it. For example, the similarity between the expected responses and the generated responses may be calculated using LLM.

[0085] The response comparison unit 112 records the calculation result (step S312). At this time, the response comparison unit 112 increments the evaluation count by 1.

[0086] Specifically, the response comparison unit 112 adds an entry to table 500 and sets identification information in ID 501 of the added entry. The response comparison unit 112 sets the selected problem ID in problem ID 502 of the added entry, sets the expression pattern of the acquired prompt in expression pattern 503, and sets the identification information of the selected LLM in model 504. The response comparison unit 112 sets the acquired generated answer in generated answer 505 of the added entry and sets the calculated similarity in similarity 506. The response comparison unit 112 sets the start date and time of the response comparison process in date and time 507 of the added entry.

[0087] The response comparison unit 112 determines whether the number of evaluations is greater than the threshold (step S313). The threshold is assumed to be set in advance. Note that the threshold can be changed as appropriate.

[0088] In this way, the response comparison unit 112 performs the evaluation of the similarity between the expected response and the generated response multiple times using the same prompt. Since the AI's response may differ even with the same prompt, the accuracy of the evaluation is ensured by performing the evaluation multiple times.

[0089] If the number of evaluations is below the threshold, the response comparison unit 112 returns to step S304.

[0090] If the number of evaluations is greater than the threshold, the answer comparison unit 112 determines whether processing has been completed for all prompt and expected answer combinations corresponding to the selected problem ID (step S314).

[0091] If processing has not been completed for all prompt and expected answer combinations corresponding to the selected problem ID, the answer comparison unit 112 returns to step S303.

[0092] In this embodiment, the response comparison unit 112 performs the evaluation of the similarity between the expected response and the generated response multiple times using prompts with the same task content but different expression patterns. Even if the task content is the same, the generated AI's response may differ if the prompt's expression pattern is different. Therefore, the accuracy of the evaluation is ensured by performing the evaluation multiple times using prompts with different expression patterns.

[0093] When processing is complete for all prompt and expected answer combinations corresponding to the selected problem ID, the answer comparison unit 112 determines whether processing has been completed for all problem IDs (step S315).

[0094] If processing is not complete for all problem IDs, the answer comparison unit 112 returns to step S302.

[0095] If processing is completed for all problem IDs, the answer comparison unit 112 determines whether processing has been completed for all LLMs in the LLM list included in the evaluation instruction (step S316).

[0096] If processing has not been completed for all LLMs in the LLM list, the answer comparison unit 112 returns to step S301.

[0097] When processing is complete for all LLMs in the LLM list, the answer comparison unit 112 terminates the answer comparison process.

[0098] Figure 11 is a flowchart illustrating an example of the proficiency determination process performed by the evaluation system 100 of Example 1.

[0099] The proficiency assessment unit 114 selects an LLM from the list of LLMs included in the evaluation instructions (step S401).

[0100] The proficiency assessment unit 114 refers to table 400 and selects a problem ID (step S402).

[0101] The proficiency assessment unit 114 calculates the similarity score of the problem ID by referring to the table 500 (step S403).

[0102] Specifically, the proficiency determination unit 114 refers to table 500 and searches for entries in which the identification information of the selected LLM is set in model 504 and the selected problem ID is set in problem ID 502. The proficiency determination unit 114 obtains the similarity stored in similarity 506 of the searched entries and calculates the average similarity score as the similarity score for the problem ID.

[0103] The proficiency assessment unit 114 calculates the variation score for the problem ID by referring to the table 500 (step S404).

[0104] The proficiency assessment unit 114 calculates the standard deviation and mean value of the similarity obtained in step S403. The proficiency assessment unit 114 calculates the variation score of the problem ID by dividing the standard deviation of the similarity by the mean value of the similarity. In this embodiment, the greater the variation in the generated answers, the larger the variation score.

[0105] The proficiency determination unit 114 determines whether processing has been completed for all problem IDs (step S405).

[0106] If processing is not complete for all problem IDs, the proficiency determination unit 114 returns to step S402.

[0107] Once processing is complete for all problem IDs, the proficiency determination unit 114 selects a domain (step S406).

[0108] The proficiency assessment unit 114 selects one domain from the list of domains. Note that when step S406 is executed for the first time, the proficiency assessment unit 114 generates a list of domains using table 400.

[0109] The proficiency determination unit 114 calculates a similarity score for the usage scenario (step S407). In this embodiment, the domain is information that specifies the usage scenario.

[0110] Specifically, the proficiency assessment unit 114 refers to table 400 to identify problem IDs corresponding to the selected domain. The proficiency assessment unit 114 calculates the average of the similarity scores of the identified problem IDs as the similarity score for the usage scenario (domain).

[0111] The proficiency assessment unit 114 calculates the variation score in the usage scenario (step S408).

[0112] Specifically, the proficiency assessment unit 114 refers to table 400 to identify problem IDs corresponding to the selected domain. The proficiency assessment unit 114 calculates the average of the variation scores of the identified problem IDs as the variation score for the usage scenario (domain).

[0113] The proficiency determination unit 114 records the calculation result (step S409).

[0114] Specifically, the proficiency determination unit 114 adds an entry to table 600 and sets identification information in ID 601 of the added entry. The proficiency determination unit 114 sets the identification information of the selected LLM in model 602 of the added entry and sets the selected domain in domain 603. The proficiency determination unit 114 sets the calculated similarity score 604 and variation score 605 in similarity score 604 and variation score 605 of the added entry. The proficiency determination unit 114 sets the start date and time of the response comparison process in date and time 606 of the added entry.

[0115] The proficiency determination unit 114 determines whether processing has been completed for all domains (step S410).

[0116] If processing is not complete for all domains, the proficiency determination unit 114 returns to step S406.

[0117] If processing is completed for all domains, the proficiency determination unit 114 determines whether processing has been completed for all LLMs in the list of LLMs included in the evaluation instruction (step S411).

[0118] If processing has not been completed for all LLMs in the LLM list, the proficiency determination unit 114 returns to step S401.

[0119] When processing is complete for all LLMs in the LLM list, the proficiency determination unit 114 terminates the proficiency determination process.

[0120] Figure 12 shows an example of a screen displayed by the evaluation system 100 of Example 1.

[0121] When the display unit 115 of the evaluation system 100 receives a request from a user to display the LLM proficiency level in a specific domain, it displays screen 1200 on the terminal or other device operated by the user. Screen 1200 includes a display field 1201 and a table 1202.

[0122] Display field 1201 displays the scores of each LLM in the specified domain in graph format. Table 1202 is a table that displays the details of each LLM score in the specified domain. The order of LLMs in the graph and table can be changed based on the release date and usage price. Also, even for the same LLM, accuracy improves with version upgrades. Therefore, it is possible to perform evaluations periodically and display the history in chronological order.

[0123] As described above, the evaluation system 100 of Example 1 makes it possible to evaluate the accuracy of the generated AI's responses in usage scenarios, taking into account the output characteristics of the generated AI, by comparing the generated responses obtained by inputting prompts with different expression patterns with the expected responses, and by comparing the generated responses obtained by inputting the same prompt multiple times with the expected responses. [Examples]

[0124] In Example 2, the score calculated as proficiency differs from that in Example 1. Furthermore, Example 2 differs from Example 1 in that it evaluates the accuracy of the generated AI's responses for each purpose (application) and social impact. The following describes Example 2, focusing on the differences from Example 1.

[0125] Figure 13 shows an example of the functional configuration of the evaluation system 100 in Example 2.

[0126] The evaluation system 100 of Example 2 differs from that of Example 1 in that it includes a fitness determination unit 116, a parameter storage unit 123, and a fitness storage unit 124. The hardware configuration of the evaluation system 100 is the same as that of Example 1.

[0127] Figure 14 shows an example of a screen displayed by the evaluation system 100 of Example 2.

[0128] The display unit 115 of the evaluation system 100 displays screen 1400 on a terminal or the like operated by the user. Screen 1400 is an interface for setting prompts and expected answers, and includes selection fields 1401, 1402, input fields 1403, 1404, 1405, 1406, and operation buttons 1407.

[0129] Selection field 1401 is for selecting the industry in which the generation AI will be introduced. Selection field 1402 is for selecting the field in which the generation AI will be introduced. In Example 2, the combination of industry and field becomes the domain. Input field 1403 is for selecting the purpose of introducing the generation AI. Selection field 1404 is for selecting the magnitude of the social impact that the generation AI's responses will have. Selection field 1405 is for selecting the LLM to be evaluated. Input field 1406 is for entering prompts and expected answers. Multiple combinations of prompts and expected answers can be entered in input field 1406. Operation button 1407 is an operation button for outputting evaluation instructions.

[0130] When the user operates the operation button 1407, evaluation instructions including the industry, field, purpose of implementation, magnitude of social impact, list of LLMs, and combinations of prompts and expected answers are output to the evaluation system 100.

[0131] Figures 15A and 15B show an example of the information managed by the prompt / answer storage unit 120 in Embodiment 2.

[0132] The prompt / answer storage unit 120 manages, for example, table 1500 and table 1510.

[0133] Table 1500 is a table for managing metadata of input prompt and expected response combinations. Table 1500 stores entries including Problem ID 1501, Domain 1502, Objective 1503, and Severity 1504.

[0134] Problem ID 1501 and Domain 1502 are the same fields as Problem ID 401 and Domain 402. In Example 2, Domain 1502 is set to include the industry and field.

[0135] Objective 1503 is a field that stores the purpose of introducing the generation AI. Importance 1504 is a field that stores the magnitude of the social impact.

[0136] Table 1510 is a table for managing combinations of prompts and expected answers. Table 1510 stores entries that include problem ID 1511, expression pattern 1512, prompt 1513, and expected answer 1514.

[0137] Problem ID 1511, expression pattern 1512, prompt 1513, and expected answer 1514 are the same fields as Problem ID 411, expression pattern 412, prompt 413, and expected answer 414.

[0138] Tables 1500 and 1510 may be treated as a single table. Furthermore, the prompt / answer storage unit 120 may manage various types of information in a data format other than a table format.

[0139] Figure 16 shows an example of the information managed by the comparison result storage unit 121 in Example 2.

[0140] The comparison result storage unit 121 manages, for example, table 1600. Table 1600 stores entries including ID 1601, problem ID 1602, representation pattern 1603, model 1604, generated answer 1605, similarity 1606, error rate 1607, missing answer rate 1608, and date and time 1609.

[0141] ID1601, Problem ID1602, Representation Pattern1603, Model1604, Generated Response1605, Similarity1606, and Date / Time1609 are the same fields as ID501, Problem ID502, Representation Pattern503, Model504, Generated Response505, Similarity506, and Date / Time507.

[0142] The error rate (1607) is a field that stores an indicator representing the degree of error in the generated response compared to the expected response. The missing response rate (1608) is a field that stores an indicator representing the degree of missing responses in the generated response compared to the expected response.

[0143] The comparison result storage unit 121 may also manage various types of information in a data format other than a table format.

[0144] Figure 17 shows an example of the information managed by the proficiency memory unit 122 of Embodiment 2.

[0145] The proficiency memory unit 122 manages, for example, table 1700. Table 1700 stores entries including ID 1701, model 1702, domain 1703, purpose 1704, importance 1705, similarity score 1706, variation score 1707, incorrect answer score 1708, missing answer score 1709, and date and time 1710.

[0146] ID1701, Model1702, Domain1703, Similarity Score1706, Variation Score1707, and Date / Time1710 are the same fields as ID601, Model602, Domain603, Similarity Score604, Variation Score605, and Date / Time606.

[0147] Objective 1704 is a field that stores the purpose of introducing the generation AI. Importance 1705 is a field that stores the magnitude of the social impact.

[0148] The error response score 1708 is a field that stores the error response score, which represents the degree of error in the generated response in the usage scenario. The missing response score 1709 is a field that stores the missing response score, which represents the degree of missing responses in the generated response in the usage scenario.

[0149] Similarity score, variation score, incorrect response score, and missing response score are examples of indicators that represent proficiency.

[0150] The proficiency level storage unit 122 may also manage various types of information in a data format other than a table format.

[0151] Figure 18 shows an example of the information managed by the parameter storage unit 123 in Embodiment 2.

[0152] The parameter storage unit 123 manages, for example, table 1800. Table 1800 stores entries including purpose 1801, importance 1802, parameter (1) 1803, parameter (2) 1804, and parameter (3) 1805.

[0153] Objective 1801 is a field that stores the purpose of introducing the generation AI. Importance 1802 is a field that stores the magnitude of the social impact.

[0154] Parameter (1) 1803, parameter (2) 1804, and parameter (3) 1805 are fields that store parameters used by the fitness determination unit 116.

[0155] The parameter storage unit 123 may manage various types of information in a data format other than a table format.

[0156] Figure 19 shows an example of the information managed by the fitness storage unit 124 of Embodiment 2.

[0157] The fitness storage unit 124 manages, for example, table 1900. Table 1900 stores entries including model 1901, purpose 1902, importance 1903, and fitness score 1904.

[0158] Model 1901 is a field that stores the identification information of the LLM. Purpose 1902 is a field that stores the purpose of introducing the generative AI. Importance 1903 is a field that stores the magnitude of the social impact. Fit 1904 is a field that stores the fit score, which represents the degree to which the generative AI fits the purpose.

[0159] The fitness storage unit 124 may manage various types of information in a data format other than a table format.

[0160] Figure 20 is a flowchart illustrating the overview of the processes performed by the evaluation system 100 in Example 2.

[0161] In Example 2, the evaluation system 100 performs a proficiency determination process, and then performs a suitability determination process (step S151).

[0162] The prompt conversion process in Example 2 is the same as in Example 1, so its explanation is omitted. In Example 2, the answer comparison process and the proficiency determination process differ in some respects from Example 1.

[0163] Figures 21A and 21B are flowcharts illustrating an example of the response comparison process performed by the evaluation system 100 in Example 2. Figure 22 shows an example of the method for calculating the similarity score in the response comparison process of Example 2.

[0164] The response comparison unit 112 calculates the similarity and then calculates the response omission rate (step S351). Specifically, the following processes are performed.

[0165] (S351-1) The answer comparison unit 112 determines whether there are pairs with a similarity greater than a threshold for each answer item of the expected answers. That is, for each column of the matrix 2200 shown in Figure 22, it is determined whether there are cells with a similarity greater than a threshold. The threshold is, for example, 80%. Note that a threshold may be set for each domain. In Figure 22, there are no pairs with a similarity greater than the threshold for the answer item "My Number Utilization Administration System".

[0166] In the following explanation, response items for which no pairs with a similarity greater than the threshold exist are referred to as missing response items.

[0167] (S351-2) The response comparison unit 112 calculates the response omission rate by dividing the number of missing response items by the number of expected response items. In this embodiment, a smaller response omission rate indicates fewer missed responses.

[0168] The response comparison unit 112 calculates the error rate (step S352). Specifically, the following processes are performed.

[0169] (S352-1) The response comparison unit 112 determines whether there are pairs with a similarity greater than a threshold for each response item in the generated responses. That is, for each row of the matrix 2200 shown in Figure 22, it is determined whether there are cells with a similarity greater than a threshold. The threshold is, for example, 80%. Note that a threshold may be set for each domain. In Figure 22, there are no pairs with a similarity greater than the threshold for the response item "System Network".

[0170] In the following explanation, a generated response item for which no pairs with a similarity score greater than the threshold exist will be referred to as an incorrect response item.

[0171] (S352-2) The answer comparison unit 112 calculates the error rate by dividing the number of incorrect answers by the number of expected answers in the generated responses. In this embodiment, a smaller error rate indicates fewer errors in the answers.

[0172] The response comparison unit 112 records the calculation result (step S353). At this time, the response comparison unit 112 increments the evaluation count by 1.

[0173] Specifically, the response comparison unit 112 adds an entry to table 1600 and sets the identification information in ID 1601 of the added entry. The response comparison unit 112 sets the selected problem ID in problem ID 1602 of the added entry, sets the expression pattern of the acquired prompt in expression pattern 1603, and sets the identification information of the selected LLM in model 1604. The response comparison unit 112 sets the acquired generated answer in generated answer 1605 of the added entry, and sets the integrated similarity 1606, incorrect answer rate 1607, and missing answer rate 1608. The response comparison unit 112 sets the start date and time of the response comparison process in date and time 1609 of the added entry.

[0174] Figures 23A and 23B are flowcharts illustrating an example of the proficiency determination process performed by the evaluation system 100 in Example 2.

[0175] The proficiency determination unit 114 refers to the table 1500 to generate combinations of domain, importance, and objective (step S451). The proficiency determination unit 114 then proceeds to step S401.

[0176] The proficiency assessment unit 114 calculates the variation score for the problem ID, and then calculates the score for missing answers for the problem ID (step S452).

[0177] Specifically, the proficiency assessment unit 114 refers to table 1600 and searches for entries in which the selected LLM identification information is set in model 1604 and the selected problem ID is set in problem ID 1602. The proficiency assessment unit 114 calculates the average of the answer omission rates stored in answer omission rate 1608 for the searched entries as the answer omission score for the problem ID.

[0178] The proficiency assessment unit 114 calculates the incorrect answer score for the problem ID (step S453).

[0179] Specifically, the proficiency assessment unit 114 refers to table 1600 and searches for entries in which the identification information of the selected LLM is set in model 1604 and the selected problem ID is set in problem ID 1602. The proficiency assessment unit 114 calculates the average of the error rates stored in error rate 1607 for the searched entries as the error score for the problem ID.

[0180] Once processing is complete for all problem IDs, the proficiency determination unit 114 selects a combination of domain, importance, and objective (step S454).

[0181] The proficiency assessment unit 114 calculates a similarity score for the usage scenario (step S455). In this embodiment, the combination of domain, importance, and purpose is the information that specifies the usage scenario.

[0182] Specifically, the proficiency assessment unit 114 refers to table 1500 to identify problem IDs corresponding to the selected combination. The proficiency assessment unit 114 calculates the average of the similarity scores of the identified problem IDs as the similarity score in the usage scenario.

[0183] The proficiency assessment unit 114 calculates the variation score in the usage scenario (step S456).

[0184] Specifically, the proficiency assessment unit 114 refers to table 1500 to identify problem IDs corresponding to the selected combination. The proficiency assessment unit 114 calculates the average of the variation scores of the identified problem IDs as the variation score in the usage scenario.

[0185] The proficiency assessment unit 114 calculates the score for missed answers in the usage scenario (step S457).

[0186] Specifically, the proficiency assessment unit 114 refers to table 1500 to identify the problem ID corresponding to the selected combination. The proficiency assessment unit 114 calculates the average of the missing answer scores for the identified problem IDs as the missing answer score in the usage scenario.

[0187] The proficiency assessment unit 114 calculates the score for incorrect answers in the usage scenario (step S458).

[0188] Specifically, the proficiency assessment unit 114 refers to table 1500 to identify the problem ID corresponding to the selected combination. The proficiency assessment unit 114 calculates the average of the incorrect answer scores for the identified problem IDs as the incorrect answer score in the usage scenario.

[0189] The proficiency assessment unit 114 records the calculation result (step S459).

[0190] Specifically, the proficiency assessment unit 114 adds an entry to table 1700 and sets identification information in the ID 1701 of the added entry. The proficiency assessment unit 114 sets the identification information of the selected LLM in the model 1702 of the added entry and sets the values ​​of the selected combination in domain 1703, purpose 1704, and importance 1705. The proficiency assessment unit 114 sets the calculated similarity score 1706, variation score 1707, incorrect answer score 1708, and missing answer score 1709 of the added entry. The proficiency assessment unit 114 sets the start date and time of the answer comparison process in the date and time 1710 of the added entry.

[0191] The proficiency determination unit 114 determines whether processing has been completed for all combinations (step S460).

[0192] If processing is not complete for all combinations, the proficiency determination unit 114 returns to step S454.

[0193] If processing is completed for all combinations, the proficiency determination unit 114 determines whether processing has been completed for all models (step S411).

[0194] If processing is not complete for all models, the proficiency determination unit 114 returns to step S401.

[0195] Once processing is complete for all models, the proficiency determination unit 114 terminates the proficiency determination process.

[0196] Figure 24 is a flowchart illustrating an example of the goodness-of-fit determination process performed by the evaluation system 100 in Example 2.

[0197] The suitability determination unit 116 generates a combination of purpose and importance by referring to the table 1500 (step S501).

[0198] The conformance determination unit 116 selects an LLM from the list of LLMs included in the evaluation instruction (step S502).

[0199] The suitability determination unit 116 selects a combination of purpose and importance (step S503).

[0200] The goodness-of-fit determination unit 116 calculates the goodness-of-fit of the LLM for a given combination of objective and importance (step S504). Specifically, the following processes are performed.

[0201] (S504-1) The suitability determination unit 116 obtains parameters corresponding to the combination of purpose and importance by referring to the table 1800.

[0202] (S504-2) The suitability determination unit 116 refers to table 1700 and searches for an entry in which the identification information of the selected LLM is set in model 1702 and the selected objective and importance are set in objective 1704 and importance 1705. The suitability determination unit 116 obtains the variation score, the incorrect response score, and the missing response score from the searched entry.

[0203] (S504-2) The goodness-of-fit determination unit 116 calculates the goodness-of-fit using various scores and parameters. For example, the goodness-of-fit determination unit 116 calculates the goodness-of-fit using equation (1). Here, S_1 represents the variation score, S_2 represents the incorrect response score, and S_3 represents the missing response score. Also, P_1 represents parameter (1), P_2 represents parameter (2), and P_3 represents parameter (3). Note that equation (1) is just one example of a method for calculating the goodness-of-fit and is not limited thereto.

[0204]

number

[0205] The suitability determination unit 116 records the calculation result (step S505).

[0206] Specifically, the goodness-of-fit determination unit 116 adds an entry to the table 1900, sets the identification information of the selected LLM in the model 1901, and sets the selected objective and importance in the objective 1902 and importance 1903. The goodness-of-fit determination unit 116 sets the calculated goodness-of-fit 1904 of the added entry.

[0207] The suitability determination unit 116 determines whether processing has been completed for all combinations (step S506).

[0208] If processing has not been completed for all combinations, the suitability determination unit 116 returns to step S503.

[0209] If processing is completed for all combinations, the fitness determination unit 116 determines whether processing has been completed for all models (step S507).

[0210] If processing is not complete for all models, the fitness determination unit 116 returns to step S502.

[0211] Once processing is complete for all models, the fitness determination unit 116 terminates the fitness determination process.

[0212] Figures 25 and 26 show examples of screens displayed by the evaluation system 100 of Example 2.

[0213] When the display unit 115 of the evaluation system 100 receives a request from the user to display the degree of fit of a model, it displays screen 2500 on a terminal or the like operated by the user. Screen 2500 includes a display field 2501, a table 2502, and operation buttons 2503.

[0214] Display field 2501 is a field that displays the degree of suitability of the LLM for each purpose in graph format. Table 2502 is a table that displays the degree of suitability of the LLM for each purpose.

[0215] Operation button 2503 is used to modify the parameters used in calculating the goodness of fit.

[0216] When the display unit 115 of the evaluation system 100 receives a request from the user to display the LLM proficiency level in a specific combination of domain, purpose, and importance, it displays screen 2600 on the terminal operated by the user. In addition, the response items of the generated response may be highlighted based on the similarity score.

[0217] As described above, the evaluation system 100 of Example 2 can present indicators representing various levels of proficiency. Furthermore, the evaluation system 100 of Example 2 can evaluate the accuracy of the generated AI's responses for each combination of objective and importance.

[0218] Furthermore, the present invention is not limited to the embodiments described above, and various modifications are included. Also, for example, the embodiments described above are detailed explanations of the configuration in order to clearly illustrate the present invention, and are not necessarily limited to those comprising all the described configurations. Additionally, some of the configurations in each embodiment can be added to, deleted from, or replaced with other configurations. Moreover, the generating AI is not limited to LLM (text generation), but includes AI that generates various data formats such as speech.

[0219] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, and the like.

[0220] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, Python, and Java (registered trademark).

[0221] Furthermore, the program code for the software that implements the functions of the embodiment may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.

[0222] In the above-described embodiment, the control lines and information lines shown are those deemed necessary for explanation and do not necessarily represent all control lines and information lines in the actual product. All components may be interconnected. [Explanation of symbols]

[0223] 100 Evaluation System 101 Natural Language Processing Services 110 Input Section 111 Prompt conversion section 112 Answer comparison section 113 Answer generation part 114 Proficiency Assessment Unit 115 Display section 116 Compatibility judgment section 120 Prompt / Answer Storage Unit 121 Comparison result storage section 122 Proficiency Memory Unit 123 Parameter Storage Unit 124 Conformance Memory Unit 200 calculator 201 Processor 202 Main storage 203 Secondary storage device 204 Network Interfaces Bus 205 300, 1200, 1400, 2500, 2600 screens

Claims

1. A computer system, The system comprises a processor, a storage device connected to the processor, and a network interface connected to the processor. It accepts a prompt containing the task details as input, executes the task defined by the prompt, and connects to a machine learning model that can communicate with it to generate an answer. The aforementioned processor, The system accepts evaluation instructions including information specifying the usage scenario of the machine learning model, input prompts, and expected answers. Multiple conversion prompts are generated by changing the representation pattern of the input prompt. Each of the input prompts and the plurality of conversion prompts is input to the machine learning model to obtain a plurality of generated responses. For each of the aforementioned multiple generated responses, the assumed response and the generated response are compared to calculate a similarity score representing the similarity between the assumed response and the generated response. By performing statistical processing using the similarity of the multiple generated responses, an index for evaluating the accuracy of the machine learning model's responses in the usage scenario is calculated. A computer system characterized by recording a first calculation result, which includes the index associated with information specifying the usage scenario, in the storage device.

2. A computer system according to claim 1, The computer system is characterized in that the processor calculates a similarity score, which represents the similarity between the assumed answer and the generated answer in the usage scenario, as an index by performing statistical processing using the similarity of the plurality of generated answers.

3. A computer system according to claim 2, The computer system is characterized in that the processor calculates a fluctuation score, which represents the degree of fluctuation of the generated responses in the usage scenario, as an index by performing statistical processing using the similarity of the plurality of generated responses.

4. A computer system according to claim 3, The computer system is characterized in that the processor calculates, as an index, an error score representing the degree of error in the generated responses in the usage scenario and an error score representing the degree of omissions in the generated responses in the usage scenario, by performing statistical processing using the similarity of the plurality of generated responses.

5. A computer system according to claim 4, The information specifying the aforementioned usage scenario includes the purpose of use. The aforementioned processor, Using the variation score, the incorrect response score, and the missing response score included in the first calculation result for which the aforementioned purpose of use is the same, a degree of fit representing the degree to which the machine learning model fits the aforementioned purpose of use is calculated. A computer system characterized by recording a second calculation result, including the degree of suitability associated with the aforementioned purpose of use, in the storage device.

6. A computer system according to claim 1, The computer system is characterized in that the processor repeatedly performs a process of inputting the same prompt or the same conversion prompt to the machine learning model and obtaining the generated response a predetermined number of times.

7. A computer system according to claim 5, It maintains parameter information that manages the parameters used to calculate the degree of suitability for each of the aforementioned purposes of use, The aforementioned processor, Select the aforementioned purpose of use, A computer system characterized by calculating the degree of fit using the parameters corresponding to the selected purpose of use, and the variation score, the incorrect response score, and the missing response score included in the first calculation result for which the purpose of use is the same.

8. A method for evaluating machine learning models executed by a computer system, The aforementioned computer system, The system comprises a processor, a storage device connected to the processor, and a network interface connected to the processor. It accepts a prompt containing the task details as input, executes the task defined by the prompt, and connects to a machine learning model that can communicate with it to generate an answer. The evaluation method for the aforementioned machine learning model is: The processor receives an evaluation instruction including information specifying the usage scenario of the machine learning model, an input prompt, and an expected response. The processor generates a plurality of conversion prompts by changing the representation pattern of the input prompt, The processor inputs each of the input prompts and the plurality of conversion prompts to the machine learning model to obtain a plurality of generated responses. The processor performs the steps of comparing the assumed answer and the generated answer for each of the plurality of generated answers and calculating a similarity score that represents the similarity between the assumed answer and the generated answer, The processor performs statistical processing using the similarity of the plurality of generated responses to calculate an index for evaluating the accuracy of the machine learning model's responses in the usage scenario. A method for evaluating a machine learning model, characterized by comprising the step of the processor recording in the storage device a calculation result including the index associated with information specifying the usage scenario.

Images