Program, information processing device, method, and system

Abstract

Objectively evaluate tuned generative AI. [Solution] A program for operating a computer including a processor and a memory causes the processor to execute the following steps: inputting a predetermined query to a first generation AI to be evaluated and causing the first generation AI to output an answer sentence to the query, causing a second generation AI to output a similarity between the answer sentence output from the first generation AI and a correct answer sentence associated with the query, and evaluating the first generation AI based on the similarity output from the second generation AI.

Description

[Technical Field]

[0001] The present disclosure relates to a program, an information processing device, a method, and a system. [Background technology]

[0002] In recent years, generative AI, typified by large-scale language models (LLMs), has rapidly developed and is expected to be used in a variety of fields. These generative AIs are often fine-tuned for specific tasks or purposes. There is a demand for technology to objectively and efficiently evaluate the performance of tuned generative AI.

[0003] Patent Document 1 proposes a technique for improving the quality of answer data when answer data to question data is acquired using a large-scale language model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2025-017338 Summary of the Invention [Problem to be solved by the invention]

[0005] However, while Patent Document 1 describes a technique for improving the quality of answer data when answer data to question data is obtained using a large-scale language model, it does not describe a technique for evaluating tuned generative AI. Generative AIs that have been tuned to a certain degree exist, and there is a demand for objective evaluation of such generative AIs.

[0006] The purpose of this disclosure is to objectively evaluate tuned generative AI. [Means for solving the problem]

[0007] In order to solve the above problem, one embodiment of a program of the present disclosure is a program for operating a computer having a processor and a memory, and is characterized in that the program causes the processor to execute the following steps: inputting a predetermined query to a first generation AI to be evaluated, and causing the first generation AI to output an answer sentence to the query; causing a second generation AI to output the similarity between the answer sentence output from the first generation AI and a correct answer sentence associated with the query; and evaluating the first generation AI based on the similarity output from the second generation AI. [Effects of the Invention]

[0008] According to the present disclosure, tuned generative AI can be objectively evaluated. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating an example of the overall configuration of an information processing system according to the present disclosure. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of a terminal device according to the present disclosure. [Figure 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of a server according to the present disclosure. [Figure 4] FIG. 2 is a diagram illustrating an example of the data structure of a learning data table 2021 according to the present disclosure. [Figure 5] FIG. 20 is a diagram illustrating an example of the data structure of a command statement table 2022 of the present disclosure. [Figure 6] 10 is a flowchart illustrating an example of the flow of an evaluation process according to the present disclosure. [Figure 7] FIG. 10 is a diagram illustrating an example of an evaluation result screen of the present disclosure. [Figure 8] FIG. 1 is a block diagram showing the basic hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In all drawings describing the embodiments, common components are designated by the same reference numerals, and repeated description will be omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Furthermore, not all components shown in the embodiments are necessarily essential components of the present disclosure. Furthermore, each drawing is a schematic diagram and is not necessarily a precise illustration.

[0011] In the following description, a "processor" refers to one or more processors. The at least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may also be another type of processor such as a GPU (Graphics Processing Unit). The at least one processor may be single-core or multi-core.

[0012] Furthermore, the at least one processor may be a processor in the broad sense, such as a hardware circuit (for example, a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) that performs part or all of the processing.

[0013] In the following explanation, information that produces an output for an input may be described using expressions such as "xxx table," but this information may be data of any structure, or may be a learning model such as a neural network that produces an output for an input. Therefore, an "xxx table" may be referred to as "xxx information."

[0014] Furthermore, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0015] In addition, in the following explanation, processing may be described using the "program" as the subject, but since a program is executed by a processor to perform specified processing while appropriately using a memory unit and / or an interface unit, etc., the subject of the processing may also be the processor (or a device such as a controller that has that processor).

[0016] The program may be installed in a device such as a computer, or may be stored in, for example, a program distribution server or a computer-readable (e.g., non-transitory) recording medium. Also, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0017] Furthermore, in the following description, identification numbers are used as identification information for various objects, but other types of identification information (for example, identifiers including alphabetic characters or symbols) may also be used.

[0018] In addition, in the following description, when describing elements of the same type without distinguishing between them, reference symbols (or common symbols among the reference symbols) may be used, and when describing elements of the same type with distinction between them, the identification numbers (or reference symbols) of the elements may be used.

[0019] In the following description, the control lines and information lines are those that are considered necessary for the description, and do not necessarily represent all the control lines and information lines in the product. All components may be interconnected.

[0020] Each information processing device is configured by a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the terminal device 10 and the server 20, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer will be omitted.

[0021] (First embodiment) <1. Overview> The system according to this embodiment objectively evaluates the performance of a first generation AI tuned for a specific task or purpose. The server inputs a predetermined query to the first generation AI to be evaluated and causes the first generation AI to output an answer to the query. The server then causes the second generation AI to output the similarity between the answer output from the first generation AI and the correct answer sentence associated with the query. The server then evaluates the first generation AI based on the similarity output from the second generation AI and presents the evaluation results to the user. This evaluation process may include collating the answer sentence and the correct answer sentence, calculating the relevance and recall based on the first and second similarities, and calculating an evaluation value based on the harmonic mean of these similarities. For example, a large language model (LLM) may be used as the first and second generation AIs.

[0022] <2. Overall system configuration> FIG. 1 is a block diagram showing an example of the overall configuration of an information processing system 1 according to this embodiment. System 1 is a system for objectively evaluating the performance of, for example, a tuned first generation AI (hereinafter referred to as "first generation AI system 30a") using a general-purpose second generation AI (hereinafter referred to as "second generation AI system 30b"). This disclosure will be described taking as an example a case where a developer or administrator evaluates a first generation AI system 30a that has undergone specific tuning.

[0023] 1 includes, for example, a terminal device 10 for inputting evaluation instructions or displaying results, a server 20 for executing the evaluation process, a first generation AI system 30a that is the evaluation target, and a second generation AI system 30b that is used for the evaluation. The terminal device 10, the server 20, the first generation AI system 30a, and the second generation AI system 30b are connected to each other via, for example, a network 80.

[0024] 1, for the sake of simplicity, an example is shown in which the system 1 includes one terminal device 10. However, in reality, the system 1 may include multiple terminal devices 10 for use by multiple administrators, etc.

[0025] 1 shows an example in which the system 1 includes one server 20, but for example, a collection of multiple devices may be one server 20. The way in which the multiple functions required to realize the server 20 are allocated to one or multiple pieces of hardware can be determined appropriately depending on the processing capacity of each piece of hardware and / or the specifications required for the server 20.

[0026] While Figure 1 shows an example in which system 1 includes one second generation AI system 30b, system 1 may include two or more second generation AI systems 30b (ensemble evaluation). Also, while Figure 1 shows an example in which first generation AI system 30a and second generation AI system 30b are independent of server 20, server 20 may include the functions of first generation AI system 30a and / or second generation AI system 30b. In other words, server 20 may store the LLM included in the first generation AI and / or second generation AI.

[0027] The terminal device 10 is an information processing device operated by, for example, an administrator or developer (hereinafter referred to as a "user") who performs the evaluation. The terminal device 10 may be realized by, for example, a mobile terminal such as a smartphone or a tablet, or a stationary PC (Personal Computer), a laptop PC, or the like.

[0028] The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The input device 13 is a device (e.g., a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.) for receiving input operations (such as an instruction to start evaluation) from a user. The output device 14 is a device (e.g., a display, a speaker, etc.) for presenting information (such as evaluation results) to the user.

[0029] The server 20 is, for example, an information processing device for providing an evaluation service of the first generation AI system, and is an information processing device realized by a computer connected to the network 80. The server 20 may be, for example, an API (Application Programming Interface) server. As shown in FIG. 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The input / output IF 23 functions as an input device for receiving input operations from a manager / operator of the evaluation service, and as an interface for an output device for outputting information to the manager / operator.

[0030] The first generative AI system 30a is a system having a generative AI tuned for a specific task or purpose to be evaluated. The first generative AI system 30a outputs a response sentence in response to an input query.

[0031] The second generation AI system 30b is, for example, a cloud server having a general-purpose LLM, etc. The number of LLMs included in the second generation AI system 30b may be one or more.

[0032] LLM is a single-modal natural language model constructed by learning from large amounts of text data, and is used in many NLG (Natural Language Generation) tasks, such as generating answers to specific questions, automatically generating sentences, and summarizing text. LLM is an example of a generative AI model. Examples of LLMs include: OpenAI: GPT-4 Google: Gemini 1.5 Flash ·Anthropic: Claude 3.5 Sonnet

[0033] The second generation AI system 30b shortens the sentence or calculates the similarity between the answer sentence and the correct sentence based on the prompt including the text data (answer sentence and correct sentence of the first generation AI system 30a) and the instruction sentence sent from the server 20, and sends the results to the server 20.

[0034] In this embodiment, data previously stored in the memory unit 202 of the server 20 is used as information used to evaluate the first generation AI system 30a. This data includes at least a plurality of "queries" used to tune the first generation AI system 30a or prepared for evaluation, and the "correct answer sentences" corresponding to each query, in association with each other (learning data table 2021). These queries and correct answer sentences are used as the basis for comparing inputs and outputs to the first generation AI system 30a in the evaluation process. The memory unit 202 also includes instruction statement templates (instruction statement table 2022) used to instruct the second generation AI system 30b.

[0035] Each information processing device, such as the terminal device 10, the server 20, the first generation AI system 30a, and the second generation AI system 30b, may be configured by a computer 90 equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer 90 and the basic functional configuration of the computer 90 realized by the basic hardware configuration will be described later (see FIG. 8). Note that for each of the terminal device 10, the server 20, the first generation AI system 30a, and the second generation AI system 30b, explanations that overlap with the basic hardware configuration and basic functional configuration of the computer 90 will be omitted.

[0036] <3. Configuration of terminal device> Fig. 2 is a block diagram showing an example of the functional configuration of the terminal device 10. As shown in Fig. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 160, a position information sensor 150, a storage unit 180, and a control unit 190. The blocks included in the terminal device 10 are electrically connected by, for example, a bus or the like.

[0037] The communication unit 120 performs processing such as modulation and demodulation for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to the outside (for example, the server 20). The communication unit 120 performs reception processing on the signal received from the outside and outputs it to the control unit 190.

[0038] The input device 13 is a device for a user (administrator, developer, etc.) operating the terminal device 10 to input instructions or information. The input device 13 is realized, for example, by a touch-sensitive device 131 or the like, which inputs instructions by touching the operation surface. If the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, keyboard, mouse, etc. The input device 13 converts instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190. The input device 13 may include, for example, a receiving port that receives electrical signals input from an external input device.

[0039] The output device 14 is a device for presenting information to a user operating the terminal device 10. The output device 14 is realized, for example, by a display 141 or the like. The display 141 displays data according to the control of the control unit 190. The display 141 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display or the like.

[0040] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of an audio signal. The audio processing unit 17 converts a signal provided from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is realized, for example, by a processor for audio processing. The microphone 171 receives an audio input and provides an audio signal corresponding to the audio input to the audio processing unit 17. The speaker 172 converts the audio signal provided from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10. Audio input and output are not essential to this evaluation system.

[0041] The camera 160 is a device that receives light with a light receiving element and outputs the light as an image capturing signal. The camera is not essential for this evaluation system.

[0042] The location information sensor 150 is a sensor, such as a GPS (Global Positioning System) module, that detects the location of the terminal device 10. Location information is not essential to this evaluation system.

[0043] The storage unit 180 is realized by, for example, the memory 15 and the storage 16, and stores data and programs used by the terminal device 10. The storage unit 180 may store, for example, account information or setting information of a user (such as an administrator).

[0044] The user information 181 includes, for example, information for identifying the user (administrator, etc.) who uses the terminal device 10, or information such as display settings.

[0045] The control unit 190 is realized by the processor 19 reading a program stored in the storage unit 180 and executing instructions included in the program. The control unit 190 controls the operation of the terminal device 10. The control unit 190 functions as an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193 by operating in accordance with the program.

[0046] The operation reception unit 191 performs processing for receiving instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives information such as an instruction to start evaluation, an instruction to select a model to be evaluated, an instruction to select an evaluation dataset, an instruction to change display settings, etc., input from the touch-sensitive device 131, a keyboard, a mouse, etc.

[0047] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives a voice signal that is input from the microphone 171 and converted into a digital signal by the voice processing unit 17.

[0048] The transmitting / receiving unit 192 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 transmits information such as an instruction to start evaluation received from a user to the server 20. In addition, the transmitting / receiving unit 192 receives information on the evaluation result transmitted from the server 20.

[0049] The presentation control unit 193 controls the output device 14 to present the information received from the server 20 to the user. Specifically, for example, the presentation control unit 193 causes the display 141 to display information on the evaluation results of the first generation AI system 30a received from the server 20 (such as the overall score and detailed results for each query). The presentation control unit 193 may also cause the speaker 172 to output information transmitted from the server 20 (such as an evaluation completion notification sound).

[0050] <4. Server configuration> Fig. 3 is a block diagram showing an example of the functional configuration of the server 20 shown in Fig. 1. As shown in Fig. 3, the server 20 performs the functions of a communication unit 201, a storage unit 202, and a control unit 203.

[0051] The communication unit 201 performs processing for the server 20 to communicate with external devices, such as the terminal device 10, the first generation AI system 30a, and the second generation AI system 30b.

[0052] The memory unit 202 is realized by the memory 25 and the storage 26, and stores data and programs used by the server 20. The programs include an application program for providing an evaluation service for the first generation AI system 30a. The memory unit 202 also stores, for example, a training data table 2021 that stores queries and correct answer sentences used for evaluation, and an instruction statement table 2022 that stores instruction statement templates for the second generation AI system 30b.

[0053] The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions included in the program. The control unit 203 controls the operation of the server 20. The control unit 203 operates in accordance with the read program to fulfill the functions of a reception control module 2031, a transmission control module 2032, a presentation control module 2033, an evaluation execution module 2034, and an operation reception module 2035.

[0054] The reception control module 2031 controls the process of the server 20 receiving signals from external devices according to a communication protocol. For example, it executes the process of receiving an evaluation start instruction from the terminal device 10, a response sentence from the first generation AI system 30a, or a similarity calculation result from the second generation AI system 30b.

[0055] The transmission control module 2032 controls the process of the server 20 transmitting signals to external devices according to a communication protocol. For example, it executes processes to control the transmission of queries to the first generation AI system 30a, the transmission of answer sentences, correct answer sentences, and instruction prompts to the second generation AI system 30b, or the transmission of evaluation results to the terminal device 10.

[0056] The presentation control module 2033 controls the process for presenting the evaluation results of the first generation AI system 30a calculated by the evaluation execution module 2034 to the terminal device 10 used by the user (administrator, etc.). Specifically, it performs a process of formatting the evaluation result information to be presented (total score, detailed list, etc.) into an appropriate display format and instructing the transmission to the terminal device 10 via the transmission control module 2032.

[0057] The evaluation execution module 2034 controls a series of processes for executing evaluation of the first generation AI system 30a. Specifically, it obtains a correct answer sentence corresponding to a query from the learning data table 2021 in the memory unit 202, sends the query to the first generation AI system 30a, and receives an answer sentence. Next, it creates a prompt using the received answer sentence, the obtained correct answer sentence, and an instruction sentence template obtained from the instruction sentence table 2022 in the memory unit 202, etc., and sends it to the second generation AI system 30b. This prompt includes, for example, a summary of the sentence or content instructing calculation of similarity (first similarity, second similarity). The evaluation execution module 2034 receives calculation results such as similarity from the second generation AI system 30b, calculates evaluation indicators (fit, recall, F1 score, etc.) based on the results, and generates a final evaluation result.

[0058] The operation reception module 2035 executes processing for receiving an instruction to start evaluation, a designation of a model to be evaluated, a designation of a learning dataset to be used, and the like, inputted by a user (administrator, etc.) via the terminal device 10.

[0059] <5. Data Structure> The data structure used in this embodiment will be described. Note that the data structure described is an example and does not exclude data not described. Furthermore, even if data is expressed as the same table, it may be stored in physically separate storage areas. The storage unit 202 of the server 20 stores, for example, a learning data table 2021 and an instruction statement table 2022.

[0060] 4 is a diagram showing an example of the data structure of a training data table 2021 that can be stored in the storage unit 202. The training data table 2021 stores data used for evaluating the first generation AI system 30a. This table can include training data used for tuning the first generation AI system 30a or a data set separately prepared for evaluation.

[0061] The learning data table 2021 can be configured as a table having columns of, for example, "query ID," "query body," and "correct answer sentence."

[0062] The item "Query ID" is an item for storing identification information (eg, a unique ID, a sequential number) for uniquely identifying each query used for evaluation.

[0063] The item "Query text" is an item for storing text data of a query, which is a question or instruction text input to the first generation AI system 30a.

[0064] The item "Correct Answer" stores text data of an answer sentence that is considered ideal or defined as the correct answer to the query indicated in the item "Query Body." This correct answer sentence is compared with the answer sentence output by the first generation AI system 30a.

[0065] In this way, the training data table 2021 stores queries used for evaluation in association with their corresponding correct answer sentences, which enables the server 20 to efficiently obtain inputs to the first generation AI system 30a and reference information to be compared with the outputs.

[0066] 5 is a diagram showing an example of the data structure of an instruction statement table 2022 that can be stored in the storage unit 202. As shown in FIG. 5, the instruction statement table 2022 can be configured as a table having columns for instruction purpose and instruction statement template, with an instruction statement ID as a key. Note that the instruction statement table 2022 is not essential, and the instruction content may be written directly in the program code.

[0067] The instruction statement ID is an item that stores an identifier for uniquely identifying each instruction statement. The instruction purpose is an item that stores information indicating what kind of processing the instruction statement requests the second generation AI system 30b to perform (e.g., "short form instruction," "similarity calculation instruction"). The instruction statement is an item that stores text data that serves as a template for a prompt to the second generation AI system 30b.

[0068] The instruction includes a string of characters that specifically instructs the second generation AI system 30b on the task to be performed. For example, a template corresponding to a "short sentence instruction" might include an instruction such as "Please shorten sentence 1 and sentence 2 below while preserving the main points. Sentence 1: {answer sentence} Sentence 2: {correct sentence}." Here, {answer sentence} and {correct sentence} are placeholders for inserting actual text data.

[0069] Similarly, a template corresponding to a "similarity calculation instruction" may include an instruction such as, "For the following two sentence sets, calculate the similarity between each sentence in Set 1 and the sentence in Set 2 that is most similar to it (first similarity), and the similarity between each sentence in Set 2 and the sentence in Set 1 that is most similar to it (second similarity), and output them in the specified format. Set 1: {abbreviated answer sentence}, Set 2: {abbreviated correct answer sentence}." Here, {abbreviated answer sentence} and {abbreviated correct answer sentence} are placeholders for inserting the text data after the abbreviation process.

[0070] <6. Operation> The operation of the information processing system in this embodiment will be described below. Fig. 6 is a flowchart showing an example of the flow of evaluation processing in this embodiment.

[0071] In step S11, the evaluation execution module 2034 of the server 20 starts executing the evaluation. This start may be triggered, for example, when an administrator or the like operates the terminal device 10 to instruct the start of evaluation and the operation reception module 2035 receives the instruction, or may be started automatically upon completion of fine-tuning of the first generation AI system 30a or at a predetermined interval. The evaluation execution module 2034 selects (or sequentially acquires) one query to be used for evaluation from the learning data table 2021 stored in the memory unit 202.

[0072] In step S12, the evaluation execution module 2034 of the server 20 prepares information corresponding to the selected query and acquires an answer sentence from the first generation AI system 30a. Specifically, the evaluation execution module 2034 transmits the query selected in step S11 to the first generation AI system 30a via the transmission control module 2032, and receives an answer sentence to the query from the first generation AI system 30a via the reception control module 2031. The evaluation execution module 2034 also reads out the correct answer sentence associated with the query from the learning data table 2021.

[0073] In step S13, the evaluation execution module 2034 of the server 20 optionally shortens the answer sentence and correct answer sentence of the first generation AI system 30a using the second generation AI system 30b. This shortening process may include, for example, a process of breaking down and organizing the individual sentences that make up the original answer sentence or correct answer sentence, or a process of extracting the main points of the entire sentence and reducing the overall amount of description. Specifically, the evaluation execution module 2034 obtains a shortened instruction template from the instruction sentence table 2022 in the memory unit 202 and creates a shortened instruction prompt by incorporating the answer sentence and correct answer sentence obtained in step S12. The created prompt is sent to the second generation AI system 30b via the transmission control module 2032, and the shortened answer sentence and the shortened correct answer sentence are received from the second generation AI system 30b via the reception control module 2031.

[0074] Here, we will explain the shortening process in detail. The answer sentence output by the first generation AI system 30a is not necessarily a short sentence, but may contain enough information that shortening process is effective. For example, the following set of query, answer sentence, and correct answer sentence can be given. Example query: "How can I share my PC's screen and audio with all participants in an online meeting?" Example of a response from the first generated AI (opening): "To share your screen with audio using an online meeting tool, first join the meeting. Next, click the 'Share' icon on the meeting control bar and turn on 'Include computer audio' in the sharing options panel. After that, select the 'screen' or 'window' you want to share and sharing will begin. To stop sharing, click the 'Stop sharing' button. Now you can share your screen with audio... (rest omitted)" Example of a correct sentence (beginning): "To share your screen with audio during an online meeting, follow these steps: First, start or join an online meeting. Next, click the 'Share' icon (the up arrow icon) on the meeting control bar. When the sharing options panel appears, turn on the 'Include computer audio' toggle switch at the top of the panel. Next, select the content you want to share from 'Screen' (your entire desktop) or a specific 'Window'. When you're finished sharing, click the 'Stop sharing' button... (Omitted)"

[0075] In step S13, the answer sentences and correct answer sentences, each of which has a certain length and amount of information, are subjected to a shortening process. As a result, "sentence 1" (e.g., individual elements of the shortened answer sentence) processed by the second generation AI system 30b in step S14 and thereafter is expected to be a concise description of a few sentences containing a few keywords or short phrases, such as "Screen sharing with audio using an online conference tool," "Need to start or join the meeting," or "Click the share icon," or to be in the form of a list. Similarly, "sentence 2" (e.g., individual elements of the shortened correct answer sentence) is expected to be a concise description corresponding to "sentence 1," such as "Screen sharing procedure with audio using an online conference tool," "Start or join the meeting," and "Click the share icon." This enables the second generation AI system 30b to process each sentence element to be compared at an appropriate granularity and perform highly accurate similarity calculations.

[0076] In step S14, the evaluation execution module 2034 of the server 20 creates a prompt for causing the second generation AI system 30b to calculate the similarity. Specifically, the evaluation execution module 2034 obtains a similarity calculation instruction template from the instruction statement table 2022 in the storage unit 202, and creates a similarity calculation instruction prompt by incorporating the answer sentence and correct answer sentence obtained in step S13 (or the original in step S12). This prompt includes, for example, content instructing the second generation AI system 30b to calculate and output the first similarity and the second similarity. In detail, this instruction is intended to cause the second generation AI system 30b to first evaluate, as the first similarity, the similarity between each sentence element included in the shortened answer sentence and all sentence elements included in the shortened correct answer sentence, and then output the similarity between the sentence element and the most similar sentence element of the shortened correct answer sentence. Similarly, as the second similarity, the intention is to evaluate the similarity of each individual sentence element contained in the shortened correct answer sentence with all sentence elements contained in the shortened answer sentence, and output the similarity with the sentence element of the most similar answer sentence among them.

[0077] In step S15, the evaluation execution module 2034 of the server 20 obtains the similarity from the second generation AI system 30b and calculates an evaluation value. Specifically, the evaluation execution module 2034 inputs the prompt created in step S14 to the second generation AI system 30b via the transmission control module 2032 and receives the calculation results of the first similarity, second similarity, etc. from the second generation AI system 30b via the reception control module 2031. Based on the received similarity, the evaluation execution module 2034 calculates evaluation values ​​such as compatibility, recall, and the F1 score, which is the harmonic mean of these. The compatibility is calculated as an index representing how well the answer sentence matches the correct answer sentence based on the first similarity for each shortened sentence. The recall is calculated as an index representing how well the answer sentence reproduces the correct answer sentence based on the second similarity for each shortened sentence.

[0078] In step S16, the presentation control module 2033 of the server 20 presents the calculated evaluation result to the user (administrator, etc.). Specifically, the presentation control module 2033 generates evaluation result information including the evaluation value (total score or score for each query, etc.) calculated in step S15, and the transmission control module 2032 transmits the information to the terminal device 10 used by the user. The presentation control unit 193 of the terminal device 10 displays the received information on the display 141, etc. This allows the user to check the evaluation result of the first generation AI system 30a.

[0079] <7. Screen Examples> An example of a screen on the display 141 of the terminal device 10 in the present disclosure will be described.

[0080] 7 shows an example of an evaluation result screen that displays the evaluation results of the first generation AI system that is the evaluation target. This screen is displayed on a terminal device 10 used by, for example, the developer or administrator of the first generation AI system.

[0081] Area 1411 is an area that displays the evaluation score for a specific query. In the example of Fig. 7, detailed evaluation indicators such as "Precision," "Recall," and their harmonic mean, "F1 score," for one query selected from the list in area 1412 are calculated and displayed.

[0082] Area 1412 is a list area that displays detailed evaluation results for each query used in the evaluation. This list may include, for example, the "query" used in the evaluation, the "first generation AI's answer" for that query, the "correct answer" for comparison, and the "similarity score" (or compatibility, recall, etc.) calculated by the second generation AI. Area 1412 in FIG. 7 includes, as an example, a "query" column, a "first generation AI's answer" column, a "correct answer" column, and a "similarity score" column. For example, specific evaluation results such as "answer 1," "correct answer 1," and "0.85" are associated with "query 1." This allows the administrator to specifically determine which queries received low evaluations. Selecting a specific query may display the full answer or correct answer, or a comparison of shortened versions, in a separate window.

[0083] Area 1413 is an interface area for performing operations related to the execution or configuration of an evaluation. In the example of Fig. 7, a "Re-evaluate" button, a drop-down list for selecting a dataset (a set of a query and a correct answer sentence) to be used for evaluation, a list for selecting a model version to be evaluated, etc. may be displayed. The user can operate these interfaces to execute an evaluation or view past evaluation results.

[0084] <8.Summary> As described above, according to the information processing system of the first embodiment, the tuned first generation AI can be objectively and efficiently evaluated using the general-purpose second generation AI. Specifically, by having the second generation AI calculate the similarity between the answer sentence of the first generation AI and the correct answer sentence, and calculating an evaluation value based on the result, evaluation can be performed without human intervention.

[0085] (Variation) In this embodiment, an example in which evaluation is performed using one second generation AI system 30b has been described, but the present disclosure is not limited to this. A configuration in which ensemble evaluation is performed using multiple different second generation AI systems 30b (e.g., different types of LLMs or different versions of LLMs) is also possible.

[0086] In this case, the evaluation execution module 2034 of the server 20 executes the processes of step S13 (short form) and step S15 (similarity calculation and evaluation value calculation) for each of the multiple second generation AI systems 30b. Then, the final evaluation value is calculated by performing statistical processing (e.g., average, median, weighted average, etc.) on the multiple evaluation values ​​(e.g., multiple F1 scores) obtained from each second generation AI system 30b. By performing ensemble evaluation in this manner, it is expected that more robust and reliable evaluation results will be obtained that are not dependent on the characteristics of a specific second generation AI.

[0087] In the above embodiment, the evaluation indexes are described as being based on the relevance, recall, and F1 score, but are not limited to these. Evaluation indexes used in other natural language processing fields may be calculated by the second generation AI system 30b or the server 20.

[0088] It is also possible to take into account the order of the sentences in the evaluation. For example, the second generation AI system 30b can calculate the difference in the order of appearance of phrases or keywords (for example, the number of reversals) between the answer sentence of the first generation AI and the correct sentence (or a shortened version of these sentences), or the server 20 can calculate it, and the result can be reflected in the evaluation value. For example, the matching score can be calculated as α·F1 score + (1-α)·R norm where R norm is the normalized inversion number (an index that approaches 1 as the degree of matching of the order increases and approaches 0 as the degree of matching decreases), and α is a weighting coefficient (0 to 1) for the F1 score and the evaluation of order.

[0089] Specifically, the reversal number (R) can be calculated as follows: A sequence of N sentences (or elements such as phrases, keywords, etc.) contained in the answer sentence to be evaluated or its shortened version is compared with the sequence of the corresponding correct sentence. When any two sentence pairs are selected from the N sentences, if the order of the pair differs (is reversed) from the order in the correct sentence, a count is added. This comparison is made for all pairs (a total of NC2 ways, i.e., N(N-1) / 2 ways), and the total number of reversed pairs is taken as the reversal number R. Therefore, if the order of the sentences is exactly the same as the correct sentence, the reversal number R will be 0. Conversely, if the order of the sentences is completely reversed from the correct sentence, the reversal number R will be the maximum value, N(N-1) / 2. Then, the normalized reversal number R is norm is calculated using this reversal number R as follows: First, the calculated reversal number R is divided by its maximum possible value, N(N-1) / 2 (where N≧2). This gives a value between 0 and 1 (let's call it R'. R'=R / (N(N-1) / 2)), and R' approaches 0 as the degree of inconsistency in the order decreases. Next, R' is subtracted from 1 to obtain R norm (R norm =1-R′). As a result, R norm is an index that is 1 if the sentence order is exactly the same as the correct sentence, and 0 if it is the exact opposite. This normalizes the value so that the higher the degree of match, the larger the value, making it easier to integrate harmoniously with the F1 score.

[0090] Furthermore, the evaluation process of this embodiment may be configured to be automatically executed each time fine-tuning of the first generation AI system 30a is performed, or at a predetermined interval (e.g., daily or weekly). This allows continuous monitoring of performance changes of the first generation AI system 30a, and efficient confirmation of the improvement effect on degradation.

[0091] In addition, in each of the above embodiments, an example has been described in which evaluation is performed using queries and correct answer sentences prepared in advance in the learning data table 2021. However, it is also possible to increase the variety of queries in order to further enrich the query set used for evaluation.

[0092] First, regarding the creation of query variations, for example, the server 20 may have a function to automatically generate multiple similar queries that are substantially identical or similar in meaning to the original query but have different expressions, based on the original query text stored in the learning data table 2021, using the second generation AI system 30b or a generation AI for query expansion prepared separately for this purpose.

[0093] For example, the second generation AI system 30b or the like generates similar queries by applying instructions to the original query text, such as replacing words with synonyms, changing word order, or adding, deleting, or changing modifiers. The correct answer sentences for the generated similar queries can be the same as the correct answer sentences for the original queries, or can be sentences that the second generation AI system 30b or the like has made minor adjustments to based on the original correct answer sentences. This allows for flexible development of evaluation query sets in accordance with the characteristics of the first generation AI system 30a to be evaluated and the purpose of the evaluation, making it possible to perform more practical evaluations.

[0094] <Basic computer hardware configuration> 8 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main memory device 902, an auxiliary memory device 903, and a communication IF 991 (interface), which are electrically connected to one another by a communication bus.

[0095] The processor 901 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0096] The main memory device 902 is used to temporarily store programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0097] The auxiliary storage device 903 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0098] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards. The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can connect to the Internet via a predetermined access point. In the case of a wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), and Bluetooth (registered trademark). In the case of a wired connection, the network also includes a direct connection using a USB (Universal Serial Bus) cable, etc.

[0099] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration across multiple computers 90 and interconnecting them via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0100] <Basic functional configuration of computer 90> The following describes the functional configuration of a computer realized by the basic hardware configuration (FIG. 8) of the computer 90. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0101] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0102] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903, expanding them in the main storage device 902, and executing processing in accordance with the programs. The control unit can realize functional units that perform various types of information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0103] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. Furthermore, the processor 901 can allocate a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with the programs. Furthermore, the control unit can cause the processor 901 to execute processes for adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0104] A database refers to a relational database, which manages data sets called masters and tables in a tabular format structurally defined by rows and columns, by relating them to each other. In a database, a table is called a table, a master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated. Typically, each table and each master has a column set as a primary key to uniquely identify a record, but setting a primary key to a column is not essential. The control unit can cause the processor 901 to add, delete, or update records in specific tables and masters stored in the storage unit according to various programs. Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0105] Note that the databases and masters in this disclosure may include any data structure (list, dictionary, associative array, object, etc.) in which information is structurally defined. The data structure also includes data that can be considered as a data structure by combining data with functions, classes, methods, etc. written in any programming language.

[0106] The communication unit is realized by the communication IF 991. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0107] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tape, non-volatile memory cards, and ROMs.

[0108] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, and Java (registered trademark).

[0109] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or the storage medium.

[0110] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes programs stored in memory. In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions. If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0111] Although several embodiments of the present disclosure have been described above, these embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are intended to be included in the scope of the inventions and their equivalents as defined in the claims, as well as in the scope and spirit of the inventions.

[0112] (Addendum) The matters described in the above embodiments will be supplemented below.

[0113] (Appendix 1) A program for operating a computer including a processor and a memory, the program causing the processor to: A step of inputting a predetermined query into a first generation AI to be evaluated and causing the first generation AI to output a response sentence to the query; a step of causing a second generation AI to output a similarity between the answer sentence output from the first generation AI and a correct answer sentence associated with the query; Evaluating the first generated AI based on the similarity output from the second generated AI; A program characterized by executing the following. (Appendix 2) In the step of outputting the similarity, The answer sentence and the correct answer sentence are abbreviated using the second generation AI; With respect to a sentence included in the shortened answer sentence that is similar to a sentence of the shortened correct answer sentence, outputting a similarity between the sentence and the shortened correct answer sentence as a first similarity to the second generation AI; a step of causing the second generation AI to output a similarity between a sentence included in the shortened correct answer sentence and the shortened answer sentence, the similarity being a second similarity; In the evaluating step, calculating a degree of suitability that indicates how well the correct answer sentence matches the answer sentence based on the first similarity; Calculating a reproducibility indicating how closely the answer sentence reproduces the correct answer sentence based on the second similarity; The program described in (Appendix 1) includes a step of calculating an evaluation value of the first generation AI by taking the harmonic mean of the conformance and the recall. (Appendix 3) The program described in (Appendix 1) or (Appendix 2), wherein the second generation AI is a general-purpose generation AI. (Appendix 4) A program described in any one of (Appendix 1) to (Appendix 3), wherein in the evaluating step, the first generation AI is evaluated by further taking into account the number of reversals calculated based on the difference in sentence order between the answer sentence and the correct sentence. (Appendix 5) In the step of outputting the similarity, the second generation AIs are caused to output the similarity; In the step of evaluating, a plurality of evaluation values ​​are calculated based on the similarities output from the plurality of second generation AIs; A program described in any of (Appendix 1) to (Appendix 4), further causing the processor to execute a step of evaluating the first generation AI by performing statistical processing on the calculated multiple evaluation values. (Appendix 6) A program described in any one of (Appendix 1) to (Appendix 5), which causes the processor to automatically execute each of the steps each time fine-tuning of the first generation AI is performed or at a predetermined interval. (Appendix 7) An information processing device including a processor and a memory, An information processing device in which the processor executes all steps in the program according to any one of (Supplementary Note 1) to (Supplementary Note 6). (Appendix 8) 1. A computer-implemented method comprising a processor and a memory, the method comprising: A method in which the processor executes all steps in the program described in any one of (Appendix 1) to (Appendix 6). (Appendix 9) A system comprising means for executing all steps in the program described in any one of (Appendix 1) to (Appendix 6). [Explanation of symbols]

[0114] 1. System 10...Terminal device 120…Communications Department 13...Input device 14...Output device 15...Memory 16…Storage 19...Processor 20...Server 22...Communication IF 23...Input / output IF 25…Memory 26…Storage 29...Processor 30a…First generation AI system 30b…Second generation AI system

Claims

1. A program for operating a computer including a processor and a memory, the program causing the processor to: A step of inputting a predetermined query to a first generation AI to be evaluated and causing the first generation AI to output a response sentence to the query; The answer sentence and the correct answer sentence associated with the query are abbreviated using a second generation AI; For a sentence included in the shortened answer sentence that is similar to the shortened correct answer sentence, outputting a similarity between the shortened correct answer sentence and the sentence as a first similarity to the second generation AI; A step of causing the second generation AI to output a similarity between a sentence included in the shortened correct answer sentence and the shortened answer sentence, the similarity being a second similarity between the sentence and the shortened answer sentence; calculating a degree of suitability that indicates how well the correct answer sentence matches the answer sentence based on the first similarity; calculating a reproducibility indicating how closely the answer sentence reproduces the correct answer sentence based on the second similarity; A program characterized by executing a step of calculating an evaluation value of the first generated AI based on the degree of conformance and the degree of reproducibility.

2. The program according to claim 1 , wherein the second generated AI is a general-purpose generated AI.

3. 2. The program according to claim 1, wherein in the step of calculating the evaluation value, the first generated AI is evaluated by further taking into account a number of inversions calculated based on a difference in sentence order between the answer sentence and the correct sentence.

4. In the step of outputting the similarity, the second generation AIs are caused to output the first similarity and the second similarity; In the step of calculating the evaluation value, a plurality of evaluation values ​​are calculated based on the first similarity and the second similarity output from a plurality of the second generation AIs; The program according to claim 1 , further causing the processor to execute a step of evaluating the first generated AI by performing statistical processing on the calculated evaluation values.

5. The program according to claim 1 , which causes the processor to automatically execute each of the steps each time fine-tuning of the first generation AI is performed or at a predetermined interval.

6. An information processing device including a processor and a memory, An information processing device, wherein the processor executes all steps of the program according to any one of claims 1 to 5.

7. 1. A computer-implemented method comprising a processor and a memory, the method comprising: A method in which the processor executes all the steps in the program of any one of claims 1 to 5.

8. A system comprising means for executing all steps of the program according to any one of claims 1 to 5.

Images