Electronic device, method, and non-transitory computer-readable storage medium for evaluating artificial intelligence model

Abstract

An electronic device is disclosed. The electronic device may acquire outputs of a plurality of artificial intelligence models for a test scenario. In order to proceed with a tournament for the test scenario, the electronic device may: input, for each of a plurality of matches included in the tournament into an evaluation model, outputs of two artificial intelligence models selected as participants from among the plurality of artificial intelligence models; and acquire a match result of the evaluation model for the two artificial intelligence models. The electronic device may update ratings indicating performances of the plurality of artificial intelligence models, on the basis of a plurality of match results of the plurality of matches acquired on the basis of proceeding with the tournament.

Description

Electronic device, method, and non-transient computer-readable storage medium for evaluating artificial intelligence models

[0001] The embodiments disclosed in this disclosure relate to an electronic device, a method, and a non-transient computer-readable storage medium for evaluating an artificial intelligence model.

[0002] Artificial intelligence (AI) is a technology that mimics the functions of the human brain, such as cognition and judgment. AI may include language models that recognize human language and characters to perform various natural language processing tasks. Language models may include massive language models pre-trained using vast amounts of data. Language models may include transformer models that include encoders and / or decoders that perform computations through attention.

[0003] Developers can evaluate the performance of many AI models during the development process. However, it may be difficult for a specific AI model to be evaluated as superior to others in every aspect. Accordingly, it is necessary to evaluate AI models from various angles.

[0004] However, comparing and evaluating these artificial intelligence models can consume significant time, resources, and costs. Consequently, there are practical limitations to extensively comparing and evaluating AI models across various aspects.

[0005] Therefore, there may be a need for an evaluation method for artificial intelligence models that provides reasonable evaluation results while consuming minimal time, resources, and costs.

[0006] The technical problems to be solved in this document are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description below.

[0007] An electronic device is disclosed. The electronic device may include at least one processor comprising a processing circuit and a memory comprising one or more storage media for storing instructions. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to obtain outputs of a plurality of artificial intelligence models for a test scenario. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to input the outputs of two artificial intelligence models selected as participants among the plurality of artificial intelligence models into an evaluation model for each of the plurality of matches included in the tournament, in order to proceed with a tournament for the test scenario, and to obtain match results of the evaluation model for the two artificial intelligence models. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to update ratings representing the performance of the plurality of artificial intelligence models based on the plurality of match results of the plurality of matches obtained based on proceeding with the tournament.

[0008] A method is disclosed. The method may be performed by an electronic device. The method may include the operation of obtaining outputs of a plurality of artificial intelligence models for a test scenario. The method may include the operation of inputting the outputs of two artificial intelligence models selected as participants among the plurality of artificial intelligence models into an evaluation model for each of the plurality of matches included in the tournament, in order to conduct a tournament for the test scenario, and the operation of obtaining the match results of the evaluation model for the two artificial intelligence models. The method may include the operation of updating ratings representing the performance of the plurality of artificial intelligence models based on the plurality of match results of the plurality of matches obtained by conducting the tournament.

[0009] A non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium may store a program comprising instructions. When the instructions are executed individually or collectively by at least one processor of an electronic device, the electronic device may cause the electronic device to obtain outputs of a plurality of artificial intelligence models for a test scenario. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to input the outputs of two artificial intelligence models selected as participants among the plurality of artificial intelligence models into an evaluation model for each of the plurality of matches included in the tournament, in order to proceed with a tournament for the test scenario, and to obtain the match results of the evaluation model for the two artificial intelligence models. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause ratings representing the performance of the plurality of artificial intelligence models to be updated based on the results of the plurality of matches obtained based on the progress of the tournament.

[0010] According to one embodiment, the electronic device can consume fewer time resources to compare and evaluate the performance of multiple artificial intelligence models by comparing and evaluating the performance of multiple artificial intelligence models through a tournament method.

[0011] According to one embodiment, the electronic device can evaluate the relative performance of a plurality of artificial intelligence models more reliably by assigning Elo ratings to the plurality of artificial intelligence models based on the match results of the match between the plurality of artificial intelligence models.

[0012] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0013] FIG. 1 is an exemplary block diagram of an electronic device according to one embodiment.

[0014] FIG. 2a is a drawing illustrating an example of modules of an electronic device that performs evaluation of artificial intelligence models according to one embodiment.

[0015] FIG. 2b is a diagram illustrating an example of a data set including model outputs of artificial intelligence models for test inputs according to one embodiment.

[0016] FIG. 3 is a drawing illustrating an example of tournaments of artificial intelligence models according to one embodiment.

[0017] FIG. 4 is a drawing illustrating an example of a user interface (UI) provided by an electronic device according to one embodiment.

[0018] FIG. 5 is a flowchart illustrating the operation of an electronic device that performs evaluation of artificial intelligence models according to one embodiment.

[0019] FIG. 6 is a flowchart illustrating the operation of an electronic device conducting a tournament according to one embodiment.

[0020] FIG. 7 is a flowchart illustrating the operation of an electronic device for updating the ratings of artificial intelligence models according to one embodiment.

[0021] The electronic device (or external electronic device) according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, a server, or a consumer electronics device. The electronic device (or external electronic device) according to the embodiments of this document is not limited to the devices described above.

[0022] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or multiple items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that the component may be connected to the other component directly (e.g., via wired or wireless) or indirectly (through a 3rd component).

[0023] As used in this document, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed as a whole, or a minimum unit of a component or part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0024] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium readable by a machine (e.g., an electronic device (101)). For example, a processor of the machine (e.g., an electronic device (101)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to at least one called instruction. One or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0025] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM , or App Store TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0026] According to various embodiments, each component (e.g., module or program) of the described components may include a singular or multiple entities. According to various embodiments, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as they were performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically; one or more of the operations may be executed in a different order; omitted; or one or more other operations may be added.

[0027] Hereinafter, various embodiments of this document will be described with reference to the attached drawings.

[0028] FIG. 1 is an exemplary block diagram of an electronic device according to one embodiment.

[0029] The electronic device (101) of FIG. 1 may be connected to an external electronic device (102) and a server (103) via wired and / or wireless communication. For example, the external electronic device (102) may receive a service provided by the electronic device (101) based on accessing a web page (or resource (e.g., a URL (uniform resource locator))) provided by the electronic device (101).

[0030] In one embodiment, the electronic device (101) may be a device for evaluating the performance of artificial intelligence (AI) models in response to a request from an external electronic device (102). In one embodiment, the electronic device (101) may be referred to as a performance evaluation device (or performance evaluation server) in terms of evaluating the performance of artificial intelligence models. In one embodiment, the electronic device (101) may be referred to as a performance evaluation service device (or performance evaluation service server) in terms of providing a service to the external electronic device (102) for evaluating the performance of artificial intelligence models.

[0031] Referring to FIG. 1, an electronic device (101) according to one embodiment may include a processor (120), memory (130), a communication circuit (140), or a combination thereof. In one embodiment, the processor (120), memory (130), and communication circuit (140) may be electrically and / or operationally connected to each other by an electronic component (or electrical component), such as a communication bus (110). Hereinafter, the hardware being operationally coupled may refer to a direct connection or an indirect connection between the hardware being established via wired or wireless means so that a second hardware is controlled by a first hardware among the hardware. Although the hardware is shown in different blocks, the embodiment is not limited thereto. Some of the hardware of FIG. 1 may be included in a single integrated circuit, such as a system on a chip (SoC). The type and / or number of hardware included in the electronic device (101) are not limited to those shown in FIG. 1. For example, the electronic device (101) may include only some of the hardware shown in FIG. 1. For example, the electronic device (101) may include other hardware not shown in FIG. 1 (e.g., input devices (e.g., microphone, mouse, and / or keyboard), output devices (e.g., speaker, display), or sensors).

[0032] According to one embodiment, the electronic device (101) may include hardware for processing data based on one or more instructions. For example, the hardware for processing data may include a processor (120). For example, the hardware for processing data may include an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The processor (120) may have the structure of a single-core processor or the structure of a multi-core processor such as a dual core, a quad core, a hexa core, or an octa core. The following operations may be performed by the processor (120) of the electronic device (101).

[0033] According to one embodiment, within the memory (130) of the electronic device (101), one or more instructions (or commands) representing operations and / or operations to be performed on data by the processor (120) of the electronic device (101) may be stored. A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the electronic device (101) and / or the processor (120) may perform at least one of the operations of FIGS. 5 through 7 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. In the following, the statement that an application is installed in the electronic device (101) may mean that one or more instructions provided in the form of an application are stored in memory (130), and that one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the electronic device (101)) by the processor (120). For example, the application may include a program and / or library related to a service provided to a user.

[0034] In one embodiment, the memory (130) may include volatile memory such as random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). For example, the volatile memory may include at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, or pseudo SRAM (PSRAM). For example, the non-volatile memory may include at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, solid state drive (SSD), or embedded multi-media card (eMMC).

[0035] According to one embodiment, a communication circuit (140) of an electronic device (101) may include hardware components for supporting the transmission and / or reception of electrical signals (or data) between the electronic device (101) and an external electronic device (102) and / or a server (103). For example, the communication circuit (140) may include at least one of a modem, an antenna, and an optic / electronic converter. The communication circuit (140) may support the transmission and / or reception of electrical signals (or data) based on various types of protocols such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), Zigbee, LTE (long term evolution), and / or 5G NR (new radio). For example, the electronic device (101) can receive information (or data) from an external electronic device (102) and / or a server (103) through a communication circuit (140). For example, the electronic device (101) can transmit information (or data) to an external electronic device (102) and / or a server (103) through a communication circuit (140).

[0036] In one embodiment, the external electronic device (102) may include a terminal owned by the user. For example, the terminal may include a personal computer (PC) such as a laptop and a desktop, a smartphone, a smart pad, a tablet PC, a smart watch, and a head-mounted device (HMD).

[0037] According to one embodiment, an external electronic device (102) may include a display (150). In one embodiment, the display (150) may output visualized information to a user. For example, the display (150) may output visualized information generated by a processor and / or a GPU (graphic processing unit). The display (150) may include a flexible display, a flat panel display (FPD), a liquid crystal display (LCD), a plasma display panel (PDP), and / or a plurality of light emitting diodes (LEDs). For example, the display (150) may be used to display an image obtained from the electronic device (101). For example, the external electronic device (102) may display an image on a part of the display (150) under the control of a display driving circuit. However, it is not limited thereto.

[0038] In one embodiment, the server (103) may be a device for providing a service based on the evaluation model (160) to the electronic device (101). In one embodiment, the evaluation model (160) may be a language model (or a large language model) that recognizes human language and characters and performs various natural language processing. In one embodiment, the evaluation model (160) may include a large language model that has been pre-trained with a vast amount of data. The evaluation model (160) may include a transformer model that includes an encoder and / or decoder that performs operations through attention. For example, the evaluation model (160) may be GPT (generative pre-trained transformer), Gemini, BERT (bidirectional encoder representations from transformers), PaLM (pathways language model), Claude, and / or LLaMA.

[0039] In one embodiment, the server (103) may be an on-premises server operated by the same operator as the operator of the electronic device (101). In one embodiment, the server (103) may be a server operated in an environment where the electronic device (101) is installed. However, it is not limited thereto. In one embodiment, the server (103) may be a server that provides cloud services.

[0040] FIG. 2a is a diagram illustrating an example of modules of an electronic device for evaluating artificial intelligence models according to one embodiment. FIG. 2b is a diagram illustrating an example of a data set including model outputs of artificial intelligence models for test inputs according to one embodiment. FIG. 3 is a diagram illustrating an example of tournaments of artificial intelligence models according to one embodiment. FIG. 4 is a diagram illustrating an example of a user interface (UI) provided by an electronic device according to one embodiment.

[0041] In FIG. 2a, the memory (130) may correspond to the memory (130) of FIG. 1. Modules (230, 240, 250, 260) included (or stored) in the memory (130) of FIG. 2a may be accessible by the electronic device (101) and / or processor (120) of FIG. 1. Modules (230, 240, 250, 260) included in the memory (130) of FIG. 2a may be executed (or controlled) by the electronic device (101) and / or processor (120) of FIG. 1.

[0042] Figures 2a through 4 are described below with reference to the electronic device (101) of Figure 1 and / or the components of the electronic device (101).

[0043] Referring to FIG. 2a, memory (130) may include (or store) a data set (201), a data set combiner (230), a match management unit (240), a record management unit (250), and a visualization unit (260). In one embodiment, the data set combiner (230), the match management unit (240), the record management unit (250), and the visualization unit (260) may be implemented as a single integrated program. In one embodiment, the program including the data set combiner (230), the match management unit (240), the record management unit (250), and the visualization unit (260) may be referred to as a manager program. In one embodiment, the program including the data set combiner (230), the match management unit (240), the record management unit (250), and the visualization unit (260) may be referred to as a VARCO ARENA.

[0044] In one embodiment, the electronic device (101) can perform functions defined by the data set combiner (230), the match management unit (240), the record management unit (250), and the visualization unit (260) based on instructions included in the data set combiner (230), the match management unit (240), the record management unit (250), and the visualization unit (260) being executed by the processor (120). In the following, the data set combiner (230), match management unit (240), record management unit (250), and visualization unit (260) are exemplified as performing operations, but it can be understood that the electronic device (101) performs functions defined by the data set combiner (230), match management unit (240), record management unit (250), and visualization unit (260) based on instructions included in the data set combiner (230), match management unit (240), record management unit (250), and visualization unit (260) being executed by the processor (120).

[0045] In one embodiment, the data set combiner (230) can acquire a data set (201). In one embodiment, the data set combiner (230) can acquire a data set (201) from an external electronic device (102) that is wired and / or wirelessly connected via a communication circuit (140). For example, the data set combiner (230) can request the data set (201) from the external electronic device (102) based on receiving a request from the external electronic device (102) for a performance evaluation of a plurality of language models (e.g., a plurality of artificial intelligence models (211, 213, 215) of FIG. 2B). For example, the data set combiner (230) can receive the data set (201) from the external electronic device (102) after requesting the data set (201) from the external electronic device (102). In one embodiment, the request for the data set (201) may include a plurality of test scenarios (e.g., a plurality of test scenarios (221, 223, 225, 227) of FIG. 2b) required for the external electronic device (102) to configure (or, generate) the data set (201).

[0046] In one embodiment, referring to FIG. 2b, the dataset (201) may include model outputs of a plurality of artificial intelligence models (211, 213, 215). For example, the dataset (201) may include model outputs of a plurality of artificial intelligence models (211, 213, 215) for test inputs. In one embodiment, the plurality of artificial intelligence models (211, 213, 215) may be artificial intelligence models (or language models) (or large language models) that recognize human language and characters and perform various natural language processing. In one embodiment, the plurality of artificial intelligence models (211, 213, 215) may be artificial intelligence models (or language models) (or large language models) (or generative artificial intelligence models) that are currently learning (or have not yet completed learning). For example, a plurality of artificial intelligence models (211, 213, 215) may include a transformer model that includes an encoder and / or decoder that performs operations through attention.

[0047] For example, at least two of the multiple artificial intelligence models (211, 213, 215) may be the same artificial intelligence model. For example, the sameness of at least two language models may include having at least two language models with the same neural network structure. For example, the sameness of at least two language models may include having the same structure (or architecture) of at least two language models. For example, the same two language models (or at least two language models with the same structure) (or at least two language models with the same neural network structure) may be language models trained on a dataset with different training times (e.g., number of epochs). For example, if language model (211) and language model (213) are the same language models, the number of times language model (211) and language model (213) have been trained on the dataset may be different. For example, if language model (211) and language model (213) are the same language models, language model (211) may have been trained on the dataset at least once, and language model (213) may not have been trained on the dataset. However, it is not limited thereto. The two identical language models (or at least two language models of the same structure) (or at least two language models having a neural network of the same structure) may be language models trained based on different learning techniques.

[0048] For example, at least two of the multiple artificial intelligence models (211, 213, 215) may be different artificial intelligence models. For example, the difference between at least two language models may include at least one of the neural network structure, the data set for learning, the number of training iterations, or the learning technique being different. For example, the difference between at least two language models may include at least two language models having neural networks of different structures. For example, the difference between at least two language models may include the structure (or architecture) of at least two language models being different. For example, the two different language models (or at least two language models of different structures) (or at least two language models having neural networks of different structures) may be language models learned based on the same data set. For example, two different language models (or at least two language models of different structures) (or at least two language models having neural networks of different structures) may be language models trained on the same dataset with different training iterations (e.g., number of epochs). However, they are not limited thereto. For example, two different language models (or at least two language models of different structures) (or at least two language models having neural networks of different structures) may be language models trained based on different datasets. For example, two different language models (or at least two language models of different structures) (or at least two language models having neural networks of different structures) may be language models trained on different datasets with different training iterations (e.g., number of epochs).For example, two different language models (or at least two language models of different structures) (or at least two language models having neural networks of different structures) may be language models learned based on different learning techniques.

[0049] In one embodiment, the test input may be an input according to a plurality of test scenarios (221, 223, 225, 227). In one embodiment, the test input may be a prompt included in each of the plurality of test scenarios (221, 223, 225, 227).

[0050] In one embodiment, a plurality of test scenarios (221, 223, 225, 227) may include prompts that consider evaluation aspects of a plurality of artificial intelligence models (211, 213, 215). For example, the evaluation aspects may be aspects for evaluating answers to questions and / or the generation of content. For example, answers to questions may include answers to problems for which a correct answer exists (e.g., math problems, physics problems). For example, answers to questions may include answers to problems for which no correct answer exists (e.g., value judgment problems). For example, the generation of content may include descriptions of existing content, summaries of existing content, the generation of content to be added to existing content, and / or the generation of new content.

[0051] In one embodiment, the prompt may be a set of words. In one embodiment, the prompt may be a set of words for a query. For example, the query may include a request to compare two or more objects (e.g., "Which is the correct sentence between A and B?", "What are the advantages and disadvantages of A and B?", "What is the difference between civil and criminal matters?"), a request for a method to achieve a goal (e.g., "What is the best way to get to A?", "What is the recommended answer regarding A?"), and / or a request for a correct answer to a problem (e.g., "What is the answer when A and B are added?"). In one embodiment, the prompt may be a set of words for generating specific content. For example, the specific content may include a transcript (e.g., "Write a transcript of the conversation with A"), writing sentences (e.g., extending text (e.g., "Describe A more richly"), continuing text (e.g., "Describe the next part of A's story"), and / or summarizing (e.g., "Summarize A").

[0052] In one embodiment, the model outputs may be the outputs of a plurality of artificial intelligence models (211, 213, 215) according to each of a plurality of test scenarios (221, 223, 225, 227). The model outputs of the plurality of artificial intelligence models (211, 213, 215) may be the outputs generated by the plurality of artificial intelligence models (211, 213, 215) based on a prompt for each of the plurality of test scenarios (221, 223, 225, 227).

[0053] In one embodiment, referring again to FIG. 2a, the data set combiner (230) can obtain a data set (201) including model outputs of a plurality of artificial intelligence models (211, 213, 215) for a plurality of test scenarios (221, 223, 225, 227).

[0054] In one embodiment, the data set combiner (230) can classify the outputs of a plurality of artificial intelligence models (211, 213, 215) according to a plurality of test scenarios (221, 223, 225, 227). For example, the data set combiner (230) can classify the outputs of a plurality of artificial intelligence models (211, 213, 215) for a plurality of prompts included in a plurality of test scenarios (221, 223, 225, 227) according to the test scenarios. For example, when outputs of multiple artificial intelligence models (211, 213, 215) for four prompts (e.g., "What is the difference between civil and criminal?", "What is the answer when A and B are added?", "Write a transcript of the conversation with A", "Summarize A") according to four test scenarios (221, 223, 225, 227) are obtained, the data set combiner (230) can classify the outputs of the multiple artificial intelligence models (211, 213, 215) for the prompt "What is the difference between civil and criminal?" of the test scenario (221) as model outputs of the test scenario (221). Likewise, the data set combiner (230) can classify the outputs of multiple artificial intelligence models (211, 213, 215) for the prompt "What is the answer when A and B are added?" of the test scenario (223). The data set combiner (230) can classify the outputs of multiple artificial intelligence models (211, 213, 215) for the prompt "Write a transcript of the conversation with A" of the test scenario (225) for the test scenario (225). The data set combiner (230) can classify the outputs of multiple artificial intelligence models (211, 213, 215) for the prompt "Summarize A" of the test scenario (227) for the test scenario (227).

[0055] In one embodiment, the data set combiner (230) can classify the outputs of a plurality of artificial intelligence models (211, 213, 215) according to evaluation aspects. For example, if the prompt for test scenario (221) is "What is the difference between civil and criminal?", the prompt for test scenario (223) is "What is the answer when A and B are added?", the prompt for test scenario (225) is "Write a transcript of the conversation with A", and the prompt for test scenario (227) is "Summarize A", the data set combiner (230) can classify the test scenarios (221, 223) as test scenarios having a specific evaluation aspect (e.g., answer to a question). Likewise, the data set combiner (230) can classify the test scenario (225) as a test scenario having a specific evaluation aspect (e.g., writing a transcript). The data set combiner (230) can classify test scenarios (227) into test scenarios having specific evaluation aspects (e.g., summary creation).

[0056] In one embodiment, the match management unit (240) may generate a plurality of tournaments (310, 330, 350, 370). In one embodiment, each of the plurality of tournaments (310, 330, 350, 370) may have a tournament bracket in which a plurality of matches are connected according to a tree structure. For example, referring to FIG. 3, a plurality of matches (311, 313, 315, 317, 321, 323, 325) included in the tournament (310) may be connected in a tree structure. In one embodiment, the plurality of tournaments (310, 330, 350, 370) may be single-elimination tournaments. For example, a single-elimination tournament may be structured such that the winning AI model (e.g., the AI ​​model labeled 'W' in FIG. 3) proceeds to the next round of matches, and the losing AI model (e.g., the AI ​​model labeled 'L' in FIG. 3) is eliminated from the tournament. In one embodiment, one or more matches (311, 313, 315, 317, 321, 323, 325) may be divided into one or more rounds. For example, among one or more matches (311, 313, 315, 317, 321, 323, 325), the matches (311, 313, 315, 317) in which each of the multiple AI models (211, 213, 215) participates as the first participant may be referred to as the matches of the first round. However, this is not limited thereto. Multiple tournaments (310, 330, 350, 370) may be double-elimination tournaments, or round-robin tournaments (or full leagues).

[0057] In one embodiment, referring again to FIG. 2a, the match management unit (240) may determine a tournament bracket for a tournament to evaluate the performance of a plurality of artificial intelligence models (211, 213, 215). In one embodiment, the electronic device (101) may determine a plurality of tournament brackets for conducting a plurality of tournaments (310, 330, 350, 370) based on the outputs of the plurality of artificial intelligence models (211, 213, 215) for each of the plurality of test scenarios (221, 223, 225, 227).

[0058] In one embodiment, the match management unit (240) may conduct a plurality of tournaments (310, 330, 350, 370). For example, the match management unit (240) may identify the match results (or winner AI model and / or loser AI model) of a plurality of matches for each of the plurality of tournaments (310, 330, 350, 370). Conducting the plurality of tournaments (310, 330, 350, 370) may include conducting a plurality of matches included in each of the plurality of tournaments (310, 330, 350, 370) according to rounds. For example, conducting a plurality of matches according to rounds may include conducting the matches sequentially according to the order of the rounds. For example, conducting the matches sequentially according to the order of the rounds may include conducting the matches of the first round, and then conducting the matches of the next round. For example, referring to FIG. 3, the match management unit (240) may proceed sequentially with the first round matches (311, 313, 315, 317) and then proceed sequentially with the next round matches (321, 323). Likewise, the match management unit (240) may proceed sequentially with the second round matches (321, 323) and then proceed with the next round match (325). In one embodiment, the match result may indicate the winning artificial intelligence model and / or the losing artificial intelligence model among the two artificial intelligence models participating in the match.

[0059] In one embodiment, conducting a match may include the match management unit (240) determining superiority between two artificial intelligence models participating in the match (or obtaining evaluation results between two artificial intelligence models) (or identifying the match result of the match). For example, the match management unit (240) may identify two artificial intelligence models participating in the match among a plurality of artificial intelligence models (211, 213, 215). For example, the match management unit (240) may identify the match result (or the winning artificial intelligence model and / or the losing artificial intelligence model) by inputting the outputs of the two artificial intelligence models into the evaluation model (160). For example, the match management unit (240) may request the evaluation model (160) to evaluate the outputs of the two artificial intelligence models included in the match. For example, the match management unit (240) may request the evaluation model (160) to evaluate the outputs of two artificial intelligence models included in the match by using the application programming interface (API) provided by the evaluation model (160). For example, the match management unit (240) may receive the match results from the evaluation model (160).

[0060] In one embodiment, the match management unit (240) can identify the match result (or, the winner AI model and / or the loser AI model) by inputting the outputs of two artificial intelligence models and the prompt used to obtain the outputs of the two artificial intelligence models into the evaluation model (160).

[0061] In one embodiment, the match management unit (240) can identify the match result (or, the winning AI model and / or the losing AI model) by inputting the outputs of two AI models, the prompt used to obtain the outputs of the two AI models, and the evaluation criteria into the evaluation model (160). In one embodiment, the evaluation criteria may be criteria for determining superiority between the outputs of two AI models. For example, the evaluation criteria may be a command (or request) (or prompt) to output the AI ​​model that produced the correct answer as the winning AI model when the evaluation aspect relates to a problem where a correct answer exists. For example, the evaluation criteria may be a command (or request) (or prompt) to output the AI ​​model that produced the output corresponding to a specified value (e.g., a value specified by the developer in a trolley dilemma) as the winning AI model when the evaluation aspect relates to a problem where no correct answer exists (e.g., a value judgment problem). In one embodiment, the specified value may correspond to the development direction of the artificial intelligence model pursued by the developer. For example, if the evaluation criterion is related to content creation, the evaluation criterion may be a command (or request) (or prompt) to output an artificial intelligence model that has produced an output consistent with the specified content creation direction (e.g., the content creation direction specified by the developer) as the winning artificial intelligence model.

[0062] In one embodiment, the match management unit (240) can predict the cost and / or time required to request the evaluation of outputs from the evaluation model (160). For example, the match management unit (240) can predict the cost and / or time required by the evaluation model (160) for evaluating each of a plurality of matches. In one embodiment, the match management unit (240) can adjust the query speed (or the cycle of requesting evaluation from the evaluation model (160)) in consideration of the predicted cost and / or time.

[0063] In one embodiment, the match management unit (240) may store match results in memory (130). For example, the match management unit (240) may store in memory (130) data in which the match results of a plurality of tournaments (310, 330, 350, 370) and a plurality of matches included in the plurality of tournaments (310, 330, 350, 370) are mapped to each other.

[0064] In one embodiment, the record management unit (250) may manage tournament results including match results. In one embodiment, the tournament result of each of the plurality of tournaments (310, 330, 350, 370) may include the match results of each of the plurality of matches of each of the plurality of tournaments (310, 330, 350, 370).

[0065] In one embodiment, the record management unit (250) can evaluate the performance of a plurality of artificial intelligence models (211, 213, 215) based on the tournament results. In one embodiment, the record management unit (250) can evaluate the ratings of the plurality of artificial intelligence models (211, 213, 215) based on the match results included in the tournament results. In one embodiment, the record management unit (250) can update the ratings representing the performance of the plurality of artificial intelligence models (211, 213, 215) based on the match results included in the tournament results.

[0066] For example, the record management unit (250) can identify an expected winning rate based on the difference in ratings between the rating of the winner AI model and the rating of the loser AI model for each of the multiple matches. For example, the record management unit (250) can update the rating of the winner AI model and the rating of the AI ​​model based on the score according to the match result and the expected winning rate. For example, the record management unit (250) can update the ratings based on the following mathematical formula 1.

[0067]

[0068] In mathematical formula 1, R updated can represent the updated rating of an artificial intelligence model. R current α may represent the rating of the artificial intelligence model before the update (or, current rating). α may represent a weight (e.g., 10). V may be a value representing the matching result. For example, if the artificial intelligence model is a winning artificial intelligence model, V may be 1. For example, if the artificial intelligence model is a losing artificial intelligence model, V may be 0. E may represent the expected win rate of the artificial intelligence model against the opponent artificial intelligence model. For example, the expected win rate (or probability of winning) may be determined based on the difference between the current rating of the artificial intelligence model and the current rating of the opponent artificial intelligence model. For example, if the difference between the above subtraction value is a specified score (e.g., 400 points), the ratio of the probability of winning (e.g., 90.90%) to the probability of losing (e.g., 9.09%) may be 10. For example, if the above subtraction value is 0, the probability of losing (e.g., 50%) and the probability of winning (e.g., 50%) may be equal.

[0069] In one embodiment, the record management unit (250) may assign one rating to each of the plurality of artificial intelligence models (211, 213, 215) for each of the evaluation aspects. For example, the record management unit (250) may assign one rating to each of the plurality of artificial intelligence models (211, 213, 215) based on match results according to one or more test scenarios for evaluating one evaluation aspect (e.g., answer to a question). For example, the record management unit (250) may assign one rating to each of the plurality of artificial intelligence models (211, 213, 215) based on match results according to one or more test scenarios for evaluating another evaluation aspect (e.g., writing a transcript). For example, each of the plurality of artificial intelligence models (211, 213, 215) may have one rating for each evaluation aspect.

[0070] In one embodiment, the record management unit (250) can generate statistical data for each of the plurality of artificial intelligence models (211, 213, 215) based on match results. For example, the record management unit (250) can generate statistical data summarizing the number of wins, number of losses, and / or match opponents for each of the plurality of artificial intelligence models (211, 213, 215).

[0071] In one embodiment, the visualization unit (260) may transmit data related to the performances of a plurality of artificial intelligence models (211, 213, 215) to an external electronic device (102) through a communication circuit (140) so that the performances of a plurality of artificial intelligence models (211, 213, 215) are displayed through a display (150) of an external electronic device (102). In one embodiment, the data related to the performances may include ratings representing the performances of a plurality of artificial intelligence models (211, 213, 215). In one embodiment, the data related to the performances may include statistical data summarizing the match results of each of the plurality of artificial intelligence models (211, 213, 215). In one embodiment, the data related to the performances may include visual objects visualizing the ratings and / or statistical data of each of the plurality of artificial intelligence models (211, 213, 215). For example, visual objects may include graphs and / or tables representing the performance of multiple artificial intelligence models (211, 213, 215) according to evaluation aspects.

[0072] In one embodiment, the external electronic device (102) can display a screen through the display (150) based on data related to performance transmitted from the visualization unit (260) of the electronic device (101).

[0073] For example, referring to FIG. 4, the screen (401) may be a result dashboard showing the performance of a plurality of artificial intelligence models (211, 213, 215) displayed through a display (150) of an external electronic device (102). For example, the screen (401) may include one or more areas (410, 430, 450). In one embodiment, the area (410) may be an area for determining information to be displayed in the areas (430, 450). For example, the area (410) may display text objects representing one or more evaluation aspects (e.g., overall, expand, evaluate, textboard, banner, quest, summary, custom task, append, character, naming, conversation generation, prompt, and / or variation).

[0074] For example, based on the selection of a text object (e.g., overall) within the area (410), the external electronic device (102) may request data related to performances corresponding to the selected text object (e.g., overall) from the electronic device (101). For example, the electronic device (101) may transmit data related to performances corresponding to the selected text object to the external electronic device (102) in response to the request. For example, the electronic device (101) may transmit data to the external electronic device (102) representing the average rating of ratings in various evaluation aspects of each of the multiple artificial intelligence models (211, 213, 215) in response to the request for overall. For example, the electronic device (101) may transmit data to the external electronic device (102) representing a graph of ratings in various evaluation aspects of each of the multiple artificial intelligence models (211, 213, 215) in response to the request for overall.

[0075] For example, in area (430), a table (440) in which multiple artificial intelligence models are sorted according to their ratings (sorted in descending order) may be displayed. For example, in response to a request for an overall table (440), in area (430), a table (440) in which the average ratings of multiple artificial intelligence models are sorted may be displayed. For example, in area (450), a graph (460) displaying the ratings of each evaluation aspect of multiple artificial intelligence models in a circular format and a legend (470) of multiple artificial intelligence models may be displayed.

[0076] As described above, the electronic device (101) can evaluate the performance of multiple artificial intelligence models (211, 213, 215) through a tournament method that consumes minimal time resources. For example, the electronic device (101) can evaluate the performance of multiple artificial intelligence models (211, 213, 215) through matches corresponding to the number of multiple artificial intelligence models (211, 213, 215) minus 1 for one test scenario. Additionally, when the electronic device (101) evaluates the performance of M multiple artificial intelligence models for N test scenarios, it may require fewer comparisons by performing N * (M-1) matches.

[0077] Additionally, the electronic device (101) can evaluate the relative performance of multiple artificial intelligence models more reliably by assigning a rating (or Elo rating) to multiple artificial intelligence models based on the match results of multiple random-match-tournaments occurring for each of the multiple test scenarios (221, 223, 225, 227) in order to prevent the ranking from being reversed by the tournament method according to the tournament bracket (i.e., easily reaching the top rank according to the bracket).

[0078] FIG. 5 is a flowchart illustrating the operation of an electronic device that performs evaluation of artificial intelligence models according to one embodiment.

[0079] At least one of the operations of FIG. 5 may be performed by the electronic device (101) of FIG. 1. For example, at least one of the operations of FIG. 5 may be controlled by the processor (120) of FIG. 1. Each of the operations of FIG. 5 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each of the operations may be changed, and at least two operations may be performed in parallel.

[0080] Hereinafter, the operations of FIG. 5 are described with reference to FIG. 1 to FIG. 4.

[0081] Referring to FIG. 5, in operation 510, the electronic device (101) can obtain outputs of a plurality of artificial intelligence models (211, 213, 215). In one embodiment, the electronic device (101) can obtain outputs of a plurality of artificial intelligence models (211, 213, 215) from an external electronic device (102) that is connected via wired and / or wireless communication through a communication circuit (140). However, it is not limited thereto. For example, the electronic device (101) can obtain outputs by inputting a prompt to the plurality of artificial intelligence models (211, 213, 215).

[0082] In operation 520, the electronic device (101) may determine a tournament (310) for a plurality of artificial intelligence models (211, 213, 215). In one embodiment, the tournament (310) may have a tournament bracket in which a plurality of matches (311, 313, 315, 317, 321, 323, 325) are connected according to a tree structure. In one embodiment, the tournament (310) may be a single-elimination tournament. In one embodiment, the plurality of matches (311, 313, 315, 317, 321, 323, 325) may be divided into one or more rounds. For example, among the multiple matches (311, 313, 315, 317, 321, 323, 325), the matches (311, 313, 315, 317) in which each of the multiple artificial intelligence models (211, 213, 215) participates as a participant for the first time may be referred to as the matches of the first round.

[0083] In operation 530, the electronic device (101) may conduct a tournament for a plurality of artificial intelligence models (211, 213, 215) according to the tournament (310). Operation 530 can be described in detail below with reference to FIG. 6.

[0084] For example, the electronic device (101) can identify the match results (or, winner AI model and / or loser AI model) of a plurality of matches of the tournament (310). Conducting the tournament (310) may include conducting a plurality of matches (311, 313, 315, 317, 321, 323, 325) included in each of the tournament (310) in a round. For example, conducting a plurality of matches (311, 313, 315, 317, 321, 323, 325) in a round may include conducting the matches sequentially in the order of the rounds. For example, proceeding with matches sequentially according to the order of the rounds may include proceeding with the matches of the first round (311, 313, 315, 317) and then proceeding with the matches of the next round (321, 323). In one embodiment, the match result may indicate the winning artificial intelligence model and / or the losing artificial intelligence model among the two artificial intelligence models participating in the match.

[0085] In one embodiment, conducting a match may include the electronic device (101) determining superiority between two artificial intelligence models that are participants in the match (or obtaining evaluation results between two artificial intelligence models) (or identifying the match result of the match). For example, the electronic device (101) may identify two artificial intelligence models that are participants in the match among a plurality of artificial intelligence models (211, 213, 215). For example, the electronic device (101) may identify the match result (or the winning artificial intelligence model and / or the losing artificial intelligence model) by inputting the outputs of the two artificial intelligence models into an evaluation model (160).

[0086] In one embodiment, the electronic device (101) can identify a match result (or a winner artificial intelligence model and / or a loser artificial intelligence model) by inputting the outputs of two artificial intelligence models and a prompt used to obtain the outputs of the two artificial intelligence models into an evaluation model (160).

[0087] In one embodiment, the electronic device (101) can identify a match result (or, a winning artificial intelligence model and / or a losing artificial intelligence model) by inputting the outputs of two artificial intelligence models, a prompt used to obtain the outputs of the two artificial intelligence models, and an evaluation criterion into an evaluation model (160). In one embodiment, the evaluation criterion may be a criterion for determining superiority between the outputs of the two artificial intelligence models.

[0088] In operation 540, the electronic device (101) can evaluate the performance of a plurality of artificial intelligence models (211, 213, 215) according to the tournament results. Operation 540 can be described in detail below with reference to FIG. 7.

[0089] In one embodiment, the electronic device (101) can evaluate the performance of a plurality of artificial intelligence models (211, 213, 215) according to the tournament results. In one embodiment, the electronic device (101) can evaluate the ratings of the plurality of artificial intelligence models (211, 213, 215) based on the match results included in the tournament results. In one embodiment, the electronic device (101) can update the ratings representing the performance of the plurality of artificial intelligence models (211, 213, 215) based on the match results included in the tournament results.

[0090] For example, the electronic device (101) can identify an expected winning rate based on the difference in ratings between the rating of the winner AI model and the rating of the loser AI model for each of the plurality of matches (311, 313, 315, 317, 321, 323, 325). For example, the electronic device (101) can update the rating of the winner AI model and the rating of the loser AI model based on the score according to the match result and the expected winning rate. For example, the electronic device (101) can update the ratings based on the mathematical formula 1.

[0091] Subsequently, the electronic device (101) can transmit data related to the performances of the multiple artificial intelligence models (211, 213, 215) to the external electronic device (102) through the communication circuit (140) so that the performances of the multiple artificial intelligence models (211, 213, 215) are displayed through the display (150) of the external electronic device (102).

[0092] FIG. 6 is a flowchart illustrating the operation of an electronic device conducting a tournament according to one embodiment.

[0093] At least one of the operations of FIG. 6 may be performed by the electronic device (101) of FIG. 1. For example, at least one of the operations of FIG. 6 may be controlled by the processor (120) of FIG. 1. Each of the operations of FIG. 6 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each of the operations may be changed, and at least two operations may be performed in parallel.

[0094] In one embodiment, the operations of FIG. 6 may be included in the operation 530 of FIG. 5. In one embodiment, the operations of FIG. 6 may be performed repeatedly until the tournament is completed.

[0095] Below, the operations of FIG. 6 are described with reference to FIG. 1 to FIG. 5.

[0096] Referring to FIG. 6, in operation 610, the electronic device (101) can identify matches according to the tournament. For example, the electronic device (101) can identify a match that satisfies a condition among a plurality of matches (311, 313, 315, 317, 321, 323, 325). For example, the electronic device (101) can identify a match among a plurality of matches (311, 313, 315, 317, 321, 323, 325) for which the match result has not been identified. For example, the electronic device (101) can identify a match among a plurality of matches (311, 313, 315, 317, 321, 323, 325) for which the participants have been determined. For example, the electronic device (101) can identify a match in which participants are determined among a plurality of matches (311, 313, 315, 317, 321, 323, 325) and the match result is not identified.

[0097] In one embodiment, the electronic device (101) can identify one match according to the order of rounds when there are two or more matches satisfying the condition. For example, a match of a lower round may be selected before a match of a higher round. For example, any one match among matches of the same round may be selected.

[0098] In operation 620, the electronic device (101) can select two artificial intelligence models from a plurality of artificial intelligence models (211, 213, 215) as participants in the identified match.

[0099] In operation 630, the electronic device (101) can input the outputs of two artificial intelligence models to the evaluation model (160). For example, the electronic device (101) can request the evaluation model (160) to evaluate the outputs of the two artificial intelligence models included in the match. For example, the electronic device (101) can request the evaluation model (160) to evaluate the outputs of the two artificial intelligence models included in the match by using an application programming interface (API) provided by the evaluation model (160). For example, the electronic device (101) can receive the match result from the evaluation model (160).

[0100] In operation 640, the electronic device (101) can identify the winning artificial intelligence model as a participant in the next match of the tournament based on the match result of the evaluation model (160). In one embodiment, the next match may be the next round match connected to the match where the match result was obtained according to the tree structure.

[0101] In one embodiment, the operations of FIG. 6 may be repeated until the electronic device (101) obtains match results for all matches (311, 313, 315, 317, 321, 323, 325) within the tournament (310).

[0102] FIG. 7 is a flowchart illustrating the operation of an electronic device for updating the ratings of artificial intelligence models according to one embodiment.

[0103] At least one of the operations of FIG. 7 may be performed by the electronic device (101) of FIG. 1. For example, at least one of the operations of FIG. 7 may be controlled by the processor (120) of FIG. 1. Each of the operations of FIG. 7 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each of the operations may be changed, and at least two operations may be performed in parallel.

[0104] In one embodiment, the operations of FIG. 7 may be included in the operation 540 of FIG. 5.

[0105] Below, the operations of FIG. 7 are described with reference to FIG. 1 to FIG. 5.

[0106] Referring to FIG. 7, in operation 710, the electronic device (101) can identify the rating of the winner artificial intelligence model and the rating of the loser artificial intelligence model. The electronic device (101) can identify the rating of the winner artificial intelligence model and the rating of the loser artificial intelligence model for a selected match among a plurality of matches (311, 313, 315, 317, 321, 323, 325) within the tournament (310). For example, the electronic device (101) can select a match that satisfies the conditions among the plurality of matches (311, 313, 315, 317, 321, 323, 325). For example, the electronic device (101) can identify a match among a plurality of matches (311, 313, 315, 317, 321, 323, 325) for which a rating update has not been performed according to the match result. In one embodiment, if there are two or more matches satisfying the condition, the electronic device (101) can select one match according to the order of the rounds. For example, a match of a lower round may be selected before a match of a higher round. For example, any one match among matches of the same round may be selected.

[0107] In operation 720, the electronic device (101) can identify a win rate based on a difference value between the ratings. For example, the win rate based on the difference value may be an expected win rate (or, probability of winning). For example, it may be determined based on the difference value obtained by subtracting the rating of the winner AI model from the rating of the loser AI model. For example, if the difference value is different by a specified score (e.g., 400 points), the ratio of the probability of winning (e.g., 90.90%) to the probability of losing (e.g., 9.09%) may be 10. For example, if the subtracted value is 0, the probability of losing (e.g., 50%) and the probability of winning (e.g., 50%) may be the same.

[0108] In operation 730, the electronic device (101) can update the rating of the winner artificial intelligence model and the rating of the loser artificial intelligence model based on the winning rate.

[0109] In one embodiment, the electronic device (101) can update the rating of the winner artificial intelligence model and the rating of the loser artificial intelligence model based on the value obtained by subtracting the winning rate from the value representing the matching result. For example, the value representing the matching result of the winner artificial intelligence model may be 1. For example, the value representing the matching result of the artificial intelligence model may be 0.

[0110] For example, the electronic device (101) can update the rating of the winner artificial intelligence model and the rating of the loser artificial intelligence model based on the above mathematical formula 1.

[0111] As described above, the electronic device (101) may include at least one processor (120) including a processing circuit and a memory (130) including one or more storage media for storing instructions. When the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause the outputs of a plurality of artificial intelligence models (211, 213, 215) for a test scenario (221). When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause the outputs of two artificial intelligence models selected as participants among the plurality of artificial intelligence models (211, 213, 215) to be input into an evaluation model (160) for each of the plurality of matches (311, 313, 315, 317, 321, 323, 325) included in the tournament (310) in order to proceed with the tournament (310) for the test scenario (221). When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause to obtain the match results of the evaluation model (160) representing the winner artificial intelligence model and / or the loser artificial intelligence model among the two artificial intelligence models for each of the plurality of matches (311, 313, 315, 317, 321, 323, 325) included in the tournament (310) for the test scenario (221).When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause the ratings representing the performance of the plurality of artificial intelligence models (211, 213, 215) to be updated based on the results of the plurality of matches (311, 313, 315, 317, 321, 323, 325) obtained based on the progress of the tournament (310).

[0112] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may be caused to identify the expected winning probability based on the difference in ratings between the first rating of the winner AI model and the second rating of the loser AI model for each of the plurality of matches (311, 313, 315, 317, 321, 323, 325), and to update the first rating and the second rating based on the score according to the match result and the expected winning probability.

[0113] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause the outputs of the plurality of artificial intelligence models (211, 213, 215) to be obtained by inputting a prompt included in the test scenario (221) to the plurality of artificial intelligence models (211, 213, 215).

[0114] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause the external electronic device (102) to receive a request for performance evaluation of the plurality of artificial intelligence models (211, 213, 215) through the communication circuit (140). When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause the external electronic device (102) to transmit a prompt included in the test scenario (221) through the communication circuit (140) based on receiving the request. When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may be caused to obtain the outputs of the plurality of artificial intelligence models (211, 213, 215) generated as the prompt is input from the external electronic device (102) through the communication circuit (140).

[0115] When the above instructions are executed by the at least one processor (120), either individually or collectively, the electronic device (101) may cause the ratings representing the performance of the plurality of artificial intelligence models (211, 213, 215) to be transmitted to the external electronic device (102) through the communication circuit (140) so that the ratings are displayed through the display (150) of the external electronic device (102).

[0116] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may be caused to identify the match result by inputting the outputs of the two artificial intelligence models and the prompt used to obtain the outputs into the evaluation model (160).

[0117] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may be caused to identify the match result by inputting the outputs of the two artificial intelligence models and the evaluation criteria into the evaluation model (160).

[0118] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause the outputs of the two artificial intelligence models to be input into the evaluation model (160) by transmitting the outputs of the two artificial intelligence models to an external server (103) through the communication circuit (140).

[0119] At least two of the plurality of artificial intelligence models (211, 213, 215) may have neural networks of the same structure. The at least two artificial intelligence models having neural networks of the same structure may be trained with different training iterations on the same dataset.

[0120] At least two of the plurality of artificial intelligence models (211, 213, 215) may have neural networks of different structures. The at least two artificial intelligence models having neural networks of different structures may be trained based on the same dataset.

[0121] At least two of the above-mentioned multiple artificial intelligence models (211, 213, 215) may differ from each other in at least one of the neural network structure, training data, number of training sessions, or training technique.

[0122] The method described above may be performed by an electronic device (101). The method may include the operation of obtaining the outputs of a plurality of artificial intelligence models (211, 213, 215) for a test scenario (221). The method may include the operation of inputting the outputs of two artificial intelligence models selected as participants among the plurality of artificial intelligence models (211, 213, 215) into an evaluation model (160) for each of the plurality of matches (311, 313, 315, 317, 321, 323, 325) included in the tournament (310) for the test scenario (221) to proceed with the tournament (310). The above method may include the operation of obtaining the match results of the evaluation model (160) representing the winner artificial intelligence model and / or the loser artificial intelligence model among the two artificial intelligence models for each of the plurality of matches (311, 313, 315, 317, 321, 323, 325) included in the tournament (310) for the above test scenario (221) in order to proceed with the tournament (310). The above method may include the operation of updating ratings representing the performance of the plurality of artificial intelligence models (211, 213, 215) based on the plurality of match results of the plurality of matches (311, 313, 315, 317, 321, 323, 325) obtained based on proceeding with the tournament (310).

[0123] For each of the plurality of matches (311, 313, 315, 317, 321, 323, 325): an operation of identifying an expected win rate based on the difference in ratings between a first rating of a winner artificial intelligence model and a second rating of a loser artificial intelligence model, and an operation of updating the first rating and the second rating based on the score according to the match result and the expected win rate.

[0124] The above method may include the operation of obtaining the outputs of the plurality of artificial intelligence models (211, 213, 215) by inputting a prompt included in the above test scenario (221) to the plurality of artificial intelligence models (211, 213, 215).

[0125] The above method may include an operation of receiving a request for performance evaluation of the plurality of artificial intelligence models (211, 213, 215) from an external electronic device (102) through a communication circuit (140) of the electronic device (101). The above method may include an operation of transmitting a prompt included in the test scenario (221) to the external electronic device (102) through the communication circuit (140) based on receiving the request. The above method may include an operation of obtaining outputs of the plurality of artificial intelligence models (211, 213, 215) generated as the prompt is input from the external electronic device (102) through the communication circuit (140).

[0126] The above method may include the operation of transmitting the ratings to the external electronic device (102) through the communication circuit (140) so that the ratings representing the performance of the plurality of artificial intelligence models (211, 213, 215) are displayed through the display (150) of the external electronic device (102).

[0127] The above method may include the operation of identifying the match result by inputting the outputs of the two artificial intelligence models and the prompt used to obtain the outputs into the evaluation model (160).

[0128] The above method may include the operation of identifying the match result by inputting the outputs of the two artificial intelligence models and the evaluation criteria into the evaluation model (160).

[0129] The above method may include the operation of inputting the outputs of the two artificial intelligence models into the evaluation model (160) by transmitting the outputs of the two artificial intelligence models to an external server (103) through the communication circuit (140) of the electronic device (101).

[0130] A non-transitory computer-readable storage medium as described above may store a program containing instructions. When the instructions are executed individually or collectively by at least one processor (120) of an electronic device (101), the electronic device (101) may cause the electronic device (101) to obtain outputs of a plurality of artificial intelligence models (211, 213, 215) for a test scenario (221). When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause the outputs of two artificial intelligence models selected as participants among the plurality of artificial intelligence models (211, 213, 215) to be input into an evaluation model (160) for each of the plurality of matches (311, 313, 315, 317, 321, 323, 325) included in the tournament (310) in order to proceed with the tournament (310) for the test scenario (221). When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause to obtain the match results of the evaluation model (160) representing the winner artificial intelligence model and / or the loser artificial intelligence model among the two artificial intelligence models for each of the plurality of matches (311, 313, 315, 317, 321, 323, 325) included in the tournament (310) for the test scenario (221).When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may cause the ratings representing the performance of the plurality of artificial intelligence models (211, 213, 215) to be updated based on the results of the plurality of matches (311, 313, 315, 317, 321, 323, 325) obtained based on the progress of the tournament (310).

[0131] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0132] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable storage media.

[0133] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may continuously store a computer-executable program, or temporarily store it for execution or download. Furthermore, the medium may be various recording or storage means in the form of a single or several combined hardware, and is not limited to a medium directly connected to a computer system, but may also exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0134] Although the embodiments have been described above with reference to limited embodiments and drawings, those skilled in the art can make various modifications and variations from the description above. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0135] Therefore, other implementations, other embodiments, and equivalents to the claims set forth below are also within the scope of the claims. According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0136] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, At least one processor including a processing circuit; and The electronic device includes a memory that stores instructions and includes one or more storage media, wherein the instructions are executed individually or collectively by the at least one processor. Obtain the outputs of multiple artificial intelligence models for a test scenario, and To conduct a tournament for the above test scenario, for each of the multiple matches included in the tournament: The outputs of two artificial intelligence models selected as participants among the aforementioned plurality of artificial intelligence models are input into an evaluation model, and Obtain the match results of the evaluation model for the two artificial intelligence models mentioned above, and Causing to update ratings representing the performance of the plurality of artificial intelligence models based on the results of the plurality of matches obtained by conducting the above tournament, Electronic device.

2. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, For each of the above multiple matches: Identifying an expected winning rate based on the difference in ratings between the first rating of the winning AI model and the second rating of the losing AI model among the two artificial intelligence models above, and Causing the updating of the first rating and the second rating based on the score according to the match result and the above expected win rate, Electronic device.

3. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Causing to obtain the outputs of the plurality of artificial intelligence models by inputting the prompt included in the above test scenario into the plurality of artificial intelligence models, Electronic device.

4. In Claim 1, Includes a communication circuit, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, A request for performance evaluation of the plurality of artificial intelligence models is received from an external electronic device through the communication circuit above, and Based on receiving the above request, a prompt included in the test scenario is transmitted to the external electronic device through the communication circuit, and Causing to obtain the outputs of the plurality of artificial intelligence models generated as the prompt is input from the external electronic device through the communication circuit, Electronic device.

5. In Claim 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Causing the ratings representing the performance of the plurality of artificial intelligence models to be displayed through the display of the external electronic device, and causing the ratings to be transmitted to the external electronic device through the communication circuit. Electronic device.

6. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Causing to identify the match result by inputting the outputs of the two artificial intelligence models and the prompt used to obtain the outputs into the evaluation model, Electronic device.

7. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Causing to identify the match result by inputting the outputs of the two artificial intelligence models and the evaluation criteria into the evaluation model, Electronic device.

8. In Claim 1, Includes a communication circuit, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, By transmitting the outputs of the two artificial intelligence models to an external server through the communication circuit, thereby causing the outputs of the two artificial intelligence models to be input into the evaluation model, Electronic device.

9. In Claim 1, At least two of the above multiple artificial intelligence models have neural networks of the same structure, At least two artificial intelligence models having the neural network of the same structure are trained on the same dataset with different training iterations. Electronic device.

10. In Claim 1, Among the plurality of artificial intelligence models mentioned above, at least two artificial intelligence models have at least one of the neural network structure, training data, number of training iterations, or training technique that is different. Electronic device.

11. In a method of an electronic device, The operation of obtaining the outputs of multiple artificial intelligence models for a test scenario, To conduct a tournament for the above test scenario, for each of the multiple matches included in the tournament: The operation of inputting the outputs of two artificial intelligence models selected as participants among the plurality of artificial intelligence models above into an evaluation model, and The operation of obtaining the match result of the evaluation model for the two artificial intelligence models, and Includes an operation of updating ratings representing the performance of the plurality of artificial intelligence models based on the results of the plurality of matches obtained by conducting the above tournament. method.

12. In Claim 11, For each of the above multiple matches: An operation to identify an expected winning rate based on the difference in ratings between the first rating of the winner artificial intelligence model and the second rating of the loser artificial intelligence model among the two artificial intelligence models above, and The operation of updating the first rating and the second rating based on the score according to the match result and the above expected win rate. method.

13. In Claim 11, The operation of obtaining the outputs of the plurality of artificial intelligence models by inputting a prompt included in the above test scenario into the plurality of artificial intelligence models. method.

14. In Claim 11, The operation of receiving a request for performance evaluation of the plurality of artificial intelligence models from an external electronic device through the communication circuit of the electronic device, Based on receiving the above request, the operation of transmitting a prompt included in the test scenario to the external electronic device through the communication circuit, and The operation of obtaining the outputs of the plurality of artificial intelligence models generated as the prompt is input from the external electronic device through the communication circuit. method.

15. In Claim 14, The operation of transmitting the ratings to the external electronic device through the communication circuit so that the ratings representing the performance of the plurality of artificial intelligence models are displayed through the display of the external electronic device. method.

16. In Claim 11, The operation of identifying the match result by inputting the outputs of the two artificial intelligence models and the prompt used to obtain the outputs into the evaluation model. method.

17. In Claim 11, The operation of identifying the match result by inputting the outputs of the two artificial intelligence models and evaluation criteria into the evaluation model. method.

18. In Claim 11, The operation of inputting the outputs of the two artificial intelligence models into the evaluation model by transmitting the outputs of the two artificial intelligence models to an external server through the communication circuit of the electronic device. method.

19. In a non-transitory computer-readable storage medium, A program containing instructions is stored, and when said instructions are executed individually or collectively by at least one processor of an electronic device, said electronic device, Obtain the outputs of multiple artificial intelligence models for a test scenario, and To conduct a tournament for the above test scenario, for each of the multiple matches included in the tournament: The outputs of two artificial intelligence models selected as participants among the aforementioned plurality of artificial intelligence models are input into an evaluation model, and Obtain the match results of the evaluation model for the two artificial intelligence models mentioned above, and Causing to update ratings representing the performance of the plurality of artificial intelligence models based on the results of the plurality of matches obtained by conducting the above tournament, Non-transient computer-readable storage media.

20. In Claim 19, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, For each of the above multiple matches: Identifying an expected winning rate based on the difference in ratings between the first rating of the winning AI model and the second rating of the losing AI model among the two artificial intelligence models above, and Causing the updating of the first rating and the second rating based on the score according to the match result and the above expected win rate, Non-transient computer-readable storage media.

Images