Prompt tuning method and system for evaluation of artificial intelligence model

Abstract

Disclosed are a prompt tuning method and system for an evaluation of an artificial intelligence model. The prompt tuning method according to an embodiment may comprise: a first step of generating an evaluation prompt; a second step of evaluating a result of execution of the same task by each of a first model and a second model, in relation to the evaluation prompt; a third step of assigning a score to the evaluation prompt on the basis of a comparison between a first evaluation result for the result of the execution by the first model and a second evaluation result for the result of the execution by the second model; and a fourth step of generating a new version of evaluation prompt by updating the evaluation prompt on the basis of the score assigned to the evaluation prompt.

Description

Prompt tuning method and system for evaluating artificial intelligence models

[0001] The following description describes a prompt tuning method and system for evaluating artificial intelligence models.

[0002] A large language model (LLM) is a type of artificial intelligence (AI) trained on large corpora of text data to generate human-like responses to natural language input. It consists of an artificial neural network with numerous parameters (typically billions of weights or more). These LLMs can be trained on a significant amount of unlabeled text using self-supervised or semi-self-supervised learning.

[0003] Recently, the number of people and companies using and / or providing AI services through LLMs has been increasing. Companies that use LLMs as a business model to provide AI services face significant challenges in verifying and improving the performance of their LLMs. Specifically, if the generation quality of the LLMs is poor, they need to improve the LLMs (e.g., through prompt tuning). However, companies typically lack data labels tailored to the domains for their AI services, making it difficult to assess the quality of the LLMs used for their AI services. Therefore, verifying the quality of the LLMs they intend to use has become a critical issue for companies.

[0004] Typically, LLMs are evaluated through human evaluations, but these human evaluations are resource-intensive and time-consuming. To address this issue, active research is underway on developing evaluation models for LLMs, enabling them to assess the quality of other LLMs. However, the problem arises that further human evaluation is required to ensure the evaluation model is functioning properly.

[0005] A prompt tuning method and system are provided that can automatically determine whether the tuning of a prompt for evaluating an artificial intelligence model has been performed well.

[0006] A method for tuning a prompt in a computer device including at least one processor, the method comprising: a first step of generating an evaluation prompt by the at least one processor; a second step of evaluating, by the at least one processor, responses of a first model and a second model to the same task in relation to the evaluation prompt; a third step of assigning a score to the evaluation prompt based on a comparison of a first evaluation result for a response of the first model and a second evaluation result for a response of the second model; and a fourth step of updating, by the at least one processor, the evaluation prompt based on the score assigned to the evaluation prompt to generate a new version of the evaluation prompt.

[0007] According to one aspect, the step of assigning the score may be characterized by assigning a relatively larger score to the evaluation prompt as the difference between the first evaluation result and the second evaluation result is greater.

[0008] According to another aspect, the step of assigning a score may be characterized by assigning a score to the evaluation prompt by comparing a distribution of the first evaluation results and a distribution of the second evaluation results for a plurality of tasks in relation to the evaluation prompt.

[0009] According to another aspect, the step of assigning the score may be characterized in that the greater the difference between the distribution of the first evaluation result and the distribution of the second evaluation result, the greater the score is assigned to the evaluation prompt.

[0010] According to another aspect, the step of assigning the score may be characterized by assigning the value of AUC obtained through an AUC (Area Under the Curve)-ROC (Receiver Operating Characteristic) evaluation of the distribution of the first evaluation result and the distribution of the second evaluation result as the score of the evaluation prompt.

[0011] According to another aspect, the prompt tuning method may further include a fifth step of repeatedly performing the second to fourth steps at least once, targeting the updated evaluation prompt, by the at least one processor.

[0012] According to another aspect, the fifth step may be characterized by (1) repeatedly performing the second to fourth steps a preset maximum number of repetitions a, (2) repeatedly performing the second to fourth steps until the number of times that the score given to the evaluation prompt of the current iteration is not observed to improve performance by a threshold value b or more compared to the score given to the previous evaluation prompt is c or more, or (3) repeatedly performing the second to fourth steps until the score given to the evaluation prompt of the current iteration is a preset threshold value d or more, wherein a and c are natural numbers, and b and d are real numbers.

[0013] According to another aspect, the method may further include providing, by the at least one processor, a set of assessment prompts including the assessment prompt and the at least one new version of the assessment prompt, wherein the set of assessment prompts includes a score assigned to each assessment prompt, wherein the score is based on the score assigned to each assessment prompt.

[0014] According to another aspect, the step of generating the evaluation prompt may be characterized by generating the evaluation prompt based on evaluation criteria of a specific task to be performed by a large language model (LLM).

[0015] According to another aspect, the fourth step may be characterized by including a step of further updating the evaluation criteria of the evaluation prompt.

[0016] A computer program stored on a computer-readable recording medium is provided for executing the above method on a computer device in combination with a computer device.

[0017] A computer-readable recording medium having recorded thereon a program for executing the above method on a computer device is provided.

[0018] A computer device is provided, comprising at least one processor configured to execute computer-readable instructions, characterized in that the computer device processes, by the at least one processor, a first step of generating an evaluation prompt; a second step of evaluating responses of a first model and a second model to the same task in relation to the evaluation prompt; a third step of assigning a score to the evaluation prompt based on a comparison of a first evaluation result for a response of the first model and a second evaluation result for a response of the second model; and a fourth step of generating a new version of the evaluation prompt by updating the evaluation prompt based on the score assigned to the evaluation prompt.

[0019] It can automatically determine whether the tuning of prompts for evaluating artificial intelligence models has been performed well.

[0020] FIG. 1 is a diagram illustrating an example of a network environment according to one embodiment of the present invention.

[0021] FIG. 2 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention.

[0022] FIG. 3 is a diagram illustrating an example of a prompt tuning process in one embodiment of the present invention.

[0023] Figures 4 to 7 are drawings showing examples of AUC-ROC evaluation results in one embodiment of the present invention.

[0024] FIG. 8 is a flowchart illustrating an example of a prompt tuning method according to one embodiment of the present invention.

[0025] Hereinafter, the detailed description will be given with reference to the attached drawings.

[0026] The prompt tuning system according to embodiments of the present invention may be implemented by at least one computer device. In this case, the computer device may have a computer program according to an embodiment of the present invention installed and run, and the computer device may perform the prompt tuning method according to embodiments of the present invention under the control of the run computer program. The computer program described above may be stored on a computer-readable recording medium, which may be coupled to the computer device and cause the computer to execute the prompt tuning method.

[0027] FIG. 1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 represents an example including a plurality of electronic devices (110, 120, 130, 140), a plurality of servers (150, 160), and a network (170). FIG. 1 is merely an example for explaining the invention, and the number of electronic devices or servers is not limited to that of FIG. 1. In addition, the network environment of FIG. 1 merely illustrates one example of environments applicable to the present embodiments, and the environments applicable to the present embodiments are not limited to the network environment of FIG. 1.

[0028] The plurality of electronic devices (110, 120, 130, 140) may be fixed terminals or mobile terminals implemented as computer devices. Examples of the plurality of electronic devices (110, 120, 130, 140) include smart phones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, etc. For example, in FIG. 1, the shape of a smart phone is shown as an example of the electronic device (110), but in embodiments of the present invention, the electronic device (110) may actually mean one of various physical computer devices that can communicate with other electronic devices (120, 130, 140) and / or servers (150, 160) via a network (170) using a wireless or wired communication method.

[0029] The communication method is not limited, and may include not only a communication method that utilizes a communication network (e.g., a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network (170) may include, but also short-range wireless communication between devices. For example, the network (170) may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. In addition, the network (170) may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network.

[0030] Each server (150, 160) may be implemented as a computer device or multiple computer devices that communicate with multiple electronic devices (110, 120, 130, 140) via a network (170) to provide commands, codes, files, contents, services, etc. For example, the server (150) may be a system that provides services to multiple electronic devices (110, 120, 130, 140) connected via a network (170).

[0031] FIG. 2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Each of the multiple electronic devices (110, 120, 130, 140) or servers (150, 160) described above can be implemented by the computer device (200) illustrated in FIG. 2.

[0032] Such a computer device (200) may include a memory (210), a processor (220), a communication interface (230), and an input / output interface (240), as illustrated in FIG. 2. The memory (210) may be a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a non-permanent mass storage device such as a disk drive. Here, the non-permanent mass storage device such as a ROM and a disk drive may be included in the computer device (200) as a separate permanent storage device distinct from the memory (210). In addition, an operating system and at least one program code may be stored in the memory (210). These software components may be loaded into the memory (210) from a computer-readable recording medium separate from the memory (210). Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (210) via a communication interface (230) other than a computer-readable recording medium. For example, the software components may be loaded into the memory (210) of the computer device (200) based on a computer program installed by files received over a network (170).

[0033] The processor (220) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (220) via the memory (210) or the communication interface (230). For example, the processor (220) may be configured to execute instructions received according to program code stored in a storage device such as the memory (210).

[0034] The communication interface (230) may provide a function for the computer device (200) to communicate with other devices (e.g., the storage devices described above) via the network (170). For example, requests, commands, data, files, etc. generated by the processor (220) of the computer device (200) according to program codes stored in a recording device such as the memory (210) may be transmitted to other devices via the network (170) under the control of the communication interface (230). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (200) via the communication interface (230) of the computer device (200) via the network (170). Signals, commands, data, etc. received via the communication interface (230) may be transmitted to the processor (220) or the memory (210), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (200) may further include.

[0035] The input / output interface (240) may be a means for interfacing with an input / output device (250). For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display or a speaker. As another example, the input / output interface (240) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. At least one of the input / output devices (250) may be configured as a single device with the computer device (200). For example, a touchscreen, a microphone, a speaker, etc. may be implemented in a form included in the computer device (200), such as a smartphone.

[0036] Additionally, in other embodiments, the computer device (200) may include fewer or more components than those illustrated in FIG. 2. However, it is not necessary to explicitly illustrate most conventional components. For example, the computer device (200) may be implemented to include at least some of the input / output devices (250) described above, or may further include other components such as a transceiver, a database, etc.

[0037] FIG. 3 is a diagram illustrating an example of a prompt tuning process according to one embodiment of the present invention. This prompt tuning process may be performed by at least one computer device (200) implementing a prompt tuning system.

[0038] In the first process (310), the computer device (200) can generate an evaluation prompt based on the evaluation criteria of a specific task (for example, a question / answer task in the embodiment of FIG. 3) to be performed by a large language model or a large language model (LLM). Here, the evaluation criteria of the task may be preset according to the domain and / or task of the artificial intelligence service to be provided, and may be updated later according to the results of the third process (330). For example, in the embodiment of FIG. 3, when providing an artificial intelligence-based question / answer service through a question / answer task, examples of preset evaluation criteria include helpfulness, relevance, accuracy, coherence, specificity, completeness, and social appropriateness. The evaluation criteria according to the embodiment of FIG. 3 are examples of evaluation criteria configured by actually updating the pre-established evaluation criteria for the question / answer task through the third process (330). However, the evaluation criteria are not limited to the evaluation criteria of the embodiment of FIG. 3 as described above, and may be freely established according to the domain and / or task of the artificial intelligence service to be provided. In this case, the computer device (200) may generate an evaluation prompt according to the established evaluation criteria. Depending on the embodiment, the initial evaluation prompt may be input from an external source.

[0039] In the second process (320), the computer device (200) can evaluate the responses of the first model and the second model to the same question in relation to the evaluation prompt. First, the first model and the second model can be selected as different models with different performances. For example, models such as GPT4 and GPT3.5, which are already well known to have a relative performance difference between the two models, can be selected as the first and second models. In this embodiment, it is assumed that the first and second models are selected such that the second model has relatively higher performance than the first model. In the prompt tuning process according to this embodiment, when the first model and the second model with different performances each generate responses to the same evaluation prompt, the higher the evaluated performance of the second model's response is than the evaluated performance of the first model's response, the better the evaluation prompt is assumed to be tuned. For example, for a question / answer task, the first model and the second model can be input with the same question and the same evaluation prompt and output responses, respectively. In this case, the computer device (200) can generate a first evaluation result for the first response of the first model and a second evaluation result for the second response of the second model, respectively. For example, the first evaluation result and the second evaluation result can each be quantified as an evaluation score.

[0040] At this time, the greater the difference between the first and second evaluation results, the more likely the computer device (200) is to determine that the evaluation prompt is well-tuned for evaluating the artificial intelligence model. As an example of determining the difference between the first and second evaluation results, the third process (330) may be performed.

[0041] Additionally, according to an embodiment, the computer device (200) can obtain multiple evaluation results according to multiple questions in relation to a single evaluation prompt by repeatedly performing the second process (320) for multiple questions. In this case, a distribution of evaluation results for the first model and a distribution of evaluation results for the second model can be generated for each evaluation prompt.

[0042] In the third process (330), the computer device (200) can perform an AUC (Area Under the Curve)-ROC (Receiver Operating Characteristic) evaluation on the distribution of the evaluation results of the first model and the second model. The AUC-ROC curve may refer to a graph measuring the classification performance of the model at various thresholds. The computer device (200) can evaluate the performance of the first model and the second model based on the AUC-ROC curve for the distribution of the evaluation results of the first model and the second model. At this time, the closer the AUC value is to 0.5, the less difference there is in the distribution of the evaluation results between the first model and the second model, and the closer the AUC value is to 1, the greater the difference there is in the distribution of the evaluation results between the first model and the second model.

[0043] In the fourth process (340), the computer device (200) provides an evaluation prompt i ) and the results of AUC-ROC evaluation (AUC-ROC i) can be used to update the evaluation prompt. As will be described below, the second process (320) to the fourth process (340) can be repeatedly performed, and various versions of the evaluation prompt can be generated according to the update of the evaluation prompt during the repetition process. At this time, the AUC may actually decrease depending on the update of the evaluation prompt. If the value of AUC decreases by more than a preset threshold value (for example, 3.0 based on the value of AUC), the computer device (200) can select the previous version of the evaluation prompt, not the current evaluation prompt, as the evaluation prompt to be updated. As a more specific example, after the first version of the evaluation prompt is updated to the second version in the previous repetition, if the value of AUC for the second version of the evaluation prompt decreases by more than the threshold value, the computer device (200) can select the first version of the evaluation prompt as the evaluation prompt to be updated. In this case, the computer device (200) can update the evaluation prompt of the first version to the third version and repeat the second process (320) to the fourth process (340) with the evaluation prompt of the third version.

[0044] Techniques for updating assessment prompts are already well known, and for example, a computer device (200) can update a selected version of an assessment prompt using at least one of the already well known techniques.

[0045] The computer device (200) can repeat the second process (320) to the fourth process (340) a preset number of times and / or until no further performance improvement is observed. For example, the computer device (200) can repeat the second process (320) to the fourth process (340) a preset number of times a (where a is a natural number) as the maximum number of repetitions. As another example, the computer device (200) can repeat the second process (320) to the fourth process (340) until the number of times during which no performance improvement (for example, an increase in the value of AUC by 0.05 or more) exceeding a threshold value b (where b is a real number) is observed is c (where c is a natural number) times or more. As another example, the computer device (200) can repeat the second process (320) to the fourth process (340) a number of times, but can stop repeating the process if the number of times during which no performance improvement exceeding the threshold value b is observed is c or more.

[0046] In another embodiment, the computer device (200) may repeat the second process (320) to the fourth process (340) until the AUC exceeds a preset threshold value d (e.g., 0.8, where d is a real number). In this case, the maximum number of repetitions may also be set and utilized.

[0047] In the fifth process (350), the computer device (200) may provide the evaluation prompt with the highest AUC value. A highest AUC value may indicate that the evaluation prompt best reflects the performance difference between the first and second models among the various evaluation prompts evaluated to date. The provided evaluation prompt may undergo additional human evaluation. In this case, qualitative evaluation can be conducted only on the final evaluation prompt provided, eliminating the need for a human to individually evaluate various evaluation prompt versions, significantly reducing time and resource consumption. Furthermore, based on the trust in the evaluation prompt, more accurate evaluation prompts for AI models can be developed.

[0048] Figures 4 to 7 are diagrams illustrating examples of AUC-ROC evaluation results according to an embodiment of the present invention. Figures 4 to 7 are diagrams illustrating examples of the results of AUC-ROC evaluations performed using GPT3.5 as the first model and GPT4 as the second model. Here, Figures 4 and 5 illustrate the results of the first AUC-ROC evaluation, and Figures 6 and 7 illustrate the results of AUC-ROC evaluations after repeating the update of the evaluation prompt N times.

[0049] FIG. 4 shows the results of generating responses by inputting each of a plurality of questions into the first and second models together with evaluation prompts, and measuring the evaluation scores for the generated responses. FIG. 5 shows an example in which the AUC-ROC evaluation result for the distribution of the evaluation scores was measured to be 0.537. In addition, FIG. 6 shows the results of generating responses by inputting each of a plurality of questions into the first and second models together with evaluation prompts, and measuring the evaluation scores for the generated responses after repeating the update of the evaluation prompts N times. FIG. 7 shows an example in which the AUC-ROC evaluation result for the distribution of the evaluation scores was measured to be 0.671.

[0050] As explained above, after repeating the automatic update of the evaluation prompt N times, the value of AUC increased from 0.537 to 0.671, which may mean that the evaluation prompt version that went through N updates is more suitable for evaluating the performance of the artificial intelligence model than the evaluation prompt version of the initial version.

[0051] In this way, the prompt tuning system automatically tunes and provides evaluation prompts that can evaluate the performance of the AI ​​model above a certain level, without requiring a human to tune the evaluation prompts by evaluating the performance of the AI ​​model for all versions of the evaluation prompts, thereby drastically reducing the time and cost for tuning the evaluation prompts compared to human evaluation.

[0052] FIG. 8 is a flowchart illustrating an example of a prompt tuning method according to an embodiment of the present invention. The prompt tuning method according to the present embodiment may be performed by at least one computer device (200). At this time, the processor (220) of the computer device (200) may be implemented to execute control instructions according to the code of the operating system included in the memory (210) or the code of at least one computer program. Here, the processor (220) may control the computer device (200) so that the computer device (200) performs steps (810 to 860) included in the method of FIG. 8 according to the control instructions provided by the code stored in the computer device (200).

[0053] At step (810), the computer device (200) may generate an evaluation prompt. For example, the computer device (200) may generate an evaluation prompt based on evaluation criteria for a specific task to be performed by the LLM.

[0054] In step (820), the computer device (200) can evaluate the performance results of the first model and the second model for the same task in relation to the evaluation prompt. For example, as described above, in order to perform a task for a question / answer service, the computer device (200) inputs an evaluation prompt and a question to the first model and the second model, respectively, checks the responses output by the first model and the second model, and evaluates the checked responses to calculate an evaluation score. An artificial intelligence model can also perform various tasks for services other than the question / answer service, and thus, checking and evaluating responses to the same question can be expanded to checking and evaluating the performance results for the same task. Meanwhile, the technology for evaluating an artificial intelligence model itself is already well known, and the evaluation of the performance results of an artificial intelligence model in the present embodiment can be performed using at least one of the already well-known technologies.

[0055] In step (830), the computer device (200) can assign a score to the evaluation prompt based on a comparison of the first evaluation result for the performance result of the first model and the second evaluation result for the performance result of the second model.

[0056] In one embodiment, the computer device (200) may assign a relatively higher score to the evaluation prompt as the difference between the first and second evaluation results increases in step (830). As described above, the larger the difference between the first and second evaluation results, the more likely the computer device (200) is to determine that the corresponding evaluation prompt is well-tuned for evaluating the artificial intelligence model. To quantify this, the computer device (200) may assign a relatively higher score to the evaluation prompt as the difference between the two evaluation results increases.

[0057] In another embodiment, the computer device (200) may assign a score to the evaluation prompt by comparing the distribution of the first evaluation results with the distribution of the second evaluation results for a plurality of tasks in relation to the evaluation prompt in step (830). In this case, the computer device (200) may assign a relatively higher score to the evaluation prompt as the difference between the distribution of the first evaluation results and the distribution of the second evaluation results increases. As an example for this, the utilization of the AUC-ROC evaluation was described above in FIG. 3. For example, the computer device (200) may assign the AUC value obtained through the AUC-ROC evaluation for the distribution of the first evaluation results and the distribution of the second evaluation results as the score for the evaluation prompt. It has already been described in detail that the AUC value increases as the difference between the distributions of the two evaluation results increases.

[0058] In step (840), the computer device (200) may generate a new version of the evaluation prompt by updating the evaluation prompt based on the score assigned to the evaluation prompt. In some embodiments, the computer device (200) may further update the evaluation criteria of the evaluation prompt. In some embodiments, the update of the evaluation prompt may be performed only for at least one evaluation criterion selected from among the evaluation criteria. For example, among the examples of the evaluation criteria described above, such as usefulness, relevance, accuracy, consistency, specificity, completeness, and social appropriateness, usefulness and relevance may be selected, and only the portions related to usefulness and relevance in the evaluation prompt may be updated. In other words, whether or not to update the evaluation prompt may be determined individually for each evaluation criterion. For example, questions related to the evaluation criterion of accuracy may be input into the first and second models, and based on the results, only the portions related to the evaluation criterion of accuracy may be individually updated in the subsequent evaluation prompt.

[0059] In step (850), the computer device (200) can repeat steps (820) to (840) at least once for the updated evaluation prompt. For example, the computer device (200) can (1) repeat steps (820) to (840) a preset maximum number of repetitions a, (2) repeat steps (820) to (840) until the number of times that a performance improvement of a threshold value b is not observed in a score assigned to an evaluation prompt of the current iteration compared to a score assigned to a previous evaluation prompt is c or more, or (3) repeat steps (820) to (840) until a score assigned to an evaluation prompt of the current iteration is equal to or greater than a preset threshold value d.

[0060] In this case, by repeating steps (820) to (840), a score may be awarded for at least one new version of the evaluation prompt.

[0061] In step (860), the computer device (200) may provide an evaluation prompt selected from a set of evaluation prompts, including an evaluation prompt and at least one new version of the evaluation prompt, based on the scores assigned to each evaluation prompt. As previously described, the provided evaluation prompts may undergo additional human evaluation. In this case, qualitative evaluations can be conducted only on the final provided evaluation prompt, eliminating the need for a human to individually evaluate various versions of the evaluation prompts, thereby significantly reducing time and resource consumption. Furthermore, more accurate evaluation prompts for the AI ​​model can be developed based on the trust in the evaluation prompts.

[0062] In this way, according to embodiments of the present invention, a prompt tuning method and system capable of automatically determining whether tuning of a prompt for evaluating an artificial intelligence model has been performed well can be provided.

[0063] The systems or devices described above may be implemented as hardware components, or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0064] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0065] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., singly or in combination. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0066] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0067] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. A method for prompt tuning of a computer device including at least one processor, A first step of generating an evaluation prompt by at least one processor; A second step of evaluating the performance results of each of the first model and the second model for the same task in relation to the evaluation prompt by at least one processor; A third step of assigning a score to the evaluation prompt based on a comparison between a first evaluation result of the performance result of the first model and a second evaluation result of the performance result of the second model, by at least one processor; and A fourth step of generating a new version of the assessment prompt by updating the assessment prompt based on the score assigned to the assessment prompt by at least one processor. A prompt tuning method including:

2. In paragraph 1, The steps for assigning the above scores are: A prompt tuning method characterized in that the greater the difference between the first evaluation result and the second evaluation result, the greater the score given to the evaluation prompt.

3. In paragraph 1, The steps for assigning the above scores are: A prompt tuning method characterized in that a score is assigned to the evaluation prompt by comparing the distribution of the first evaluation results and the distribution of the second evaluation results for a plurality of tasks in relation to the evaluation prompt.

4. In paragraph 3, The steps for assigning the above scores are: A prompt tuning method characterized in that the greater the difference between the distribution of the first evaluation result and the distribution of the second evaluation result, the greater the score given to the evaluation prompt.

5. In paragraph 3, The steps for assigning the above scores are: A prompt tuning method characterized in that the value of AUC obtained through AUC (Area Under the Curve)-ROC (Receiver Operating Characteristic) evaluation for the distribution of the first evaluation result and the distribution of the second evaluation result is given as a score of the evaluation prompt.

6. In paragraph 1, A fifth step of repeating the second to fourth steps at least once, targeting the updated evaluation prompt, by at least one processor. A prompt tuning method that includes more.

7. In paragraph 6, The fifth step above is, (1) Repeating the second to fourth steps a preset maximum number of repetitions a, or (2) Repeating the second to fourth steps until the number of times that the score given to the evaluation prompt of the current iteration is not observed to improve performance by a threshold value b or more compared to the score given to the previous evaluation prompt is c or more, or (3) Repeating the second to fourth steps until the score given to the evaluation prompt of the current iteration is equal to or greater than a preset threshold value d, The above a and c are natural numbers, The above b and d are real numbers A prompt tuning method characterized by .

8. In paragraph 6, At least one new version of the evaluation prompt is scored by the above iteration, A step of providing, by said at least one processor, an assessment prompt selected from a set of assessment prompts including said assessment prompt and said at least one new version of said assessment prompt, based on a score assigned to each assessment prompt. A prompt tuning method that includes more.

9. In paragraph 1, The steps to create the above evaluation prompt are: A prompt tuning method characterized by generating evaluation prompts based on evaluation criteria of a specific task to be performed by a large language model (LLM).

10. In paragraph 9, The fourth step above is, Steps to further update the evaluation criteria of the above evaluation prompt A prompt tuning method characterized by including:

11. A computer program stored on a computer-readable recording medium for executing the method of any one of claims 1 to 10 on a computer device by being combined with a computer device.

12. A computer-readable recording medium having recorded thereon a computer program for executing the method of any one of claims 1 to 10 on a computer device.

13. At least one processor implemented to execute computer-readable instructions. Including, By at least one processor, Step 1: Create an evaluation prompt; A second process of evaluating the performance results of each of the first and second models for the same task in relation to the above evaluation prompt; A third process of assigning a score to the evaluation prompt based on a comparison of the first evaluation result for the performance result of the first model and the second evaluation result for the performance result of the second model; and Step 4: Create a new version of the assessment prompt by updating the assessment prompt based on the scores given to the assessment prompt. to handle A computer device characterized by:

14. In paragraph 13, To process the above third process, by at least one processor, The greater the difference between the first and second evaluation results, the greater the score given to the evaluation prompt. A computer device characterized by:

15. In paragraph 13, To process the above third process, by at least one processor, Comparing the distribution of the first evaluation results and the distribution of the second evaluation results for multiple tasks in relation to the above evaluation prompt and assigning a score to the above evaluation prompt. A computer device characterized by:

16. In paragraph 15, To process the above third process, by at least one processor, Assigning the value of AUC obtained through AUC (Area Under the Curve)-ROC (Receiver Operating Characteristic) evaluation for the distribution of the first evaluation result and the distribution of the second evaluation result as the score of the evaluation prompt. A computer device characterized by:

17. In paragraph 13, By at least one processor, A fifth process that repeats the first to fourth processes at least once for the updated evaluation prompt. to process more A computer device characterized by:

18. In paragraph 17, At least one new version of the evaluation prompt is scored by the above iteration, By at least one processor, A sixth process for providing an evaluation prompt selected from a set of evaluation prompts including the above evaluation prompt and at least one new version of the evaluation prompt, based on a score assigned to each evaluation prompt. to process more A computer device characterized by:

Images