Method of optimizing prompt for classification of unlabeled datas based on multi-model and computing apparatus for performing the same

Abstract

A multi-model-based prompt optimization method for classifying non-labeled data performed by a computing device according to one embodiment comprises: a step of obtaining an initial prompt containing a plurality of texts of a set unit; a step of generating variant prompts having different overall configurations, each containing at least one of the plurality of set unit texts; a step of inputting the variant prompts into N artificial neural network models to obtain N classification results for the non-labeled data for each variant prompt as the output of the artificial neural network models; a step of calculating a contribution indicating the degree to which the plurality of set unit texts influence classification by the artificial neural network models based on the result of calculating mutual agreement for the obtained classification results; and a step of generating an updated prompt through highlighting, maintaining, or removing processing of the text within the initial prompt based on the calculated contribution.

Description

Technology Field

[0001] The present invention relates to a method for optimizing a prompt to be used for classifying unlabeled data based on a multi-model basis, and a computing device for performing the method. Background Technology

[0002] Large-scale labeling tasks involving unlabeled data, such as unlabeled text, images, and videos, are subject to significant constraints due to high time and economic costs. While approaches utilizing unsupervised or weakly supervised learning to discover latent patterns in unlabeled data or applying clustering techniques to learn data structures can be considered, limitations in terms of scalability and cost-efficiency remain when unlabeled data constitutes the majority of the workload.

[0003] Although prompt-based data classification using artificial neural network models, such as large language models, can be applied to such unlabeled data, there are difficulties in actual implementation due to a high dependency on the design of appropriate prompts. Prior art literature

[0004] Registered Patent Publication No. 10-2672166, Method for Optimizing Prompt Information for Generative AI, Publication Date June 7, 2024. The problem to be solved

[0005] According to an embodiment, a multi-model based prompt optimization method for classifying non-labeled data and a computing device for performing the same are provided, which calculates the degree to which text within an initial prompt influences classification based on an initial prompt and a classification result for non-labeled data using a plurality of artificial neural network models, and generates an updated prompt optimized for the classification of non-labeled data based thereon.

[0006] The problems to be solved by the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the description below. means of solving the problem

[0007] A multi-model-based prompt optimization method for classifying non-labeled data performed by a computing device according to a first aspect comprises: a step of obtaining an initial prompt containing multiple texts of a set unit; a step of generating variant prompts that have different overall compositions and include at least one of the multiple set unit texts; a step of inputting the variant prompts into N artificial neural network models to obtain N classification results for the non-labeled data for each variant prompt as the output of the artificial neural network models; a step of calculating a contribution indicating the degree to which the multiple set unit texts influence classification by the artificial neural network models based on the result of calculating mutual agreement for the obtained classification results; and a step of generating an updated prompt through highlighting, maintaining, or removing processing of the text within the initial prompt based on the calculated contribution.

[0008] Here, the non-label data may include at least one of non-label text, non-label image, and non-label video.

[0009] When sequentially referring to units from word to sentence, line, and paragraph as lower to upper units, at least one of the plurality of setting unit texts can be selected based on the result of first calculating the contribution from the upper unit, and the update prompt can be generated based on the result of second calculating the contribution of the lower unit for the selected setting unit text.

[0010] The above emphasis processing may include at least one method of adding meaning to the set unit text among the plurality of set unit texts whose contribution is greater than or equal to a threshold value.

[0011] Based on the result of calculating the mutual agreement of the classification results obtained above, the classification results by at least one of the N artificial neural network models can be excluded from the calculation of the contribution.

[0012] A computing device according to a second aspect comprises: a memory in which a computer program including at least one instruction is stored; and a processor that executes said instruction; wherein the processor performs a multi-model-based prompt optimization method for classifying non-labeled data, comprising: a step of obtaining an initial prompt containing a plurality of texts of setting units by executing said instruction; a step of generating variant prompts having different overall configurations, each containing at least one of said set unit texts; a step of inputting said variant prompts into N artificial neural network models to obtain N classification results for non-labeled data for each variant prompt as the output of said artificial neural network models; a step of calculating a contribution indicating the degree to which said plurality of set unit texts influence classification by said artificial neural network models based on the result of calculating mutual agreement for said classification results; and a step of generating an updated prompt through highlighting, maintaining, or removing processing of text within said initial prompt based on said contribution.

[0013] According to a third perspective, a computer program stored on a computer-readable recording medium includes instructions for the computer program to cause a processor to perform a multi-model-based prompt optimization method for classifying the unlabeled data.

[0014] According to the fourth aspect, a computer-readable recording medium storing a computer program includes instructions for the computer program to cause a processor to perform a multi-model-based prompt optimization method for classifying the unlabeled data. Effects of the invention

[0015] According to an embodiment, based on the classification results of non-labeled data using an initial prompt and multiple artificial neural network models, a contribution indicating the degree to which text within the initial prompt influences classification is calculated, and based on this, an updated prompt optimized for the classification of non-labeled data is generated, thereby reducing dependence on the initial prompt design while improving the accuracy and reliability of classification for non-labeled data.

[0016] In addition, when sequentially referring to units from word to sentence, line, and paragraph as lower to higher units, for the initial prompt, the text of the corresponding unit can be selected based on the result of the first calculation of text contribution per higher unit, and then an update prompt can be generated based on the result of the second calculation of text contribution per lower unit. When applying this second calculation of contribution, the update prompt can be generated more quickly compared to the case where only the first calculation of contribution is performed.

[0017] In addition, based on the results of calculating the mutual agreement of classification outcomes by N artificial neural network models, models that fall outside the acceptable range of results are excluded from the contribution calculation, thereby further improving the accuracy and reliability of classification for unlabeled data. Brief explanation of the drawing

[0018] FIG. 1 is a configuration diagram of a computing device for performing a multi-model-based prompt optimization method for classifying non-labeled data according to an embodiment of the present invention. FIG. 2 is a configuration diagram showing the functional classification of computer programs loaded by a processor of a computing device according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating a multi-model-based prompt optimization method for classifying non-labeled data according to an embodiment of the present invention. Specific details for implementing the invention

[0019] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0020] The terms used in this specification will be briefly explained, and the invention will be described in detail.

[0021] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.

[0022] When a part of a specification is described as 'comprising' a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0023] Additionally, the term "part" as used in the specification refers to software or hardware components, such as FPGAs or ASICs, and the "part" performs certain roles. However, the meaning of "part" is not limited to software or hardware. The "part" may be configured to reside in an addressable storage medium or configured to operate one or more processor parts. Thus, by example, the "part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts."

[0024] Below, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the invention. Additionally, parts of the drawings that are irrelevant to the description are omitted to clearly explain the invention.

[0025] FIG. 1 is a configuration diagram of a computing device for performing a multi-model-based prompt optimization method for classifying non-labeled data according to an embodiment of the present invention, and FIG. 2 is a configuration diagram functionally classifying a computer program loaded by a processor of a computing device according to an embodiment of the present invention.

[0026] Referring to FIGS. 1 and 2, a computing device (100) according to an embodiment includes a memory (110) in which a computer program (111) is stored and a processor (120), and may further include an input unit (130) and / or an output unit (140).

[0027] A computer program (111) stored in the memory (110) of such a computing device (100) includes at least one instruction that can be executed by a processor (120). By executing the instruction of such a computer program (111) by the processor (120), the computer program (111) is loaded so that a multi-model based prompt optimization method for classifying non-labeled data according to an embodiment of the present invention can be performed. Additionally, various data necessary for the processor (120) to perform a multi-model based prompt optimization method for classifying non-labeled data according to an embodiment of the present invention may be stored in the memory (110).

[0028] The processor (120) executes instructions of a computer program (111) stored in memory (110) to load the computer program (111), thereby performing a multi-model based prompt optimization method for classifying non-labeled data according to the embodiment. The functions of the modified prompt generation unit (210), the first to N artificial neural network models (221, 222, 223), the text contribution calculation unit (230), and the update prompt generation unit (240) by the computer program (111) thus loaded will be explained again below.

[0029] The input unit (130) receives various data necessary for the processor (120) of the computing device (100) to perform a multi-model-based prompt optimization method for classifying non-labeled data according to an embodiment, and provides this data to the processor (120). For example, the input unit (130) may receive an initial prompt and provide it to the processor (120).

[0030] The output unit (140) can provide various data acquired and / or generated as the processor (120) of the computing device (100) performs a multi-model-based prompt optimization method for classifying non-labeled data according to the embodiment. For example, the output unit (140) can provide an update prompt generated by the update prompt generation unit (240) to the outside. For example, providing the update prompt to the outside by the output unit (140) may include outputting to a peripheral device connected via a serial interface, etc., or transmitting via a communication channel.

[0031] When a computer program (111) loaded by a processor (120) performs the function of a variation prompt generation unit (210), it generates variation prompts with different overall configurations that include at least one of the multiple setting unit texts within the initial prompt. Here, a setting unit may be a word unit, a sentence unit, a line unit, and a paragraph unit, and the units from word unit to sentence unit, line unit, and paragraph unit may be sequentially referred to as lower units to higher units. For example, if the setting unit is a line unit, multiple text lines such as a first text line, a second text line, a third text line, etc., may be included in the initial prompt. For example, the variation prompts may be designed not to include any one of the different text lines.

[0032] The first to Nth artificial neural network models (221, 222, 223) loaded into the computer program (111) loaded by the processor (120) each receive variant prompts generated by the variant prompt generation unit (210) and obtain and output N classification results for unlabeled data for each variant prompt. Although three artificial neural network models are shown in FIG. 2, this is merely an example and the number is not limited; each artificial neural network model may be a pre-trained large language model, but is not specifically limited.

[0033] When a computer program (111) loaded by a processor (120) performs the function of a text contribution calculation unit (230), a contribution is calculated based on the result of calculating the mutual agreement of classification results obtained by the first to N artificial neural network models (221, 222, 223), indicating the degree to which a plurality of set unit texts influence classification by the first to N artificial neural network models (221, 222, 223). For example, if the set unit is a line unit, the agreement of a classification result by a variation prompt containing the first text line with another classification result can be compared with the agreement of a classification result by a variation prompt not containing the first text line with another classification result, and the contribution of the first text line can be determined based on the difference between the two agreements. Additionally, the text contribution calculation unit (230) may exclude classification results by at least one of the first to N artificial neural network models (221, 222, 223) from the calculation of contribution based on the result of calculating mutual agreement for classification results obtained by the first to N artificial neural network models (221, 222, 223).

[0034] When a computer program (111) loaded by a processor (120) performs the function of an update prompt generation unit (240), an update prompt is generated through highlighting, maintaining, or removing text within an initial prompt based on the contribution calculated by a text contribution calculation unit (230). Here, the highlighting may include at least one method of adding meaning to a set unit text among a plurality of set unit texts whose contribution is greater than or equal to a threshold. For example, the highlighting may include a first method of adding specific symbols such as '**' or '##' before and after the text for the corresponding set unit text, a second method of converting the entire text to uppercase, a third method of adding predefined highlighting adjectives or adverbs such as 'very important' or 'essentially', and a fourth method of inputting the corresponding set unit text into a pre-set large language model (LLM) to emphasize the meaning. Here, the fourth method may include, for example, inputting the corresponding setting unit text into a large language model, generating a rewritten emphasis text unit to highlight the meaning by providing an emphasis prompt such as ‘express it more clearly and strongly while maintaining the core meaning of the next unit’ or ‘rewrite this content with emphasis from an expert’s perspective,’ and replacing the original setting unit text within the initial prompt with the generated emphasis text unit. Additionally, the update prompt generation unit (240) may select at least one setting unit text among a plurality of setting unit texts based on the result of first calculating the contribution from the upper unit, and generate an update prompt based on the result of second calculating the contribution of the lower unit for the selected setting unit text.

[0035] FIG. 3 is a flowchart illustrating a multi-model-based prompt optimization method for classifying non-labeled data according to an embodiment of the present invention.

[0036] Hereinafter, a multi-model based prompt optimization method for classifying non-labeled data performed by a computing device (100) according to an embodiment will be described in detail with reference to FIGS. 1 to 3.

[0037] First, the processor (120) of the computing device (100) can perform a multi-model-based prompt optimization method for classifying non-labeled data according to the embodiment by executing instructions of a computer program (111) stored in memory (110) and loading the computer program (111). The computer program (111) thus loaded performs the functions of a variation prompt generation unit (210), a first to Nth artificial neural network model (221, 222, 223), a text contribution calculation unit (230), and an update prompt generation unit (240).

[0038] The input unit (130) of the computing device (100) can receive an initial prompt containing multiple texts of a setting unit and provides the received initial prompt to the processor (120) (S310).

[0039] Then, a computer program (111) loaded by the processor (120) can perform the function of a variation prompt generating unit (210), and the variation prompt generating unit (210) generates variation prompts with different overall configurations, including at least one of a plurality of setting unit texts in the initial prompt. Here, a setting unit may be a word unit, a sentence unit, a line unit, and a paragraph unit, and the units from the word unit to the sentence unit, line unit, and paragraph unit may be sequentially referred to as lower units to higher units. For example, if the setting unit is a line unit, a plurality of text lines, such as a first text line, a second text line, a third text line, etc., may be included in the initial prompt. For example, the variation prompt generating unit (210) may design the variation prompts so as not to include any one of the different text lines among the first text line to the third text line (S320).

[0040] The variant prompts generated by the variant prompt generation unit (210) are each input to the first to N artificial neural network models (221, 222, 223), and the first to N artificial neural network models (221, 222, 223) obtain and output N classification results for non-labeled data for each variant prompt. Here, the non-labeled data to be classified may be stored in memory (110), and the processor (120) may read it from memory (110) to use for classification. Alternatively, the non-labeled data may be input through the input unit (130) and transmitted to the processor (120) (S330).

[0041] And, when a computer program (111) loaded by a processor (120) performs the function of a text contribution calculation unit (230), the text contribution calculation unit (230) calculates a contribution indicating the degree to which a plurality of set unit texts influence classification by the first to N artificial neural network models (221, 222, 223) based on the result of calculating mutual agreement for classification results obtained by the first to N artificial neural network models (221, 222, 223). Here, when calculating mutual agreement for classification results, any one of known algorithms may be used and is not particularly limited. For example, the text contribution calculation unit (230) may calculate mutual agreement for classification results using Cohen's kappa algorithm, rand index / adjusted rand index algorithm, normalized mutual information algorithm, confusion matrix + accuracy / F1 score algorithm, etc.

[0042] Additionally, when the text contribution calculation unit (230) calculates the contribution that a set unit text has to classify, for example, when the set unit is a line unit, it can compare the degree of agreement between a classification result by a variation prompt containing the first text line and another classification result by a variation prompt not containing the first text line with the degree of agreement between the two classification results, and determine the contribution of the first text line based on the difference between the two degrees of agreement (S340).

[0043] Additionally, the text contribution calculation unit (230) may exclude classification results by at least one of the first to N artificial neural network models (221, 222, 223) from the calculation of contribution based on the result of calculating mutual agreement for classification results obtained by the first to N artificial neural network models (221, 222, 223). For example, if a specific classification result has a singularity that falls outside the allowable range, the classification result by the corresponding artificial neural network model may be excluded from the calculation of contribution, thereby further improving the accuracy and reliability of classification for non-labeled data.

[0044] Next, when a computer program (111) loaded by a processor (120) performs the function of an update prompt generation unit (240), the update prompt generation unit (240) generates an update prompt by highlighting, maintaining, or removing text within an initial prompt based on the contribution calculated by the text contribution calculation unit (230). Here, highlighting may include adding modifiers that add meaning to the object to be highlighted. Maintaining may not modify the text, and removing may exclude the text from the prompt.

[0045] Additionally, the update prompt generation unit (240) can select at least one setting unit text among a plurality of setting unit texts based on the result of first calculating the contribution from the upper unit, and generate an update prompt based on the result of second calculating the contribution of the lower unit for the selected setting unit text. When applying the second contribution calculation in this way, the amount of data to be processed is reduced compared to the case where only the first contribution is calculated, so the update prompt can be generated more quickly (S350).

[0046] Next, the output unit (140) of the computing device (100) may provide an update prompt to the outside, which is the final result of the processor (120) performing multi-model-based prompt optimization for the classification of non-labeled data as described above, under the control of the processor (120). For example, the output unit (140) may output the update prompt generated by the update prompt generation unit (240) to a peripheral device connected via a serial interface, etc., or transmit it via a communication channel.

[0047] Meanwhile, a computer program stored on a computer-readable recording medium according to an embodiment includes at least one instruction for a processor of a computing device to perform each step included in a multi-model-based prompt optimization method for classifying non-labeled data according to the aforementioned embodiments.

[0048] Additionally, according to an embodiment, a computer-readable recording medium storing a computer program includes at least one instruction in the computer program for causing a processor of a computing device to perform each step included in a multi-model-based prompt optimization method for classifying non-labeled data according to the aforementioned embodiments.

[0049] As explained above, according to an embodiment of the present invention, a contribution indicating the degree to which text within an initial prompt influences classification is calculated based on the classification results of non-labeled data using an initial prompt and a plurality of artificial neural network models, and an updated prompt optimized for the classification of non-labeled data is generated based on this, thereby improving the accuracy and reliability of classification of non-labeled data while reducing dependence on the initial prompt design.

[0050] In addition, when sequentially referring to units from word to sentence, line, and paragraph as lower to higher units, for the initial prompt, the text of the corresponding unit can be selected based on the result of the first calculation of text contribution per higher unit, and then an update prompt can be generated based on the result of the second calculation of text contribution per lower unit. When applying this second calculation of contribution, the update prompt can be generated more quickly compared to the case where only the first calculation of contribution is performed.

[0051] In addition, based on the results of calculating the mutual agreement of classification outcomes by N artificial neural network models, models that fall outside the acceptable range of results are excluded from the contribution calculation, thereby further improving the accuracy and reliability of classification for unlabeled data.

[0052] Combinations of each step of each flowchart attached to the present invention may be performed by computer program instructions. Since these computer program instructions may be loaded into the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions performed through the processor of the computer or other programmable data processing equipment create means for performing the functions described in each step of the flowchart. Since these computer program instructions may also be stored in a computer-available or computer-readable recording medium that can be directed toward the computer or other programmable data processing equipment to implement the function in a specific manner, the instructions stored in the computer-available or computer-readable recording medium may also produce a manufactured item containing instruction means for performing the function described in each step of the flowchart. Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that execute a computer or other programmable data processing equipment by performing a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in each step of the flowchart.

[0053] Additionally, each step may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the steps may occur out of order. For example, two steps described in succession may actually be performed substantially simultaneously, or the steps may sometimes be performed in reverse order according to the corresponding function.

[0054] The above description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0055] 100: Computing device 110: Memory 111: Computer program 120: Processor 130: Input section 140: Output section 210 : Variant prompt generation section 221: The First Artificial Neural Network Model 222: The Second Artificial Neural Network Model 223: The Nth Artificial Neural Network Model 230: Text Contribution Calculator 240: Update prompt creation section

Claims

Claim 1 A multi-model-based prompt optimization method for classifying non-labeled data performed by a computing device, comprising: a step of obtaining an initial prompt containing multiple texts of setting units; a step of generating variant prompts having different overall compositions, each containing at least one of the multiple texts of setting units; a step of inputting the variant prompts into N artificial neural network models to obtain N classification results for the non-labeled data for each variant prompt as the output of the artificial neural network models; a step of calculating a contribution indicating the degree to which the multiple texts of setting units influence classification by the artificial neural network models based on the result of calculating mutual agreement for the obtained classification results, wherein if the classification result by at least one of the N artificial neural network models has a singularity that exceeds an allowable range based on the result of calculating mutual agreement for the obtained classification results, the contribution is excluded from the calculation; and a step of generating an update prompt from the initial prompt and outputting it externally based on the calculated contribution. Claim 2 A multi-model-based prompt optimization method for classifying non-label data, wherein the non-label data comprises at least one of non-label text, non-label image, and non-label video. Claim 3 A multi-model based prompt optimization method for classifying non-labeled data according to claim 1, wherein, when sequentially referring to lower units to upper units, word units, sentence units, line units, and paragraph units, at least one of the plurality of setting unit texts is selected based on the result of first calculating the contribution from the upper unit, and the update prompt is generated based on the result of second calculating the contribution of the lower unit for the selected setting unit text. Claim 4 A multi-model-based prompt optimization method for classifying non-labeled data, wherein, in claim 1, the update prompt is generated through highlighting processing of text within the initial prompt, and the highlighting processing includes at least one method of adding meaning to a set unit text among the plurality of set unit texts whose contribution is greater than or equal to a threshold. Claim 5 In claim 1, the mutual agreement is a multi-model based prompt optimization method for classifying unlabeled data, wherein the mutual agreement is calculated using any one of the following algorithms: Cohen's kappa algorithm, Rand index or adjusted Rand index algorithm, normalized mutual information algorithm, and F1 score algorithm. Claim 6 A computing device for performing a multi-model-based prompt optimization method for classifying non-labeled data, comprising: a memory storing a computer program including at least one instruction; a processor executing said instruction; and an output unit, wherein the processor, by executing said instruction, obtains an initial prompt containing a plurality of texts of setting units; generates variant prompts having different overall configurations, each containing at least one of said set unit texts; inputs said variant prompts into N artificial neural network models to obtain N classification results for non-labeled data for each variant prompt as outputs of said artificial neural network models; calculates a contribution indicating the degree to which said plurality of set unit texts influence classification by said artificial neural network models based on the result of calculating mutual agreement for said classification results, wherein if a classification result by at least one of said N artificial neural network models has a singularity that exceeds an allowable range based on the result of calculating mutual agreement for said classification results, such that such singularity is excluded from the calculation of the contribution; and generates an update prompt from said initial prompt based on said contribution and outputs it externally through said output unit. Claim 7 In claim 6, the non-label data comprises at least one of non-label text, non-label image, and non-label video, in a computing device. Claim 8 A computing device according to claim 6, wherein, when the units from word to sentence, line, and paragraph are sequentially referred to as lower units to upper units, at least one of the plurality of setting unit texts is selected based on the result of first calculating the contribution from the upper unit, and the update prompt is generated based on the result of second calculating the contribution of the lower unit for the selected setting unit text. Claim 9 A computing device according to claim 6, wherein the update prompt is generated through highlighting processing of text within the initial prompt, and the highlighting processing includes at least one method of adding meaning to a set unit text among the plurality of set unit texts whose contribution is greater than or equal to a threshold value. Claim 10 In claim 6, the mutual agreement is calculated using any one of the following algorithms: Cohen's kappa algorithm, Rand index or adjusted Rand index algorithm, normalized mutual information algorithm, and F1 score algorithm. Claim 11 A computer program stored on a computer-readable recording medium, wherein the computer program comprises instructions for the processor to perform a multi-model-based prompt optimization method for classifying non-labeled data according to any one of claims 1 to 5 by the processor executing at least one instruction. Claim 12 A computer-readable recording medium storing a computer program, wherein the computer program includes instructions for the processor to perform a multi-model-based prompt optimization method for classifying unlabeled data according to any one of claims 1 to 5 by the processor executing at least one instruction.

Images