Method and system for regenerating question by using prompt optimization through iterative procedures

Abstract

A method and a system for regenerating a question by using prompt optimization through iterative procedures are disclosed. A question regeneration method according to an embodiment may comprise the steps of: receiving a first query; determining whether the input first query includes an open-ended question without a correct answer; generating a tuned second query by tuning the first query including the open-ended question without the correct answer through an artificial intelligence model; and providing the tuned second query.

Description

Method and system for regenerating questions using prompt optimization through an iterative process

[0001] The following describes a method and system for regenerating questions using prompt optimization through an iterative process.

[0002] Large language models (LLMs) are a type of artificial intelligence (AI) trained on large corpora of text data to generate human-like responses to natural language input. These models consist of artificial neural networks with numerous parameters (typically billions of weights or more). These LLMs can be trained on large amounts of unlabeled text using self-supervised or semi-self-supervised learning.

[0003] The process of tuning the queries (prompts) input to AI models, such as LLM, to obtain high-quality answers is called prompt engineering. Even when using the same AI model, the quality of the answers users receive depends on how well the prompts are written and input to the model, making prompt engineering increasingly important.

[0004] A method and system for regenerating questions using prompt optimization through an iterative process are provided.

[0005] A question regeneration method of a question regeneration system implemented by at least one computer device is provided, comprising: a step of receiving a first question by at least one processor included in the at least one computer device; a step of determining by the at least one processor whether the input first question includes a subjective question without a correct answer; a step of tuning by the at least one processor the first question including the subjective question without a correct answer through an artificial intelligence model to generate a tuned second question; and a step of providing by the at least one processor the tuned second question.

[0006] According to one aspect, the step of generating the tuned second query may include a first step of generating an evaluation result for the first query and a first answer generated for the first query; and a second step of inputting a prompt based on the first query, the first answer, and the evaluation result into a first artificial intelligence model to generate a tuned question output by the first artificial intelligence model as the tuned second query.

[0007] According to another aspect, the first answer generated for the first query may be characterized by including an answer generated by inputting the first query into a second artificial intelligence model that is pre-trained to output an answer to the query.

[0008] According to another aspect, the first artificial intelligence model and the second artificial intelligence model may be characterized as different models.

[0009] According to another aspect, the step of generating the tuned second query may further include a third step of repeatedly performing the first step and the second step at least once for the tuned second query and the second answer generated for the tuned second query.

[0010] According to another aspect, the step of generating the tuned second query may include a first step of generating sub-queries included in the framework of the first query using the artificial intelligence model; a second step of generating answers to each of the sub-queries within the framework of the first query using the artificial intelligence model; a third step of generating an evaluation result of the answers to each of the sub-queries; a fourth step of generating an analysis result of the answers to each of the sub-queries by inputting a prompt based on the evaluation result into the artificial intelligence model; and a fifth step of generating the tuned second query for the first query by inputting a prompt based on the analysis result into the artificial intelligence model.

[0011] According to another aspect, the step of generating the tuned second query may further include a sixth step of repeatedly performing the second to fifth steps at least once for the tuned second query.

[0012] According to another aspect, the step of receiving the first query may be characterized in that the first query is transmitted from a service system that includes the query regeneration system or communicates with the query regeneration system via a network, and the first query includes a query input through the service from a user using the service of the service system.

[0013] According to another aspect, the service system may include an instant messaging service server that provides an instant messaging service, and the service of the service system may be characterized by including a conversation service using the instant messaging service between an artificial intelligence assistant account and the user.

[0014] According to another aspect, the step of providing the tuned second query may be characterized in that it is implemented such that the tuned second query is transmitted to the service system, the service system inputs the tuned second query into a pre-trained artificial intelligence model that receives a question and outputs an answer, and the answer output by the pre-trained artificial intelligence model is transmitted to the user.

[0015] According to another aspect, the pre-learned artificial intelligence model may be implemented in an artificial intelligence model-based system that communicates with the service system through a network, and the service system may be implemented to transmit the tuned second query to the artificial intelligence model-based system through an API (Application Programming Interface) request and receive an answer to the tuned second query from the artificial intelligence model-based system through an API response.

[0016] According to another aspect, the determining step may be characterized by determining whether the input query includes a subjective question without a correct answer by using a pre-trained artificial intelligence model that receives a query and outputs information on whether the input query includes a subjective question without a correct answer.

[0017] According to another aspect, the step of generating the tuned second query may be characterized by generating a list of a plurality of tuned second queries by repeating the tuning multiple times, and the step of providing the tuned second query may be characterized by providing a list of the plurality of tuned second queries.

[0018] According to another aspect, it may be characterized in that a recommended query is provided to a user who inputs the first query based on a list of the tuned plurality of second queries.

[0019] A computer program stored on a computer-readable recording medium is provided for executing the above method on a computer device in combination with a computer device.

[0020] A computer-readable recording medium having recorded thereon a program for executing the above method on a computer device is provided.

[0021] A question regeneration system implemented by at least one computer device, comprising at least one processor implemented to execute instructions readable by the at least one computer device, wherein the question regeneration system is characterized in that the at least one processor receives a first question, determines whether the input first question includes a subjective question without a correct answer, tunes the first question including the subjective question without a correct answer through an artificial intelligence model to generate a tuned second question, and provides the tuned second question.

[0022] A method and system for regenerating questions using prompt optimization through an iterative process can be provided.

[0023] FIG. 1 is a diagram illustrating an example of a network environment according to one embodiment of the present invention.

[0024] FIG. 2 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention.

[0025] FIG. 3 is a diagram illustrating an example of an environment in which a question regeneration system operates according to one embodiment of the present invention.

[0026] FIG. 4 is a diagram illustrating an example of a chat room of an instant messaging service in which an artificial intelligence assistant official account and a user participate, according to one embodiment of the present invention.

[0027] FIG. 5 is a diagram illustrating an example of providing a recommended question in one embodiment of the present invention.

[0028] FIG. 6 is a diagram illustrating an example of a process for regenerating a question according to an explicit request of a user in one embodiment of the present invention.

[0029] FIG. 7 is a diagram illustrating an example of a process for regenerating all questions classified as subjective questions without a correct answer without an explicit request from a user, in one embodiment of the present invention.

[0030] FIG. 8 is a flowchart illustrating an example of a question regeneration method according to one embodiment of the present invention.

[0031] FIGS. 9 and 10 are graphs showing examples of the results of quantitative evaluation for the conventional method, the SPE method, and the IPC method, respectively, in one embodiment of the present invention.

[0032] Hereinafter, the detailed description will be given with reference to the attached drawings.

[0033] The question regeneration system according to embodiments of the present invention may be implemented by at least one computer device. In this case, the computer device may have a computer program according to an embodiment of the present invention installed and run, and the computer device may perform the question regeneration method according to embodiments of the present invention under the control of the run computer program. The computer program described above may be stored on a computer-readable recording medium, which may be coupled to the computer device and cause the computer to execute the question regeneration method.

[0034] FIG. 1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 represents an example including a plurality of electronic devices (110, 120, 130, 140), a plurality of servers (150, 160), and a network (170). FIG. 1 is merely an example for explaining the invention, and the number of electronic devices or servers is not limited to that of FIG. 1. In addition, the network environment of FIG. 1 merely illustrates one example of environments applicable to the present embodiments, and the environments applicable to the present embodiments are not limited to the network environment of FIG. 1.

[0035] The plurality of electronic devices (110, 120, 130, 140) may be fixed terminals or mobile terminals implemented as computer devices. Examples of the plurality of electronic devices (110, 120, 130, 140) include smart phones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, etc. For example, in FIG. 1, the shape of a smart phone is shown as an example of the electronic device (110), but in embodiments of the present invention, the electronic device (110) may actually mean one of various physical computer devices that can communicate with other electronic devices (120, 130, 140) and / or servers (150, 160) via a network (170) using a wireless or wired communication method.

[0036] The communication method is not limited, and may include not only a communication method that utilizes a communication network (e.g., a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network (170) may include, but also short-range wireless communication between devices. For example, the network (170) may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. In addition, the network (170) may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network.

[0037] Each of the servers (150, 160) may be implemented as a computer device or multiple computer devices that communicate with multiple electronic devices (110, 120, 130, 140) via a network (170) to provide commands, codes, files, contents, services, etc. For example, the server (150) may be a system that provides services to multiple electronic devices (110, 120, 130, 140) connected via a network (170).

[0038] FIG. 2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Each of the multiple electronic devices (110, 120, 130, 140) or servers (150, 160) described above can be implemented by the computer device (200) illustrated in FIG. 2.

[0039] Such a computer device (200) may include a memory (210), a processor (220), a communication interface (230), and an input / output interface (240), as illustrated in FIG. 2. The memory (210) may be a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a non-permanent mass storage device such as a disk drive. Here, the non-permanent mass storage device such as a ROM and a disk drive may be included in the computer device (200) as a separate permanent storage device distinct from the memory (210). In addition, an operating system and at least one program code may be stored in the memory (210). These software components may be loaded into the memory (210) from a computer-readable recording medium separate from the memory (210). Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (210) via a communication interface (230) other than a computer-readable recording medium. For example, the software components may be loaded into the memory (210) of the computer device (200) based on a computer program installed by files received over a network (170).

[0040] The processor (220) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (220) via the memory (210) or the communication interface (230). For example, the processor (220) may be configured to execute instructions received according to program code stored in a storage device such as the memory (210).

[0041] The communication interface (230) may provide a function for the computer device (200) to communicate with other devices (e.g., the storage devices described above) via the network (170). For example, requests, commands, data, files, etc. generated by the processor (220) of the computer device (200) according to program codes stored in a recording device such as the memory (210) may be transmitted to other devices via the network (170) under the control of the communication interface (230). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (200) via the communication interface (230) of the computer device (200) via the network (170). Signals, commands, data, etc. received via the communication interface (230) may be transmitted to the processor (220) or the memory (210), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (200) may further include.

[0042] The input / output interface (240) may be a means for interfacing with an input / output device (250). For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display or a speaker. As another example, the input / output interface (240) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. At least one of the input / output devices (250) may be configured as a single device with the computer device (200). For example, a touchscreen, a microphone, a speaker, etc. may be implemented in a form included in the computer device (200), such as a smartphone.

[0043] Additionally, in other embodiments, the computer device (200) may include fewer or more components than those illustrated in FIG. 2. However, it is not necessary to explicitly illustrate most conventional components. For example, the computer device (200) may be implemented to include at least some of the input / output devices (250) described above, or may further include other components such as a transceiver, a database, etc.

[0044] FIG. 3 is a diagram illustrating an example of an environment in which a question regeneration system operates according to one embodiment of the present invention. The question regeneration system (300) according to this embodiment may be implemented to be included in a service system (310) or implemented separately from the service system (310) and communicate with the service system (310) via a network (e.g., a network (170)).

[0045] The service system (310) may be implemented using at least one computer device (for example, a computer device (200)) to provide a service to a plurality of user terminals (320) accessing the service system (310). In the present embodiment, the service system (310) may provide a service that provides answers to user inquiries in conjunction with an artificial intelligence model-based system (330). Here, the user may include a service member who has an account for the service of the service system (310). For example, the service provided by the service system (310) may provide an input interface for receiving a user inquiry. In addition, the service system (310) may transmit the user inquiry input through the input interface to the artificial intelligence model-based system (330) using a function such as an API (Application Programming Interface) call.

[0046] The artificial intelligence model-based system (330) can input a user's query into the artificial intelligence model (331) and transmit the output of the artificial intelligence model (331) as an answer to the user's query to the service system (310), and the service system (310) can transmit the answer transmitted from the artificial intelligence model-based system (330) as an answer to the user's query to the corresponding user terminal.

[0047] In this process, the question regeneration system (300) can tune (e.g., optimize) the prompt to be input into the artificial intelligence model (331) as the user's query. For example, the service system (310) can transmit the user's query to the question regeneration system (300), and the question regeneration system (300) can tune the prompt corresponding to the user's query and transmit the improved prompt to the service system (310). In this case, the service system (310) can request the artificial intelligence model-based system (330) to generate an answer using the improved prompt, and the artificial intelligence model-based system (330) can input the prompt improved by the question regeneration system (300) into the artificial intelligence model (331) to generate and provide an answer. In addition, the service system (310) can transmit the answer received from the artificial intelligence model-based system (330) to the corresponding user terminal. Therefore, the user can receive a higher quality answer.

[0048] At this time, the question regeneration system (300) according to the present embodiment can tune "subjective questions without correct answers" among the prompts. Prompts can be divided into system prompts and human prompts. Here, a system prompt may refer to a prompt that is included by default in all user queries, and a human prompt may refer to a prompt such as a query or instruction entered by the user.

[0049] Table 1 below provides examples of system prompts and human prompts. Depending on the embodiment, system prompts may include user information (age, gender, region, occupation, etc.) and prompts that require the user to generate a response using user information. In this case, system prompts may additionally include prompts that request that the user not directly mention user information, i.e., not include user information in the response.

[0050] Terminology Examples Prompt System prompt A default prompt for all user queries You are a competent translator who can translate English into Korean. Please follow the rules below when translating English text into Korean. Human prompt (=query, instruction) A prompt such as a query or instruction entered by the user ○ Everyday conversation - Hello, what is your name? ○ Labeled questions - Tell me yes / no if this text contains an advertisement - Tell me yes / no if this text is hate speech ○ Generating prompts - Generate a movie review ○ Open-ended questions without labels - How can I write a good prompt? - How can I talk to my friend about something I did wrong without hurting their feelings? - Explain the impact of COVID-19 on the global economy

[0051] Furthermore, the quality of an answer can be defined differently depending on the type of human prompt, as shown in Table 2 below. For example, for questions with a correct answer, the quality of the answer can be evaluated by calculating the accuracy of the answer based on the score for a correct answer and the score for an incorrect answer. On the other hand, for open-ended questions without a correct answer, scores can be calculated for various criteria such as accuracy, consistency, completeness, usefulness, relevance, social appropriateness, and specificity. These scores can be obtained through a function created to output item-specific scores for a specific instruction (prompt). Alternatively, the scores can be obtained through a pretrained AI model that outputs item-specific scores for question-answer pairs.

[0052] Terminology Example Quality is defined differently depending on the type of human prompt. ▷ Questions with correct answers· Classification problems → accuracy, F1 score, etc. ▷ Quality for open-ended questions (evaluated in embodiments of the present invention) (questions, answers)· Accuracy: Evaluate whether the answer is factually accurate and consistent with the details of the question.· Consistency: Check whether the answer maintains a logical flow and consistency in the idea.· Completeness: Determine whether the answer completely covers all aspects of the question.· Usefulness: Evaluate whether the answer provides valuable and practical information that satisfies the user's needs or solves the user's problem. This includes checking whether the answer is useful, relevant, and actionable for the user. Rather than simply stating facts, you should instruct users on how to apply those facts effectively. · Relevance: Ensure that answers directly and appropriately answer the question. · Social appropriateness: Evaluate whether answers adhere to moral, ethical, and professional standards; maintain fairness and avoid discrimination; uphold honesty and transparency; respect the rights and dignity of individuals; foster accountability; have a beneficial impact; and avoid inappropriate or offensive sexual content. · Specificity: Ensure that answers provide detailed and accurate information, not general statements.

[0053] Table 3 below shows examples of subjective questions without correct answers, including academic questions, everyday questions / concerns, and examples that have been improved through optimization.

[0054] Glossary of Terms: Example: Academic Question: How does politics impact everyday life? Explain with specific examples how political decisions impact the economic, social, and cultural aspects of everyday life. Everyday Question / Concern: Do you have any regrets about past relationships? If you could go back in time, how would you change your behavior? Reflect on a specific past relationship and provide a detailed example of a behavior you regret. Discuss the emotional impact of this behavior and how you would leverage the opportunity. Consider cultural differences, communication strategies, and personal growth in your answer.

[0055] In this way, the question regeneration system (300) can first identify prompts that include "subjective questions without correct answers" among the input prompts. For example, the question regeneration system (300) may include an artificial intelligence model that receives a prompt and outputs information on whether the input prompt includes a "subjective question without correct answers." Thereafter, the question regeneration system (300) can set the prompt including the "subjective question without correct answers" as a target prompt for optimization and tune this target prompt for optimization. At this time, the optimization direction can be performed in a direction that improves quality while minimizing cost and time.

[0056] Table 4 below shows examples of human prompts and responses before and after optimization, while Table 5 shows examples of the cost required to provide responses and the response quality evaluation scores. The LLM response model was used.

[0057]

[0058] Case Human Prompt Total Cost ($) Evaluation Score (Score Scale: 0-10) Before Improvement (as-is) What is Harry Potter? 0.0001485 [9,10,10,9,8,8,10] After Improvement (to-be) Can you provide a detailed explanation about the Harry Potter series, including information about its author, main characters, plot, setting, and cultural impact? Optimization Cost: 0.0080985 Answer Cost: 0.0006975 Total Cost: 0.008796 [9,10,10,9,9,9,10]

[0059] Let {(Q, A)} be a set of input data Q and output data A sampled from the population, D train , and when a specific single instruction (or prompt) is ρ, the score can be calculated as f(ρ, Q, A). At this time, from the perspective of optimization, the objective function of finding ρ that maximizes the expected value of the score f(ρ, Q, A) for possible (Q, A) data sets can be defined as in mathematical expression 1 below. ρ can be a concept corresponding to a model parameter.

[0060]

[0061] If optimization problems are not solved algorithmically, the optimal ρ can be found through trial and error. As previously explained, these optimization problems can also be solved using artificial intelligence models like LLM.

[0062] In one embodiment, the question regeneration system (300) can utilize a method (hereinafter, Simple Prompt Engineering (SPE)) in which the artificial intelligence model optimizes the question by simply inputting a pair of (question, answer) into an LLM-based artificial intelligence model and prompting the model to improve the existing question.

[0063] First, the question regeneration system (300) can obtain evaluation results for existing question and answer pairs as shown in Table 6 below.

[0064]

[0065] Afterwards, the question regeneration system (300) can obtain a prompt (improved question) that is expected to receive a better quality answer through prompt engineering as shown in Table 7 below.

[0066]

[0067] At this time, "question" as input may mean the previous prompt (question), "previous_response" may mean the answer of the artificial intelligence model to the previous prompt, and "eval_result" may mean the evaluation result (evaluation score) of the (question, answer) obtained as in Table 6.

[0068] Table 8 below shows examples of questions and answers improved through optimization using SPE, and Table 9 shows examples of changes in scores.

[0069]

[0070]

[0071] The question regeneration system (300) can also iteratively perform optimization by inputting additional optimizations into an LLM-based artificial intelligence model using improved question and answer pairs. In another embodiment, the question regeneration system (300) can utilize a method of performing optimization based on synthetic boundary cases (hereinafter, Intent-based Prompt Calibration (IPC)). For example, a good quality benchmark data set is required to compare different prompts, but such a good quality benchmark data set is difficult to obtain. In addition, it is not realistic to access a large amount of data in a situation where optimization is iteratively performed due to cost considerations.

[0072] First, the question regeneration system (300) has a subjective question without a correct answer as an optimization target, unlike the optimization target in a classification problem (e.g., a prompt with a correct answer) or a text generation problem (a prompt without a correct answer and generating a specific text).

[0073] At this time, the question regeneration system (300) can optimize (tune) questions (prompts) classified as subjective questions without a correct answer among the user's queries through the processes of initial prompt generation, prediction, evaluation, analysis, and improvement. At this time, each process of initial prompt generation, prediction, analysis, and improvement, excluding the evaluation process, can be performed through an artificial intelligence model.

[0074] Table 10 below provides examples of prompts input to the AI ​​model for the initial prompt generation process. The prompts for the initial prompt generation process indicate that the AI ​​model should generate {num_samples} unique sub-questions for the main question being optimized. These sub-questions should be complex enough to pose questions not covered by the main question, while also being challenging and relevant to the task description. This is designed to optimize for open-ended questions with no correct answer.

[0075] 초기 프롬프트 생성As an advanced language model you should create {num_samples} challenging and unique sub-questions for the task outlined below.These samples should be intricately designed to throw the questions that the task's main question does not cover, challenging yet relevant to the task description.The task description and main question is phrased as a generative task. The results prompts samples should be input to the model.The model will be able then to generate an answer given the main question and the sub-question input.### Task Description:{task_description}### Main Question:{instruction}### Requirements for Challenging Samples:1. Each prompt must present a unique and intricate challenge.2. The prompts should cover a diverse range of scenarios within the scope of the task, avoiding repetition and predictability.3. Each prompt should contain only the prompt part, without generating also the results.4.Each prompt should contain only the prompt part, without any mention of the task description or instructions!!Generate the prompt samples keeping these requirements in mind.###.

[0076] Table 11 below provides examples of prompts input to the AI ​​model for the prediction process. During the prediction process, prompts can be used to have the AI ​​model generate answers to each of the sub-questions generated during the initial prompt generation process. Because these sub-questions are intertwined with the main question, the prompts are designed to generate answers based on the main question as a large framework.

[0077] Assistant is a large language model designed to generate answers from challenging language task.Given a list of {batch_size} sub-questions, you should generate answers following main question as a larger framework.### Main Question:{task_instruction}### List of sub-questions:{samples}##Each generated text should be end with <eos>token,Answer exactly in the following format for each sample:#### Sample <id>: <text generation> <eos>################ Sample <id>: <text generation> <eos>############

[0078] The evaluation process, as previously described, involves generating scores for various criteria (e.g., accuracy, consistency, completeness, usefulness, relevance, social appropriateness, and specificity). Table 12 below provides examples of prompts input into the AI ​​model for analysis. During the analysis process, prompts (sub-questions) with low scores can be analyzed.

[0079] 분석Assistant is a large language model designed to provide a high quality analysis for every task.You are given the following task description{task_description}Here is the main question that was given to the model:{prompt}An expert evaluated the quality of answer on the main question according to the following scale: {labels}The mean score for this question is: {accuracy}##Here is a list of challenging cases for the given question and their score:## Challenging Cases:{failure_cases}###Note that the score labels are __absolutely correct__, but the prompts (main question) may be incorrect and need modification.Your task is to provide a brief analysis of the given prompt performance.Guidelines:1. The analysis should contain only the following information:- A summary of the common mistakes of the prompt and the ways he can be improve his generation, try to cluster the failure cases into groups and describe each group.2.The total length of your analysis should be less than 200 tokens!###Analysis:.

[0080] Table 13 below provides examples of prompts input to the AI ​​model for the improvement process. During the improvement process, the prompt (main question) can be refined based on the analysis results to generate improved questions.

[0081] 개선Assistant is a large language model designed to provide the best instructions for every task.Below are a few suggested instructions for the task and score (mean of answer quality), for the following task description:{task_description}## Examples{history}######This is the analysis for the last instruction:{error_analysis}######Your task is to generate:1. Anew instruction that is- Different from all the instructions above.- Follows exactly the error analysis modification suggestions, and fix the instruction to improve the quality of the instruction.- Has a higher score than all the instructions above.2. The predicted score of this instructionsYou must adhere the error analysis instructions! Even in case it seems there is a contradiction between these instructions, and the task. The error analysis was evaluate by an expert ranker, thus represent the exact intent of the task.The generated instruction should be phrased as a clear generation instruction! It should not include any instructions and descriptions on the modification that should be done to the instruction. Note that the previous instruction contains implicit assumptions on the intent of the task that might be incorrect. You should replace this assumption with more accurate assumptions using the score of the previous instructions and the error analysis.

[0082] In addition, the question regeneration system (300) can generate improved questions at each iteration by repeatedly performing the prediction, evaluation, analysis, and improvement processes among the processes of initial prompt generation, prediction, evaluation, analysis, and improvement. The question regeneration system (300) can be implemented to provide a question with the highest quality score among the plurality of improved questions generated through the above-described iterations. According to an embodiment, the question regeneration system (300) can also provide a list of a plurality of improved questions. In this case, the list of a plurality of improved questions can further include the quality scores of each of the plurality of improved questions.

[0083] Meanwhile, the service system (310) may transmit the question with the highest quality score to the artificial intelligence model-based system (330). The artificial intelligence model-based system (330) may generate an answer to the transmitted question and transmit it to the service system (310), and the service system (310) may transmit the transmitted answer to the corresponding user terminal. In another embodiment, the service system (310) may provide at least some of a plurality of improved questions as recommended questions to the corresponding user terminal. In this case, the service system (310) may transmit a recommended question selected by the user from among the recommended questions to the artificial intelligence model-based system (330). In this case, the artificial intelligence model-based system (330) may generate an answer to the transmitted question and transmit it to the service system (310), and the service system (310) may transmit the transmitted answer to the corresponding user terminal.

[0084] Meanwhile, the service system (310) may include an instant messaging service server that provides an instant messaging service, and the service provided by the service system (310) may include an instant messaging service. For example, the service system (310) may provide functions for an instant messaging service-based conversation between an artificial intelligence (AI) assistant's official account (OA) and a user.

[0085] FIG. 4 is a diagram illustrating an example of a chat room of an instant messaging service in which an artificial intelligence assistant official account and a user participate, according to an embodiment of the present invention. The embodiment of FIG. 4 illustrates an example of a chat room screen (410) of an artificial intelligence assistant official account and an instant messaging service in which a user participates. When a user inputs a query through the input interface of the chat room, the artificial intelligence assistant official account can generate and provide an answer through the service system (310) and the artificial intelligence model-based system (330). At this time, the service system (310) can provide a user interface for regenerating a question, such as the dotted box (420), through the chat room. If the user selects the user interface (for example, in a touchscreen environment, the user touches the area where the user interface is displayed with a finger), the service system (310) can transmit the user's question, "Who is BBB of Group AAA?" to the question regeneration system (300). In this case, the question regeneration system (300) can generate an improved question for the transmitted question and transmit it to the service system (310). The service system (310) can receive an improved answer to the improved question through the artificial intelligence model-based system (330) and display it as an instant message of the artificial intelligence assistant's official account in the chat room.

[0086] FIG. 5 is a diagram illustrating an example of providing recommended questions according to an embodiment of the present invention. The embodiment of FIG. 5 illustrates an example of a chat room screen (510) of an instant messaging service in which an AI assistant official account and a user participate. When a user selects a user interface for regenerating a question indicated in a dotted box (520), the service system (310) may transmit the user's question, "Who is BBB of Group AAA?", to the question regeneration system (300). At this time, the question regeneration system (300) may generate a list of improved questions for the transmitted question and transmit it to the service system (310). In this case, the service system (310) may expose at least some of the improved questions to the user, as shown in the square box (530), and provide a user interface for selecting one of the exposed questions. When the user selects a specific question, the service system (310) may receive an answer to the selected question through the AI ​​model-based system (330) and display it as an instant message of the AI ​​assistant official account in the chat room. Improved questions can also be exposed to users as instant messages from the AI ​​assistant's official account. In this case, a user interface may also be displayed to prompt users to select a specific question from the displayed list. Alternatively, a user interface may be provided in the chatroom input interface, allowing users to select a specific question by entering text, such as "n times." In another embodiment, a user interface may be provided to prompt users to enter the content (text) of a specific question through the chatroom input interface.

[0087] Meanwhile, as previously explained, since the question regeneration system (300) targets subjective questions without a correct answer for optimization, the question regeneration system (300) can receive and classify all user queries from the service system (310). At this time, the question regeneration system (300) can perform question optimization on queries classified as subjective questions without a correct answer among the user's queries. In this case, the user interface, such as the "Regenerate" button, may operate in different ways depending on the type of the user's query. For example, if the user's query is a question without a correct answer, the question regeneration system (300) may operate to regenerate the question (generate a tuned query). On the other hand, if the user's query is not a question without a correct answer, the query may be regenerated according to the existing logic for regenerating queries.

[0088] Additionally, the above embodiment illustrates an example in which question optimization utilizing the question regeneration system (300) is performed when the user explicitly selects a user interface such as the "Regenerate" button. However, depending on the embodiment, question optimization utilizing the question regeneration system (300) may be automatically performed for all questions classified as subjective questions without a correct answer among the user's questions.

[0089] FIG. 6 is a diagram illustrating an example of a process for regenerating questions according to an explicit request of a user in one embodiment of the present invention, and FIG. 7 is a diagram illustrating an example of a process for regenerating all questions classified as subjective questions without a correct answer without an explicit request of a user in one embodiment of the present invention.

[0090] In the embodiments of FIGS. 6 and 7, when a user (610 or 710) inputs a query through the input interface of the chat room, the service system (310) can first determine whether the query is classified as a subjective question without a correct answer through the question regeneration system (300). However, in the embodiments of FIGS. 6 and 7, this process is omitted for convenience of explanation.

[0091] When a user (610) inputs a question (a subjective question without a correct answer), the service system (310) can transmit the question to the artificial intelligence model-based system (330) via an API request. Thereafter, the artificial intelligence model-based system (330) can generate an answer to the user's (610) question using the artificial intelligence model (331) and transmit the answer to the service system (310) via an API response, and the service system (310) can transmit the transmitted answer back to the user (610). Thereafter, when the user (610) requests regeneration of the question, the service system (310) can optimize the user's (610) question through the question regeneration system (300) to generate and provide an improved question (620). At this time, the service system (310) can transmit the improved question (620) to the artificial intelligence model-based system (330) via an API request using the improved question (620). Thereafter, the artificial intelligence model-based system (330) can generate an improved answer to the improved question (620) using the artificial intelligence model (331) and transmit it to the service system (310) via an API response, and the service system (310) can transmit the improved answer transmitted from the artificial intelligence model-based system (330) back to the user (610).

[0092] In the embodiment of FIG. 7, when a user (710) inputs a question (a subjective question without a correct answer), the service system (310) can optimize the question of the user (710) through the question regeneration system (300) to generate and provide an improved question (720). At this time, the service system (310) can transmit the improved question (720) to the artificial intelligence model-based system (330) through an API request using the improved question (720). Thereafter, the artificial intelligence model-based system (330) can generate an improved answer for the improved question (720) using the artificial intelligence model (331) and transmit the improved answer to the service system (310) through an API response, and the service system (310) can transmit the improved answer transmitted from the artificial intelligence model-based system (330) back to the user (710).

[0093] FIG. 8 is a flowchart illustrating an example of a question regeneration method according to an embodiment of the present invention. The question regeneration method according to the present embodiment can be performed by a question regeneration system (300). The question regeneration system (300) can be implemented by at least one computer device (200). At this time, the processor (220) of the computer device (200) can be implemented to execute control instructions according to the code of the operating system included in the memory (210) or the code of at least one computer program. Here, the processor (220) can operate according to the control instructions provided by the code stored in the computer device (200) to control the computer device (100) so that the question regeneration system (300) implemented by the computer device (200) performs steps (810 to 840) included in the method of FIG. 8.

[0094] In step (810), the question regeneration system (300) may receive a first query. For example, the question regeneration system (300) may receive the first query from a service system (310) that includes the question regeneration system (300) or communicates with the question regeneration system (300) via a network. In this case, the first query may include a query inputted through a service from a user using the service of the service system (310). As described above, the service system (310) may include an instant messaging service server that provides an instant messaging service. In this case, the service of the service system (310) may include a conversation service using an instant messaging service between an artificial intelligence assistant account and a user.

[0095] In step (820), the question regeneration system (300) can determine whether the input first query includes a subjective question without a correct answer. For example, the question regeneration system (300) can determine whether the input query includes a subjective question without a correct answer by using a pre-trained artificial intelligence model that receives a query and outputs information on whether the input query includes a subjective question without a correct answer. The pre-trained artificial intelligence model here may correspond to the artificial intelligence model (331) included in the artificial intelligence model-based system (330), but may also be a separate model different from the artificial intelligence model (331) depending on the embodiment.

[0096] In step (830), the question regeneration system (300) can generate a tuned second question by tuning a first question including a subjective question without a correct answer through an artificial intelligence model.

[0097] In one embodiment, the question regeneration system (300) may generate a tuned second query according to the SPE method described above. For example, the question regeneration system (300) may perform a first step of generating an evaluation result for a first question and a first answer generated for the first question, and a second step of inputting a prompt based on the first question, the first answer, and the evaluation result into a first artificial intelligence model to generate a tuned question output by the first artificial intelligence model as a tuned second query, in order to generate the tuned second query in step (830). Here, the first answer generated for the first query may include an answer generated by inputting the first query into a second artificial intelligence model that is pre-trained to output an answer to the question. Here, the second artificial intelligence model may correspond to the artificial intelligence model (331) included in the artificial intelligence model-based system (330) described above. On the other hand, the first artificial intelligence model may correspond to the artificial intelligence model (331), but may also be a separate model different from the artificial intelligence model (331) depending on the embodiment. Depending on the embodiment, the question regeneration system (300) may repeat the above-described first and second steps at least once for the tuned second query and the second answer generated for the tuned second query, thereby finally generating the tuned second query.

[0098] In another embodiment, the question regeneration system (300) may generate a tuned second query according to the IPC method described above. For example, the question regeneration system (300) may perform a first step of generating sub-queries included in the framework of the first query using an artificial intelligence model to generate the tuned second query in step (830), a second step of generating answers to each of the sub-queries within the framework of the first query using the artificial intelligence model, a third step of generating evaluation results of answers to each of the sub-queries, a fourth step of generating analysis results of answers to each of the sub-queries by inputting a prompt based on the evaluation results into the artificial intelligence model, and a fifth step of generating a tuned second query for the first query by inputting a prompt based on the analysis results into the artificial intelligence model. Here, the artificial intelligence model used for the IPC method may correspond to the artificial intelligence model (331), but may also be a separate model different from the artificial intelligence model (331) depending on the embodiment. Additionally, according to an embodiment, the question regeneration system (300) may repeat steps 2 to 5 for the IPC method at least once for the tuned second query to finally generate the tuned second query.

[0099] In step (840), the question regeneration system (300) may provide a tuned second query. For example, the question regeneration system (300) may transmit the tuned second query to the service system (310). In this case, the service system (310) may be implemented to input the tuned second query into a pre-trained artificial intelligence model that receives a question and outputs an answer, and to transmit the answer output by the pre-trained artificial intelligence model to the user. Here, the pre-trained artificial intelligence model may correspond to the artificial intelligence model (331) included in the artificial intelligence model-based system (330). For example, the pre-trained artificial intelligence model may be implemented in the artificial intelligence model-based system (330) that communicates with the service system (310) via a network. In this case, the service system (310) may be implemented to transmit the tuned second query to the artificial intelligence model-based system (330) through an API request, and receive an answer to the tuned second query from the artificial intelligence model-based system (330) through an API response. The artificial intelligence model-based system (330) may input the tuned second query transmitted from the service system (310) into the artificial intelligence model (331) to generate an answer, and may transmit the generated answer to the service system (310). In this case, the service system (310) may provide an improved answer to the user's query by transmitting the transmitted answer to the user.

[0100] As previously described, the question regeneration system (300) may generate a list of tuned second queries generated in each iteration at step (830). In this case, the question regeneration system (300) may also provide a list of tuned second queries at step (840). Examples of providing recommended queries through a plurality of tuned second queries have been previously described in detail.

[0101] Figures 9 and 10 are graphs illustrating examples of quantitative evaluation results for the conventional method, the SPE method, and the IPC method, respectively, according to one embodiment of the present invention. Here, the conventional method may refer to a method that utilizes pre-optimization questions and answers to pre-optimization questions.

[0102] The experiment was conducted on three types of datasets (AI QA dataset, relationships generated using LLM, and questions related to various fields), and Table 14 below shows the results.

[0103]

[0104] Meanwhile, the graph in Fig. 9 shows examples of quantitative evaluation results for various subject question datasets from LLMs of the conventional, SPE, and IPC methods. Here, (asis_asis) can correspond to the conventional method, (pe_asis_tobe) to the SPE method, and (ipc_asis_tobe) to the IPC method, respectively. In addition, the graph in Fig. 10 shows examples of quantitative evaluation results for relational subject question datasets from LLMs of the conventional, SPE, and IPC methods.

[0105] Overall, the IPC method shows greater performance improvements than the SPE method. The SPE method improved quality in accuracy, completeness, usability, and specificity compared to the existing method, while the IPC method improved quality in all categories except social relevance.

[0106] In this way, according to embodiments of the present invention, a method and system for regenerating questions using prompt optimization through an iterative process can be provided.

[0107] The systems or devices described above may be implemented as hardware components, or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0108] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0109] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., singly or in combination. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0110] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0111] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.< / eos> < / id> < / eos> < / id> < / eos>

Claims

1. A question regeneration method of a question regeneration system implemented by at least one computer device, A step of receiving a first query by at least one processor including the at least one computer device; A step of determining, by the above at least one processor, whether the input first query includes a subjective question with no correct answer; A step of generating a tuned second query by tuning a first query including a subjective question without a correct answer through an artificial intelligence model by the above at least one processor; and The step of providing the tuned second query by the above at least one processor Question regeneration method including 2. In paragraph 1, The step of generating the above-mentioned tuned second query is, A first step of generating an evaluation result for the first query and the first answer generated for the first query; and A second step of inputting a prompt based on the first query, the first answer, and the evaluation result into a first artificial intelligence model to generate a tuned question output by the first artificial intelligence model as the tuned second query. A question regeneration method characterized by including 3. In paragraph 2, A question regeneration method characterized in that the first answer generated for the above first question includes the answer generated by inputting the above first question into a second artificial intelligence model that has been pre-trained to output an answer to the question.

4. In Paragraph 3, A question regeneration method characterized in that the first artificial intelligence model and the second artificial intelligence model are different models from each other.

5. In paragraph 2, The step of generating the above-mentioned tuned second query is, A third step of repeating the first step and the second step at least once with respect to the tuned second query and the second answer generated for the tuned second query. A question regeneration method characterized by further including 6. In paragraph 1, The step of generating the above-mentioned tuned second query is, A first step of generating sub-queries included in the framework of the first query using the artificial intelligence model above; A second step of generating answers for each of the sub-queries within the framework of the first query using the artificial intelligence model; A third step of generating evaluation results for the answers to each of the above sub-queries; A fourth step of generating an analysis result of an answer for each of the sub-queries by inputting a prompt based on the evaluation result into the artificial intelligence model; and Step 5: generating the tuned second query for the first query by inputting a prompt based on the analysis result into the artificial intelligence model. A question regeneration method characterized by including 7. In paragraph 6, The step of generating the above-mentioned tuned second query is, Step 6, which repeats Steps 2 through 5 at least once for the tuned second query. A question regeneration method characterized by further including 8. In paragraph 1, The step of receiving the above-mentioned first query is, A service system that includes the above-mentioned question regeneration system or communicates with the above-mentioned question regeneration system via a network receives the above-mentioned first query, and The above first query includes a query entered through the service by a user using the service of the above service system. A question regeneration method characterized by 9. In Paragraph 8, The above service system includes an instant messaging service server that provides instant messaging services, and The service of the above service system includes a conversation service using an instant messaging service between an artificial intelligence assistant account and the user. A question regeneration method characterized by 10. In paragraph 8, The step of providing the above-mentioned tuned second query is, The above-mentioned tuned second query is transmitted to the above-mentioned service system, and The above service system is implemented to input the above-mentioned tuned second query into a pre-trained artificial intelligence model that receives the question and outputs an answer, and to deliver the answer output by the pre-trained artificial intelligence model to the above-mentioned user. A question regeneration method characterized by 11. In Paragraph 10, The above pre-learned artificial intelligence model is implemented in an artificial intelligence model-based system that communicates with the service system through a network, The above service system is implemented to transmit the tuned second query to the artificial intelligence model-based system through an API (Application Programming Interface) request and to receive an answer to the tuned second query from the artificial intelligence model-based system through an API response. A question regeneration method characterized by 12. In Paragraph 1, The above-mentioned determining step is, A question regeneration method characterized by receiving a query and using a pre-trained artificial intelligence model to output information on whether the input query includes a subjective question without a correct answer, thereby determining whether the input query includes a subjective question without a correct answer.

13. In paragraph 1, The step of generating the above-mentioned tuned second query is, Repeating the above tuning multiple times to generate a list of tuned second queries, The step of providing the above-mentioned tuned second query is, Providing a list of the above tuned plurality of second queries A question regeneration method characterized by 14. In paragraph 13, A question regeneration method characterized by providing a recommended query to a user who entered the first query based on a list of the tuned plurality of second queries.

15. A computer program stored on a computer-readable recording medium combined with a computer device to execute the method of any one of claims 1 to 14 on the computer device.

16. A computer-readable recording medium having recorded thereon a computer program for executing the method of any one of claims 1 to 14 on a computer device.

17. In a question regeneration system implemented by at least one computer device, At least one processor configured to execute instructions readable by at least one computer device, By at least one processor, Receive the first query, and Determine whether the first query entered above contains a subjective question without a correct answer, A first query including a subjective question without a correct answer is tuned using an artificial intelligence model to generate a tuned second query, Providing the above-mentioned tuned second query A question regeneration system featuring .

18. In Paragraph 17, To generate the tuned second query, by the at least one processor, A first process for generating an evaluation result for the first query and the first answer generated for the first query; and A second process of inputting a prompt based on the first query, the first answer, and the evaluation result into a first artificial intelligence model to generate a tuned question output by the first artificial intelligence model as the tuned second query. A question regeneration system characterized by processing.

19. In Paragraph 17, To generate the tuned second query, by the at least one processor, A first process of generating subqueries included in the framework of the first query using the artificial intelligence model; A second process of generating answers to each of the sub-queries within the framework of the first query using the artificial intelligence model; A third process for generating evaluation results of answers to each of the above sub-queries; A fourth process of generating analysis results of answers to each of the sub-queries by inputting a prompt based on the evaluation results into the artificial intelligence model; and A fifth process of generating a tuned second query for the first query by inputting a prompt based on the analysis result into the artificial intelligence model. A question regeneration system characterized by processing .

20. In paragraph 17, By the at least one processor to input the above first query, A service system that includes the above-mentioned question regeneration system or communicates with the above-mentioned question regeneration system via a network receives the above-mentioned first query, and The above first query includes a query entered through the service by a user using the service of the above service system. A question regeneration system featuring .

Images