System, method, and program for constructing dataset for evaluating user information personalization capabilities of retrievers

Abstract

A system, a method, and a program for constructing a dataset for evaluating user information personalization capabilities of retrievers are disclosed. The method comprises the steps of: extracting a plurality of queries and targets corresponding to each of the plurality of queries from sample data; inputting a first prompt into an AI model and outputting an instruction set comprising a plurality of instructions including a virtual user scenario; further associating the instruction set with each of the plurality of corresponding queries and targets, and outputting the associated data as element data; inputting the element data into the AI model along with a second prompt, and tuning the targets included in the element data to match the virtual user scenario included in the plurality of instructions; and storing the plurality of tuned element data as a dataset.

Description

Systems, methods, and programs for constructing a data set for evaluating the user information customization capabilities of RETRIEVERS

[0001] The present invention relates to a system, method, and program for constructing a data set for evaluating a user information customization function of a retriever, and more particularly, to a system, method, and program for constructing a data set for evaluating a user information customization function of a retriever using an AI model.

[0002] Large Language Models (LLMs) are also additionally trained in the form of instruction tuning for various generation tasks to match user instructions and preferences {Ouyang, Long, et al. Training language models to follow instructions with human feedback. NeurIPS (2022)}.

[0003] In addition to LLMs, it's also desirable to tune information retrievers to reflect user intent (here, retrievers are lightweight filters that scan document repositories to select a set of candidate documents relevant to a query, and can be retrieval systems that use language models). For example, if a user intends to write a blog post for children and searches for documents on climate change, it's more helpful to retrieve easily understandable articles than complex scientific articles.

[0004] However, most search engines simply focus on the user's query and do not reflect the user's information (search intent, preferences, etc.). This makes it difficult to produce search results that reflect the user's intent. For this reason, there is a lack of appropriate means to evaluate whether a search engine reflects the user's information.

[0005] BEIR (Thakur et al., 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models) is a conventional technique for evaluating whether a search engine outputs search results that reflect the user's information. However, the BEIR benchmark evaluates search engines for each search task, rather than for each user instance. Therefore, it is not suitable for evaluating the instruction-following function, which indicates whether the search results through the search engine reflect the actual user's intent. Furthermore, the BEIR benchmark is not suitable for evaluating the instruction-tracing function because the number of instances used for evaluation is too small.

[0006] (Non-patent literature 1) PlotQA: Reasoning over Scientific Plots, Nitesh Methani et al., 2020.

[0007] (Non-patent document 2) Thakur et al., 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models

[0008] The problem to be solved by the present invention is to provide a system, method and program for constructing a data set for evaluating the user information customization function of retrievers.

[0009] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0010] A system for constructing a data set for retrievers using a language model of the present invention may include at least one processor; at least one server; and at least one memory storing instructions or information for causing the at least one processor to perform operations, wherein the instructions perform operations including: extracting a plurality of queries and targets corresponding to each of the plurality of queries from sample data stored in the memory; inputting a first prompt stored in the memory into an AI model stored in a server to output an instruction set composed of a plurality of instructions including a virtual user scenario; further matching the instruction set to each of the plurality of corresponding queries and targets and outputting the instruction set as element data; inputting the element data together with a second prompt into the AI ​​model to tune the targets included in the element data to match the virtual user scenarios included in the plurality of instructions; and storing the tuned plurality of element data as a data set in the memory.

[0011] In the above system, the first prompt may be a command that is input into the AI ​​model and outputs a virtual user scenario containing various information related to the user in the form of a sentence.

[0012] In the above system, the virtual user scenario may include information about the background, location, occupation, hobbies, interests, search goals, or preferred sources of the virtual user.

[0013] In the above system, the step of inputting the data set into an AI model together with a third prompt stored in the memory, and removing the element data that obtained a score lower than a predetermined score from the data set; wherein the third prompt may include an instruction configured to assign a score through the AI ​​model based on whether the target matches the query and whether the target matches the plurality of instructions.

[0014] In the above system,

[0015] The step of tuning to the above virtual user scenario may further include inputting the element data together with the second prompt into the AI ​​model to tune the target including the element data to the query.

[0016] A method for constructing a data set for retrievers using a language model of the present invention may include extracting a plurality of queries and targets corresponding to each of the plurality of queries from sample data; inputting a first prompt into an AI model to output an instruction set composed of a plurality of instructions including a virtual user scenario; further corresponding the instruction set to each of the plurality of queries and the targets and outputting the instruction set as element data; inputting the element data together with a second prompt into the AI ​​model to tune the targets included in the element data to match the virtual user scenarios included in the plurality of instructions; and storing the tuned plurality of element data as a data set.

[0017] In the above method, the first prompt may be a command that is input into the AI ​​model and outputs a virtual user scenario containing various information related to the user as a sentence.

[0018] In the above method, the virtual user scenario may include information about the background, location, occupation, hobbies, interests, search goals, or preferred sources of the virtual user.

[0019] In the above method, the method further includes a step of inputting the data set into an AI model together with a third prompt, and removing the element data that obtained a score lower than a predetermined score from the data set; wherein the third prompt may include an instruction configured to assign a score through the AI ​​model based on whether the target matches the query and whether the target matches the plurality of instructions.

[0020] In the above method, the step of tuning to the virtual user scenario is:

[0021] The method may further include inputting the element data into the AI ​​model together with the second prompt to tune the target including the element data to suit the query.

[0022] According to another aspect of the present invention, a program may be stored in a computer-readable recording medium for constructing a data set for retrievers using a language model according to embodiments of the present invention, in combination with a computer.

[0023] According to the present invention, by securing a large number of virtual user scenarios using an AI model and constructing a large-scale data set, a data set can be derived that more accurately evaluates whether a search engine outputs search results that reflect user information.

[0024] Additionally, according to the present invention, by improving consistency between data, it is possible to more accurately evaluate whether the search engine outputs search results that reflect the user's information.

[0025] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0026] FIG. 1 is a schematic diagram of a system for implementing a method for constructing a data set for evaluating user information customization capabilities of retrievers, according to one embodiment of the present disclosure.

[0027] FIG. 2 is a block diagram illustrating the configuration of a device for constructing a data set for evaluating a user information customization function of retrievers according to one embodiment of the present disclosure.

[0028] FIG. 3 is a block diagram illustrating a method for constructing a data set for evaluating the user information customization function of retrievers according to embodiments of the present invention.

[0029] Figures 4 to 7 are examples of first to third prompt commands according to embodiments of the present invention.

[0030] The following examples are provided as examples to ensure that those skilled in the art can fully grasp the spirit of the present invention. Therefore, the present invention is not limited to the embodiments described below and may be embodied in other forms.

[0031] Throughout the present invention, the same reference numerals denote the same components. The present invention does not describe all elements of the embodiments, and any content that is general in the technical field to which the present invention pertains or that overlaps between the embodiments is omitted. The terms 'part, module, element, block' used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple 'parts, modules, elements, blocks' may be implemented as a single component, or a single 'part, module, element, block' may include multiple components.

[0032] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0033] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0034] Throughout the specification, when we say that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0035] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0036] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0037] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0038] A system for constructing a data set for evaluating the user information customization function of a retriever according to the present invention may include a device, and the device may include any of various devices capable of performing computational processing and providing results to a user. For example, the system for constructing a data set for evaluating the user information customization function of a retriever according to the present invention may include at least one of a computer, a server device, and a portable terminal, or may be any form having the same or similar functions as these. However, the present invention is not limited thereto.

[0039] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0040] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0041] The above portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, ring, bracelet, anklet, necklace, glasses, contact lens, or head-mounted-device (HMD).

[0042] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0043] The present invention relates to a system, method, and program for constructing a data set for evaluating a user information customization function of retrievers, and more particularly, to a system, method, and program for constructing a data set for evaluating a user information customization function of retrievers using an AI model.

[0044] FIG. 1 is a schematic diagram of a system for constructing a data set for evaluating the user information customization function of retrievers according to one embodiment of the present invention.

[0045] As illustrated in FIG. 1, the system (1000) may include a device (100) and a server (200), and the server (200) may include an AI model (300).

[0046] The device (100) and server (200) included in the system (1000) can communicate via a network (W). Here, the network (W) may include a wired network and a wireless network. For example, the network may include various networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0047] Additionally, the network (W) may include the well-known World Wide Web (WWW). However, the network (W) according to an embodiment of the present invention is not limited to the networks listed above, and may include at least part of a well-known wireless data network, a well-known telephone network, or a well-known wired / wireless television network.

[0048] The device (100) can input a first prompt stored in memory into an AI model stored on a server to output a plurality of instruction sets including virtual user scenarios. In addition, the device (100) can generate a plurality of targets by corresponding the plurality of output instruction sets to a plurality of queries, and store the plurality of targets as a data set.

[0049] FIG. 1 illustrates a case where a server (200) is implemented outside of a device (100). In this case, the server (200) may be connected to the device (100) via wired or wireless connections. However, this is merely an example, and the server (200) may also be implemented as a component of the device (100).

[0050] FIG. 1 illustrates a case where the AI ​​model (400) is implemented outside the device (100), but is not limited thereto and may be implemented as a component in the device (100).

[0051] FIG. 2 is a block diagram illustrating the configuration of a device for constructing a data set for evaluating a user information customization function of retrievers according to one embodiment of the present invention.

[0052] As illustrated in FIG. 2, the device (100) may include a memory (110), a communication module (120), a display (130), an input module (140), and a processor (150). However, the present invention is not limited thereto, and the device (100) may have its software and hardware configurations modified / added / omitted within a range apparent from a perspective of ordinary skill in the art, depending on the required operation. In addition, the device (100) may be replaced with a system, and the device (100) may include a plurality of devices, in which case each component included in the device (100) may be included in at least one of the plurality of devices.

[0053] The memory (110) can store data supporting various functions of the device (100), programs for the operation of the processor (150), input / output data, and a plurality of application programs or applications run on the device, data for the operation of the device (100), commands, and AI models. At least some of these application programs can be downloaded from an external server via wireless communication.

[0054] The memory (110) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD (Solid State Disk type), an SDD (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0055] Additionally, the memory (110) may be separate from the device and may include a database connected wired or wirelessly.

[0056] The communication module (120) may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0057] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard 232), power line communication, or plain old telephone service (POTS).

[0058] The wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, in addition to a WiFi module and a Wireless Broadband module.

[0059] The display (130) displays (outputs) information or data processed in the device (100), data input or output through the AI ​​model (400), etc. In addition, the display (130) can display execution screen information of an application program (e.g., an application) running in the device (100), or UI (User Interface) or GUI (Graphical User Interface) information according to such execution screen information.

[0060] The input module (140) is for receiving information from a user. When a user inputs information through the input unit, the processor (150) can control the operation of the device (100) to correspond to the input information.

[0061] The input module (140) may include hardware physical keys (e.g., buttons located on at least one of the front, rear, and side of the device, dome switches, jog wheels, jog switches, etc.) and software touch keys. For example, the touch keys may be formed as virtual keys, soft keys, or visual keys displayed on a touchscreen type display (130) through software processing, or as touch keys placed on a part other than the touchscreen. Meanwhile, the virtual keys or visual keys may have various forms and be displayed on the touchscreen, and may be formed as, for example, graphics, text, icons, videos, or a combination thereof.

[0062] The processor (150) may be implemented as a memory that stores data on an algorithm for controlling the operation of components within the device (100) (including learning or executing an AI model) or a program that reproduces the algorithm, and at least one processor (not shown) that performs the aforementioned operation using the data stored in the memory. In this case, the memory and the processor may be implemented as separate chips, or may be implemented as a single chip.

[0063] In one embodiment, the system (1000) or device (100) according to the present invention may include at least one processor, and when including multiple processors, the multiple processors may be included in different devices (100).

[0064] In addition, the processor (150) can control any one or a combination of the components described above to implement various embodiments according to the present disclosure described below on the device (100).

[0065] FIG. 3 is a block diagram illustrating a method for constructing a data set for evaluating the user information customization function of retrievers according to an embodiment of the present invention.

[0066] Referring to FIG. 3, a plurality of queries and a target corresponding to each of the plurality of queries are extracted from sample data (S210). Each query may correspond to one target, or each query may correspond to multiple targets. The sample data may be stored in the memory (110) of the device (100). The sample data may be configured to include a query, a target, and / or an instruction. The sample data may be already known data or data arbitrarily created by a user, but is not limited thereto.

[0067] A query is a string of text that allows a user to search for information by entering it into a search engine. For example, a query may take the form of a question, such as "What does a bloated stomach feel like?", but is not limited to this.

[0068] A target is the result of a query entered into a search engine, and its relevance may vary depending on the case. For example, a query like "What is the pain of a bloated stomach like?" might output a target for professional opinion, such as "The subject reports significant abdominal discomfort consistent with the bloating associated with abdominal distension. Specifically, the patient experiences a gassy sound ('popping'), which temporarily relieves the pain. This symptom may represent temporary relief from intra-abdominal pressure. Additional diagnostic details, such as pain scale ratings and comparative analysis with other abdominal conditions, are needed to fully characterize the experience to aid in clinical evaluation and treatment planning. Recent research highlights the importance of differentiating between these pain profiles for accurate diagnosis." On the other hand, a target for personal opinion, such as "In today's post, I'm going to address the discomfort I've experienced due to abdominal bloating, which has been quite painful. Oddly enough, I've occasionally experienced a 'popping' sensation, followed by a period of relief from the pressure. I've been trying natural remedies and lifestyle changes to alleviate this abdominal bloating. Stay tuned as I share my experiences with these methods and their results." It can be printed.

[0069] Queries extracted from sample data are called seed queries. Ideally, seed queries can be tailored to various targets as instructions evolve through tuning and filtering, as described in the following steps. Furthermore, seed queries tend to become less clear the longer they are, so they should preferably be comprised of text between 255 and 371 characters, but this is not a limitation.

[0070] Next, the first prompt is input into the AI ​​model to output an instruction set consisting of multiple instructions, which is a virtual user scenario (S220).

[0071] The first prompt may be stored in the memory (110) of the device (100). As shown in Fig. 4, the first prompt refers to a command that is input into the AI ​​model and outputs a virtual user scenario containing various information related to the user (e.g., occupation, search context, search location, search goal, preferred source, etc.) in the form of a sentence. For example, the instructions of the first prompt can be structured such that when the first prompt is input to the AI ​​model, the sentences output can include "My job is {A}", "I am in a situation called {B}", "I am doing {C}", and more specifically, as in the instruction (410), "Your task is to generate a set of scenarios for the provided search query. The following is a specification for the scenario generation task: The scenarios should reflect very specific scenarios in which a user interacts with the AI ​​search engine. Within the scenarios, the user can write about his or her job, background, situation, location, profession, hobbies, interests, or goals for performing the search. Additionally, the user can explicitly reflect his or her preferences for the documents to be searched. The scenarios should be written from a first-person perspective. For example, they should begin with phrases such as "I am {job}", "I am in a situation...", "During my {situation}...".

[0072] The multiple instructions output through the first prompt could include hypothetical user scenarios that seek expertise in a specific area, such as, "As a healthcare professional specializing in gastrointestinal disorders, I have noticed an increase in patients complaining of abdominal discomfort. To better advise you, I am searching for medical articles or patient testimonials that specifically describe the pain associated with abdominal bloating and distinguish it from other types of abdominal pain. I would prioritize finding recent medical studies or articles that include symptoms, patient experiences, and pain scale ratings related to abdominal bloating." Or, hypothetical user scenarios that seek personal opinions, such as, "As a healthcare professional who struggles with digestive health issues, I am looking for home remedies and lifestyle changes that may help alleviate abdominal bloating and the pain it causes. I am particularly interested in personal blogs or holistic health forums that discuss the physical sensation of bloating and provide testimonials on the effectiveness of various treatments."

[0073] Next, for each of the corresponding multiple queries and targets, an instruction set is additionally assigned and output as element data (S230).

[0074] For example, a query corresponds to a single target, and an instruction set consisting of multiple instructions is additionally associated with this target. In this case, in a later element data tuning step, the single target may be tuned to a virtual user scenario included in the multiple instructions, and the single target may be modified into multiple targets corresponding to each of the multiple instructions. However, this is not limited thereto, and for example, a query may correspond to multiple targets, and an instruction set consisting of multiple instructions may be associated with this target.

[0075] Next, the element data is input into the AI ​​model along with the second prompt, so that the target including the element data is tuned to a virtual user scenario included in multiple instructions (S240).

[0076] Here, the second prompt may be stored in the memory (110) of the device (100). The second prompt refers to a command that tunes the target input to the AI ​​model to fit the virtual user scenario of the instruction set, as shown in FIG. 5. For example, the command of the second prompt may include a command such as command (510): "User Instructions: Your task is to generate a revised document for the provided search query and scenario pair. The following is a specification for the document revision task: The revised document should reflect the unique scenario in which the user interacts with the AI ​​search engine. Within the revised document, you should revise details that reflect the user's background, situation, location, occupation, hobbies, interests, or search goals. It is also important to include information related to the user's preferences. You should directly revise the provided document to be of good quality, as found in the AI ​​search engine. It does not end with simply suggesting it! The same keywords from the provided scenario should not be included in the revised document. They should be worded differently."

[0077] For example, let's say the pre-tuning target was "Recent academic discussions and clinical studies have shown that abdominal bloating, the uncomfortable feeling of abdominal distension, is associated with certain dietary choices. Distension of the gastrointestinal tract, commonly described as 'bloating', can cause temporary abdominal pain, which is sometimes relieved by excessive gas passing. This experience has been corroborated by empirical studies in which certain foods, including those high in fermentable oligo-, disaccharides-, monosaccharides-, and polyols (FODMAP), have been identified as triggers for abdominal bloating and the resulting temporary pain." And let's assume the hypothetical user scenario was about a user who wanted a personal opinion. In this case, if you feed the element data into the AI ​​model along with a second prompt, the target content could be tuned to be more about a user who wants personal opinions, such as, "I struggle with digestive health issues and am looking for home remedies and lifestyle changes to help relieve bloating and the pain it causes. I'm particularly interested in personal blogs or holistic health forums that discuss the physical sensation of bloating and provide testimonials on the effectiveness of various treatments."

[0078] Through this, the present invention utilizes an AI model to secure a large number of virtual user scenarios, thereby constructing a large-scale dataset. This allows for the creation of a dataset that more accurately assesses whether a search engine outputs search results that reflect user information. Furthermore, by improving the consistency between target information contained in the dataset and corresponding virtual user scenarios, the present invention enables a more accurate assessment of whether a search engine outputs search results that reflect user information.

[0079] Next, the tuned plurality of element data are stored in the memory as a data set (S250).

[0080] Next, the data set is input into the AI ​​model along with the third prompt to remove element data that have scored lower than a predetermined score from the data set (S260).

[0081] Here, the third prompt may include instructions configured to assign a score to the AI ​​model based on whether the target matches the query and whether the target matches the instruction, and preferably to output a description thereof along with the score.

[0082] More specifically, the third prompt can be configured with a command that assigns a score for whether the target matches the query, as in FIG. 6. For example, the command can be configured to assign 1 point if the target and query appear to have very little or no relationship, 2 points if the target and query appear to have some relationship but are quite distinct, 3 points if the target and query appear to have a moderate relationship, 4 points if the target and query appear to have a high relationship, and 5 points if the target and query appear to have a very high relationship or are nearly identical. More specifically, the third prompt is something like the command (610): "User Instructions: You are a similarity evaluator! Your task is to compute the similarity based on the relatedness between QUERY and DOCUMENT, as shown below. During the evaluation, I want you to rate the relatedness of the pairs according to the following scoring scale: Score 1: DOCUMENT and QUERY are very little or not at all related to each other. The compared elements have few common properties or contexts. Score 2: DOCUMENT and QUERY are somewhat related, but are quite distinct. They share a few properties or context details, but differ significantly in most respects. Score 3: DOCUMENT and QUERY are moderately related to each other. They share some properties or contexts, but there are still notable differences that prevent a high similarity score. Score 4: DOCUMENT and QUERY are highly related. They share many properties or context details, with only a few differences that do not have a major impact. Score 5: DOCUMENT and QUERY are very highly related or nearly identical. They share almost all properties or have almost the same context, and There are only very minor or negligible differences. You will be provided with a QUERY and DOCUMENT pair." command.

[0083] Also, the third prompt is to output a description along with the score, as in the command (620): "Create in the following format: <description> {description of the score}< / 설명> <score> {score}< / 점수> " may contain commands.

[0084] Additionally, the third prompt can be configured as a command that awards points based on whether the target matches the instruction (virtual user scenario), as shown in FIG. 7. For example, the command can be configured such that if the target and the virtual user scenario have little or no connection, 1 point is awarded; if the target and the virtual user scenario have limited connection, 2 points are awarded; if the target and the virtual user scenario have moderate connection, 4 points are awarded; and if the target and the virtual user scenario have perfect connection and are accurate in all aspects, 5 points are awarded. More specifically, the third prompt is a command (710) that says, "User Instructions: You are a ranking agent! Each potential document has a corresponding document ID, and you are tasked with ranking the answers based on their relevance to the QUERY, SCENARIO pair. During the evaluation process, I want you to rank the relevance of the pairs according to the following scoring scale: Score 1: The document has little or no relevance to the user's SCENARIO, and has little to do with the user's occupation, background, situation, location, job, hobbies, interests, or goals. It does not take preferences and context into account, making it an overall poor fit. Score 2: The document has limited relevance, and some elements match the user's SCENARIO and QUERY. It may have some contextual understanding and preference consideration, but does not provide a comprehensive and appropriate response. Score 3: The document is moderately relevant. It captures some aspects of the user's SCENARIO well. It shows adequate contextual relevance and takes user-stated preferences into account. However, it could improve in terms of depth and clarity. Score 4: The document is highly relevant. It does not take into account user's It aligns well with SCENARIO and covers most of the relevant aspects.It exhibits strong contextual relevance and effectively addresses the user's preferences. However, there may be some room for improvement. Score 5: The document is fully relevant and accurately addresses all aspects of the user's SCENARIO, QUERY, and preferences. It integrates perfectly with the user's context and demonstrates a deep understanding. The document is exceptionally clear and concise, and presents information in a well-organized manner, resulting in perfectly tailored results. Given a search document (ID: 1-10), only the top-ranked IDs should be generated, with no additional comments such as [ID]. This is very important! You can include a command like "You will be given a list of documents and a pair of QUERY and SCENARIO."

[0085] Through this, the present invention can more accurately evaluate whether the search engine outputs search results that reflect the user's information by improving consistency between data.

[0086] In another embodiment, the AI ​​model may further include tuning the target included in the element data to a query by inputting the element data together with a second prompt, thereby tuning the target included in the element data to a hypothetical user scenario included in the plurality of instructions. For example, the second prompt may include a command such as "However, the revised document must be relevant to the provided query. In other words, it must be applicable to general queries," such as command (520) of FIG. 5, to enable tuning the target to a query.

[0087] Through this, the present invention can more accurately evaluate whether the search engine outputs search results that reflect the user's information by improving the consistency between the target information included in the data set and the corresponding query.

[0088] Meanwhile, a method for constructing a data set for evaluating the user information customization function of retrievers according to embodiments of the present invention can be implemented by the system described with reference to FIG. 1.

[0089] AI models according to embodiments of the present invention can be controlled, executed, trained, driven, etc. by a processor, and thus, at least one of the tasks of executing, training, and driving the AI ​​models can be performed by at least one processor. Furthermore, the AI ​​models can be stored in memory, and feature data according to the present invention can also be stored in memory.

[0090] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0091] Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0092] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present disclosure can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. A system for building a data set for retrievers using a language model, comprising: at least one processor; at least one server; and at least one memory storing instructions or information that cause the at least one processor to perform an operation, wherein the instructions perform an operation. A step of extracting a plurality of queries and a target corresponding to each of the plurality of queries from sample data stored in the memory; A step of inputting a first prompt stored in the above memory into an AI model stored in a server to output an instruction set consisting of a plurality of instructions including a virtual user scenario; A step of additionally corresponding the instruction set to each of the corresponding plurality of queries and the target and outputting it as element data; A step of inputting the element data into the AI ​​model along with a second prompt, thereby tuning the target included in the element data to fit the virtual user scenario included in the plurality of instructions; and A step of storing the tuned plurality of element data as a data set in the memory; A system including.

2. In claim 1, The above first prompt is a system command that is input into the AI ​​model and outputs a virtual user scenario containing various information related to the user as a sentence.

3. In claim 1, The above hypothetical user scenario is a system that includes information about the background, location, occupation, hobbies, interests, search goals, or preferred sources of the hypothetical user.

4. In claim 1, Further comprising a step of inputting the data set into an AI model together with a third prompt stored in the memory, and removing the element data that obtained a score lower than a predetermined score from the data set; The system wherein the third prompt comprises instructions configured to assign a score to the target based on whether the target matches the query and whether the target matches the plurality of instructions through the AI ​​model.

5. In claim 1, The steps to tune to the above hypothetical user scenario are: A system further comprising inputting the element data into the AI ​​model together with the second prompt, thereby tuning the target including the element data to suit the query.

6. A method for constructing a data set for retrievers using a language model, Extracting multiple queries and targets corresponding to each of the multiple queries from sample data; By inputting a first prompt into the AI ​​model, it outputs an instruction set consisting of multiple instructions containing a virtual user scenario; For each of the corresponding plurality of queries and targets, the instruction set is additionally matched and output as element data; By inputting the above element data into the AI ​​model together with the second prompt, the target included in the above element data is tuned to the virtual user scenario included in the plurality of instructions; A method for storing tuned multiple element data as a data set.

7. In claim 6, The above first prompt is a method in which a command is input into an AI model to output a virtual user scenario containing various information related to the user as a sentence.

8. In claim 6, The above virtual user scenario includes information about the virtual user's background, location, occupation, hobbies, interests, search goals, or preferred sources.

9. In claim 6, Further comprising a step of inputting the above data set into an AI model together with a third prompt, thereby removing the element data that obtained a score lower than a predetermined score from the above data set; A method wherein the third prompt comprises instructions configured to assign a score to the AI ​​model based on whether the target matches the query and whether the target matches the plurality of instructions.

10. In claim 6, The steps to tune to the above hypothetical user scenario are: A method further comprising inputting the element data into the AI ​​model together with the second prompt, thereby tuning the target including the element data to suit the query.

11. A program stored in a computer-readable recording medium for executing the method of any one of claims 6 to 10, in combination with a computer.

Images