System, method, and program for constructing data set for evaluating user information customization function of retriever

Abstract

Disclosed are a system, a method, and a program for constructing a data set for evaluating a user information customization function of a retriever. The method comprises the following steps: extracting a plurality of queries (Query) and targets (targets) respectively corresponding to the plurality of queries (Query) from sample data; inputting the first cue word into the AI model and outputting an instruction set composed of a plurality of instructions including a virtual user scene; a plurality of queries (Query) and the target (target) corresponding to the plurality of queries and the target (target), respectively, further corresponding to the instruction set, and outputting the plurality of queries and the target (target) as element data; inputting the element data together with a second cue word into the AI model, and performing fine tuning so as to match the target included in the element data with the virtual user scene included in the plurality of instructions; and storing the plurality of fine-tuned element data as a data set.

Description

TECHNICAL FIELD

[0001] The present invention relates to a system, method, and program for constructing a data set for evaluating a user information customization function of retrievers, and more particularly, to a system, method, and program for constructing a data set for evaluating a user information customization function of retrievers using an AI (Artificial Intelligence) model. BACKGROUND

[0002] LLM (Large Language Models) also receives additional training in the form of instruction tuning for various generation tasks to align the user's instruction (instruction) with the preference degree {Ouyang, Long, et al. Training language models to follow instructions with human feedback. NeurIPS (2022)}.

[0003] In addition to LLM, retrievers also preferably tune the user's preference degree to reflect the user's intention {here, retrievers refer to lightweight filters that retrieve a document repository to select a candidate document set related to a query (query), and can be a retriever system using a language model}. For example, in the case where a user retrieves a document related to the issue of climate change with the intention of describing a blog for children, it is more helpful to retrieve news that is easy to understand than to retrieve complex scientific news.

[0004] However, most retrievers cannot reflect the user's information (search intention, inclination, etc.), but simply focus on the user's query and output the search result, so it is difficult to output a search result that reflects the user's intention. For this reason, appropriate means to evaluate whether the retriever outputs a search result that reflects the user's information is also lacking.

[0005] As a conventional technique for evaluating whether a retriever outputs a search result reflecting user information, BEIR (Thakur et al., 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models) is disclosed. However, BEIR benchmark evaluates retrievers not according to the user's instance but for each search job, and thus is not suitable for evaluating an instruction-following function indicating whether a search result obtained by a retriever contains an actual user's intention. In addition, the number of instances for evaluation in BEIR benchmark is too small, and thus is also not suitable for evaluating the instruction-following function.

[0006] (Non-Patent Literature 1) PlotQA: Reasoning over Scientific Plots, Nitesh Methani et al., 2020.

[0007] (Non-Patent Literature 2) Thakur et al., 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models SUMMARY

[0008] PROBLEMS TO BE SOLVED BY THE INVENTION

[0009] The present application is to solve the problem of providing a system, method, and program for constructing a dataset for evaluating the user information customization function of a retriever.

[0010] The problem to be solved by the present application is not limited to the above-mentioned problem, and other problems not mentioned herein can be clearly understood by those skilled in the art from the following description.

[0011] MEANS FOR SOLVING THE PROBLEM

[0012] A system for constructing a dataset of retrievers using a language model according to the present invention includes at least one processor, at least one server, and at least one memory having stored therein commands or information for causing the at least one processor to perform operations including the steps of: extracting a plurality of queries (Query) and targets (target) corresponding to the plurality of queries (Query), respectively, from sample data stored in the memory; inputting a first prompt stored in the memory to an AI model stored in the server and outputting an instruction set composed of a plurality of instructions including a virtual user scenario; outputting the plurality of queries (Query) and the targets (target), respectively, as element data further corresponding to the instruction set; inputting the element data together with a second prompt to the AI model and fine-tuning the targets included in the element data to match the virtual user scenario included in the plurality of instructions; and storing the fine-tuned plurality of element data as a dataset in the memory.

[0013] In the above system, the first prompt is a command input to the AI model to output a virtual user scenario including various information related to a user as an article.

[0014] In the above system, the virtual user scenario includes information related to a background, a location, a profession, a hobby, an interest, a search purpose, or a preferred source of a virtual user.

[0015] In the above system, further including the steps of: inputting the dataset together with a third prompt stored in the memory to the AI model and removing the element data obtaining a score lower than a predetermined score from the dataset, the third prompt including a command configured to assign a score according to whether the target is consistent with the query and whether the target is consistent with the plurality of instructions through the AI model.

[0016] In the above system, the step of fine-tuning to match the virtual user scenario further includes inputting the element data together with the second prompt to the AI model and fine-tuning the targets included in the element data to match the queries (Query).

[0017] The method of constructing a data set for retrievers using a language model according to the present invention includes the steps of extracting a plurality of queries and targets corresponding to the plurality of queries from sample data, inputting a first prompt to an AI model and outputting an instruction set composed of a plurality of instructions including a virtual user scenario, outputting the plurality of queries and the targets as element data corresponding to the instruction set, inputting the element data to the AI model together with a second prompt, fine-tuning the targets included in the element data to match the virtual user scenario included in the plurality of instructions, and storing the fine-tuned element data as a data set.

[0018] In the above method, the first prompt is a command input to the AI model to output a virtual user scenario including various information related to a user as an article.

[0019] In the above method, the virtual user scenario includes information related to a background, a location, a profession, a hobby, an interest, a search purpose, or a preferred source of a virtual user.

[0020] In the above method, the method further includes inputting the data set to the AI model together with a third prompt including a command configured to assign a score according to whether the target is consistent with the query and whether the target is consistent with the plurality of instructions through the AI model, and removing the element data obtaining a score lower than a predetermined score from the data set.

[0021] In the above method, the step of fine-tuning the virtual user scenario further includes inputting the element data to the AI model together with the second prompt and fine-tuning the target included in the element data to match the query.

[0022] Another aspect of the present invention is a program stored in a computer-readable recording medium to construct a data set for retrievers using a language model.

[0023] Effects of the Invention

[0024] According to the present invention, a large-scale data set is constructed by ensuring a large number of virtual user scenarios using an AI model, so that a data set that more accurately evaluates whether a retriever outputs a search result reflecting user information can be derived.

[0025] In addition, according to the present application, matching between data is improved, so that it is possible to more accurately evaluate whether a retriever outputs a search result reflecting user information.

[0026] Effects of the present application are not limited to the above-mentioned effects, and other effects not mentioned herein will be apparent to those skilled in the art from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a schematic diagram of a system for implementing a method of constructing a data set for evaluating user information customization functions of retrievers according to one embodiment of the present disclosure.

[0028] Figure 2 is a block diagram for explaining the structure of an apparatus for constructing a data set for evaluating user information customization functions of retrievers according to one embodiment of the present disclosure.

[0029] Figure 3 is a block diagram for explaining a method of constructing a data set for evaluating user information customization functions of retrievers according to one embodiment of the present application.

[0030] Figures 4 to 7 is an example of the first to third prompt word commands according to one embodiment of the present application. DETAILED DESCRIPTION

[0031] The embodiments described below are presented by way of illustration to fully set forth the principles of the present application to those skilled in the art. Thus, the present application is not limited to the embodiments described below, but can be embodied in other forms.

[0032] In the entire specification of the present application, the same symbols represent the same constituent elements. The present application does not describe all elements of the embodiments, and general matters in the art or matters repeated between the embodiments are omitted. The terms such as "part, module, component, block" used in the specification can be implemented by software or hardware, and according to the embodiments, a plurality of "parts, modules, components, blocks" can be implemented by one constituent element or one "part, module, component, block" can include a plurality of constituent elements.

[0033] In the entire specification, when referring to a certain part being "connected" with other parts, this includes not only the case where the certain part is directly connected with the other parts, but also the case where the certain part is indirectly connected with the other parts, including the case where they are connected through a wireless communication network.

[0034] In addition, when referring to a certain part "including" a certain constituent element, this does not exclude other constituent elements unless specifically stated otherwise, and other constituent elements can also be included.

[0035] Throughout the specification, when referring to a certain component being "on" another component, not only the case where the certain component is in contact with the other component is included, but also the case where another component is present between the two components is included.

[0036] The first, second, and the like are used to distinguish one constitutional element from other constitutional elements, and these are used for the constitutional element without a limiting effect.

[0037] In the case where it is not explicitly defined in the specification, the singular includes the plural.

[0038] The identification symbol of each step is used for convenience of explanation, and the identification symbol does not indicate the order of each step, and in the case where a specific order is not explicitly described in the specification, each step can be implemented in an order different from the described order.

[0039] The system for constructing a data set for evaluating a user information customization function of retrievers according to the present application can include a device, and the device can include various devices that perform an operation process to provide a result to a user. For example, the system for constructing a data set for evaluating a user information customization function of retrievers according to the present application includes at least one computer, a server device, a portable terminal, and the like, or can be any form that implements the same or similar functions. The present application is not limited to this.

[0040] Here, the computer includes, for example, a notebook computer, a desktop, a laptop, a tablet PC, a Slate PC, and the like, in which a web browser is built-in.

[0041] The server device, which is a server that processes information by communicating with an external device, can include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, a web server, and the like.

[0042] The aforementioned portable terminals, as wireless communication devices that ensure portability and mobility, may include all types of handheld wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet), and smartphones, as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0043] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0044] This invention relates to systems, methods, and procedures for constructing datasets for evaluating the user information customization capabilities of retrievers; more specifically, this invention relates to systems, methods, and procedures for constructing datasets for evaluating the user information customization capabilities of retrievers using AI models.

[0045] Figure 1 This is a schematic diagram of a system for constructing a dataset for evaluating the user information customization function of retrievers, according to an embodiment of the present invention.

[0046] like Figure 1 As shown, system 1000 may include device 100 and server 200, and server 200 may include AI model 300.

[0047] The device 100 and server 200 included in system 1000 can communicate via network W. Here, network W can include wired networks and wireless networks. For example, the network can include various networks such as local area network (LAN), metropolitan area network (MAN), and wide area network (WAN).

[0048] Additionally, network W may include the well-known World Wide Web (WWW). However, network W in embodiments of the present invention is not limited to the networks listed above, and may also include at least a portion of well-known wireless data networks, well-known telephone networks, and well-known wired and wireless television networks.

[0049] Device 100 inputs a first prompt stored in memory into an AI model stored on a server and outputs a set of multiple instructions that include a virtual user scenario. Additionally, device 100 maps the output instruction sets to multiple queries to generate multiple targets, and stores these targets as a dataset.

[0050] Figure 1 This illustration shows a scenario where the server 200 is implemented externally to the device 100. In this case, the server 200 can be connected to the device 100 via wired / wireless means. However, this is only one embodiment, and the server 200 can also be implemented as a component of the device 100.

[0051] Figure 1 The example shown illustrates the implementation of AI model 400 outside of device 100, but it is not limited to this; AI model 400 can also be implemented as a component of device 100.

[0052] Figure 2 This is a block diagram illustrating the structure of an apparatus for constructing a dataset for evaluating user information customization functions of retrievers, according to an embodiment of the present invention.

[0053] like Figure 2 As shown, device 100 may include a memory 110, a communication module 120, a display 130, an input module 140, and a processor 150. However, it is not limited to this; depending on the required operation, software and hardware structures within the scope that are obvious to those skilled in the art may be modified, added to, or omitted from device 100. Furthermore, device 100 may be replaced by a system, and device 100 may include multiple devices, in which case the various components included in device 100 may be included in at least one of the multiple devices.

[0054] The memory 110 may store data supporting various functions of the device 100, programs for operating the processor 150, input / output data, multiple applications or applications driven in the device, data, commands, and AI models required for the operation of the device 100. At least some of such applications may be downloaded from an external server via wireless communication.

[0055] Such a memory 110 may include at least one type of storage recording medium selected from flash memory type, hard disk type, SSD type, SDD type, multimedia card micro type, card type memory (e.g., SD or XD memory), RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic storage, magnetic disk, and optical disk.

[0056] Additionally, the memory 110 may be detached from the device and may include a database connected via wired or wireless means.

[0057] The communication module 120 may include one or more components for communicating with external devices, such as at least one of a broadcast receiving module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0058] Wired communication modules can include not only various wired communication modules such as local area network (LAN) modules, wide area network (WAN) modules, or value-added network (VAN) modules, but also various cable communication modules such as USB (Universal serial bus), DHMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

[0059] In addition to Wi-Fi modules and wireless broadband modules, wireless communication modules can also include wireless communication modules that support various wireless communication methods such as GSM (global system for mobile communication), CDMA (code division multiple access), WCDMA (wideband code division multiple access), UMTS (universal mobile telecommunications system), TDMA (time division multiple access), LTE (long term evolution), 4G, 5G, and 6G.

[0060] The display 130 displays (outputs) information or data processed in the device 100, data input or output through the AI ​​model 400, etc. In addition, the display 130 may display execution screen information of an application (e.g., a program) driven in the device 100 or UI (User Interface) or GUI (Graphical User Interface) information generated based on such execution screen information.

[0061] The input module 140 is used to receive information from the user. When the user inputs information through the input section, the processor 150 controls the operation of the device 100 in a manner corresponding to the input information.

[0062] Such an input module 140 may include hardware physical keys (e.g., buttons, dome switches, jog wheels, jog switches, etc. located on at least one of the front, rear, and side surfaces of the device) and software touch keys. As an example, the touch keys may consist of virtual keys, soft keys, or visual keys displayed on a touchscreen-type display 130 through software processing, or touch keys located outside the aforementioned touchscreen. On the other hand, the aforementioned virtual keys or visual keys may be displayed on the touchscreen in various forms, such as graphics, text, icons, videos, or combinations thereof.

[0063] The processor 150 may be implemented by a memory storing data related to an algorithm or a program for reproducing an algorithm used to control the operations of the constituent elements within the device 100 (including training or executing an AI model) and at least one processor (not shown) that performs the aforementioned operations using the data stored in the memory. In this case, the memory and the processor may be implemented by separate chips or by a single chip.

[0064] In one embodiment, the system 1000 or apparatus 100 of the present invention may include at least one processor; in the case of multiple processors, the multiple processors may be included in apparatus 100 that are different from each other.

[0065] In addition, the processor 150 may control one or more combinations of the above-described components in order to implement the various embodiments of the present disclosure described below on the device 100.

[0066] Figure 3 This is a block diagram illustrating a method for constructing a dataset for evaluating the user information customization function of a retriever according to an embodiment of the present invention.

[0067] Reference Figure 3 The sample data is used to extract multiple queries and corresponding targets for each query (S210). One query may correspond to one target or multiple targets. The sample data may be stored in the memory 110 of the device 100. The sample data is configured to include queries, targets, and / or instructions. The sample data may be publicly known data or user-created data, but is not limited to these.

[0068] A query is a string that a user enters into a retrieval system to retrieve information. For example, a query can be a question like "How does bloating and abdominal pain feel?", but it is not limited to this.

[0069] The goal is to output results for a search engine input query, and the degree of relevance will vary depending on the context. For example, when a query like "How does bloating and abdominal pain feel?" is entered, the output might be something like, "Subjects report significant abdominal discomfort consistent with a feeling of distension associated with bloating. In particular, patients may hear a gas-like ("bursting") sound, which temporarily relieves the pain. This symptom can manifest as temporary relief due to pressure within the abdominal cavity. To fully characterize experiences that contribute to clinical assessment and treatment, additional diagnostic details such as pain rating scales and comparative analysis with other abdominal conditions are needed. Recent studies have emphasized the need to differentiate such pain characteristics for accurate diagnosis." On the other hand, the output might be something like, "In today's post, I want to share my experience with bloating, which was very painful. Surprisingly, I occasionally felt a 'bursting' sensation, and strangely, there was a period of relief from the pressure later. I tried natural remedies and lifestyle changes to relieve this bloating. I will continue to share these methods and my experience with the results, so please stay tuned." This is a more personal opinion.

[0070] The query extracted from the sample data is called a seed query. Preferably, the seed query can be matched with various targets by evolving instructions through fine-tuning or filtering according to the steps below. In addition, as the length of the seed query increases, the main idea becomes unclear. Therefore, the seed query is preferably composed of text of 255 to 371 characters, but is not limited to this.

[0071] Next, the first prompt word is input into the AI ​​model, and an instruction set consisting of multiple instructions representing a virtual user scenario is output (S220).

[0072] The first prompt word can be stored in the memory 110 of device 100. For example... Figure 4As shown, the first suggestion word refers to the command input into the AI ​​model that outputs a virtual user scenario, including various information related to the user (e.g., occupation, search context, search location, search purpose, preferred source, etc.), as an article. For example, the command for the first suggestion word is structured such that when the first suggestion word is input into the AI ​​model, the output article includes "My occupation is {A}", "I am in state {B}", "I am performing {C}". More specifically, it can also be structured as command 410: "Generating a set of scenarios for the provided search query is everyone's job. Below are the instructions for the scenario generation task: The scenario needs to reflect a very specific scenario of the user's interaction with the AI ​​search engine. Within the scenario, the user can write their own job title, background, status, location, position, hobbies, interests, or the purpose of performing the search. In addition, the user can clearly express their preference for the documents to be searched. The scenario needs to be written from a first-person perspective. For example, it needs to start with statements such as "My {occupation} is", "My current state is...", "During my {state}..."

[0073] The first prompt can be used to output multiple commands, such as: "As a medical specialist treating gastrointestinal disorders, I've noticed an increase in patients suffering from abdominal discomfort. To provide better consultation, I'm searching for medical papers or patient testimonies that specifically describe pain associated with bloating and can differentiate it from other types of abdominal pain. I'd prefer to find recent medical research or news reports that include symptoms related to bloating, patient experiences, and pain rating scales," which reflects a desire for expertise in a specific field. Alternatively, it could be used to output commands like: "While suffering from digestive health issues, I'm looking for home remedies and lifestyle changes to relieve bloating and the resulting pain. In particular, I'm very interested in personal blogs or holistic health forums that discuss the physical sensations of bloating and provide testimonials about the effectiveness of various treatments," which reflects a desire for personal opinions.

[0074] Next, the corresponding multiple queries and targets are further mapped to the instruction set and output as feature data (S230).

[0075] For example, a query might correspond to a single target, which in turn corresponds to a set of instructions comprising multiple commands. In this case, during subsequent fine-tuning of the feature data, the single target is adjusted to match the virtual user scenario included in the multiple commands, thereby modifying the single target into multiple targets corresponding to the multiple commands respectively. However, this is not the only possibility; for instance, a query could also correspond to multiple targets, which in turn correspond to a set of instructions comprising multiple commands.

[0076] Next, the feature data and the second prompt word are input into the AI ​​model to fine-tune the matching between the target included in the feature data and the virtual user scenario included in the multiple instructions (S240).

[0077] Here, the second prompt word can be stored in the memory 110 of device 100. For example... Figure 5 As shown, the second prompt word refers to a command that fine-tunes the target by matching the target input into the AI ​​model with the virtual user scenario of the instruction set. For example, a command with a second prompt word might include a command like command 510: "User Instructions: Generating a revised document based on the provided search query and scenario is your task. Here are the instructions for the document revision task: The revised document needs to reflect the inherent scenario of the user's interaction with the AI ​​search engine. Within the revised document, detailed information reflecting the user's background, situation, location, occupation, hobbies, interests, or the target of the search needs to be modified. In addition, including information related to the user's preferences is very important. High-quality documents that can be found from the AI ​​search engine need to be directly modified. Simply providing suggestions is not enough! The same keywords for the provided scenario cannot be included in the revised document. The corresponding content needs to be expressed in a different way."

[0078] For example, suppose the initial target is something like, "Based on recent academic discussions and clinical studies, bloating, a feeling of discomfort due to abdominal distension, is related to the choice of specific diets. Distension of the gastrointestinal tract, generally described as 'bloating,' can trigger temporary abdominal pain, sometimes relieved by excessive flatulence. This experience has been validated by an empirical study showing that certain meals high in fermentable oligosaccharides, disaccharides, monosaccharides, and FODMAPs can trigger bloating and the resulting temporary pain." The virtual user scenario is one relevant to users seeking personalized advice. In this case, when the element data is input into the AI ​​model along with the second cue word, the target content is fine-tuned to something like, "During periods of suffering from digestive health issues, I am looking for home remedies and lifestyle changes to alleviate bloating and the resulting pain. In particular, I am very interested in personal blogs or holistic health forums that discuss the physical sensations of bloating and provide postscripts on the effects of various treatments," to match the scenario relevant to users seeking personalized advice.

[0079] Therefore, this invention utilizes AI models to construct a large-scale dataset using numerous virtual user scenarios, thereby deriving a dataset that more accurately evaluates whether the search engine outputs search results reflecting user information. Furthermore, this invention improves the matching between target information included in the dataset and the corresponding virtual user scenarios, thus more accurately evaluating whether the search engine outputs search results reflecting user information.

[0080] Next, the fine-tuned data of the aforementioned elements are stored as a dataset in the aforementioned memory (S250).

[0081] Next, the dataset and the third prompt word are input into the AI ​​model, and the feature data that receives a lower score than the predetermined score is removed from the dataset (S260).

[0082] Here, the third prompt may include a command configured to assign a score based on whether the target matches the query and whether the target matches the instruction, preferably, a command that outputs a description of this along with the score.

[0083] More specifically, such as Figure 6As shown, the third prompt word can be composed of a command that assigns a score regarding whether the target is consistent with the query. For example, the command is composed as follows: assign 1 point when the target and the query seem to have very little or no relevance; assign 2 points when the target and the query seem to have a certain degree of relevance but have significant differences; assign 3 points when the target and the query seem to have a general degree of relevance to each other; assign 4 points when the target and the query seem to have a relatively high degree of relevance to each other; assign 5 points when the target and the query seem to have a very high degree of relevance or seem to be almost the same. More specifically, the third prompt word can include a command such as Command 610: "User instruction: You are a similarity evaluator! It is your task to calculate the similarity based on the relevance between the query (QUERY) and the document (DOCUMENT) shown below. During the evaluation process, I hope to evaluate the relevance of the corresponding pairs according to the following score criteria: Score 1: The document (DOCUMENT) and the query (QUERY) have very little or no relevance to each other. There are almost no common attributes or contexts in the comparison elements. Score 2: The document (DOCUMENT) and the query (QUERY) have a certain degree of relevance, but have significant differences. Share a few detailed matters of attributes or contexts, but there are quite large differences in most aspects. Score 3: The document (DOCUMENT) and the query (QUERY) have a medium degree of relevance to each other. Share some attributes or contexts, but there are still differences that need attention that prevent a high similarity score. Score 4: The document (DOCUMENT) and the query (QUERY) have a high degree of relevance. Share more detailed matters of attributes or contexts, and there are only a few differences that do not have a major impact. Score 5: The document (DOCUMENT) and the query (QUERY) have a very high degree of relevance or are almost the same. Share almost all attributes or contexts that are almost the same, and there are only very weak or negligible differences. You will be given the query (QUERY) and document (DOCUMENT) pair."

[0084] In addition, in order to output an explanation for this together with the score, the third prompt word can include a command such as Command 620: "Please generate according to the following form: <explanation>{Explanation of the score}< / explanation><score>{Score}< / score>".

[0085] In addition, as Figure 7As shown, the third prompt can be composed of commands that assign scores based on whether the target matches the instruction (virtual user scenario). For example, commands can be constructed as follows: 1 point for insufficient relevance and no connection between the target and the virtual user scenario; 2 points for limited relevance; 3 points for moderate relevance; 4 points for high relevance; and 5 points for perfect relevance and accuracy in all aspects. More specifically, the third prompt may include a prompt such as Command 710: "User Instruction: You are the scoring agent! This document ID exists in each potential document. You are responsible for scoring the answer based on the relevance of the query / scenario pair. During the evaluation process, I expect you to assess the relevance of the corresponding pair according to the following scoring criteria: Score 1: The document is almost or completely irrelevant to the user's scenario, and almost irrelevant to the user's job, background, situation, location, occupation, hobbies, interests, or purpose. Preferences and context are not considered, therefore it is not suitable overall. Score 2: The document has limited relevance; some elements are consistent with the user's scenario and query. While there is some understanding of the context and consideration of preferences, it does not provide an overall appropriate response. Score 3: The document shows a moderate degree of relevance. It captures the user's scenario well." Some aspects. Shows appropriate contextual adaptability, taking into account the user's explicit preferences. However, there is room for improvement in depth and clarity. Score 4: The document shows high relevance. It is very consistent with the user's scenario and covers most of the relevant aspects. Shows high contextual adaptability and effectively takes into account the user's preferences. However, there may be minor room for improvement. Score 5: The document is completely relevant and accurately covers all aspects of the user's scenario, query, and preferences. It blends perfectly with the user's context, demonstrating a deep understanding. The document is unexpectedly clear and concise, providing information systematically and offering perfectly fitting results. In the given search documents (ID: 1~10), only the highest-scoring ID should be generated, and there should be no additional suggestions such as [ID]. This is very important! The command will provide a document directory and query / scenario pairs.

[0086] Therefore, the present invention improves the matching between data, thereby enabling a more accurate assessment of whether the retrieval machine outputs retrieval results that reflect user information.

[0087] As another embodiment, the feature data and the second prompt word are input into the AI ​​model. When fine-tuning to match the target included in the feature data with the virtual user scenario included in the aforementioned multiple instructions, the method further includes a step of fine-tuning to match the target included in the feature data with the query. For example, this may include... Figure 5 Command 520, such as "But the revised document should be relevant to the provided query. That is, it should be applicable to general queries," allows the second prompt word's command to be fine-tuned to match the query for the target.

[0088] Therefore, the present invention improves the matching between target information included in the dataset and the corresponding query, thereby enabling a more accurate assessment of whether the retrieval machine outputs retrieval results that reflect user information.

[0089] On the other hand, the method for constructing a dataset for evaluating the user information customization function of retrievers, as described in embodiments of the present invention, can be derived by referring to... Figure 1 The system described is implemented.

[0090] The AI ​​model in embodiments of the present invention can be controlled, operated, learned, and driven by a processor, thereby enabling the AI ​​model to perform at least one task among operation, learning, and driving via at least one processor. Furthermore, the AI ​​model can be stored in a memory, and the feature data of the present invention can also be stored in a memory.

[0091] On the other hand, the disclosed embodiments can be implemented in the form of a recording medium for storing computer-executable commands. Commands can be stored in the form of program code, and when executed by a processor, a program module is generated to perform the operations of the disclosed embodiments. The recording medium can be implemented as a computer-readable recording medium.

[0092] Computer-readable recording media include all kinds of recording media that store computer-readable commands. For example, they can be ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, optical data storage devices, etc.

[0093] The disclosed embodiments have been described above with reference to the accompanying drawings. Those skilled in the art can implement this disclosure in different forms than the disclosed embodiments without altering the technical concept or essential features. The disclosed embodiments are merely illustrative, and the present invention is not limited thereto.

Claims

1. A system for constructing a dataset for retrievalrs using a language model, comprising: At least one processor; at least one server; and at least one memory, wherein the at least one memory stores commands or information that cause the at least one processor to perform operations. The operations performed by the above command include the following steps: Extract multiple queries and targets corresponding to the multiple queries from the sample data stored in the aforementioned memory. The first prompt stored in the aforementioned memory is input into the AI ​​model stored on the server, and an instruction set consisting of multiple instructions including virtual user scenarios is output. The corresponding queries and targets are further mapped to the instruction set and output as feature data. The aforementioned element data, along with the second prompt word, is input into the aforementioned AI model, and fine-tuned to match the aforementioned target included in the aforementioned element data with the aforementioned virtual user scenario included in the aforementioned multiple instructions; and The fine-tuned data of the aforementioned elements are stored as a dataset in the aforementioned memory.

2. The system according to claim 1, wherein, The first prompt word mentioned above is a command input into the AI ​​model that outputs a virtual user scenario, including various information related to the user, as an article.

3. The system according to claim 1, wherein, The aforementioned virtual user scenarios include information related to the virtual user's background, location, occupation, hobbies, interests, search purpose, or preferred source.

4. The system according to claim 1, further comprising the following steps: The aforementioned dataset, along with the third prompt words stored in the aforementioned memory, is input into the AI ​​model, and the aforementioned feature data that receives a score lower than the predetermined score is removed from the aforementioned dataset. The third prompt word mentioned above includes commands configured to assign scores based on whether the target matches the query and whether the target matches the multiple instructions, using the AI ​​model mentioned above.

5. The system according to claim 1, wherein, The steps for fine-tuning to match the aforementioned virtual user scenarios also include: The aforementioned element data, along with the aforementioned second prompt, are input into the aforementioned AI model, and the aforementioned target included in the aforementioned element data is fine-tuned to match the query.

6. A method for constructing a dataset for retrievalrs using a language model, comprising the following steps: Extract multiple queries and their corresponding targets from the sample data. The first prompt word is input into the AI ​​model, and the output consists of a set of instructions, including multiple instructions based on a virtual user scenario. The corresponding queries and targets are further mapped to the instruction set and output as feature data. The aforementioned element data, along with the second prompt word, is input into the aforementioned AI model, and the aforementioned target included in the aforementioned element data is fine-tuned to match the aforementioned virtual user scenario included in the aforementioned multiple instructions; The fine-tuned data of the above-mentioned elements are stored as a dataset.

7. The method according to claim 6, wherein, The first prompt word mentioned above is a command input into the AI ​​model that outputs a virtual user scenario, including various information related to the user, as an article.

8. The method according to claim 6, wherein, The aforementioned virtual user scenarios include information related to the virtual user's background, location, occupation, hobbies, interests, search purpose, or preferred source.

9. The method according to claim 6, wherein, Also includes: The aforementioned dataset, along with the third prompt word, was input into the AI ​​model, and the data of the aforementioned elements that received scores lower than the predetermined scores were removed from the aforementioned dataset. The third prompt word mentioned above includes commands configured to assign scores based on whether the target matches the query and whether the target matches the multiple instructions, using the AI ​​model mentioned above.

10. The method according to claim 6, wherein, The steps for fine-tuning to match the aforementioned virtual user scenarios also include: The aforementioned element data, along with the aforementioned second prompt, are input into the aforementioned AI model, and the aforementioned target included in the aforementioned element data is fine-tuned to match the query.

11. A program stored in a computer-readable recording medium for use with a computer to perform the method according to any one of claims 6 to 10.

Images