Method for constructing training data set for training an information retrieval model, and computer program and electronic device supporting the same

By classifying candidate documents and masking hard negatives, the method enhances the learning performance and stability of information retrieval models, addressing inefficiencies in differentiating between positive and negative documents.

KR103047074B1Active Publication Date: 2026-11-11SIONIC AI INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
KR1020260008134
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2026-01-15
Publication Date
2026-11-11
Estimated Expiration
2046-01-15

AI Technical Summary

Technical Problem

Existing information retrieval models face inefficiencies in differentiating between positive and negative documents due to the limited use of semantically similar negative documents in training, leading to label noise and reduced learning efficiency.

Method used

A method to construct a training dataset by identifying and classifying candidate documents as genuine or false negative documents, incorporating them into the training process, and masking hard negative documents during loss function calculation, while adjusting learning parameters based on epochs.

Benefits of technology

Improves the learning performance and stability of information retrieval models by minimizing label contamination and mitigating loss oscillations, enhancing the model's ability to distinguish between similar documents.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112026006025476-PAT00008_ABST
    Figure 112026006025476-PAT00008_ABST
Patent Text Reader

Abstract

A method for constructing a training dataset for training an information retrieval model according to one embodiment of the present disclosure may include: a step of calculating a first similarity between a query and a positive document for each of a plurality of query-positive document pairs; a step of identifying at least one candidate document for each of a plurality of queries from a corpus based on the first similarity; a step of calculating a correlation with at least one candidate document for each of a plurality of queries; a step of classifying at least one candidate document as a genuine negative document or a false negative document based on the correlation; a step of constructing a training dataset based on the classification result; and a step of training an information retrieval model using the training dataset.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] Various embodiments of the present disclosure relate to a method for constructing a training dataset for training an information retrieval model, and a computer program and electronic device supporting the same. Background Technology

[0002] Corresponding to the advancement of digital convergence, in which various information and communication technologies are integrated, electronic devices provide various functions and / or services that are integrated into their pivotal functions. For example, an electronic device can support information retrieval services and / or search result-based application services by utilizing an information retrieval model to search for documents related to a user query among multiple documents included in a corpus and providing the search results. Such an information retrieval model can be trained using pairs of a query and a corresponding positive document (e.g., a correct answer document) so that the query and the positive document have high semantic similarity in the embedding space. Additionally, negative documents (e.g., incorrect answer documents) contrasting with the positive documents may be utilized in the training process of the information retrieval model. For example, a so-called in-batch negatives method may be used, which calculates a loss value by considering documents included in pairs other than the corresponding query-positive document pair within the batch as negative documents for the query.

[0003] However, due to the nature of the in-batch negative method, which utilizes negative documents within a batch in a limited manner, negative documents that are semantically similar to queries and thus prone to confusion for the information retrieval model in actual service environments may not be sufficiently reflected in the training. In other words, training using the in-batch negative method may be limited to negative documents within a batch that are relatively easy for the information retrieval model to distinguish; consequently, the efficiency of enhancing the model's ability to precisely differentiate documents similar to positive documents within the corpus may be reduced. Accordingly, a solution that incorporates negative documents prone to confusion into the information retrieval model's training dataset may be considered; however, if these negative documents are actually false negatives equivalent to positives for the query, label noise may occur in the training dataset, thereby hindering the learning efficiency of the information retrieval model.

[0004] The foregoing is provided as background technology for the purpose of aiding understanding of the present disclosure, and no claim or determination is made as to whether it can be applied as prior art related to the present disclosure. The problem to be solved

[0005] The present disclosure provides a method for constructing a training dataset for training an information retrieval model to solve the above-mentioned problem.

[0006] The present disclosure provides a computer program that supports a method for constructing a learning dataset for learning an information retrieval model to solve the above-mentioned problem.

[0007] The present disclosure provides an electronic device that supports a method for configuring a learning dataset for learning an information retrieval model to solve the above-mentioned problem.

[0008] The technical problems that this disclosure aims to solve are not limited to those described above, and other unmentioned problems will be clearly understood by those skilled in the art from the various embodiments described below. means of solving the problem

[0009] The present disclosure may be implemented in various ways, including methods, electronic devices, and / or computer programs stored on computer-readable recording media.

[0010] A method for constructing a training dataset for training an information retrieval model executed by at least one processor according to one embodiment of the present disclosure may include, for each of a plurality of query-positive document pairs, a step of calculating a first similarity between a query and a positive document; a step of identifying at least one candidate document for each of the plurality of queries from a corpus based on the first similarity; a step of calculating a correlation with at least one candidate document for each of the plurality of queries; a step of classifying at least one candidate document as a genuine negative document or a false negative document based on the correlation; a step of constructing a training dataset based on the classification result; and a step of training an information retrieval model using the training dataset.

[0011] According to one embodiment of the present disclosure, the step of identifying at least one candidate document may include: calculating a second similarity between a specific query and a document included in a corpus; comparing a first similarity and a second similarity calculated for a specific query; and identifying a corpus document as a candidate document for a specific query if the value obtained by subtracting the first similarity from the second similarity is greater than or equal to a first threshold.

[0012] According to one embodiment of the present disclosure, the step of classifying at least one candidate document as a genuine negative document or a false negative document may include comparing the relevance between a specific query and a candidate document identified for the specific query with a second threshold, classifying the candidate document as a genuine negative document if the relevance is less than the second threshold, and classifying the candidate document as a false negative document if the relevance is greater than or equal to the second threshold.

[0013] According to one embodiment of the present disclosure, the step of constructing a training dataset may include the step of updating a set of positive documents of a specific query including a first positive document corresponding to a specific query, and the step of generating a set of hard negative documents of a specific query.

[0014] According to one embodiment of the present disclosure, the step of updating a positive document set may include adding a candidate document classified as a false negative document for a specific query to the positive document set as a second positive document for a specific query.

[0015] According to one embodiment of the present disclosure, the step of generating a set of hard negative documents may include determining a candidate document classified as a genuine negative document for a specific query as a hard negative document and generating a set of hard negative documents including the hard negative document.

[0016] According to one embodiment of the present disclosure, the step of training an information retrieval model may include, at each step of the training loop, a step of masking hard negative documents selected from a set of hard negative documents to be used in calculating a loss function and excluding them from the calculation of the loss function.

[0017] According to one embodiment of the present disclosure, the step of excluding from the calculation of the loss function may include: forming a batch from a training dataset; for each of a plurality of queries included in the batch, calculating a third similarity with a positive document corresponding to each of the plurality of queries included in the batch; calculating a fourth similarity between a specific query included in the batch and a selected hard negative document; comparing the third similarity and the fourth similarity calculated for the specific query; and masking the hard negative document when the value obtained by subtracting the fourth similarity from the third similarity is greater than or equal to a third threshold.

[0018] According to one embodiment of the present disclosure, the step of training an information retrieval model may include determining a learning control parameter value based on a pre-set learning schedule according to the learning epoch of the information retrieval model, and applying the learning control parameter value to at least one of batch configuration or loss function calculation.

[0019] According to one embodiment of the present disclosure, a computer program stored on a computer-readable recording medium may be provided for executing a method for constructing a learning dataset for learning an information retrieval model on a computer.

[0020] An electronic device according to one embodiment of the present disclosure includes a memory for storing instructions and at least one processor, and when the instructions are executed by at least one processor, the electronic device may calculate a first similarity between a query and a positive document for each of a plurality of query-positive document pairs, identify at least one candidate document for each of the plurality of queries from a corpus based on the first similarity, calculate a correlation with at least one candidate document for each of the plurality of queries, classify at least one candidate document as a genuine negative document or a false negative document based on the correlation, construct a training dataset based on the classification result, and train an information retrieval model using the training dataset. Effects of the invention

[0021] According to various embodiments of the present disclosure, a mechanism may be provided to improve the learning performance and stability of an information retrieval model by systematically reflecting candidate documents that the information retrieval model may confuse with positive documents for a query into a training dataset and training the information retrieval model using said training dataset.

[0022] According to various embodiments of the present disclosure, a candidate document is classified as a true negative document or a false negative document based on the correlation between a query and a candidate document, and by reflecting the false negative document as a positive document when constructing a training dataset, label contamination in which a document equivalent to a positive document is included as a negative document in the training dataset can be minimized.

[0023] According to various embodiments of the present disclosure, while training an information retrieval model using a training dataset, at least some of the negative documents included in the batch are excluded from the calculation of the loss function according to a margin criterion, thereby mitigating loss oscillations caused by excessive negative training signals and improving the convergence stability of the information retrieval model.

[0024] According to various embodiments of the present disclosure, by applying a schedule to learning control parameters according to a learning epoch to control batch configuration and / or loss function calculation, the learning performance of an information retrieval model can be improved through the gradual adjustment of learning difficulty.

[0025] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims. Brief explanation of the drawing

[0026] Various embodiments of the present disclosure will be described with reference to the drawings below, and identical or similar reference numbers may be assigned to identical or corresponding components in connection with the description of the drawings. FIG. 1 is a drawing illustrating an exemplary electronic device in a network environment according to one embodiment of the present disclosure. FIG. 2 is a drawing illustrating, by way of example, a memory of an electronic device according to one embodiment of the present disclosure. FIG. 3 is a diagram illustrating an exemplary processing pipeline for identifying candidate documents for a query according to one embodiment of the present disclosure. FIG. 4 is a diagram illustrating an exemplary processing pipeline for classifying candidate documents for a query according to one embodiment of the present disclosure. FIG. 5 is a diagram illustrating, in an exemplary manner, a processing pipeline that constitutes a training dataset of an information retrieval model according to one embodiment of the present disclosure. FIG. 6 is a diagram illustrating an exemplary processing pipeline for training an information retrieval model according to one embodiment of the present disclosure. FIG. 7 is a diagram illustrating, in an exemplary manner, a processing pipeline for scheduling learning control parameters related to the learning of an information retrieval model according to one embodiment of the present disclosure. FIG. 8 is a diagram illustrating, in an exemplary manner, a method for configuring a learning dataset for learning an information retrieval model according to one embodiment of the present disclosure. Specific details for implementing the invention

[0027] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions regarding widely known functions or configurations will be omitted if there is a risk that the gist of the present disclosure may be unnecessarily obscured.

[0028] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Additionally, in the description of the following embodiments, the description of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0029] The advantages and features of the disclosed embodiments and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various different forms, and the embodiments provided are merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the invention.

[0030] The terms used in this disclosure will be briefly explained, and the disclosed embodiments will be described in detail. The terms used in this disclosure have been selected to be as generally used as possible, taking into account their functions within the disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms may be selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0031] In this disclosure, singular expressions include plural expressions unless the context clearly specifies them as singular. Additionally, plural expressions include singular expressions unless the context clearly specifies them as plural. Throughout this disclosure, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0032] As used in this disclosure, the terms “module” or “part” refer to software or hardware components, and the “module” or “part” performs certain roles. However, the meaning of “module” or “part” is not limited to software or hardware. The “module” or “part” may be configured to reside in an addressable storage medium or configured to run on one or more processors. Accordingly, as an example, the “module” or “part” may include components such as software components, object-oriented software components, class components, and task components, and at least one of processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The components and the functions provided within the “module” or “part” may be combined into a smaller number of components and “modules” or “parts,” or further separated into additional components and “modules” or “parts.”

[0033] According to one embodiment, a 'module' or 'part' may be implemented as a processor and memory. The term 'processor' should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc.

[0034] In some contexts, 'processor' may refer to an Application-Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), etc. 'Processor' may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other combination of such configurations.

[0035] Furthermore, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' may also refer to various types of processor-readable media, such as Random Access Memory (RAM), Read-Only Memory (ROM), Non-Volatile Random Access Memory (NVRAM), Programmable Read-Only Memory (PROM), Erasable-Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), Flash Memory, Magnetic or Marked Data Storage Devices, Registers, etc. If a processor can read information from memory and / or write information to memory, the memory is said to be in an electronic communication state with the processor. Memory integrated into a processor is in an electronic communication state with the processor.

[0036] The terms 1, 2, A, B, (a), (b), etc. used in this disclosure are used merely to distinguish one component from another, and the essence, order, or sequence of said component is not limited by such terms.

[0037] Where in this disclosure it is stated that one component is 'connected,' 'coupled,' or 'connected' to another component, it should be understood that the component may be directly connected or connected to the other component, but that another component may also be 'connected,' 'coupled,' or 'connected' between each component.

[0038] As used in this disclosure, “comprising” and / or “comprising” does not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0039] FIG. 1 is a drawing illustrating an exemplary electronic device in a network environment according to one embodiment of the present disclosure.

[0040] Referring to FIG. 1, an electronic device (100) according to one embodiment may include various types of devices capable of storing, providing, and executing computer-executable programs and data related to a data processing service (for example, a service in which an information retrieval model classifies candidate documents that may be confused with positive documents corresponding to a query into genuine negative documents or false negative documents based on their relevance to the query, constructs a training dataset reflecting the classification results, and trains an information retrieval model using the training dataset). For example, the electronic device (100) may include at least one of a system device, a server device, a cloud service-based distributed computing device, or a mobile communication device.

[0041] In one embodiment, the electronic device (100) can communicate with at least one external electronic device (300) through a network (200). According to various embodiments, the network (200) may be configured, depending on the installation environment, as a wired network including at least one of Ethernet, a wired home network (power line communication), a telephone line communication device, or RS-serial communication, a mobile communication network, a wireless network including at least one of a WLAN (wireless LAN), Wi-Fi, Bluetooth, or ZigBee, or a combination of a wired network and a wireless network.

[0042] In the present disclosure, the communication method between the electronic device (100) and at least one external electronic device (300) is not limited and may include a communication method utilizing a communication network (e.g., mobile communication network, wired internet, wireless internet, broadcasting network and / or satellite network) that the network (200) can support, as well as a short-range wireless communication method between the electronic device (100) and at least one external electronic device (300).

[0043] In one embodiment, the electronic device (100) may include components related to the execution and / or provision of data processing services. For example, the electronic device (100) may include a memory (110), a communication module (120), and at least one processor (130). According to various embodiments, at least some of the components included in the electronic device (100) may be connected to each other via a bus, GPIO (general purpose input and output), SPI (serial peripheral interface) and / or MIPI (mobile industry processor interface) to exchange signals, data and / or information.

[0044] In one embodiment, the memory (110) may include a non-transient recording medium readable by any computer. According to one embodiment, the memory (110) may include a non-perishable permanent mass storage device comprising at least one of a read-only memory (ROM), a disk drive, a solid state drive (SSD), or a flash memory. Alternatively, such a non-perishable permanent mass storage device may be included in the electronic device (100) as a separate permanent storage device distinct from the memory (110).

[0045] In one embodiment, the memory (110) may store at least one of an operating system, at least one application program, or at least one program code. According to various embodiments, these software components may be loaded from a computer-readable recording medium distinct from the memory (110). In various embodiments, the recording medium may include at least one of a floppy drive, a disk, a tape DVD / CD-ROM drive, or a memory card that can be directly connected to the electronic device (100). Alternatively, the software components may be loaded into the memory (110) via a communication module (120). For example, at least one application program may be installed and loaded into the memory (110) based on files provided by a file distribution system over a network (200).

[0046] In one embodiment, the memory (110) may store instructions related to the functional operation of the components of the electronic device (100). For example, the memory (110) may store instructions that cause the components of the electronic device (100) to perform a defined functional operation when executed by at least one processor (130).

[0047] In one embodiment, the communication module (120) may provide a configuration or function for the electronic device (100) to communicate with at least one external electronic device (300) through a network (200). For example, the communication module (120) may establish communication (or a communication channel) with at least one external electronic device (300) according to a defined communication protocol and perform the transmission and reception of signals, data and / or information through said communication.

[0048] According to one embodiment, the communication module (120) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module and / or a GNSS (global navigation satellite system) communication module) and a wired communication module (e.g., a LAN (local area network) communication module and / or a power line communication module). In various embodiments, the wireless communication module and the wired communication module may be composed of separate chips or integrated into a single chip.

[0049] In one embodiment, at least one processor (130) may be configured to execute instructions of a computer program provided by memory (110) and / or a communication module (120) by performing arithmetic, logic, and input / output operations. For example, at least one processor (130) may be configured to execute instructions received according to program code stored in memory (110) and / or instructions received from at least one external electronic device (300) through the communication module (120).

[0050] According to one embodiment, at least one processor (130) can execute a program to provide data processing services by processing instructions of a computer program. In one embodiment, at least one processor (130) can receive and process signals, data, and / or information transmitted from at least one external electronic device (300) through a communication module (120) while executing the program. Additionally, at least one processor (130) can store signals, data, and / or information generated while executing the program in a memory (110) or transmit them to at least one external electronic device (300) through the communication module (120).

[0051] The functional operation of the electronic device (100) according to the various embodiments described below can be implemented by at least one processor (130) executing (or processing) instructions of a computer program stored in (or loaded into) memory (110).

[0052] FIG. 2 is a drawing illustrating, by way of example, a memory of an electronic device according to one embodiment of the present disclosure.

[0053] Referring to FIG. 2, the memory (110) of an electronic device according to one embodiment (e.g., the electronic device (100) of FIG. 1) may include a program (111) and a database (113) that support a data processing service (e.g., a service that identifies candidate documents that a dense embedding-based information retrieval model may confuse with a positive document corresponding to a query, and trains an information retrieval model based on a training dataset that reflects the candidate documents).

[0054] In one embodiment, the program (111) may include an information retrieval model (111a) and a cross-encoder model (111b), at least one of these models may include a transformer-based artificial neural network model implemented by software instructions, firmware and / or logic. According to one embodiment, at least one processor of the electronic device (100) (e.g., processor (130) of FIG. 1) may be configured to execute the functions of the information retrieval model (111a) and the cross-encoder model (111b) respectively by executing the program (111) and processing instructions and / or parameters.

[0055] In one embodiment, the information retrieval model (111a) may be configured to receive a query and a document as input, calculate a similarity for a query-document pair, and output the result of the similarity calculation. For example, the information retrieval model (111a) may include an encoder (or dual encoder) that embeds each of the query and the document and converts them into a vector representation, and may be configured to calculate a similarity between the query vector and the document vector (e.g., a dot product, cosine similarity, and / or an equivalent score). In this regard, the information retrieval model (111a) may have a pre-trained state to calculate (or output) a similarity for a query-document pair. For example, the information retrieval model (111a) may be pre-trained based on an in-batch negatives method in which, for a query of a first pair among a plurality of query-positive document pairs included in a batch, the positive document of the first pair is considered the correct answer, and at least some of the positive documents of the other pair are considered negative documents. Additionally, the information retrieval model (111a) may be pre-trained by updating parameters according to an InfoNCE-based loss function so that the similarity score between the query of the first pair and the positive document is increased, and the similarity score between the query and the negative document is relatively lowered.

[0056] In one embodiment, the cross encoder model (111b) may be configured to calculate semantic relevance for query-document pairs and output the result of the relevance calculation. For example, the cross encoder model (111b) may be configured to receive a query-document pair in a combined form (e.g., token sequence combination) and to calculate the semantic relevance of the query-document pair (e.g., logit, probability value and / or equivalent score) by performing cross-attention-based encoding on the pair.

[0057] In one embodiment, the database (113) may store a training data pool. For example, the database (113) may store a training data pool comprising a plurality of query-positive document pairs, each of which may include a query and a positive document corresponding to the query (or confirmed to be relevant to the query). Additionally, the database (113) may store high-dimensional vector representations (or embedding vectors) for the queries and / or positive documents included in each of the plurality of query-positive document pairs in a cache form to support the calculation of similarity of the information retrieval model (111a) for at least some of the plurality of query-positive document pairs included in the training data pool. According to various embodiments, at least some of the vector representations stored in the database (113) may be updated periodically or asynchronously in response to parameter updates of the information retrieval model (111a).

[0058] In various embodiments, the database (113) may store a corpus containing multiple documents. Alternatively, the database (113) may store identification information (e.g., document ID), metadata, vector representation, and / or vector index for each of the multiple documents included in the corpus, and in this case, the electronic device (100) may obtain (or query) the corresponding document from an external electronic device (e.g., the external electronic device (300) of FIG. 1) based on the identification information and / or metadata of the document stored in the database (113).

[0059] FIG. 3 is a diagram illustrating an exemplary processing pipeline for identifying candidate documents for a query according to one embodiment of the present disclosure.

[0060] Referring to FIG. 3, an electronic device according to one embodiment (e.g., the electronic device (100) of FIG. 1) can identify candidate documents for each of a plurality of queries included in a batch by using an information retrieval model (e.g., the information retrieval model (111a) of FIG. 2). For example, the electronic device (100) can identify candidate documents from a corpus that are difficult to distinguish from positive documents corresponding to the query using the information retrieval model (111a).

[0061] In this regard, the electronic device (100) can obtain (410) different multiple query-positive document pairs from a training data pool stored in a database (e.g., the database (113) of FIG. 2). For example, the electronic device (100) can obtain multiple query-positive document pairs from the training data pool based on random sampling or a sampling rate set by domain, based on a preset batch size and / or batch configuration rules.

[0062] In one embodiment, the electronic device (100) may configure a first batch using a plurality of acquired query-positive document pairs. For example, the electronic device (100) may configure a first batch comprising a plurality of query-positive document pairs, such as a first pair of a first query and a corresponding first positive document, and a second pair of a second query and a corresponding second positive document.

[0063] According to one embodiment, the electronic device (100) can calculate (430) similarity for each of the plurality of query-positive document pairs included in the first batch. For example, the electronic device (100) can input the query and positive document included in each of the plurality of query-positive document pairs into an information retrieval model (111a) to obtain a similarity for each query-positive document pair (e.g., inner product, cosine similarity and / or equivalent score) calculated from the information retrieval model (111a). In this regard, the information retrieval model (111a) may be configured to calculate similarity between a query vector and a positive document vector by embedding each of the input query and positive document. Alternatively, the information retrieval model (111a) may be configured to calculate similarity between a query vector and a positive document vector by obtaining an embedding vector for the query and / or positive document from a database (113). According to one embodiment, the electronic device (100) may store the similarity of each of a plurality of query-positive document pairs obtained from an information retrieval model (111a) in a memory (e.g., memory (110) of FIG. 1 and / or FIG. 2) or a database (113) at least temporarily.

[0064] In one embodiment, the electronic device (100) can identify (450) candidate documents from a corpus stored in a database (113) or an external electronic device (e.g., the external electronic device (300) of FIG. 1) for each of the plurality of queries included in the first batch (e.g., queries included in each of the plurality of query-positive document pairs). For example, for each of the plurality of queries, the electronic device (100) can identify at least one candidate document that is highly likely to be confused with the positive document corresponding to the query by the information retrieval model (111a). In this regard, the electronic device (100) can calculate the similarity between each of the plurality of queries and the plurality of documents included in the corpus using the information retrieval model (111a). In various embodiments, the electronic device (100) may determine all of the plurality of documents included in the corpus as targets for calculating similarity with each of the plurality of queries. Alternatively, the electronic device (100) may select some of the plurality of documents included in the corpus as targets for calculating similarity with each of the plurality of queries. For example, for each of a plurality of queries, the electronic device (100) may identify a plurality of parent documents (or a plurality of parent document vectors) that are estimated to have high similarity to the corresponding query (or query vector) based on a vector index stored in the database (113), and determine the plurality of parent documents as targets for calculating similarity with each of the plurality of queries.

[0065] In one embodiment, the electronic device (100) can determine whether a document in a corpus is a candidate document by comparing, for each of a plurality of queries included in a first batch, a first similarity of a query-positive document pair corresponding to the query with a second similarity between each of the query and a plurality of documents included in the corpus. For example, if the second similarity is greater than the first similarity, the electronic device (100) can identify a corpus document associated with the second similarity as a candidate document for the query. Alternatively, the electronic device (100) can identify a corpus document associated with the second similarity as a candidate document for the query if the value obtained by subtracting the first similarity from the second similarity is greater than or equal to a preset first threshold (or margin).

[0066] In one embodiment, the electronic device (100) may manage a corpus document identified as a candidate document for each of a plurality of queries among a plurality of documents of a corpus stored in a database (113) by associating it with the corresponding query. For example, the electronic device (100) may map identification information (e.g., document ID) of a corpus document identified as a candidate document and identification information (e.g., query ID) of a query associated therewith to the database (113) and store them in the database (113). Alternatively, the electronic device (100) may obtain original data of a corpus document identified as a candidate document for each of a plurality of queries from an external electronic device (300), store it in the database (113), and map identification information of the corpus document and identification information of the query associated therewith to the database (113).

[0067] FIG. 4 is a diagram illustrating an exemplary processing pipeline for classifying candidate documents for a query according to one embodiment of the present disclosure.

[0068] Referring to FIG. 4, an electronic device according to one embodiment (e.g., the electronic device (100) of FIG. 1) can classify at least one candidate document identified for each of a plurality of queries included in a first batch by using a cross-encoder model (e.g., the cross-encoder model (111b) of FIG. 2). For example, the electronic device (100) can classify the candidate document as a true negative document, meaning a document that is actually less relevant to the query, based on the degree of relevance between the query and the candidate document identified for the query, or as a false negative document, meaning a document that was identified as a candidate document for the query but is actually highly relevant to the query (e.g., having relevance equivalent to a positive document corresponding to the query).

[0069] In this regard, the electronic device (100) can calculate (510) the degree of correlation between a query and at least one candidate document identified for each of the plurality of queries included in the first batch. For example, the electronic device (100) may form a pair of a query and a corresponding candidate document for each of the plurality of queries, and input the query-candidate document pair into a cross-encoder model (111b) to obtain the degree of correlation between the query-candidate document pairs calculated from the cross-encoder model (111b). In this regard, the cross-encoder model (111b) may be configured to perform cross-attention-based encoding on the input query-candidate document pairs to calculate the degree of semantic correlation of the pairs (e.g., logits, probability values ​​and / or equivalent scores).

[0070] In one embodiment, the electronic device (100) may classify (530) a candidate document included in a query-candidate document pair as a genuine negative document or a false negative document based on the relevance of the query-candidate document pair. In this regard, the electronic device (100) may classify a candidate document by comparing the relevance of the query-candidate document pair with a preset second threshold. For example, if the relevance of the query-candidate document pair is less than the second threshold, the electronic device (100) may classify a candidate document included in the query-candidate document pair as a genuine negative document for the query. Alternatively, if the relevance of the query-candidate document pair is greater than or equal to the second threshold, the electronic device (100) may classify a candidate document included in the query-candidate document pair as a false negative document for the query.

[0071] In one embodiment, the electronic device (100) may record classification results for at least one candidate document for each of a plurality of queries in a database (113). For example, the electronic device (100) may store relevance values ​​(or scores) and / or classification information (e.g., true negative or false negative) for the candidate documents of each query by mapping them to identification information (e.g., query ID) of the corresponding query.

[0072] FIG. 5 is a diagram illustrating, in an exemplary manner, a processing pipeline that constitutes a training dataset of an information retrieval model according to one embodiment of the present disclosure.

[0073] Referring to FIG. 5, an electronic device according to one embodiment (e.g., the electronic device (100) of FIG. 1) can construct a training dataset of an information retrieval model (e.g., the information retrieval model (111a) of FIG. 2) based on the classification results of candidate documents for each of a plurality of queries included in a first batch.

[0074] In this regard, for each of the plurality of queries included in the first batch, the electronic device (100) can update the set of positive documents for the corresponding query and generate a set of hard negative documents (610) that includes negative documents that are easily confused by the information retrieval model (111a) because they have a high similarity to the query. For example, for each of the plurality of queries, the electronic device (100) can update the set of positive documents for the query by adding at least one candidate document classified as a false negative document for the query as a positive document to the set of positive documents that includes the positive document for the corresponding query. Additionally, for each of the plurality of queries, the electronic device (100) can determine at least one candidate document classified as a true negative document for the corresponding query as a hard negative document and generate a set of hard negative documents that includes the hard negative document.

[0075] In various embodiments, the electronic device (100) may define a data structure for a list of documents corresponding to each of a plurality of queries in the database (113). For example, the electronic device (100) may manage a set of positive documents in the database (113) with a key-value structure in which the identification information of a query (e.g., query ID) is used as the key and the identification information of a document that has a positive document relationship with the query (e.g., document ID) is used as the value. Additionally, the electronic device (100) may append to the database (113) a set of hard negative documents with a key-value structure in which the identification information of a query is used as the key and the identification information of a document that has a hard negative document relationship with the query is used as the value.

[0076] In one embodiment, the electronic device (100) may construct a training dataset (630) based on an updated set of positive documents and a generated set of hard negative documents for each of a plurality of queries. For example, the electronic device (100) may construct a training dataset of an information retrieval model (111a) by, for each of a plurality of queries, pairing each of the query and at least one positive document included in the set of positive documents to create a query-positive document pair, and pairing each of the query and at least one hard negative document included in the set of hard negative documents to create a query-hard negative document pair.

[0077] In one embodiment, the electronic device (100) may store a set of positive documents and a set of hard negative documents for each of a plurality of queries included in a training dataset in a database (113). For example, the electronic device (100) may store identification information (e.g., set ID) of a set of positive documents corresponding to a query and identification information of a set of hard negative documents by mapping them to identification information (e.g., query ID) of the corresponding query.

[0078] FIG. 6 is a diagram illustrating an exemplary processing pipeline for training an information retrieval model according to one embodiment of the present disclosure.

[0079] Referring to FIG. 6, an electronic device according to one embodiment (e.g., the electronic device (100) of FIG. 1) can train an information retrieval model (e.g., the information retrieval model (111a) of FIG. 2) based on a configured training dataset, and in the training process, at least some of the hard negative documents included in the loss function calculation are masked according to a preset criterion (e.g., excluded from the loss function calculation) to update the parameters of the information retrieval model (111a).

[0080] In this regard, the electronic device (100) may form a second batch from a training dataset based on an updated set of positive documents and a generated set of hard negative documents for each of the multiple queries, and train an information retrieval model (111a) (710) using the second batch. According to one embodiment, the electronic device (100) may obtain different multiple queries from the training dataset in relation to the formation of the second batch. For example, the electronic device (100) may obtain multiple queries from the training dataset based on random sampling or a sampling rate set by domain, based on a preset batch size and / or batch configuration rules. Additionally, for each of the multiple queries, the electronic device (100) may form multiple query-positive document pairs and / or multiple query-hard negative document pairs to be used for training by sampling at least one positive document from the set of positive documents and at least one hard negative document from the set of hard negative documents.

[0081] In one embodiment, the electronic device (100) may configure a second batch comprising a plurality of query-positive document pairs. In this case, the electronic device (100) may train an information retrieval model (111a) based on an in-batch negative method, which, for each query included in the second batch, uses a positive document corresponding to the query as the correct answer document and considers at least some of the positive documents corresponding to other queries within the second batch as negative documents to calculate a loss function. Additionally, the electronic device (100) may, in addition to the in-batch negative method, incorporate hard negative documents sampled from a set of hard negative documents for each query included in the second batch into the calculation of the loss function. For example, the electronic device (100) can train an information retrieval model (111a) by calculating a loss function for each query included in the second batch, by including a hard negative document sampled from a set of hard negative documents corresponding to the query as a negative document, and considering a positive document corresponding to another query as a negative document.

[0082] In various embodiments, the electronic device (100) may form a plurality of query-positive document-hard negative document tuples based on positive documents and hard negative documents sampled for each of a plurality of queries, and may form a second batch including the plurality of query-positive document-hard negative document tuples.

[0083] In one embodiment, the electronic device (100) may mask (730) at least some of the hard negative documents included in the loss function calculation corresponding to the second batch according to a margin criterion at each step of the learning loop that trains the information retrieval model (111a) using the second batch. In this regard, for each query included in the second batch, the electronic device (100) may obtain from the information retrieval model (111a) a third similarity between the query and a positive document and a fourth similarity between the query and a hard negative document (e.g., a document sampled from a set of hard negative documents and included in the loss function calculation, or a document included in a query-positive document-hard negative document tuple).

[0084] According to one embodiment, the electronic device (100) may determine masking for at least a portion of hard negative documents based on a comparison between a third similarity and a fourth similarity. For example, the electronic device (100) may mask the hard negative documents corresponding to the fourth similarity so that they are excluded from the loss function calculation if the value obtained by subtracting the fourth similarity from the third similarity is greater than or equal to a preset third threshold (or margin). That is, the electronic device (100) may mask at least a portion of the hard negative documents at each step of the learning loop to exclude them from the loss function calculation, so that the information retrieval model (111a) prevents the learning signal from being excessively accumulated by hard negative documents that are sufficiently distinguishable from positive documents corresponding to the query, and so that learning is more concentrated on hard negative documents that are relatively difficult to distinguish. Such masking processing for hard negative documents can provide a technical effect of saving computational resources of at least one processor (e.g., processor (130) of FIG. 1) and accelerating the learning convergence speed of the information retrieval model (111a) by blocking unnecessary gradient backpropagation for training samples that the information retrieval model (111a) can already sufficiently distinguish.

[0085] In one embodiment, the electronic device (100) may update (750) the parameters of the information retrieval model (111a) based on a loss function that reflects the result of the masking process. In this regard, the electronic device (100) may use the information retrieval model (111a) to calculate a loss function based on the similarity to positive documents and the similarity to negative documents (e.g., in-batch negative documents and / or hard negative documents) for each of the plurality of queries included in the second batch. In this operation, the electronic device (100) may calculate the loss such that the contribution of hard negative documents determined to be excluded from the loss function calculation according to the masking process is not reflected.

[0086] According to one embodiment, the electronic device (100) may calculate a gradient for a loss function and update parameters of an information retrieval model (111a) (e.g., weights of an encoder) based on said gradient. For example, for each query included in a second batch, the electronic device (100) may update parameters of the information retrieval model (111a) such that the similarity between the query and a positive document increases and the similarity between the query and a negative document decreases. In one embodiment, the electronic device (100) may repeatedly perform the parameter update of the information retrieval model (111a) at each step of the learning loop.

[0087] FIG. 7 is a diagram illustrating, in an exemplary manner, a processing pipeline for scheduling learning control parameters related to the learning of an information retrieval model according to one embodiment of the present disclosure.

[0088] Referring to FIG. 7, an electronic device according to one embodiment (e.g., the electronic device (100) of FIG. 1) can determine a learning control parameter value according to the schedule by applying a preset schedule according to the learning epoch of an information retrieval model (e.g., the information retrieval model (111a) of FIG. 2) and reflect the learning control parameter value in the batch configuration and / or loss function calculation.

[0089] According to one embodiment, the electronic device (100) may apply a preset schedule (810) to the learning loop as the epoch of the learning loop progresses. For example, the electronic device (100) may identify the index (or progress rate) of the current learning epoch and call a schedule according to a schedule interval (e.g., an early learning interval, a mid-term learning interval, or a late learning interval) corresponding to the index of the epoch. In this regard, the learning schedule of the information retrieval model (111a) may be defined as a linear function, a step function, an exponential function and / or a function equivalent thereto and may be pre-stored in a database (e.g., the database (113) of FIG. 2), and at least some different learning control parameter values ​​may be set to adjust the learning difficulty for the information retrieval model (111a) for each schedule interval.

[0090] In one embodiment, the electronic device (100) may determine (830) a learning control parameter value based on a schedule corresponding to an index of an identified epoch. According to various embodiments, the learning control parameter determined based on the learning schedule interval may include a parameter for controlling the configuration of a second batch and / or the calculation of a loss function used for learning the information retrieval model (111a). For example, the learning control parameter may include a parameter related to at least one of the number of queries to be included in the second batch, the number of hard negative documents to be sampled from a set of hard negative documents corresponding to the queries, a third threshold (or margin) used as a masking criterion for hard negative documents, or the ratio (or maximum number) of hard negative documents to be masked to be excluded from the calculation of the loss function.

[0091] According to one embodiment, the electronic device (100) may determine a learning control parameter value to set the number of hard negative documents relatively low or to set the third threshold relatively high when the epoch index (or progress rate) for the learning of the information retrieval model (111a) corresponds to a first schedule interval (e.g., an initial learning interval). Alternatively, the electronic device (100) may determine a learning control parameter value to set the number of hard negative documents to gradually increase or to set the third threshold to gradually decrease when the epoch index for the learning of the information retrieval model (111a) corresponds to a second schedule interval after the first schedule interval (e.g., a mid-term learning interval or a late learning interval).

[0092] In one embodiment, the electronic device (100) may reflect the determined learning control parameter value in the second batch configuration and / or loss calculation (850). That is, the electronic device (100) may dynamically adjust the learning control parameter value so that the adjustment of the learning difficulty is reflected in the second batch configuration and / or loss function calculation as the learning epoch of the information retrieval model (111a) progresses, thereby mitigating loss oscillations caused by excessive negative learning signals and improving the convergence stability and learning performance of the information retrieval model (111a).

[0093] According to various embodiments, dynamic adjustment of the second batch configuration can maximize the learning efficiency of the information retrieval model (111a) within a limited memory capacity (e.g., memory (110) of FIG. 1 and / or FIG. 2). For example, in various embodiments of the present disclosure, instead of physically increasing the batch size, a sophisticated learning effect greater than that achieved using a large batch can be achieved with a small memory (110) occupancy by intensively reflecting valid hard negative documents selected using a cross-encoder model (e.g., cross-encoder model (111b) of FIG. 2) and positive documents reflecting false negative documents into the training dataset.

[0094] FIG. 8 is a diagram illustrating, in an exemplary manner, a method for configuring a learning dataset for learning an information retrieval model according to one embodiment of the present disclosure.

[0095] The steps of the method for constructing a training dataset for training an information retrieval model described in the embodiment of FIG. 8 may be performed sequentially or non-sequentially. For example, the order of execution of the steps described in the embodiment of FIG. 8 may be changed, some steps may be performed repeatedly, some steps may be omitted, or at least two steps may be performed in parallel. In addition, at least some of the steps described in the embodiment of FIG. 8 may include operations identical or similar to the operation of the electronic device (e.g., the electronic device (100) of FIG. 1) described above with reference to the drawings, and redundant descriptions of identical or similar operations may be omitted below.

[0096] Referring to FIG. 8, in step S910, an electronic device according to one embodiment (e.g., the electronic device (100) of FIG. 1) can calculate the similarity between a query and a positive document for each of a plurality of query-positive document pairs. In this regard, the electronic device (100) can obtain different plurality of query-positive document pairs from a pool of training data stored in a database (e.g., the database (113) of FIG. 2) according to random sampling or a sampling rate set by domain, and can form a first batch using the plurality of query-positive document pairs.

[0097] In one embodiment, the electronic device (100) inputs each query and positive document of a plurality of query-positive document pairs included in a first batch into an information retrieval model (e.g., the information retrieval model (111a) of FIG. 2) to obtain a similarity (e.g., inner product, cosine similarity and / or equivalent score) for each query-positive document pair calculated from the information retrieval model (111a).

[0098] In step S920, an electronic device (100) according to one embodiment can identify at least one candidate document for each of a plurality of queries from a corpus based on the similarity of each of a plurality of query-positive document pairs. For example, for each of a plurality of queries included in a first batch, the electronic device (100) can identify at least one candidate document that is likely to be confused with the positive document corresponding to the query by the information retrieval model (111a). In this regard, the electronic device (100) can calculate the similarity between each of the plurality of queries and the plurality of documents included in the corpus using the information retrieval model (111a).

[0099] In one embodiment, the electronic device (100) can determine whether a document in a corpus is a candidate document for a query by comparing, for each of a plurality of queries, a first similarity of a query-positive document pair corresponding to the query and a second similarity between each of the query and a plurality of documents included in the corpus. For example, the electronic device (100) can identify a corpus document associated with the second similarity as a candidate document for the query if the second similarity is greater than the first similarity, or if the value obtained by subtracting the first similarity from the second similarity is greater than or equal to a preset first threshold (or margin).

[0100] In step S930, an electronic device (100) according to one embodiment can calculate the degree of relevance to at least one candidate document for each of a plurality of queries included in a first batch. For example, the electronic device (100) may form a pair of each of a plurality of queries and at least one candidate document corresponding to the query, and input the query-candidate document pair into a cross-encoder model (e.g., the cross-encoder model (111b) of FIG. 2) to obtain the degree of relevance (e.g., logit, probability value and / or equivalent score) for each query-candidate document pair calculated from the cross-encoder model (111b).

[0101] In step S940, an electronic device (100) according to one embodiment may classify at least one candidate document for a query as a genuine negative document or a false negative document based on the relevance of each query-candidate document pair. According to one embodiment, the electronic device (100) may classify candidate documents included in a pair by comparing the relevance of the query-candidate document pair with a preset second threshold. For example, the electronic device (100) may classify a candidate document as a genuine negative document for the query if the relevance of the query-candidate document pair is less than the second threshold, and may classify a candidate document as a false negative document for the query if the relevance of the query-candidate document pair is greater than or equal to the second threshold.

[0102] In step S950, an electronic device (100) according to one embodiment may construct a training dataset of an information retrieval model (111a) based on the classification results of candidate documents for each of the plurality of queries included in the first batch. In this regard, for each of the plurality of queries included in the first batch, the electronic device (100) may update the set of positive documents for the corresponding query and generate a set of hard negative documents. For example, for each of the plurality of queries, the electronic device (100) may update the set of positive documents for the query by adding at least one candidate document classified as a false negative document for the corresponding query as a positive document to the set of positive documents containing the positive documents of the corresponding query. Additionally, for each of the plurality of queries, the electronic device (100) may generate a set of hard negative documents containing at least one candidate document classified as a true negative document for the corresponding query.

[0103] In one embodiment, the electronic device (100) may construct a training dataset based on an updated set of positive documents and a generated set of hard negative documents for each of a plurality of queries. For example, the electronic device (100) may construct a training dataset by, for each of a plurality of queries, generating a query-positive document pair by pairing each of the corresponding query with at least one positive document included in the set of positive documents, and generating a query-hard negative document pair by pairing each of the corresponding query with at least one hard negative document included in the set of hard negative documents.

[0104] In step S960, an electronic device (100) according to one embodiment may train an information retrieval model (111a) using a training dataset. In this regard, the electronic device (100) may obtain a plurality of different queries from the training dataset according to random sampling or a sampling rate set by domain. Additionally, for each of the plurality of queries, the electronic device (100) may configure a plurality of query-positive document pairs and / or a plurality of query-hard negative document pairs to be used for training by sampling at least one positive document from a set of positive documents and at least one hard negative document from a set of hard negative documents.

[0105] In one embodiment, the electronic device (100) may configure a second batch comprising a plurality of query-positive document pairs, and in this case, the electronic device (100) may train an information retrieval model (111a) based on an in-batch negative method for the second batch. Additionally, the electronic device (100) may, in addition to the in-batch negative method, incorporate hard negative documents sampled from a set of hard negative documents for each query included in the second batch into the calculation of the loss function. Alternatively, the electronic device (100) may configure a plurality of query-positive document-hard negative document tuples based on positive documents and hard negative documents sampled for each of the plurality of queries, and may configure a second batch comprising a plurality of query-positive document-hard negative document tuples.

[0106] In one embodiment, the electronic device (100) can train an information retrieval model (111a) using a second batch, and at each step of the training loop, at least some of the hard negative documents included in the loss function calculation corresponding to the second batch can be masked. In this regard, for each query included in the second batch, the electronic device (100) can obtain a third similarity between the query and a positive document and a fourth similarity between the query and a hard negative document (e.g., a document sampled from a set of hard negative documents and included in the loss function calculation, or a document included in a query-positive document-hard negative document tuple).

[0107] According to one embodiment, the electronic device (100) may determine masking for at least a portion of hard negative documents based on a comparison between a third similarity and a fourth similarity. For example, the electronic device (100) may mask the hard negative documents corresponding to the fourth similarity so that they are excluded from the loss function calculation if the value obtained by subtracting the fourth similarity from the third similarity is greater than or equal to a preset third threshold (or margin). In one embodiment, the electronic device (100) may update the parameters of the information retrieval model (111a) based on the loss function that reflects the result of the masking process.

[0108] The method described above may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or multiple hardware components combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Furthermore, other examples of media may include recording or storage media managed by sites or servers that supply or distribute various software, such as app stores that distribute applications.

[0109] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof.

[0110] Those skilled in the art will understand that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate such interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functional aspects. Whether such functions are implemented in hardware or in software depends on the design requirements imposed on the specific application and the overall system. Those skilled in the art may implement the described functions in various ways for each specific application, but such implementations should not be construed as departing from the scope of the disclosure.

[0111] In a hardware implementation, the processing units used to perform the techniques may be implemented in one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure, computers, or a combination thereof.

[0112] Accordingly, the various exemplary logic blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors coupled with a DSP core, or any other combination of configurations.

[0113] In firmware and / or software implementations, techniques may be implemented as instructions stored on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or marked data storage device, etc. The instructions may be executable by one or more processors, and may cause the processor(s) to perform specific aspects of the functions described in this disclosure.

[0114] When implemented in software, the techniques described above may be stored on a computer-readable medium as one or more instructions or code, or transmitted through a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transmission of a computer program from one place to another. Storage media may be any available media accessible by a computer.

[0115] As a non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to transfer or store desired program code in the form of instructions or data structures and can be accessed by a computer. Additionally, any connection is appropriately referred to as a computer-readable medium.

[0116] For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of a medium.

[0117] The disks and discs used in this disclosure include CDs, laser discs, optical discs, DVDs (digital versatile discs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically, while discs reproduce data optically using a laser. The above combinations should also be included within the scope of computer-readable media.

[0118] The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other known form of storage medium. An exemplary storage medium may be connected to a processor so that the processor can read information from the storage medium or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and the storage medium may exist within an ASIC. The ASIC may exist within a user terminal. Alternatively, the processor and the storage medium may exist as separate components within the user terminal.

[0119] Although the embodiments described above have been described as utilizing aspects of the subject matter disclosed herein in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or a distributed computing environment. Furthermore, aspects of the subject matter in the present disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and portable devices.

[0120] Although the present disclosure has been described in relation to some embodiments, various modifications and changes may be made without departing from the scope of the present disclosure as understood by a person skilled in the art to which the invention of the present disclosure pertains. Furthermore, such modifications and changes should be considered to fall within the scope of the claims appended to the present disclosure. Explanation of the symbols

[0121] 100: Electronic device 110: Memory 111: Program 111a: Information retrieval model 111b: Cross-encoder model 113: Database 120: Communication module 130: Processor

Claims

Claim 1 A method for constructing a training dataset for training an information retrieval model executed by at least one processor, comprising: a step of calculating a first similarity between a query and a positive document for each of a plurality of query-positive document pairs; a step of identifying at least one candidate document for each of a plurality of queries from a corpus based on the first similarity; a step of constructing a query-candidate document pair by pairing the query and the identified at least one candidate document for each of the plurality of queries, and inputting the query-candidate document pair into a cross-encoder model to calculate a relevance value representing semantic relevance for each query-candidate document pair; a step of comparing the relevance value for each query-candidate document pair with a second threshold value, classifying the candidate document corresponding to the relevance value as a true negative document if the relevance value is less than the second threshold value, and classifying the candidate document corresponding to the relevance value as a false negative document if the relevance value is greater than or equal to the second threshold value; and constructing a training dataset based on the classification result. A method for constructing a training dataset, comprising the steps of: a step; and a step of training an information retrieval model using the training dataset. Claim 2 A method for constructing a training dataset according to claim 1, wherein the step of identifying at least one candidate document comprises: a step of calculating a second similarity between a specific query among the plurality of queries and a document included in the corpus; a step of comparing the first similarity and the second similarity calculated for a query-positive document pair corresponding to the specific query; and a step of identifying a document included in the corpus as a candidate document for the specific query when the value obtained by subtracting the first similarity from the second similarity is greater than or equal to a first threshold. Claim 3 delete Claim 4 A method for constructing a training dataset according to claim 1, wherein the step of constructing the training dataset comprises: a step of updating a set of positive documents of a specific query including a first positive document corresponding to a specific query among a plurality of queries; and a step of determining a candidate document classified as a true negative document for the specific query as a hard negative document, and generating a set of hard negative documents of the specific query including the hard negative document. Claim 5 A method for constructing a training dataset according to claim 4, wherein the step of updating the set of positive documents includes the step of adding a candidate document classified as a false negative document for the specific query to the set of positive documents as a second positive document for the specific query. Claim 6 A method for constructing a training dataset according to claim 5, wherein the step of training the information retrieval model comprises, at each step of the training loop, a step of masking at least one hard negative document selected as a target for loss function calculation from the set of hard negative documents and excluding it from the loss function calculation. Claim 7 A method for constructing a training dataset according to claim 6, wherein the step of excluding from the calculation of the loss function comprises: a step of constructing a batch from the training dataset; a step of calculating a third similarity with a positive document corresponding to each of the plurality of queries included in the batch for each of the plurality of queries included in the batch; a step of calculating a fourth similarity between a specific query included in the batch and the selected hard negative document; a step of comparing the third similarity and the fourth similarity calculated for the specific query; and a step of masking the hard negative document when the value obtained by subtracting the fourth similarity from the third similarity is greater than or equal to a third threshold. Claim 8 A method for constructing a training dataset according to claim 1, wherein the step of training the information retrieval model comprises: determining a learning control parameter value based on a pre-set learning schedule according to the learning epoch of the information retrieval model; and applying the learning control parameter value to at least one of batch configuration or loss function calculation. Claim 9 A computer program stored on a computer-readable recording medium for executing on a computer a method for constructing a learning dataset for learning an information retrieval model according to any one of paragraphs 1, 2, 4 through 8. Claim 10 In an electronic device, a memory that stores instructions; The electronic device comprises at least one processor, wherein, when the instructions are executed by the at least one processor, the electronic device calculates a first similarity between a query and a positive document for each of a plurality of query-positive document pairs, identifies at least one candidate document for each of the plurality of queries from a corpus based on the first similarity, pairs the query and the identified at least one candidate document for each of the plurality of queries to form a query-candidate document pair, inputs the query-candidate document pair into a cross-encoder model to calculate a relevance value representing semantic relevance for each query-candidate document pair, compares the relevance value for each query-candidate document pair with a second threshold, classifies the candidate document corresponding to the relevance value as a genuine negative document if the relevance value is less than the second threshold, and classifies the candidate document corresponding to the relevance value as a false negative document if the relevance value is greater than or equal to the second threshold, and constructs a training dataset based on the classification result, and trains an information retrieval model using the training dataset. Device.

Citation Information

Patent Citations

  • Training data creation method and training data creation apparatus

    US20190362187A1