Method and system for constructing redundancy-aware retrieval evaluation dataset for high-similarity and high-redundancy corpora

Abstract

The present invention relates to a method and system for constructing a redundancy-aware search evaluation dataset to fairly evaluate the performance of a searcher in a unique and highly redundant corpus with high document similarity and information redundancy. The method of the present invention divides input documents into a plurality of document fragments, extracts atomic information of a single fact unit from each document fragment using a language model, and selects valid atomic information satisfying validity criteria among the extracted atomic information. Subsequently, independent evaluation and ranking fusion are performed on the valid atomic information according to a plurality of quality criteria to determine target atomic information, and an equivalent fact set is constructed by performing candidate search based on embedding similarity and equivalence determination based on a language model on the target atomic information. Additionally, query candidates are generated based on the target atomic information or the equivalent fact set, and a representative query is determined among the query candidates that have passed a plurality of logical verification criteria. Finally, a search evaluation data item including a representative query, a correct answer, and an evidence group is stored, wherein the evidence group includes source evidence and alternative evidence containing factual content equivalent to the source evidence. Accordingly, even if the searcher fails to recover source evidence, if it recovers alternative evidence, it can be recognized as having recovered correct evidence, allowing for a realistic evaluation of the searcher's actual performance in unique and highly redundant corpora.

Description

Technology Field

[0001] The present invention relates to the field of information retrieval and Retrieval-Augmented Generation (RAG) technology, and more specifically, to a method, system, and computer-readable recording medium for automatically constructing a search evaluation dataset that recognizes duplicate information in a unique and highly redundant corpus with high document similarity and information redundancy, and for fairly evaluating the performance of a retriever using the constructed search evaluation dataset.

[0002] Furthermore, the present invention relates to a data construction and evaluation technology for realistically evaluating the actual search performance of a searcher in a unique and highly redundant environment by using a Large Language Model (LLM) to extract atomic information of a single fact unit from document fragments, controlling the quality of evaluation data through separation evaluation and ranking fusion based on multiple quality criteria, and constructing an evidence group that includes alternative evidence semantically equivalent to the source evidence using embedding similarity and language model-based equivalence determination. Background Technology

[0003] With the recent advancements in Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) technology—which searches for documents related to user queries from external document repositories and generates answers based on the retrieved documents—is being utilized across various industries. In these RAG systems, the performance of the retriever, which accurately recovers document fragments highly relevant to the user's query, significantly impacts the overall quality of the answer. Consequently, the importance of retrieval evaluation datasets and benchmarks for objectively assessing retriever performance is increasing.

[0004] Conventional search evaluation datasets or Q&A benchmarks are generally constructed under the premise of environments where document redundancy is low and a specific correct answer document or paragraph corresponds relatively clearly to a single query. For example, public Q&A datasets or general-purpose search benchmarks typically designate a single reference document or paragraph containing the correct answer as "golden evidence" and evaluate performance based on whether the search engine has retrieved that reference document or paragraph.

[0005] However, document corpora used in actual corporate environments—such as financial disclosures, statutes, contracts, case law, patent literature, and technical documents—often feature similar phrasing across documents, and identical or essentially the same factual content frequently appears repeatedly in multiple documents or document fragments. In such unique and highly redundant corpora, a single correct fact can be included in other document fragments in a semantically equivalent form, not just in the source document fragment. Research papers also explain that document similarity and information redundancy are significantly higher in enterprise domains such as Finance, Patent, and Legal compared to standard benchmarks.

[0006] In such environments, even if the searcher fails to retrieve the initially designated correct document fragment, it can retrieve semantically equivalent alternative evidence sufficient to derive the correct answer. Nevertheless, in conventional evaluation methods centered on a single correct document or paragraph, such alternative evidence is not recognized as correct evidence, leading to a problem where the searcher's actual performance is unfairly underestimated. This issue is particularly pronounced in highly redundant domains, where the same facts are distributed across multiple documents; consequently, the searcher is treated as an incorrect answer in evaluation metrics even though it has actually retrieved sufficient evidence.

[0007] Furthermore, conventional methods for constructing question-and-answer evaluation datasets often rely on manual annotations or language model-based automatic generation using a single prompt. Manual methods are costly and time-consuming to build, and it is difficult to construct large-scale, domain-specific customized datasets. On the other hand, single-prompt-based automatic generation methods have limitations, such as inconsistent quality of generated atomic information or queries, and difficulty in reliably satisfying multiple quality criteria simultaneously, including contextual independence, correct answer probability, appropriateness of information scope, and query clarity.

[0008] Furthermore, when a language model is required to evaluate multiple quality criteria simultaneously, problems may arise where judgments between criteria become mixed or the correction of scores for each criterion is inconsistent. Consequently, some of the generated data items may consist of queries regarding trivial facts, queries where the correct answer is exposed within the question itself, queries that cannot be answered based solely on the given evidence, or queries that request information that is excessive or insufficient compared to what is necessary. If such degraded data is included in the evaluation dataset, the reliability of the searcher performance evaluation is compromised.

[0009] Therefore, in real-world industrial corpora with high document similarity and information redundancy, there is a need for a technology to construct a redundancy cognitive search and evaluation dataset that can accept not only source evidence but also alternative evidence as correct evidence by breaking down factual information contained in document fragments into atomic units and systematically tracking semantically equivalent factual information.

[0010] In addition, there is a need for technology capable of automatically constructing sets of queries, answers, and evidence that are logically valid and suitable for search performance evaluation, by mitigating the instability of language model judgments and evaluating multiple quality criteria separately by criterion and fusing the ranking results of the evaluation results by criterion during the data generation process using a language model. The problem to be solved

[0011] The objective of the present invention is to provide a method and system for constructing a redundancy-aware search evaluation dataset capable of fairly evaluating the actual search performance of a searcher in a unique and highly redundant corpus with high document similarity and information redundancy.

[0012] In addition, another objective of the present invention is to construct a search evaluation dataset capable of recognizing as correct evidence not only source evidence but also alternative evidence containing factual content equivalent to source evidence, by extracting atomic information of a single fact unit from input documents and tracking semantically equivalent factual information that repeatedly exists in multiple document fragments based on said atomic information.

[0013] Furthermore, another objective of the present invention is to prevent the problem of search results retrieving alternative evidence from a highly redundant corpus being unfairly evaluated as incorrect answers by combining embedding similarity-based candidate search and language model-based equivalence determination to form document fragments containing substantially identical factual content, despite differing expressions, into an equivalent fact set or evidence group.

[0014] In addition, another objective of the present invention is to mitigate the instability of language model judgment and improve the quality and consistency of search evaluation data items by separating and evaluating multiple quality criteria by criterion and fusing the ranking results for each criterion during the process of automatically generating atomic information, queries, and answers using a language model.

[0015] Furthermore, another objective of the present invention is to automatically determine a set of query, answer, and evidence that can be clearly answered solely with given evidence and is suitable for search performance evaluation by applying logical verification criteria, such as contextual independence, prohibition of answer exposure, information equivalence, question clarity, and answer probability, to generated query candidates.

[0016] In addition, another objective of the present invention is to enable a search performance evaluation closer to an actual RAG environment than a conventional evaluation method centered on a single correct document fragment, by evaluating the performance of a searcher based on how much the search results cover the evidence group using the aforementioned duplicate cognitive search evaluation dataset. means of solving the problem

[0017] To achieve the above objective, a method for constructing a duplicate cognitive search evaluation dataset according to an embodiment of the present invention is a method performed by at least one processor, comprising: a step of dividing input documents included in an input corpus into a plurality of document fragments; a step of extracting atomic information of a single fact unit from each of the plurality of document fragments using a language model; a step of selecting atomic information satisfying a predetermined validity criterion among the extracted atomic information as valid atomic information; a step of performing independent evaluations for each of the valid atomic information according to a plurality of quality criteria and determining target atomic information by ranking and fusing the evaluation results for the plurality of quality criteria; a step of sequentially performing embedding similarity-based candidate search and language model-based equivalence determination on the target atomic information to construct an equivalence fact set including one or more atomic information semantically equivalent to the target atomic information; a step of generating query candidates based on the target atomic information or the equivalence fact set and performing verification for the query candidates according to a plurality of logical verification criteria; and a step of determining a representative query by ranking and fusing the evaluation results for the query candidates that have passed the plurality of logical verification criteria. and may include the step of storing a search evaluation data item including the representative query, the correct answer to the representative query, and an evidence group corresponding to the set of equivalent facts.

[0018] The above evidence group may include source evidence used to derive the correct answer to the above representative query and alternative evidence containing factual content equivalent to the source evidence. Accordingly, even if the searcher does not directly retrieve the source evidence, if it retrieves the alternative evidence, the search result may be evaluated as having retrieved the correct answer evidence.

[0019] The step of dividing the above input documents into a plurality of document fragments may include: a step of parsing the above input documents; a step of generating document fragments by dividing the parsed input documents based on the number of tokens while maintaining sentence boundaries; and a step of assigning a document fragment identifier to each document fragment, the document identifier, the page identifier, and the fragment identifier.

[0020] The step of extracting the atomic information may include a step of decomposing the content contained in each document fragment such that one atomic information contains only one indivisible factual claim. The step of selecting the valid atomic information may include a step of filtering the extracted atomic information according to a criterion including atomicity, validity, and non-ambiguity.

[0021] Additionally, the step of selecting the valid atomic information may include the step of excluding atomic information corresponding to at least one of missing essential elements, fragmented information, information that is insufficient in context, information lacking utility, and meta-information. The missing essential elements may include cases where at least one of the subject, object, time, place, and manner is omitted.

[0022] The step of determining the target atomic information may include: a step of calculating evaluation scores for each of the valid atomic informations based on quality criteria of validity, completeness, specificity, clarity, and questionability; a step of calculating a rank for each quality criterion based on the evaluation scores for each quality criterion; and a step of calculating a mutual rank fusion score based on the ranks for each quality criterion. The mutual rank fusion score may be calculated using the inverse of the rank for each criterion, for example, based on the following formula.

[0023] s(x)=Σ(1 / (rank_i(x)+k))

[0024] Here, x is the atomic information or query candidate to be evaluated, rank_i(x) is the rank of x at the i-th quality criterion, and k can be a constant greater than or equal to 0.

[0025] The step of constructing the above equivalence fact set may include: calculating a cosine similarity between an embedding vector for the target atomic information and embedding vectors of atomic information included in a document fragment other than the target atomic information; selecting atomic information whose cosine similarity is greater than or equal to a predetermined threshold as candidate atomic information; and performing an equivalence determination using a language model for each of the candidate atomic information and including the candidate atomic information determined to be equivalent to the target atomic information in the above equivalence fact set.

[0026] The determination of equivalence using the above language model may include a step of determining whether the target atom information and the candidate atom information satisfy at least one of semantic equivalence, logical equivalence, and factual equivalence. The factual equivalence may be determined based on whether at least one of entities, attributes, relationships, time points, places, and quantitative values ​​corresponds to each other.

[0027] The step of generating the above-mentioned query candidates and performing verification on the above-mentioned query candidates according to a plurality of logical verification criteria may include: a step of sampling a plurality of atomic information among valid atomic information; a step of selecting M atomic information having interconnectedness among the sampled plurality of atomic information; a step of generating a plurality of query candidates based on the selected M atomic information as a common basis; and a step of applying logical verification criteria of contextual independence, prohibition of revealing the correct answer, information equivalence, item clarity, and correct answer possibility to each of the plurality of query candidates. A query candidate that does not satisfy any of the above-mentioned logical verification criteria may be discarded.

[0028] The step of determining the representative query may include: a step of calculating evaluation scores for each query quality criterion of connectivity, fluency, keyness, and validity for each query candidate that has passed the logic verification criteria; a step of calculating a ranking for each query quality criterion based on the evaluation scores for each query quality criterion; and a step of calculating a query ranking fusion score based on the rankings for each query quality criterion. The query candidate with the highest query ranking fusion score may be determined as the representative query.

[0029] The above search evaluation data item may include at least one of a query identifier, a representative query, a correct answer, an evidence group, a source document identifier, a document fragment identifier, an atomic information identifier, and domain information. The evidence group may be a group formed by grouping multiple document fragments containing the same factual content, and if the search result of the searcher includes any one of the document fragments included in the evidence group, the evidence group may be determined to have been recovered.

[0030] In addition, a method for constructing a duplicate recognition search evaluation dataset according to an embodiment of the present invention may further include a step of evaluating the performance of a searcher using a search evaluation dataset containing the search evaluation data items. The step of evaluating the performance of the searcher may include: a step of determining whether at least one document fragment included in each evidence group is included within the top K document fragments of the search results; a step of calculating the ratio of evidence groups covered by the search results; and a step of calculating a complete recovery rate indicating whether all evidence groups are covered by the search results.

[0031] In addition, to achieve the above objective, a duplicate recognition search evaluation dataset construction system according to one embodiment of the present invention may include: a data preprocessing module that divides input documents into a plurality of document fragments; an atomic information extraction module that extracts atomic information of a single fact unit from each of the plurality of document fragments; a validity verification module that selects valid atomic information satisfying a predetermined validity criterion among the atomic information; a criterion-based ranking module that performs independent evaluations according to a plurality of quality criteria on the valid atomic information and rank-fuses the evaluation results according to the plurality of quality criteria to determine target atomic information; a duplicate detection module that constructs an equivalent fact set by performing candidate search based on embedding similarity and equivalence determination based on a language model on the target atomic information; a query generation module that generates query candidates based on the target atomic information or the equivalent fact set; a logic verification module that performs verification according to a plurality of logic verification criteria on the query candidates; a final ranking module that determines a representative query by rank-fuses the evaluation results according to a plurality of query quality criteria on the query candidates that have passed the logic verification criteria; and a data storage unit that stores search evaluation data items including the representative query, the correct answer, and the evidence group.

[0032] In addition, a computer-readable recording medium according to one embodiment of the present invention may have a program recorded thereon that enables a computer to perform the method for constructing a duplicate recognition search evaluation dataset. Effects of the invention

[0033] According to the present invention, there is an effect of fairly evaluating the actual search performance of a searcher even in unique and highly redundant corpora with high document similarity and information redundancy.

[0034] In addition, according to the present invention, by extracting atomic information of a single fact unit from input documents and tracking semantically equivalent fact information that repeatedly exists in a plurality of document fragments based on said atomic information, there is an effect of utilizing not only source evidence but also alternative evidence containing factual content equivalent to source evidence as correct evidence.

[0035] In addition, according to the present invention, even if the searcher fails to retrieve the source evidence initially designated, if it retrieves alternative evidence sufficient to derive the correct answer, it can be recognized as a legitimate search result, thereby preventing the problem of underestimating the performance of the searcher that occurred in conventional evaluation methods centered on a single correct answer document fragment.

[0036] Furthermore, according to the present invention, by combining embedding similarity-based candidate search and language model-based equivalence determination, document fragments containing substantially identical factual content, although differing in expression, can be effectively detected. Accordingly, there is an effect of simultaneously achieving recall-based candidate search and precision-based equivalence verification.

[0037] Furthermore, according to the present invention, when evaluating the quality of atomic information, query candidates, or data items, multiple quality criteria are evaluated separately by criterion, and by ranking and fusing the evaluation results for each criterion, it is possible to construct more stable and consistent data than when relying on a single judgment or single score of a language model.

[0038] In addition, according to the present invention, atomic information is selected using multiple quality criteria such as validity, completeness, specificity, clarity, and questionability, and query candidates are filtered using logical verification criteria such as contextual independence, prohibition of exposure of correct answers, information equivalence, item clarity, and correct answer possibility, thereby preventing atomic information or queries unsuitable for search evaluation from being included in the dataset.

[0039] In addition, according to the present invention, by generating a multi-hop query based on a plurality of atomic information as a common basis and verifying whether the generated query accurately corresponds to the necessary evidence information, it is possible to automatically construct not only simple fact-checking type queries but also search evaluation items that require combining multiple pieces of evidence.

[0040] In addition, according to the present invention, search evaluation data items including a query, a correct answer, and an evidence group can be automatically generated, thereby enabling the efficient construction of domain-specific search evaluation datasets for various domains such as finance, law, patents, technical documents, and internal corporate documents.

[0041] In addition, according to the present invention, since the performance of the searcher can be evaluated based on how much the search results cover each evidence group, the partial recovery performance and complete recovery performance of the searcher in unique and highly redundant corpora can be measured more realistically.

[0042] In addition, according to the present invention, since performance differences based on a searcher, re-ranker, generative model, or a combination thereof in a RAG system can be compared based on redundant cognitive evidence groups, the improvement effect of the RAG pipeline can be objectively verified under conditions close to an actual industrial environment.

[0043] The effects of the present invention are not limited to those mentioned above, and other effects that a person skilled in the art can predict from the composition of the present invention may also be included within the scope of the present invention. Brief explanation of the drawing

[0044] FIG. 1 is a diagram illustrating the overall flow of a duplicate recognition search evaluation dataset construction framework according to one embodiment of the present invention. FIG. 2 is a diagram illustrating examples of judgment criteria used in valid information filtering, valid information ranking, query logic filtering, and query ranking according to an embodiment of the present invention. Figure 3 is a graph showing the performance comparison results of the separation evaluation and ranking fusion method by criterion according to one embodiment of the present invention. Figure 4 is a graph illustrating the change in search performance by domain according to an increase in the number of hops of a query according to one embodiment of the present invention. FIG. 5 is a block diagram of a duplicate recognition search evaluation dataset construction system according to one embodiment of the present invention. Specific details for implementing the invention

[0045] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. However, the embodiments described below are merely illustrative to aid in understanding the present invention, and the scope of the present invention is not limited by the following embodiments. A person skilled in the art to which the present invention pertains may modify, substitute, add, or omit parts of the components within the scope of the technical spirit of the present invention without departing from it, and such modification, substitution, addition, or omission may be included within the scope of the present invention.

[0046] The terms used in this specification are intended to describe embodiments of the present invention and are not intended to limit the present invention to specific embodiments. Components described in the singular in this specification may include a plurality of components unless the context clearly indicates otherwise, and components described in the plural are not to be interpreted as excluding a single component. Furthermore, expressions such as “include,” “have,” and “have” mean that the described components are present and are not to be interpreted as excluding the existence or addition of other components not described.

[0047] In this specification, the term “corpus” may refer to a set of input data for constructing a search evaluation dataset. The corpus may include a plurality of input documents, and the input documents may include, for example, financial disclosure documents, annual reports, statutes, contracts, case law, patent literature, technical documents, internal corporate documents, Wiki documents, web documents, or combinations thereof. However, the types of input documents are not limited thereto, and documents from various domains with high similarity or information redundancy between documents may be used as the input corpus of the present invention.

[0048] In this specification, “uniqueness corpus” may refer to a corpus in which expressions, structures, topics, or positions in embedding space among multiple documents or document fragments are similar to one another. For example, a uniqueness corpus may be a corpus in which different documents contain similar sentence structures, similar terms, similar clause formats, similar financial items, or similar technical descriptions. In one embodiment, similarity between documents may be calculated based on the mean cosine similarity between the embedding vectors of the document fragments.

[0049] In this specification, a “highly redundant corpus” may refer to a corpus in which the same or substantially identical factual content is repeatedly present in multiple documents or fragments of documents. For example, if a single company name, financial figures, legal requirements, patent components, technical effects, or procedural requirements are repeatedly described in multiple fragments of documents with only different expressions, the corpus may be considered a highly redundant corpus. In one embodiment, the degree of information redundancy may be calculated based on the ratio of atomic information in which the same or equivalent factual content exists in other fragments of documents.

[0050] In this specification, “document fragment” may refer to a unit divided to be used as a unit for searching, analyzing, or evaluating an input document. Document fragments may be generated by paragraph units, sentence units, clause units, page units, token count units, or a combination thereof. In one embodiment, the input document may be divided based on the number of tokens, while maintaining sentence boundaries. For example, the input document may be divided into 512 token units, and the overlap between document fragments may be set to 0. However, the token count and overlap values ​​are merely examples and may be varied depending on the type of input document, the search unit of the searcher, or the purpose of evaluation.

[0051] In this specification, “atomic information” or “atomic fact” may refer to a unit of information containing a single indivisible factual claim. A single atomic information may preferably contain a single factual content regarding a single subject, a single object, a single relationship, a single attribute, a single numerical value, a single point in time, or a combination thereof. For example, the sentence “Tesla’s long-term debt as of December 31, 2024 was $5.26 billion” may be atomic information because it contains a single independent factual claim. On the other hand, the sentence “Tesla’s headquarters are in Austin, and Apple’s headquarters are in Cupertino” may be broken down into multiple atomic information because it contains two or more independent factual claims.

[0052] In this specification, “valid atomic information” may refer to atomic information among extracted atomic information that satisfies minimum validity criteria suitable for constructing a search evaluation dataset. The minimum validity criteria may include atomicity, validity, non-ambiguity, self-sufficiency, non-obviousness, factuality, or a combination thereof. For example, if atomic information contains only a single factual claim, is understandable without external context, and includes substantial knowledge capable of supporting search queries or the generation of correct answers, such atomic information may be selected as valid atomic information.

[0053] In this specification, “target atomic information” may refer to atomic information selected through evaluation and ranking fusion based on quality criteria among valid atomic information. Target atomic information may be used as reference information for duplicate detection, configuration of equivalent fact sets, query generation, or configuration of evidence groups. In one embodiment, a predetermined number of top valid atomic information may be selected as target atomic information for each document fragment. For example, the top three valid atomic information for each document fragment may be selected as target atomic information, but are not limited thereto.

[0054] In this specification, the term “equivalent fact set” may mean a set containing one or more atomic information that is semantically, logically, or factually equivalent to a single target atomic information. Atomic information included in the equivalent fact set may convey the same or substantially the same factual content, even if they differ in expression, sentence structure, document source, or location. For example, the atomic information “The company’s revenue in the third quarter of 2023 increased by 15%” and the atomic information “The company’s earnings in the third quarter of 2023 grew by 15 percent” may be determined to be equivalent facts if they represent the same entity, the same period, the same financial item, and the same quantitative change.

[0055] In this specification, “source evidence” may mean a fragment of document, atomic information, or a combination thereof that is directly used to generate a query or a correct answer. Additionally, “alternative evidence” may mean a fragment of document, atomic information, or a combination thereof that is contained in a fragment of document different from the source evidence but contains factual content equivalent to that of the source evidence and can support the derivation of the same correct answer. In this specification, “evidence group” may mean a unit that groups source evidence and one or more alternative evidence satisfying one correct fact or one essential evidence.

[0056] In this specification, the “duplicate awareness search evaluation dataset” may refer to a set of search evaluation data items including a query, a correct answer, and one or more evidence groups. The duplicate awareness search evaluation dataset may be configured such that, during the evaluation of a searcher, it determines not only whether the searcher has recovered source evidence, but also whether it has recovered alternative evidence included in the same evidence group. Accordingly, even if the searcher fails to recover the initially designated source document fragment, if it recovers an alternative document fragment sufficient to derive the correct answer, the evidence group may be evaluated as covered.

[0057] In this specification, “searcher” may refer to a component that searches for relevant documents, document fragments, phrases, sentences, or atomic information from a corpus or document repository in response to an input query. The searcher may be implemented as a search pipeline including a sparse searcher such as BM25, an embedding-based dense searcher, a language model-based searcher, and a reranker, or a combination thereof. The specific type of searcher does not limit the scope of the invention.

[0058] In this specification, the term “language model” may refer to a model capable of receiving natural language text as input and performing text generation, classification, judgment, summarization, query generation, answer generation, or equivalence determination. The language model may be a large-scale language model, a model called via an external API, a model executed on a local server, an open-source model, or a commercial model. Furthermore, in the present invention, the language model may perform at least one of atomic information extraction, validation, evaluation based on quality criteria, equivalence determination, query candidate generation, logic verification, answer generation, or query quality evaluation.

[0059] In this specification, “embedding model” may mean a model that converts text, document fragments, or atomic information into embedding vectors in a vector space. The embedding model may be a sentence embedding model, a document embedding model, a language model-based embedding model, or a combination thereof. In one embodiment, the similarity between embedding vectors may be calculated based on cosine similarity, and the embedding vectors may be L2 normalized prior to similarity calculation.

[0060] In this specification, “rank fusion” may mean combining ranks calculated for a plurality of evaluation criteria to produce a single integrated rank or integrated score. In one embodiment, rank fusion may be performed using a mutual rank fusion method that utilizes the inverse of the rank for each criterion. For example, the integrated score s(x) for an evaluation target x may be calculated based on the rank_i(x) for each criterion. In this case, rank fusion may be performed by combining the rank information for each criterion rather than by directly averaging the magnitudes of the scores for each criterion calculated by the language model, thereby mitigating the impact of inconsistencies in score correction between criteria.

[0061] In this specification, “criterion-wise separate evaluation” may mean performing a separate evaluation or judgment for each criterion, rather than simultaneously evaluating multiple quality criteria or logical criteria in a single prompt or judgment procedure. For example, the quality of atomic information may be independently evaluated according to criteria of validity, completeness, specificity, clarity, and questionability, and the quality of query candidates may be independently evaluated according to criteria of connectivity, fluency, keyness, and validity. This combination of criterion-wise separate evaluation and rank fusion may be referred to in this specification as CRRF (Criterion-wise Prompting with Reciprocal Rank Fusion) or criterion-wise prompt-based reciprocal rank fusion.

[0062] The method for constructing a duplicate recognition search evaluation dataset according to an embodiment of the present invention may be performed, overall, by dividing an input document into document fragments, extracting atomic information from the document fragments, selecting valid atomic information, determining target atomic information through separation evaluation by criteria and ranking fusion, constructing an equivalent fact set based on the target atomic information, and generating search evaluation data items including a query, a correct answer, and an evidence group based on the equivalent fact set. Additionally, the generated search evaluation dataset may be used to evaluate the performance of a searcher based on whether the search results of the searcher cover each evidence group.

[0063] A system according to an embodiment of the present invention may be implemented as a computing device comprising one or more processors, memory, storage devices, and communication interfaces. The system may be implemented as a single server, multiple servers, a cloud server, a GPU server, a local workstation, a distributed computing environment, or a combination thereof. Additionally, each function of the system may be implemented as a software module, a hardware module, or a combination of software and hardware. For example, a data preprocessing module, an atomic information extraction module, a validation module, a ranking module by criteria, a duplicate detection module, a query generation module, a logic verification module, a final ranking module, an evaluation module, and a data storage module may be implemented by program code executed by one or more processors.

[0064] In embodiments of the present invention, each module may be executed within the same computing device or may be executed on different computing devices and interconnected via a network. For example, language model calling and embedding vector generation may be performed via an external API, and data storage and retrieval may be performed via a database or a search engine. The data storage may be implemented as a relational database, a document database, a vector database, a search engine, a file system, or a combination thereof.

[0065] A method according to an embodiment of the present invention may be implemented as a computer program, and said computer program may be stored on a computer-readable recording medium. When said computer program is executed by one or more processors, said processors may perform each step of the method for constructing a duplicate recognition search evaluation dataset according to an embodiment of the present invention. A computer-readable recording medium may include an electronic, magnetic, optical, or semiconductor-based storage medium, and may include, for example, a hard disk, SSD, memory, ROM, RAM, flash memory, optical disk, or a storage medium accessible via a network.

[0066] Hereinafter, the specific configuration of the method and system for constructing a duplicate recognition search evaluation dataset according to an embodiment of the present invention will be described with reference to the attached drawings.

[0068] FIG. 1 is a diagram illustrating the overall flow of a redundancy recognition search evaluation dataset construction framework according to an embodiment of the present invention. As shown in FIG. 1, the framework according to an embodiment of the present invention may largely include a Valid Information Selection step, a Redundancy Tracking step, a Question-Answer Generation step, and a RedQA Dataset generation step.

[0069] First, in the valid information selection step, the input document or input corpus is divided into document fragments, and atomic information can be extracted from each document fragment. Here, a document fragment may be a text unit that serves as a search or evaluation unit of the searcher, and may be, for example, text divided into paragraphs, sentences, pages, or a predetermined number of tokens. The atomic information may be information in the form of a single factual unit containing a single indivisible factual claim.

[0070] In the valid information selection step of Fig. 1, validity filtering may be performed on atomic information extracted from document fragments. Validity filtering may be a process of determining whether atomic information can be used as a component of a search evaluation data item. For example, it may be determined whether the atomic information can be understood without external context, whether it contains only a single factual claim, or whether it contains substantive information capable of supporting query or answer generation. Information missing essential elements, fragmentary information, meta-information, and trivial or low-evaluation-value information may be excluded during the validity filtering process.

[0071] Atomic information that has passed validity filtering can undergo an information ranking process. In the information ranking process, multiple quality criteria can be evaluated separately by criterion. For example, independent evaluations can be performed for each criterion, such as validity, completeness, specificity, clarity, and questionability. The evaluation results for each criterion are converted into rankings, and the rankings for each criterion can be integrated using a mutual ranking fusion method. Accordingly, top atomic information suitable for constructing a search evaluation dataset from each document fragment or the entire pool of valid information can be selected as target atomic information.

[0072] Next, in the duplicate tracking step, equivalent factual information existing in other document fragments can be detected for the target atomic information selected in the valid information selection step. The duplicate tracking step illustrated in FIG. 1 allows for a more precise determination of whether the same fact is repeated than a coarse similarity judgment at the document fragment level by tracking duplicates at the atomic information level. That is, even if an entire document fragment is not completely identical to another document fragment, if specific atomic information within it represents a fact that is identical or substantially identical to the atomic information of another document fragment, the atomic information can be determined as duplicate or equivalent fact.

[0073] The duplicate tracking step may include candidate search based on embedding similarity and equivalence determination based on a language model. In one embodiment, target atomic information and atomic information contained in other document fragments may each be converted into embedding vectors by an embedding model. Subsequently, the similarity between the embedding vector of the target atomic information and the embedding vectors of other atomic information is calculated, and atomic information with a similarity greater than or equal to a predetermined threshold may be selected as candidate atomic information. This candidate search based on embedding similarity may be a primary search process to increase recall.

[0074] Subsequently, a language model-based equivalence determination can be performed on the candidate atomic information. The language model can determine whether the target atomic information and the candidate atomic information represent the same factual content, even if their expressions differ. In this case, at least one of semantic equivalence, logical equivalence, or factual equivalence may be considered. For example, if two pieces of atomic information represent the same entity, the same attribute, the same relationship, the same time point, or the same quantitative value, they can be determined to be equivalent facts. Candidate atomic information that passes the equivalence determination can be configured as an equivalent fact set together with the target atomic information.

[0075] Next, in the query-answer generation step, a query candidate can be generated based on valid atomic information or an equivalent set of facts. In the query-answer generation step illustrated in FIG. 1, a plurality of atomic information may be selected as basis information for query generation, and one or more query candidates based on the selected atomic information may be generated by a language model. In one embodiment, a plurality of atomic information having interconnectedness may be selected to generate a multi-hop query. In this case, the generated query may be a query in which the answer can be derived by searching or combining the plurality of atomic information together.

[0076] The generated query candidates may undergo a logical filtering process. Logical filtering is a process of verifying whether the generated query is suitable as a search evaluation data item. For example, it may be determined whether the query does not depend on specific document structures such as tables, pages, or sentence numbers; whether the query itself contains correct answers or intermediate inference clues; whether the range of information required by the query corresponds exactly to the range of atomic information provided; whether the query expression is clear; and whether the correct answer can be derived solely from the given evidence. In one embodiment, a query candidate that does not satisfy any one of the multiple logical verification criteria may be discarded.

[0077] Query ranking can be performed on query candidates that have passed logical filtering. During the query ranking process, query quality criteria such as connectivity, fluency, keyness, and validity can be evaluated by criterion. The evaluation results for each criterion are converted into rankings, and the rankings by criterion can be integrated using a ranking fusion method. The query candidate with the highest integrated score can be selected as the representative query. Along with this, the correct answer corresponding to the query can be generated or finalized.

[0078] In the final stage of FIG. 1, the representative query, correct answer, and evidence generated through the process described above can be stored as search evaluation data items. At this time, the evidence may not consist solely of a single source document fragment, but may consist of an evidence group that includes source evidence and alternative evidence containing factual content equivalent to the source evidence. Accordingly, the final search evaluation dataset may consist of a set of data items including a query, a correct answer, and a duplicate cognitive evidence group.

[0079] Such search evaluation datasets can be used to evaluate the performance of a searcher. In conventional search evaluation methods, if a searcher fails to retrieve the initially designated correct document fragment, it may be evaluated as incorrect. However, in an embodiment of the present invention, even if the searcher fails to directly retrieve the source evidence, if it retrieves alternative evidence included in the same evidence group, the evidence group may be determined to be covered. Accordingly, the actual search performance of the searcher in unique and highly redundant corpora can be evaluated more fairly and realistically.

[0080] The framework illustrated in FIG. 1 can be implemented as program code executed by one or more processors. For example, the processes of valid information screening, duplicate tracking, query-correct answer generation, and search evaluation dataset storage can each be implemented as software modules, and each module can be executed on the same server or on multiple distributed servers. Additionally, at least some of atomic information extraction, evaluation by quality criteria, equivalence determination, query generation, and logical filtering can be performed through an external language model API or a local language model, and embedding similarity-based candidate search can be performed using an embedding model and a vector search system.

[0082] FIG. 2 is a diagram illustrating examples of judgment criteria used in valid information filtering, valid information ranking, query logic filtering, and query ranking according to an embodiment of the present invention.

[0083] As illustrated in FIG. 2, the method for constructing a duplicate recognition search evaluation dataset according to an embodiment of the present invention may not merely generate atomic information or query candidates, but may include a process of verifying and ranking whether the generated atomic information or query candidates are suitable as search evaluation data items according to a plurality of criteria. Through this, it is possible to prevent items that are incomplete, ambiguous, or unsuitable for search evaluation from being included in the dataset among the results generated by the language model.

[0084] Figure 2 (A) illustrates an example of valid information filtering. Valid information filtering is a process of determining whether atomic information extracted from document fragments satisfies the minimum requirements for use in constructing a search evaluation dataset. In one embodiment, valid information filtering can determine whether atomic information is complete information that can be understood without external context, contains a single independent factual claim, or contains substantial information that can support query or correct answer generation.

[0085] For example, if atomic information can be understood independently by clearly including a specific person, organization, document, period, number, event, or relationship, such atomic information may be judged as valid information. On the other hand, information that lacks essential elements such as subject, object, time, place, or manner, information with incomplete meaning due to the separation of only parts of a sentence, meta-information such as document titles, page numbers, table numbers, URLs, etc., or trivial information that has low practical utility in search evaluation may be judged as invalid information and excluded.

[0086] Figure 2 (B) illustrates an example of a quality criterion used for ranking valid information. In an embodiment of the present invention, an evaluation based on a plurality of quality criteria may be performed to select atomic information that is more suitable for constructing a search evaluation dataset even among atomic information that has passed validity filtering. The quality criteria may include, for example, validity, completeness, specificity, clarity, and questionability.

[0087] Validity may indicate whether the atomic information is substantial and meaningful knowledge capable of supporting search queries or the generation of correct answers. Completeness may indicate whether the atomic information can be understood self-sufficiently without external context. Specificity may indicate whether the atomic information includes specific entity names, dates, numbers, places, attributes, or relationships. Clarity may indicate whether the atomic information does not contain ambiguous pronouns, unclear indicators, or expressions that can be interpreted in multiple senses. Questionability may indicate whether the atomic information can be converted into a natural language query with a clear correct answer.

[0088] In one embodiment, each quality criterion may be evaluated independently of one another. For example, a language model may perform a first evaluation that judges only validity, a second evaluation that judges only completeness, a third evaluation that judges only specificity, a fourth evaluation that judges only clarity, and a fifth evaluation that judges only questionability for a single piece of atomic information. Subsequently, the evaluation results for each criterion may be converted into a ranking by criterion, and the rankings by criterion may be integrated through mutual ranking fusion. Accordingly, atomic information that is generally superior across multiple quality criteria may be selected as the target atomic information.

[0089] Figure 2 (C) illustrates an example of a logical validation criterion used for query logic filtering. Query logic filtering is a process of determining whether a query candidate generated by a language model is logically valid to be used as a search evaluation data item. In one embodiment, query logic filtering may be performed according to a plurality of criteria including contextual independence, prohibition of revealing the correct answer, information equivalence, item clarity, and correct answer probability.

[0090] Contextual independence may indicate whether a query can be understood independently without relying on specific document structures or external contexts, such as “the table above,” “the sentence above,” “this document,” or “the relevant page.” The prohibition of revealing the answer may indicate whether the query itself contains the correct answer or the intermediate inference results necessary to derive the answer. Informational equivalence may indicate whether the scope of information required by the query exactly corresponds to the scope of the provided atomic information or evidentiary information. Item clarity may indicate whether the query does not contain ambiguous references, incomplete comparisons, unclear objects, or polysemous expressions. Possibility of a correct answer may indicate whether a clear and unique correct answer to the query can be derived solely from the provided document fragments or evidentiary information.

[0091] In particular, information equivalence may serve as a criterion to prevent a generated query from requiring excessive information beyond the provided evidence information, or from being reduced to a simple query that can be answered with only a portion of the provided evidence information. For example, when generating a multi-hop query based on multiple atomic information, the query must require all of the selected multiple atomic information without requiring external information that exceeds that scope. Accordingly, a single-hop query disguised as a multi-hop query or a query that cannot be answered with only the given evidence may be eliminated.

[0092] In one embodiment, query logic filtering can be performed using a zero-tolerance method. That is, if a query candidate does not satisfy any of the multiple logic verification criteria, the query candidate may be excluded from the search evaluation data items. Through this, queries included in the search evaluation dataset can be maintained in a state where they are context-independent, do not reveal the correct answer, correspond exactly to the provided evidence, are clear, and can be answered solely with the given evidence.

[0093] Figure 2 (D) illustrates an example of a query quality criterion used for query ranking. When there are multiple query candidates that have passed logical filtering, an embodiment of the present invention may perform an evaluation based on query quality criteria to select a representative query among the query candidates. The query quality criteria may include, for example, connectivity, fluency, keyness, and validity.

[0094] Connectivity can indicate whether a query naturally links logical relationships between multiple atomic pieces of information to form a multi-hop inference structure. Fluency can indicate whether the query possesses natural and smooth expressions that an actual user would likely query in natural language. Coreity can indicate whether the query removes unnecessary modifiers or incidental information and focuses on key information. Validity can indicate whether the query possesses substantial value and a legitimate query motivation as a search evaluation data item.

[0095] In one embodiment, query quality criteria may also be evaluated separately by criterion. For each query candidate, evaluations regarding connectivity, fluency, keyness, and validity are performed independently, and a ranking may be calculated based on the evaluation results for each criterion. Subsequently, the rankings by criterion may be combined using a mutual ranking fusion method, and the query candidate with the highest integrated score may be confirmed as the representative query.

[0096] The judgment criteria illustrated in FIG. 2 are merely examples for explaining an embodiment of the present invention, and the scope of the present invention is not limited to the specific criteria or specific examples illustrated in FIG. 2. For example, quality criteria used for valid information ranking, logic verification criteria used for query logic filtering, or query quality criteria used for query ranking may be partially modified, added, or omitted depending on the domain, type of corpus, number of query hops, type of searcher, or purpose of evaluation.

[0097] However, according to an embodiment of the present invention, by separating and evaluating multiple criteria for atomic information and query candidates by criterion and fusing the evaluation results by criterion into ranks, a more stable and consistent search evaluation dataset can be constructed than a method relying on a single judgment or single score of a language model. In addition, by removing invalid atomic information and logically inappropriate query candidates in advance, the quality of the final search evaluation data items can be improved.

[0099] FIG. 3 is a graph illustrating the results of a performance comparison between a criterion-based separate evaluation and ranking fusion method according to an embodiment of the present invention. Specifically, FIG. 3 shows the results of a performance comparison between a method of evaluating multiple quality criteria together in a single prompt and a method of evaluating each quality criterion separately and then performing ranking fusion when evaluating candidate information using a language model.

[0100] In the graph shown in Fig. 3, the vertical axis represents the NDCG@3 value, and the horizontal axis represents the evaluation method and aggregation method. NDCG@3 is an indicator representing the ranking quality of the top three candidates, and a higher value may indicate that candidate information suitable for building a search evaluation dataset is placed at the top. In Fig. 3, for example, evaluation results using GPT5-Nano and GPT5 are shown, and aggregation results using Base, MinMax, and RRF methods are compared.

[0101] In FIG. 3, “Combined” may refer to a method of evaluating candidate information by considering multiple quality criteria together in a single prompt or a single judgment procedure. On the other hand, “Separate” may refer to a method of evaluating multiple quality criteria independently by separating them by criterion. For example, in the evaluation of atomic information, the criteria of validity, completeness, specificity, clarity, and questionability may be evaluated separately, and in the evaluation of query candidates, the criteria of connectivity, fluency, keyness, and validity may be evaluated separately.

[0102] Additionally, in FIG. 3, “Base” may refer to a method of averaging the evaluation scores by criterion as is or performing basic aggregation, and “MinMax” may refer to a method of averaging the scores by criterion after normalizing them. “RRF” may refer to a method of calculating a rank fusion score based on the inverse of the rank by criterion, rather than directly combining the evaluation results by criterion based on the magnitude of the scores. In one embodiment, the rank fusion score of the RRF method may be calculated based on the following formula.

[0103] s(x)=Σ(1 / (rank_i(x)+k))

[0104] Here, x is candidate information or query candidate to be evaluated, rank_i(x) is the rank of x at the i-th quality criterion, and k may be a constant greater than or equal to 0. In one embodiment, k may be set to 0, but is not limited thereto.

[0105] As shown in Figure 3, the method combining separate evaluation by criteria and RRF can show a higher NDCG@3 value than other evaluation and aggregation methods. For example, when using GPT5-Nano, the Separate+RRF method shows an NDCG@3 value of 0.463, and when using GPT5, the Separate+RRF method shows an NDCG@3 value of 0.467. This means that the method combining separate evaluation by criteria and ranking can provide superior candidate ranking performance compared to the Combined method, which evaluates criteria together, or the Base or MinMax methods based on average scores.

[0106] The reason for using separate evaluation by criterion in the embodiments of the present invention is that when a language model judges multiple quality criteria simultaneously, judgments between criteria may become mixed or results may be generated that are biased toward a specific criterion. For example, a single piece of candidate information may have high specificity because it contains specific numerical values, but may have low completeness because it is difficult to understand without external context. In this case, if multiple criteria are evaluated at once, the strengths and weaknesses of each criterion may not be clearly separated. On the other hand, by performing separate evaluation by criterion, evaluation results for each criterion can be obtained independently for each piece of candidate information, thereby allowing for a more stable judgment of the candidate's quality.

[0107] In addition, the reason for using rank fusion in the embodiments of the present invention is that the absolute values ​​of scores calculated by the language model may not have the same meaning for each criterion. For example, it is difficult to conclude that a score of 0.8 in the validity criterion and a score of 0.8 in the clarity criterion represent the same reliability or the same judgment strength. Therefore, in the embodiments of the present invention, rather than directly averaging the magnitudes of the scores for each criterion, the rankings of the candidates within each criterion are calculated, and the rankings for each criterion are fused to calculate an integrated score.

[0108] As such, the experimental results of Figure 3 demonstrate that the criterion-based prompt-based mutual ranking fusion, i.e., the CRRF method, of the present invention can improve the ranking quality of candidate information or query candidates during the process of constructing a language model-based dataset. In particular, in the embodiment of the present invention, this criterion-based separate evaluation and ranking fusion method can be applied in both the atomic information ranking stage and the query candidate ranking stage.

[0109] In one embodiment, in the atomic information ranking step, for each atomic information that has passed the validity filtering, an evaluation based on five criteria—validity, completeness, specificity, clarity, and questionability—may be performed. The evaluation result for each criterion is converted into a criterion-specific ranking, and the criterion-specific rankings may be fused using the RRF method. Accordingly, target atomic information suitable for duplicate detection and query generation may be selected from each document fragment or the entire valid information pool.

[0110] In another embodiment, in the query candidate ranking step, for each of the multiple query candidates that have passed logical filtering, an evaluation based on four criteria—connectivity, fluency, keyness, and validity—may be performed. The evaluation result for each criterion is converted into a ranking based on the criterion, and the rankings based on the criterion may be fused using the RRF method. Accordingly, among the logically valid query candidates, the representative query most suitable as a search evaluation data item may be selected.

[0111] The results illustrated in FIG. 3 relate to an embodiment using a specific language model and a specific dataset, and the scope of the present invention is not limited to the specific model name, specific numerical value, or specific evaluation data illustrated in FIG. 3. For example, in an embodiment of the present invention, a language model other than a GPT family model may be used, and MRR, Top-1, NDCG@k, or other ranking quality indicators may be used in addition to NDCG@3. Furthermore, the parameters of the RRF method, the method of calculating evaluation scores by criterion, the method of handling ties, and the number of candidates may be changed depending on the domain or implementation environment.

[0112] However, as illustrated in FIG. 3, when combining separate evaluation by criteria and ranking fusion, atomic information or query candidates suitable for constructing a search evaluation dataset can be selected more stably while mitigating the instability of judgment in the language model and the problem of score correction between criteria. Therefore, according to an embodiment of the present invention, when automatically constructing a duplicate recognition search evaluation dataset from a unique and highly redundant corpus, the quality and consistency of data items can be improved.

[0114] FIG. 4 is a graph illustrating the change in search performance by domain according to an increase in the number of hops of a query according to an embodiment of the present invention. Specifically, FIG. 4 shows how the PerfRecall@10 performance of the searcher changes in 1-hop to 4-hop queries for the General-Wiki, Patent, Finance, and Legal domains.

[0115] In Figure 4, the horizontal axis represents the number of hops for a query, and the vertical axis represents the PerfRecall@10 value. Here, the number of hops may refer to the number of different atomic information or evidence information required to derive the correct answer for a single query. For example, a 1-hop query may be a query that can be answered with only one atomic information or one evidence group, and a 2-hop query may be a query that can be answered by combining two atomic information or two evidence groups. Similarly, a 3-hop query and a 4-hop query may be queries that can be answered by retrieving or combining three or four atomic information or evidence groups, respectively.

[0116] PerfRecall@10 may be an indicator indicating whether all evidential information necessary to derive the correct answer to a query is included within the top 10 document fragments of the search results. In one embodiment, if the evidential information required for the query consists of multiple evidence groups, PerfRecall@10 may be evaluated as successful if the top 10 document fragments of the search results cover all of the multiple evidence groups. On the other hand, if the top 10 search results cover only some evidence groups and do not cover the remaining evidence groups, the query may be evaluated as having failed to achieve complete retrieval.

[0117] In an embodiment of the present invention, an evidence group may include source evidence and one or more alternative evidence. Therefore, when calculating PerfRecall@10, the searcher is not required to retrieve the initially designated source document fragment; the evidence group may be deemed covered even if any of the alternative evidence included in the same evidence group is retrieved. Accordingly, PerfRecall@10 according to an embodiment of the present invention can more realistically reflect the searcher's actual answer probability in unique and highly redundant corpora than conventional evaluation metrics centered on a single correct document fragment.

[0118] As illustrated in Figure 4, in the General-Wiki domain, even if PerfRecall@10 decreases as the number of hops increases, it may show a relatively gradual decreasing trend. For example, in the General-Wiki domain, PerfRecall@10 may appear as 100.0% at 1-hop, 96.0% at 2-hop, 84.9% at 3-hop, and 66.4% at 4-hop. This may mean that in low-redundancy or low-similarity environments, even if the number of hops increases, necessary evidence exists relatively distinguishably, so the searcher is relatively likely to retrieve multiple pieces of evidence together.

[0119] On the other hand, in unique and highly redundant domains such as Patent, Finance, and Legal, PerfRecall@10 may decrease sharply as the number of hops increases. For example, in the Patent domain, PerfRecall@10 may be 94.5% at 1-hop, 71.0% at 2-hop, 54.1% at 3-hop, and 27.9% at 4-hop. In the Finance domain, PerfRecall@10 may be 90.1% at 1-hop, 59.4% at 2-hop, 32.4% at 3-hop, and 8.5% at 4-hop. In the Legal domain, PerfRecall@10 may be 85.7% at 1-hop, 46.5% at 2-hop, 25.0% at 3-hop, and 5.0% at 4-hop.

[0120] These results demonstrate that the difficulty of searching for multi-hop queries can significantly increase in unique-word and highly redundant corpora. Since unique-word corpora contain numerous similar document fragments, the top results of a search engine may be biased toward similar document fragments related to specific facts. In this case, the search engine may repeatedly retrieve some evidence necessary for the query but fail to include other groups of evidence required to derive the correct answer in the top results. In other words, a problem may arise where the top K search results are filled with similar or redundant evidence rather than containing a diverse range of complementary evidence.

[0121] Furthermore, in highly redundant corpora, identical or equivalent factual information may be repeated across multiple document fragments. While this redundancy can increase the probability of a search success by providing alternative evidence, it can also hinder the balanced retrieval of multiple evidence groups by causing the searcher to repeatedly retrieve various document fragments containing the same facts. Particularly in multi-hop queries, since different factual information or different evidence groups are all required to derive the correct answer, the PerfRecall@10 can be significantly degraded if only duplicate document fragments related to a specific evidence group are repeatedly retrieved.

[0122] The results of Figure 4 support the necessity of a redundant cognitive search evaluation dataset according to an embodiment of the present invention. In a conventional evaluation method centered on a single correct document fragment, even if a searcher recovers alternative evidence equivalent to the source evidence, it may not be recognized as correct evidence. In this case, the performance of the searcher in a highly redundant corpus may be evaluated as lower than it actually is. On the other hand, in an embodiment of the present invention, an equivalent fact set is constructed at the atomic information level, and the source evidence and alternative evidence are managed as a single evidence group, so that even if the searcher recovers alternative document fragments belonging to the same evidence group, the evidence can be evaluated as having been recovered.

[0123] In addition, the results of Figure 4 show that it is difficult to sufficiently evaluate the actual RAG performance in a unique and highly redundant environment based solely on the average related document retrieval performance of the searcher. In multi-hop queries, it is important not only to retrieve a single related document fragment but also to retrieve all multiple evidence groups necessary for deriving the correct answer. Therefore, in an embodiment of the present invention, the performance of the searcher can be evaluated using Coverage@K, which indicates how much the search result covers each evidence group, and PerfRecall@K, which indicates whether all essential evidence groups have been covered.

[0124] In one embodiment, Coverage@K may be calculated as the ratio of the evidence group covered by the top K document fragments of the search results to the total evidence group required for the query. For example, if 4 evidence groups are required for one query and the top 10 document fragments of the search results cover 2 of those evidence groups, Coverage@10 may be calculated as 50%. On the other hand, PerfRecall@K may be calculated as success if the top K document fragments of the search results cover all evidence groups required for the query. Accordingly, Coverage@K may represent partial recovery performance, and PerfRecall@K may represent full recovery performance.

[0125] According to the embodiment illustrated in FIG. 4, as the number of hops increases, PerfRecall@10 decreases sharply, particularly in the Finance, Patent, and Legal domains; thus, it can be confirmed that recovering all multiple evidence groups in unique and highly redundant domains is a difficult technical challenge. To solve this technical challenge, the embodiment of the present invention combines atomic information-based redundancy tracking, the construction of equivalent fact sets, evidence group-based evaluation labeling, and redundancy recognition evaluation indicators.

[0126] Specifically, embodiments of the present invention do not construct evaluation labels based solely on the relevance of the entire document fragment, but rather track identical or equivalent factual information based on atomic information within the document fragment. Additionally, multiple document fragments containing equivalent factual content are grouped and stored as a single evidence group. Accordingly, even if a searcher retrieves a substitute document fragment instead of the source document fragment, the corresponding evidence group can be evaluated as covered, and the retrieval status of each of the multiple evidence groups required in a multi-hop query can be precisely determined.

[0127] The graph shown in FIG. 4 illustrates results obtained under specific domains, specific search engines, and specific experimental conditions. Therefore, the scope of the present invention is not limited to the specific numerical values, specific domains, or specific number of hops shown in FIG. 4. For example, embodiments of the present invention may be applied not only to 1-hop to 4-hop queries but also to queries with more hops, and may be applied to various unique and highly redundant corpora such as medical, scientific and technological, corporate internal knowledge management, customer support documents, product manuals, or regulatory documents, in addition to financial, patent, and legal domains.

[0128] In addition, although PerfRecall@10 was used as an example in FIG. 4, embodiments of the present invention are not limited thereto. For example, the K value targeting the top K search results can be changed to 1, 3, 5, 20, 50, etc., and the evaluation indicator may include Coverage@K, PerfRecall@K, MRR, NDCG@K, or a combination thereof. In addition, the unit of the search result may be a document fragment, paragraph, sentence, atomic information, file, or a combination thereof.

[0129] As such, Figure 4 shows that the complete evidence retrieval performance of a searcher can rapidly deteriorate as the number of hops increases in a unique and highly redundant corpus, and indicates that the redundancy cognitive search evaluation dataset construction and evidence group-based evaluation method according to an embodiment of the present invention are useful for fairly and realistically evaluating the performance of a searcher in an actual industrial environment.

[0131] FIG. 5 is a block diagram of a duplicate recognition search evaluation dataset construction system according to one embodiment of the present invention.

[0132] As illustrated in FIG. 5, a duplicate recognition search evaluation dataset construction system (100) according to one embodiment of the present invention may include a data preprocessing module (110), an atomic information extraction module (120), a validity verification module (130), a ranking module by criteria (140), a duplicate detection module (150), a query generation module (160), a logic verification module (170), a final ranking module (180), an evaluation module (190), and a data storage module (195).

[0133] The above-described duplicate recognition search evaluation dataset construction system (100) may be implemented as a computing device including one or more processors and memory. For example, the system (100) may be implemented as a single server, multiple servers, a cloud server, a GPU-based server, a local workstation, a distributed computing environment, or a combination thereof. Each module may be implemented as program code executed by one or more processors, and some or all of the functions of each module may be implemented in software, hardware, or a combination thereof.

[0134] The data preprocessing module (110) receives an input corpus and can divide the input documents included in the input corpus into multiple document fragments. For example, the data preprocessing module (110) can extract text data by parsing PDF documents, word processor documents, HTML documents, text documents, database records, or a combination thereof. Additionally, the data preprocessing module (110) can divide the extracted text data into multiple document fragments according to sentence units, paragraph units, page units, clause units, token count units, or a combination thereof.

[0135] In one embodiment, the data preprocessing module (110) may assign a document fragment identifier to each of the document fragments. The document fragment identifier may include a document name, a page number, a fragment number, domain information, or a combination thereof. The document fragment identifier may subsequently be used to track which document fragment the atomic information, equivalence fact set, evidence group, and search evaluation data item originated from.

[0136] The atomic information extraction module (120) can extract atomic information of a single fact unit from each of the plurality of document fragments generated by the data preprocessing module (110). The atomic information extraction module (120) can use a language model (200) to decompose the plurality of fact claims contained in the document fragments into one or more atomic information. Here, one atomic information can be generated to include only one indivisible fact claim.

[0137] For example, if a single document fragment contains facts regarding multiple companies, multiple figures, or multiple events, the atomic information extraction module (120) can separate them into independent fact units. Accordingly, the duplicate detection module (150) can then precisely determine whether identical or equivalent facts are repeated at the atomic information level rather than at the entire document fragment.

[0138] The validation module (130) can select atomic information suitable for constructing a search evaluation dataset from among the atomic information extracted by the atomic information extraction module (120) as valid atomic information. The validation module (130) can determine the validity of atomic information based on atomicity, validity, non-ambiguity, self-sufficiency, non-obviousness, factuality, or a combination thereof.

[0139] Additionally, the validation module (130) may exclude atomic information corresponding to at least one of missing essential elements, fragmented information, information that is insufficient in context, information that lacks utility, and meta-information. For example, information that cannot be understood independently because the subject, object, time, place, or manner is missing, information that cannot form a complete fact because only a part of the sentence is extracted, and meta-information such as document titles or page numbers may be excluded from valid atomic information.

[0140] The criterion-based ranking module (140) can perform evaluations based on multiple quality criteria on valid atomic information selected by the validity verification module (130), and determine target atomic information by ranking fusion of the evaluation results based on the multiple quality criteria. In one embodiment, the multiple quality criteria may include validity, completeness, specificity, clarity, and questionability.

[0141] The criterion-based ranking module (140) can evaluate each quality criterion independently using a language model (200). For example, validity evaluation, completeness evaluation, specificity evaluation, clarity evaluation, and questionability evaluation can be performed by different prompts or different judgment procedures. Subsequently, the criterion-based ranking module (140) converts the evaluation results for each quality criterion into a criterion-based ranking, and combines the criterion-based rankings in a mutual ranking fusion manner to calculate an integrated ranking of valid atomic information.

[0142] The duplicate detection module (150) can construct an equivalent fact set by performing candidate search based on embedding similarity and equivalence determination based on a language model for the target atom information determined by the standard ranking module (140). Specifically, the duplicate detection module (150) can convert the target atom information and the comparison target atom information into embedding vectors using an embedding model (210). Subsequently, the duplicate detection module (150) can calculate the similarity between the embedding vector of the target atom information and the embedding vectors of the comparison target atom information.

[0143] In one embodiment, the similarity may be cosine similarity. The duplicate detection module (150) may select atomic information whose cosine similarity is greater than or equal to a predetermined threshold as candidate atomic information. The candidate atomic information may be selected from among atomic information contained in other document fragments, excluding atomic information contained in the same document fragment as the target atomic information.

[0144] Subsequently, the duplicate detection module (150) can perform an equivalence determination between target atomic information and candidate atomic information using a language model (200). The equivalence determination can be performed based on at least one of semantic equivalence, logical equivalence, and factual equivalence. For example, if two atomic information represents the same entity, the same attribute, the same relationship, the same time, the same place, or the same quantitative value, the duplicate detection module (150) can determine the two atomic information as equivalent facts.

[0145] The duplicate detection module (150) can include candidate atomic information determined to be equivalent in an equivalent fact set corresponding to target atomic information. Accordingly, multiple atomic information containing substantially identical factual content, although expressed differently, can be managed as a single equivalent fact set.

[0146] The query generation module (160) can generate a query candidate based on at least one of valid atomic information, target atomic information, or an equivalent fact set. In one embodiment, the query generation module (160) may sample a plurality of atomic information among the valid atomic information and select M atomic information that have connectivity among the sampled plurality of atomic information. Here, M may represent the number of hops of the query.

[0147] The query generation module (160) can generate one or more query candidates based on a common basis of selected M atomic information. The query candidates may be single-hop queries that can be answered with only one atomic information, or multi-hop queries that can be answered by using multiple atomic information together. The query generation module (160) can generate multiple query candidates in the form of natural language using a language model (200).

[0148] The logic verification module (170) can perform verification according to a plurality of logic verification criteria on query candidates generated by the query generation module (160). In one embodiment, the plurality of logic verification criteria may include context independence, prohibition of revealing the correct answer, information equivalence, question clarity, and correct answer possibility.

[0149] Contextual independence may indicate whether a query can be understood independently without relying on specific document structures, tables, drawings, page numbers, or external context. The prohibition of revealing the answer may indicate whether the query itself contains the correct answer or the intermediate inference results necessary to derive the answer. Information equivalence may indicate whether the scope of information required by the query exactly corresponds to the scope of selected atomic or evidentiary information. Item clarity may indicate whether the wording of the query is unambiguous. Correctness may indicate whether a clear correct answer can be derived solely from the provided atomic or evidentiary information.

[0150] In one embodiment, the logic verification module (170) can filter query candidates using a zero-tolerance method. That is, if a query candidate does not satisfy any of the multiple logic verification criteria, the logic verification module (170) can discard the query candidate. Accordingly, the logical validity and correctness of the query included in the search evaluation dataset can be improved.

[0151] The final ranking module (180) can determine a representative query by ranking and fusing evaluation results for multiple query quality criteria for query candidates that have passed the logic verification module (170). In one embodiment, the query quality criteria may include connectivity, fluency, keyness, and validity. The final ranking module (180) can independently evaluate the query quality criteria for each query candidate, calculate a ranking for each criterion, and then mutually rank and fuse the rankings for each criterion to calculate a query ranking fusion score. The query candidate with the highest query ranking fusion score may be determined as the representative query.

[0152] The data storage (195) may store search evaluation data items including a representative query, a correct answer, and an evidence group. The evidence group may include source evidence and alternative evidence. Source evidence may be a document fragment or atomic information directly used to generate the query or correct answer, and alternative evidence may be another document fragment or atomic information containing factual content equivalent to the source evidence.

[0153] In one embodiment, the data store (195) may store a query identifier, a representative query, a correct answer, an evidence group, a source document identifier, a document fragment identifier, an atomic information identifier, a domain information, a hop count, a quality score, a rank fusion score, or a combination thereof. Additionally, the data store (195) may store the document fragments, atomic information, equivalence fact sets, query candidates, logical verification results, and evaluation results of the input corpus together with the search evaluation dataset.

[0154] The evaluation module (190) can evaluate the performance of a searcher using a search evaluation dataset stored in a data repository (195) and the search results of the searcher (300) to be evaluated. Specifically, the evaluation module (190) inputs a representative query into the searcher (300) to be evaluated and obtains the top K search results from the searcher (300). Subsequently, the evaluation module (190) can determine whether the top K search results cover each evidence group included in the search evaluation data item.

[0155] The above-mentioned coverage status can be determined on an evidence group basis. That is, if either the source evidence or the substitute evidence included in a single evidence group is included in the top K search results, the evaluation module (190) can determine that the evidence group is covered. Accordingly, even if the evaluation target searcher (300) cannot directly retrieve the source evidence, if it retrieves substitute evidence containing the same factual content, it can recognize that evidence as correct evidence.

[0156] The evaluation module (190) can calculate Coverage@K, which represents the proportion of covered evidence groups. Additionally, the evaluation module (190) can calculate PerfRecall@K, which represents whether all evidence groups required for a query are covered by the top K search results. Furthermore, the evaluation module (190) can calculate MRR, NDCG@K, Top@K, Precision@K, Recall@K, or a combination thereof.

[0157] The language model (200) may be linked with at least one of an atomic information extraction module (120), a validity verification module (130), a ranking module by criteria (140), a duplicate detection module (150), a query generation module (160), a logic verification module (170), and a final ranking module (180). For example, the language model (200) may extract atomic information from document fragments, determine the validity of atomic information, calculate a score for atomic information by quality criteria, determine equivalence between candidate atomic information and target atomic information, generate query candidates and correct answers, or perform logic verification and quality evaluation of query candidates.

[0158] The embedding model (210) can be linked with the duplicate detection module (150). The embedding model (210) can convert target atomic information and comparison target atomic information into embedding vectors, and the duplicate detection module (150) can select candidate atomic information based on the similarity between the embedding vectors. In one embodiment, the embedding model (210) can be implemented as an internal component of the system (100), and in another embodiment, it can be provided through an external API or an external server.

[0159] The evaluation target searcher (300) may be a target for performance evaluation using a search evaluation dataset constructed by the system (100) of the present invention. The evaluation target searcher (300) may be a BM25-based searcher, an embedding-based dense searcher, a language model-based searcher, a search pipeline including a reranker, a search module included in a RAG system, or a combination thereof. The evaluation target searcher (300) may be an internal component of the system (100) or an external system.

[0160] In FIG. 5, the language model (200), the embedding model (210), and the evaluation target searcher (300) may be shown outside the system (100), but the scope of the invention is not limited thereto. In other embodiments, at least one of the language model (200), the embedding model (210), and the evaluation target searcher (300) may be included as an internal component of the system (100). Additionally, the data flow and processing order between the modules are not limited to the example shown in FIG. 5, and some steps may be performed in parallel or repeatedly depending on the implementation environment or processing purpose.

[0161] According to the system (100) illustrated in FIG. 5, atomic information, valid atomic information, target atomic information, equivalent fact sets, query candidates, representative queries, correct answers, and evidence groups can be sequentially generated and stored from an input corpus. Additionally, using the stored evidence groups, the searcher (300) to be evaluated can recognize alternative evidence as correct answer evidence even when it recovers not only source evidence but also alternative evidence. Accordingly, the actual search performance of the searcher in a unique and highly redundant corpus can be evaluated more fairly and realistically.

[0163] The search evaluation dataset constructed according to the above-described embodiment may be composed of a plurality of search evaluation data items including a query, a correct answer, and an evidence group. Here, the evidence group may include a plurality of document fragments satisfying a single essential evidence, and the plurality of document fragments may contain identical or substantially identical factual content even if they exist in different documents or different locations. Accordingly, if the search result of the searcher includes any one of the document fragments included in the evidence group, the evidence group may be determined to have been recovered.

[0164] In one embodiment, a search evaluation data item may include at least one of a query identifier, a representative query, a correct answer, one or more evidence groups, a source document identifier, a document fragment identifier, an atomic information identifier, domain information, a hop count, a query quality score, a ranking fusion score, and a log of a generation or verification process. For example, a search evaluation data item may be stored in the following structure.

[0165] {

[0166] "query_id": "query_001",

[0167] "question": "What is the trading symbol of the company led by Jensen Huang?"

[0168] "answer": "NVDA",

[0169] "gold_chunk_groups": [

[0170] [

[0171] "ANNUAL_REPORT_NVIDIA_2023_page018_chunk002",

[0172] "ANNUAL_REPORT_NVIDIA_2024_page028_chunk001"

[0173] ],

[0174] [

[0175] "ANNUAL_REPORT_NVIDIA_2023_page090_chunk001"

[0176] ]

[0177] ],

[0178] "domain": "finance"

[0179] }

[0180] In the above example, each internal list of gold_chunk_groups may represent a single evidence group. Multiple document fragments included within a single evidence group may be source evidence or substitute evidence satisfying the same essential evidence. Accordingly, if the top search result of the searcher includes any one of the document fragments included in the first evidence group, the first evidence group may be determined to be covered, and if the top search result of the searcher includes a document fragment included in the second evidence group, the second evidence group may be determined to be covered.

[0181] In one embodiment, the search evaluation dataset may be stored in a structure including a metadata area, a query area, and a corpus area. The metadata area may include the total number of queries, the total number of document fragments, the total number of evidence, the average number of evidence per query, domain information, or dataset type. The query area may include a query identifier, the query phrase, the correct answer, and the evidence group for each query. The corpus area may include document fragment content and source information for the document fragments by document fragment identifier.

[0182] According to this data structure, for a single query, instead of designating only a single correct document fragment as correct evidence, multiple document fragments containing the same factual content can be managed as a single evidence group. Accordingly, even if the searcher fails to recover the source evidence used to generate the initial query, if it recovers alternative evidence included in the same evidence group, it can be evaluated as having recovered the corresponding essential evidence.

[0183] In one embodiment, the evaluation module can calculate the performance of the searcher to be evaluated using a search evaluation dataset. The evaluation module inputs a representative query into the searcher to be evaluated and obtains the top K search results from the searcher to be evaluated. Subsequently, the evaluation module can determine whether the top K search results each cover one or more evidence groups included in the search evaluation data items.

[0184] In one embodiment, Coverage@K may be calculated as the ratio of the evidence groups covered by the top K search results out of the total evidence groups required for a single query. For example, if four evidence groups are required for a single query and the top K search results cover two of those evidence groups, Coverage@K may be calculated as 50%. Coverage@K may indicate how much of the evidence required to derive the correct answer has been partially recovered by the searcher.

[0185] In one embodiment, PerfRecall@K may be calculated based on whether all evidence groups required for a query are covered by the top K search results. For example, if all evidence groups required for a query are covered by the top K search results, PerfRecall@K for that query may be calculated as success. Conversely, if one or more evidence groups are not covered, PerfRecall@K for that query may be calculated as failure. PerfRecall@K may indicate whether the searcher has completely recovered all evidence required to generate the correct answer.

[0186] Additionally, the evaluation module may further calculate MRR, NDCG@K, Top@K, Precision@K, Recall@K, or a combination thereof. Even in this case, the determination of relevance may be performed at the level of evidence groups rather than at the level of a single document fragment. That is, if either source evidence or substitute evidence within the same evidence group is included in the search results, it may be determined that the relevant evidence corresponding to that evidence group has been recovered.

[0187] Table 1 is a table showing domain-specific statistics of a duplicate awareness search evaluation dataset, e.g., the RedQA dataset, according to one embodiment of the present invention. In Table 1, the number of chunks may refer to the number of document fragments used as search units in each domain corpus, and the number of samples may refer to the number of search evaluation data items including queries, correct answers, and evidence groups. Additionally, the duplication may indicate the degree to which identical or equivalent factual content exists in other document fragments, and the similarity may refer to the embedding-based average similarity between document fragments.

[0188] Corpus Chunk count Redundancy (%) Similarity (%) Sample count Finance 3,281 63.2 35.1 645 Patent 3,339 49.7 29.0 480 Legal 4,090 25.1 40.7 605 General-Wiki 3,218 1.4 8.8 688

[0190] As shown in Table 1, the Finance, Patent, and Legal domains may have high overlap or high similarity compared to the General-Wiki domain. For example, the Finance domain may show 63.2% overlap and 35.1% similarity, while the Patent domain may show 49.7% overlap and 29.0% similarity. The Legal domain may show 25.1% overlap and 40.7% similarity. On the other hand, the General-Wiki domain may show 1.4% overlap and 8.8% similarity.

[0191] As such, corpora used in actual corporate environments or specialized domains may have higher document similarity and information redundancy compared to general public question-and-answer benchmarks. Therefore, if a searcher is evaluated based solely on a single correct document fragment or a single piece of gold evidence, a problem may arise where the searcher treats alternative evidence as incorrect even though it has recovered sufficient alternative evidence to derive the correct answer. To solve this problem, embodiments of the present invention may organize equivalent fact sets at the atomic information level and store source evidence and alternative evidence together as an evidence group.

[0192] Table 2 is a table comparing the similarity and information redundancy between standard benchmarks and enterprise domain documents according to one embodiment of the present invention. In Table 2, standard benchmarks may include, for example, MS MARCO, HotpotQA, and FiQA, and enterprise domains may include, for example, Patent, Finance, and Legal.

[0193] division dataset Similarity (%) Redundancy (%) Standard Benchmarks MS MARCO 6.6 10.2 Standard Benchmarks HotpotQA 8.8 1.4 Standard Benchmarks FiQA 15.7 12.4 Enterprise Domains Patent 29.0 49.7 Enterprise Domains Finance 35.1 63.2 Enterprise Domains Legal 40.7 25.1

[0195] As shown in Table 2, standard benchmarks generally have low document similarity and information redundancy, whereas enterprise domains such as Patent, Finance, and Legal may have relatively high document similarity or information redundancy. For example, HotpotQA may show a similarity of 8.8% and redundancy of 1.4%, while Finance may show a similarity of 35.1% and redundancy of 63.2%. This difference implies that conventional general-purpose search evaluation datasets may not adequately reflect the unique and highly redundant characteristics of the actual industrial environment.

[0196] In an embodiment of the present invention, similarity may be calculated by converting each of a plurality of document fragments into an embedding vector and based on pairwise cosine similarity or average cosine similarity between said embedding vectors. For example, after calculating the embedding vectors of document fragments included in a corpus, the cosine similarity for different pairs of document fragments may be calculated, and the average value of the calculated cosine similarity may be calculated as the inter-document similarity of the corpus. However, the method of calculating similarity is not limited thereto, and inner product similarity, Euclidean distance-based similarity, BM25-based text similarity, sentence embedding-based similarity, or a combination thereof may be used.

[0197] Additionally, in an embodiment of the present invention, the redundancy can be calculated based on the results of redundancy detection at the atomic information unit level. For example, among all atomic information extracted within a corpus, the ratio of atomic information that exists with the same or equivalent factual content in other document fragments can be calculated as the redundancy. In this case, the existence of the same or equivalent factual content can be determined by combining embedding similarity-based candidate search and language model-based equivalence determination.

[0198] In one embodiment, when searching for candidates based on embedding similarity, atomic information corresponding to itself may be excluded from candidate atomic information, and atomic information included in the same document fragment as the target atomic information may also be excluded from candidate atomic information. Accordingly, duplicate detection can be performed to track equivalent factual information existing between different document fragments, rather than repeated representations within the same document fragment.

[0199] The figures in Tables 1 and 2 described above are merely examples to illustrate the effects of the present invention, and the scope of the present invention is not limited to a specific domain, a specific number of chunks, a specific degree of overlap, a specific degree of similarity, or a specific number of samples. Embodiments of the present invention may be applied to various corpora with high document similarity or information overlap, such as medical documents, scientific and technological literature, corporate internal knowledge documents, customer support documents, product manuals, regulatory documents, educational content, administrative documents, or combinations thereof, in addition to finance, law, and patents.

[0200] In one embodiment, the method for constructing a duplicate cognitive search evaluation dataset of the present invention may include at least one step among input document splitting, atomic information extraction, selection of valid atomic information, fusion of evaluation and ranking by criteria, construction of an equivalent fact set, generation of query candidates, logical verification, determination of a representative query, storage of search evaluation data items, and searcher evaluation. These steps may be performed sequentially by the components described in FIGS. 1 to 5, but the scope of the present invention is not limited to a specific order of execution. For example, some steps may be performed in parallel, some steps may be performed repeatedly, and some steps may be omitted depending on the domain or the purpose of constructing the dataset.

[0201] In one embodiment, the judgment or generation process performed by the language model may be performed on a prompt basis. For example, a prompt for extracting atomic information may instruct to extract atomic information containing only one factual claim from a document fragment. A prompt for validation may be configured to determine whether the atomic information corresponds to missing essential elements, fragmentation, lack of context, lack of utility, or meta-information. A prompt for quality evaluation may be configured to independently evaluate only one criterion among validity, completeness, specificity, clarity, or questionability.

[0202] In one embodiment, the evaluation result by criterion may be a score between 0 and 1 produced by a language model, but is not limited thereto. In another embodiment, the evaluation result by criterion may be produced as a high, medium, or low grade, a pass / fail result, a pairwise comparison result, or a relative preference ranking among candidates. When the evaluation result by criterion is produced in the form of a score, the ranking by criterion may be determined according to the descending order of the scores by criterion.

[0203] Additionally, the prompt for determining equivalence may be configured to determine whether target atomic information and candidate atomic information convey identical or substantially identical factual content. The prompt for query validation may be configured to determine whether a query candidate satisfies specific criteria among contextual independence, prevention of answer disclosure, informational equivalence, item clarity, and correctness. The prompt for query quality evaluation may be configured to independently evaluate only one criterion among connectivity, fluency, keyness, or validity.

[0204] In an embodiment of the present invention, evaluating multiple criteria separately is intended to improve the judgment stability of the language model. When a language model evaluates multiple criteria simultaneously, judgments between criteria may be mixed or a specific criterion may be excessively reflected. On the other hand, by performing separate evaluations by criterion, judgment results for each criterion can be obtained independently, and multiple criteria can be reflected in a balanced manner through subsequent ranking fusion.

[0205] In one embodiment, rank fusion can be performed based on the inverse of the rank for each criterion. The rank fusion score s(x) for evaluation target x can be calculated based on the following equation.

[0206] s(x)=Σ(1 / (rank_i(x)+k))

[0207] Here, x is the atomic information or query candidate to be evaluated, rank_i(x) is the rank of x at the i-th evaluation criterion, and k may be a constant greater than or equal to 0. In one embodiment, k may be set to 0, and in another embodiment, k may be set to a positive constant to adjust the influence of the rank by criterion. Additionally, the rank fusion score may be calculated as an average value divided by the number of criteria, and weights may be assigned by criteria as needed.

[0208] In embodiments of the present invention, tie handling can be performed in various ways. For example, multiple items having the same evaluation score by criterion may be assigned different consecutive ranks according to the input order. In another embodiment, items having the same evaluation score may be assigned a joint rank, and an average rank corresponding to that joint rank may be used. In yet another embodiment, a tiebreak based on the order of appearance within a document, an atomic information identifier, a time of creation, or a random value may be used.

[0209] The language model, embedding model, tokenizer, chunk size, similarity threshold, number of candidates, number of query candidates, number of hops, evaluation criteria, and ranking fusion parameters used in embodiments of the present invention may be set to exemplary values ​​and do not limit the scope of the present invention. For example, the chunk size may be set to 512 tokens, but may be set to 256 tokens, 1,024 tokens, or other sizes. The embedding similarity threshold may be set to 0.5, but may be automatically adjusted based on the average similarity of all atomic information pairs, the average similarity by domain, the top percentile, or a learned threshold.

[0210] Additionally, the number of target atomic information selected from a single document fragment can be 3, but it can be changed to 1, 5, or more values ​​depending on the characteristics of the corpus, the length of the document fragment, the number of atomic information, the size of the evaluation dataset, or the purpose of query generation. The number of atomic information sampled during query generation, the number of selected hops M, and the number of query candidates generated can also be changed depending on the implementation purpose.

[0211] A system according to an embodiment of the present invention may be linked with an external language model API, an external embedding model API, a local language model, a local embedding model, a vector database, a search engine, or a combination thereof. In one embodiment, the language model or embedding model exists outside the system, and the system may call the model via a network. In another embodiment, the language model or embedding model may be installed inside the system and executed in a local environment.

[0212] A data repository according to an embodiment of the present invention may be implemented as a relational database, a document database, a vector database, a search engine, a file system, or a combination thereof. For example, the data repository may store search evaluation data items in JSON format and may use a search engine or a vector database together for searching document fragments and atomic information. Additionally, the data repository may store execution logs of each step, judgment results of the language model, embedding similarity scores, equivalence determination results, query validation results, and final evaluation results together.

[0213] Embodiments of the present invention can be used not only for evaluating a searcher alone but also for evaluating the entire RAG pipeline. For example, the redundant cognitive search evaluation dataset of the present invention can be used to compare performance differences based on a searcher, a reranker, a generative model, or a combination thereof. In this case, by analyzing together whether the evidence group recovered by the searcher is covered and whether the answer generated by the generative model is correct, it is possible to distinguish which factor—search failure, evidence utilization failure, or generation failure—contributes to the degradation of overall RAG performance.

[0214] Embodiments of the present invention may be implemented in the form of a computer program. The computer program may be stored on a computer-readable recording medium and may be configured to perform at least one step among input document segmentation, atomic information extraction, selection of valid atomic information, evaluation by criteria and ranking fusion, construction of equivalent fact sets, generation of query candidates, logical verification, determination of representative query, storage of search evaluation data items, and searcher evaluation when executed by one or more processors.

[0215] The computer-readable recording medium may include electronic, magnetic, optical, or semiconductor-based storage media. For example, the computer-readable recording medium may include a hard disk, SSD, flash memory, ROM, memory card, optical disk, magnetic disk, or a storage medium accessible via a network. Additionally, the computer program may be executed on a server, a cloud system, a user terminal, an internal enterprise server, or in a distributed computing environment.

[0216] In this specification, a computer-readable recording medium may be a non-transient computer-readable recording medium that excludes the transient signal itself.

[0217] The embodiments described above are merely examples to illustrate the technical concept of the present invention, and the scope of the present invention is not limited to the said embodiments. A person skilled in the art to which the present invention pertains may make various changes to the order of each step, some components, judgment criteria, evaluation indicators, data storage structure, model type, or implementation environment within the scope of the technical concept of the present invention without departing from it. Such changes and variations may also be included within the scope of the rights defined by the claims of the present invention.

Claims

Claim 1 A method for constructing a redundant cognitive search evaluation dataset performed by at least one processor comprises: a step of dividing input documents included in an input corpus into a plurality of document fragments; a step of extracting atomic information of a single fact unit from each of the plurality of document fragments using a language model; a step of selecting atomic information satisfying a predetermined validity criterion among the extracted atomic information as valid atomic information; a step of performing independent evaluations according to a plurality of quality criteria for each of the valid atomic information and determining target atomic information by ranking and fusing the evaluation results according to the plurality of quality criteria; a step of sequentially performing embedding similarity-based candidate search and language model-based equivalence determination on the target atomic information to construct an equivalence fact set including one or more atomic information semantically equivalent to the target atomic information; a step of generating query candidates based on the target atomic information or the equivalence fact set and performing verification according to a plurality of logical verification criteria on the query candidates; and a step of determining a representative query by ranking and fusing the evaluation results according to a plurality of query quality criteria for the query candidates that have passed the plurality of logical verification criteria. A method for constructing a duplicate cognitive search evaluation dataset, comprising the step of storing a search evaluation data item including the above representative query, the answer to the above representative query, and one or more evidence groups necessary to derive the answer to the above representative query, wherein the evidence group includes source evidence used to derive the answer to the above representative query and alternative evidence containing factual content equivalent to the source evidence. Claim 2 A method for constructing a duplicate cognitive search evaluation dataset according to claim 1, wherein the step of dividing the input documents into a plurality of document fragments comprises: a step of parsing the input documents; a step of generating document fragments by dividing the parsed input documents based on the number of tokens while maintaining sentence boundaries; and a step of assigning a document fragment identifier to each document fragment, the document fragment identifier including a document identifier, a page identifier, and a fragment identifier. Claim 3 A method for constructing a redundant cognitive search evaluation dataset according to claim 1, wherein the step of extracting atomic information comprises the step of decomposing the content contained in each document fragment such that one atomic information contains only one indivisible factual claim, and the step of selecting valid atomic information comprises the step of filtering the extracted atomic information according to a criterion including atomicity, validity, and non-ambiguity. Claim 4 A method for constructing a duplicate cognitive search evaluation dataset according to claim 1, wherein the step of selecting valid atomic information includes the step of excluding atomic information corresponding to at least one of essential element missing, fragmented information, contextually insufficient information, information lacking utility, and meta-information, wherein the essential element missing includes cases where at least one of subject, object, time, place, and manner is omitted. Claim 5 In claim 1, the step of determining the target atomic information comprises: a step of calculating evaluation results for each of the valid atomic informations according to quality criteria of validity, completeness, specificity, clarity, and questionability; a step of calculating a rank for each quality criterion based on the evaluation results for each quality criterion; and a step of calculating a mutual rank fusion score based on the rank for each quality criterion, wherein the mutual rank fusion score is calculated based on the following formula: s(x)=Σ(1 / (rank_i(x)+k)) where x is the atomic information to be evaluated, rank_i(x) is the rank of atomic information x at the i-th quality criterion, and k is a constant greater than or equal to 0. Claim 6 A method for constructing a duplicate cognitive search evaluation dataset according to claim 1, wherein the step of constructing the equivalence fact set comprises: a step of calculating a cosine similarity between an embedding vector for the target atomic information and embedding vectors of atomic information included in a document fragment different from the target atomic information; a step of selecting atomic information whose cosine similarity is greater than or equal to a predetermined threshold as candidate atomic information, wherein atomic information identical to the target atomic information and atomic information included in a document fragment identical to the target atomic information are excluded from the candidate atomic information; and a step of performing an equivalence determination using a language model for each of the candidate atomic information and including the candidate atomic information determined to be equivalent to the target atomic information in the equivalence fact set. Claim 7 A method for constructing a duplicate cognitive search evaluation dataset according to claim 6, wherein the determination of equivalence using the language model includes the step of determining whether the target atom information and the candidate atom information satisfy at least one of semantic equivalence, logical equivalence, and factual equivalence, and the factual equivalence is determined based on whether at least one of an entity, attribute, relationship, time point, place, and quantitative value corresponds to each other. Claim 8 A method for constructing a duplicate cognitive search evaluation dataset according to claim 1, wherein the step of generating the query candidate and performing verification according to a plurality of logical verification criteria on the query candidate comprises: a step of sampling a plurality of atomic information among valid atomic information; a step of selecting M atomic information having connectivity among the sampled plurality of atomic information; a step of generating a plurality of query candidates based on the selected M atomic information as a common basis; and a step of applying logical verification criteria of context independence, prohibition of exposure of the correct answer, information equivalence, item clarity, and correct answer possibility to each of the plurality of query candidates, wherein a query candidate that does not satisfy any one of the logical verification criteria is discarded. Claim 9 A system for constructing a duplicate recognition search evaluation dataset for a corpus with high document similarity and information redundancy, comprising: a data preprocessing module that divides input documents into multiple document fragments; an atomic information extraction module that extracts atomic information of a single fact unit from each of the multiple document fragments; a validation module that selects valid atomic information satisfying predetermined validity criteria among the atomic information; a criterion-based ranking module that performs independent evaluations according to multiple quality criteria on the valid atomic information and rank-fuses the evaluation results according to the multiple quality criteria to determine target atomic information; a duplicate detection module that constructs an equivalent fact set by performing embedding similarity-based candidate search and language model-based equivalence determination on the target atomic information; a query generation module that generates query candidates based on the target atomic information or the equivalent fact set; a logic verification module that performs verification according to multiple logic verification criteria on the query candidates; and a final ranking module that determines a representative query by rank-fuses the evaluation results according to multiple query quality criteria on the query candidates that have passed the logic verification criteria. A system for constructing a duplicate cognitive search evaluation dataset, comprising a data repository for storing search evaluation data items including the representative query, correct answer, and evidence group, wherein the evidence group includes source evidence and alternative evidence containing factual content equivalent to the source evidence. Claim 10 A non-transient computer-readable recording medium, wherein a program is recorded thereon that enables a computer to perform a method for constructing a duplicate recognition search evaluation dataset according to any one of claims 1 to 8.

Images