Re-ranking system, method, and program for extracting passages having high relevance to query

Abstract

Disclosed are a re-ranking system, method, and program for extracting passages having high relevance to a query. The system comprises at least one processor and a memory for storing at least one instruction. The at least one processor executes the at least one instruction stored in the memory to execute re-ranking of a plurality of passages by performing a tournament sort a plurality of number of times. Performing the tournament sort each time may include: a step of pairing a specific number of first passages included in the passages to output a plurality of groups, evaluating the relevance between the query and the first passages included in each of the groups, extracting passages having high relevance up to a predetermined rank from among the first passages, and outputting the extracted passages as second passages; and a step of pairing a specific number of the second passages to output a plurality of groups, evaluating the relevance between the query and the second passages included in each of the groups, extracting passages having high relevance up to a predetermined rank from among the second passages, and outputting the extracted passages as third passages.

Description

Re-ranking system, method, and program for extracting paragraphs highly relevant to a query

[0001] The present invention relates to a re-ranking system, method, and program for extracting paragraphs highly related to a query, and more particularly, to a re-ranking system, method, and program for extracting paragraphs highly related to a query from among paragraphs included in a document stored in a database.

[0002] Recently, the importance of building universally applicable, efficient generative retrieval systems—more specifically, domain-specific, zero-shot, and unstructured knowledge-based retrieval systems—has been emphasized. One method for achieving this goal is to extract and re-rank passages highly relevant to a query.

[0003] Referring to Fig. 1, re-ranking proceeds in two steps. The first step (initial search) (12) searches for 100-1,000 documents (12a) relevant to the query in a large database (11) (e.g., 11M documents in Wikipedia, 8.8M documents in MS MARCO, etc.). For this purpose, statistical search methods such as BM25 or techniques such as Maximum Inner Product Search (MIPS) are used. The second step {re-ranking} (13) re-ranks the documents extracted from the initial search to extract 1-10 most relevant paragraphs (13a). This allows the output (14) of the paragraphs most relevant to each query to be selected, even if the query is input to a retriever or large-scale language model (LLM) that has not been fine-tuned for a specific domain.

[0004] Traditionally, cross-encoder models such as MonoT5 (Document Ranking with a Pretrained Sequence-to-Sequence Model, Rodrigo Nogueira et al., 2020) or RankT5 (RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses, Honglei Zhuang et al., 2022) have been used for reranking. However, these models are relatively incapable of comparing relevance because they use pointwise reranking of each document. To improve these shortcomings, a listwise re-ranking model using a large-scale language model (LLM) (The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models, Ronak Pradeep 3, 2021.) was developed, but it has a disadvantage of low efficiency due to the large parametric model size and causes the problem of lost in the middle (lost in the middle means that the language model cannot understand information about documents located in the middle because the location bias for documents or paragraphs located in the first and last parts becomes strong).

[0005] Another conventional technique, DuoT5 (The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models, Ronak Pradeep et al., 2021), compares the relevance between documents through pairwise re-ranking, which is a method that compares each document, but has the disadvantage of having a high time complexity of O(n^2).

[0006] (Non-patent Document 1) Document Ranking with a Pretrained Sequence-to-Sequence Model, Rodrigo Nogueira and 3 others, 2020.

[0007] (Non-patent document 2) RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses, Honglei Zhuang et al., 2022.

[0008] (Non-patent document 3) The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models, Ronak Pradeep, 3 people, 2021.

[0009] (Non-patent Document 4) The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models, Ronak Pradeep et al., 2021.

[0010] The problem to be solved by the present invention is to provide a re-ranking system, method, and program for extracting paragraphs highly related to a query from among paragraphs included in a document stored in a database.

[0011] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0012] The re-ranking system of the present invention is a system including at least one processor and a memory storing one or more commands, wherein the at least one processor executes the one or more commands stored in the memory to perform a plurality of tournament sorts to perform re-ranking of a plurality of paragraphs, wherein the tournament sorting includes the steps of: dividing first paragraphs included in the paragraphs into a plurality of groups by pairing them by a specific number, evaluating the relevance between the first paragraphs included in each group and a query, extracting paragraphs having a high relevance up to a predetermined rank among the first paragraphs, and outputting them as second paragraphs; and dividing second paragraphs into a plurality of groups by pairing them by a specific number, evaluating the relevance between the second paragraphs included in each group and the query, and extracting paragraphs having a high relevance up to a predetermined rank among the second paragraphs, and outputting them as third paragraphs.

[0013] In the above system, the first paragraph, which is identical to the third paragraph output in at least one of the above tournament sortings, may not be used as a target for evaluating the relevance in any one or more of the above tournament sortings performed thereafter.

[0014] In the above system, the result of at least one relevance evaluation performed in at least one of the tournament alignments can be reused in a step of evaluating the relevance of any one or more of the tournament alignments performed thereafter.

[0015] In the above system, the step of evaluating the relevance can generate output in the order of low relevance to high relevance.

[0016] In the above system, the relevance is evaluated through an encoder-decoder structure, and the encoder-decoder structure may be a FiD or T5-base architecture.

[0017] In the above system, the query or the paragraph may be data stored in a database converted into a vector.

[0018] In the above system, the specific number may be 5.

[0019] In the above system, the predetermined ranking may be top 1 or top 2.

[0020] A re-ranking method according to another aspect of the present invention is a method for performing re-ranking of a plurality of paragraphs by performing a plurality of tournament sorts performed by at least one processor, the method comprising: dividing first paragraphs included in the paragraphs into a plurality of groups by pairing them by a specific number, evaluating the relevance between the first paragraphs included in each group and a query, extracting paragraphs having a high relevance up to a predetermined rank among the first paragraphs, and outputting the paragraphs as second paragraphs; dividing second paragraphs into a plurality of groups by pairing them by a specific number, evaluating the relevance between the second paragraphs included in each group and the query, and extracting paragraphs having a high relevance up to a predetermined rank among the second paragraphs, and outputting the paragraphs as third paragraphs.

[0021] In the above method, the first paragraph, which is identical to the third paragraph output in at least one of the tournament alignments, may not be used as a target for evaluating the relevance in any one or more of the tournament alignments performed thereafter.

[0022] In the above method, the result of at least one relevance evaluation performed in at least one of the tournament alignments can be reused in the step of evaluating the relevance of any one or more of the tournament alignments performed thereafter.

[0023] In the above method, the step of evaluating the relevance can generate output in the order of low relevance to high relevance.

[0024] In the above method, the relevance is evaluated through an encoder-decoder structure, and the encoder-decoder structure may be a FiD or T5-base architecture.

[0025] In the above method, the query or the paragraph may be data stored in a database converted into a vector.

[0026] In the above method, the specific number may be 5.

[0027] In the above method, the predetermined ranking may be top 1 or top 2.

[0028] A program according to another aspect of the present invention may be a program stored in a computer-readable recording medium to cause a computer to execute a re-ranking method according to embodiments of the present invention.

[0029] Figure 1 is a conceptual diagram of extracting and re-ranking paragraphs highly related to open-domain information queries.

[0030] FIG. 2 is a schematic diagram of a re-ranking system for extracting paragraphs highly relevant to a query, according to one embodiment of the present disclosure.

[0031] FIG. 3 is a re-ranking device for extracting paragraphs with high relevance to a query according to one embodiment of the present invention.

[0032] FIG. 4 is a block diagram illustrating a method for selecting, as output, paragraphs whose relevance to a query is higher than a predetermined rank using an encoder-decoder structure according to one embodiment of the present invention.

[0033] FIG. 5 is a block diagram illustrating a process in which documents output from an encoder-decoder structure are output in order from documents with low relevance to documents with high relevance, according to one embodiment of the present invention.

[0034] FIG. 6 is a block diagram illustrating a method for re-ranking paragraphs by performing multiple tournament alignments according to one embodiment of the present invention (when the basic operating unit (r) is '1' (up to the top 1 in relevance)).

[0035] FIG. 7 is a block diagram of extracting multiple paragraphs highly related to a query (310) by performing tournament sorting multiple times according to one embodiment of the present invention.

[0036] FIG. 8 is a block diagram for re-ranking paragraphs by performing multiple tournament alignments according to embodiments of the present invention (when the basic operating unit (r) is '2' (from 1st to 2nd in terms of relevance)).

[0037] Figure 9 shows the results of comparing the complexity of the present invention with other models.

[0038] Figure 10 shows the results of comparing the computational performance (FLOPs; Floating Point Operations per Second) (x-axis) of the present invention and the BEIR performance (y-axis) for evaluating paragraph relevance with other models.

[0039] Figure 11 shows the results of comparing the present invention and pointwise models (MonoT5 and RankT5) at NDCG@10 based on BEIR.

[0040] Figure 12 shows the results of comparing the present invention with the prior art listwise re-ranking model (RankGPT, RankVicuna, RankZephyr) and pairwise re-ranking model (DuoT5) based on NDCG@10.

[0041] The following examples are provided as examples to ensure that those skilled in the art can fully grasp the spirit of the present invention. Therefore, the present invention is not limited to the embodiments described below and may be embodied in other forms.

[0042] Throughout the present invention, the same reference numerals denote the same components. The present invention does not describe all elements of the embodiments, and any content that is general in the technical field to which the present invention pertains or that overlaps between the embodiments is omitted. The terms 'part, module, element, block' used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple 'parts, modules, elements, blocks' may be implemented as a single component, or a single 'part, module, element, block' may include multiple components.

[0043] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0044] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0045] Throughout the specification, when we say that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0046] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0047] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0048] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0049] A re-ranking system for extracting paragraphs highly relevant to a query from among paragraphs contained in documents stored in a database according to the present invention may include a device, and the device may include various devices capable of performing computational processing and providing results to a user. For example, the re-ranking system for extracting paragraphs highly relevant to a query from among paragraphs contained in documents stored in a database may include at least one of a computer, a server device, and a portable terminal, or may be any form having the same or similar functions as these. However, the present invention is not limited thereto.

[0050] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0051] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0052] The above portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD).

[0053] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0054] The present invention relates to a re-ranking system, method, and program for extracting paragraphs highly related to a query, and more particularly, to a re-ranking system, method, and program for extracting paragraphs highly related to a query from among paragraphs included in a document stored in a database.

[0055] FIG. 2 is a schematic diagram of a re-ranking system for extracting paragraphs highly relevant to a query according to one embodiment of the present invention.

[0056] As illustrated in FIG. 2, the system (1000) may include a device (210), a database (220), an AI model (230), and a retriever (240).

[0057] The device (210), database (220), AI model (230), and retriever (240) included in the system (1000) can communicate via a network (W). Here, the network (W) may include a wired network and a wireless network. For example, the network may include various networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0058] Additionally, the network (W) may include the well-known World Wide Web (WWW). However, the network (W) according to an embodiment of the present invention is not limited to the networks listed above, and may include at least part of a well-known wireless data network, a well-known telephone network, or a well-known wired / wireless television network.

[0059] The database (220) may include queries, documents, and paragraphs input to a retriever or a large-scale language model (LLM) in the form of data. Furthermore, the database (220) may include an encoder-decoder structure that converts queries, documents, and paragraphs included in the form of data into vectors or evaluates the relevance between a 'query' and a 'document or paragraph.' Here, the encoder-decoder structure may be based on, but is not limited to, a Fusion-in-Decoder (FiD) or T5-base architecture. Furthermore, the database (220) may include a model for tournament sorting that performs re-ranking of multiple paragraphs.

[0060] The device (210) can convert queries, documents, and paragraphs stored in the database (220) into vectors using an encoder-decoder structure. In addition, the device (210) performs multiple tournament alignments to re-rank multiple paragraphs.

[0061] Here, the tournament sorting process is as follows. First, multiple paragraphs are paired by a specific number of pairs and divided into multiple groups. Next, a decoder evaluates the relevance of each paragraph to the query for each group, extracting only paragraphs that rank up to a predetermined rank (e.g., 1st or 2nd). The extracted paragraphs are then paired again by a specific number of pairs, and the relevance evaluation process is repeated. Finally, through repeated evaluations, a single paragraph is extracted as the final paragraph. Since tournament sorting is performed multiple times (k times), the top k paragraphs are ultimately output through re-ranking.

[0062] The k paragraphs finally outputted can be inputted to another AI model (230) or retriever (240) that has not been fine-tuned along with the query, which is a dialogue whose relevance has been evaluated, but this is not limited thereto. Even if the other AI model (230) or retriever (240) is not fine-tuned, the paragraphs outputted through re-ranking exhibit improved zero-shot performance because only paragraphs related to the query are inputted.

[0063] FIG. 2 illustrates a case where a database (220) is implemented outside of a device (210). In this case, the database (220) may be connected to the device (210) via wired or wireless connections. However, this is merely an example, and the database (220) may also be implemented as a component of the device (210).

[0064] FIG. 2 illustrates a case where the AI ​​model (230) is implemented outside the device (210) (e.g., cloud-based), but is not limited thereto, and may be implemented as a component of the device (210).

[0065] FIG. 3 is a block diagram illustrating a configuration of a re-ranking device for extracting paragraphs highly relevant to a query according to one embodiment of the present invention.

[0066] As illustrated in FIG. 3, the device (210) may include a memory (310), a communication module (320), a display (330), an input module (340), and a processor (350). However, the present invention is not limited thereto, and the device (210) may have its software and hardware configurations modified / added / omitted within a range apparent from a perspective of ordinary skill in the art, depending on the required operation. In addition, the device (210) may be replaced with a system, and the device (210) may include a plurality of devices, in which case each component included in the device (210) may be included in at least one of the plurality of devices.

[0067] The memory (310) can store data supporting various functions of the device (210), programs for the operation of the processor (350), input / output data, and a plurality of application programs or applications run on the device, data for the operation of the device (210), commands, and AI models. At least some of these application programs can be downloaded from an external server via wireless communication.

[0068] The memory (310) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD (Solid State Disk type), an SDD (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0069] Additionally, the memory (310) may be separate from the device and may include a database connected wired or wirelessly. The database (220) illustrated in FIG. 1 may be implemented as a component of the memory (310).

[0070] The communication module (320) may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0071] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard 232), power line communication, or plain old telephone service (POTS).

[0072] The wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, in addition to a WiFi module and a Wireless Broadband module.

[0073] The display (330) displays (outputs) information or data processed in the device (210), data input or output through the AI ​​model (230), etc. In addition, the display (330) can display execution screen information of an application program (e.g., an application) running in the device (210), or UI (User Interface) or GUI (Graphical User Interface) information according to such execution screen information.

[0074] The input module (340) is for receiving information from a user. When a user inputs information through the input unit, the processor (350) can control the operation of the device (210) to correspond to the input information.

[0075] The input module (340) may include hardware physical keys (e.g., buttons located on at least one of the front, rear, and side of the device, dome switches, jog wheels, jog switches, etc.) and software touch keys. For example, the touch keys may be formed as virtual keys, soft keys, or visual keys displayed on a touchscreen-type display (330) through software processing, or as touch keys placed on a part other than the touchscreen. Meanwhile, the virtual keys or visual keys may be displayed on the touchscreen in various forms, and may be formed as, for example, graphics, text, icons, videos, or a combination thereof.

[0076] The processor (350) may be implemented as a memory that stores data on an algorithm for controlling the operation of components within the device (210) (including learning or executing an AI model) or a program that reproduces the algorithm, and at least one processor (not shown) that performs the aforementioned operation using the data stored in the memory. In this case, the memory and the processor may be implemented as separate chips, or may be implemented as a single chip.

[0077] In one embodiment, the system (1000) or device (210) according to the present invention may include at least one processor, and when including multiple processors, the multiple processors may be included in different devices (210).

[0078] In addition, the processor (350) can control any one or a combination of the components described above to implement various embodiments according to the present disclosure described below on the device (210).

[0079] FIG. 4 is a block diagram illustrating a method of selecting paragraphs whose relevance to a query is higher than a predetermined rank as output using an encoder-decoder structure according to an embodiment of the present invention.

[0080] In re-ranking, it is desirable to use an encoder-decoder structure based on Fusion-in-Decoder (FiD) or T5-base architecture, where a specific number (m) of paragraphs (e.g., 5) are input and the top r paragraphs (e.g., 1 or 2) are re-ranked as basic operation units (r).

[0081] Specifically, first, the paragraphs [p1, ..., p m ] is input to the encoder in the form of identifiers (e.g. '1, 2, 3, 4, 5') and queries (410) combined (421, 422, 423, 424, 425), and paragraphs [p1, ..., p m ] as many result values ​​as the number of [h1, ..., h m ](440) is output. For example, if a query (410) such as "When did Thomas Edison invent the light bulb?" is given, the query can be output in the form of a combination of the paragraph (p1) "Lightning struck in Seoul" (421), the paragraph (p2) "Thomas Edison talked about cars" (422), the paragraph (p3) "Coffee is good for dieting" (423), the paragraph (p4) "KEPCO solved the lighting problem" (424), and the paragraph (p5) "Thomas Edison invented the light bulb in 1879" (425) along with identifiers 1 to 5. In this case, the result value (hi) can be expressed as follows.

[0082]

[0083] In the above equation, q represents a query, i represents an identifier, and pi represents a paragraph.

[0084] The next encoded result value (hi)(440) is a sequence [h1, ..., h m ](450) and are input to the decoder (430b). The decoder (430b) outputs the sequence [h1, ..., h m ](450) compares the relevance of the sorted sequence of paragraph indices (identifiers) [i1', ..., i m '](460) is created.

[0085]

[0086] The decoder (430b) generates output in order from paragraphs with low relevance to paragraphs with high relevance. Compared to the conventional listwise re-ranking model, which generates an index starting with paragraphs with high relevance, the present invention generates an index in order from paragraphs with low relevance to paragraphs with high relevance, thereby providing an advantageous inference chain. This is because the elimination method, which first eliminates paragraphs with low relevance and then ensures that the last remaining paragraph is the correct answer, can be an advantageous strategy for extracting paragraphs with high relevance.

[0087] Next, the paragraph index (identifier) ​​sequence [i1,...,i] generated by the decoder (430b) m ](460) extracts a paragraph of the basic operating unit (r). The basic operating unit (r) may be 1 or 2, but is not limited thereto.

[0088] If the basic operating unit (r) is 1, the paragraph index (identifier) ​​sequence generated by the decoder [i1,...,i m], only the paragraph [pi'_m] with the highest (1st) rank set in advance is extracted and selected as output (470a). If the index is created in the order of paragraphs with low relevance to paragraphs with high relevance as in the preferred embodiment of the present invention, the paragraph with the last rank is selected as output as shown in Fig. 5 (i.e., when r=1 in Fig. 5).

[0089] When the basic operating unit (r) is 2, the paragraph index (identifier) ​​sequence generated by the decoder [i1, ..., i m ] The paragraph with the highest (1st) rank determined in advance [p i '_m] and the next highest (2nd) paragraph[p i ′(m-1)] is extracted and selected as output (470b). If the index is created in the order of paragraphs with low relevance to paragraphs with high relevance, as in the preferred embodiment of the present invention, one paragraph in the last order or two paragraphs in the last order are selected as output, as shown in Fig. 5 (i.e., when r=2 in Fig. 5).

[0090] FIG. 6 is a block diagram illustrating a method for re-ranking paragraphs by performing multiple tournament alignments according to an embodiment of the present invention (when the basic operating unit (r) is '1' (up to the top 1 in relevance)).

[0091] The present invention relates to re-ranking a larger number (n) of paragraphs (e.g., 5) than a specific number (m) of paragraphs into a final k-paragraph group. To this end, the present invention proposes an algorithm for re-ranking k paragraphs from n paragraphs by extending a method of extracting a basic operating unit (r) (e.g., 1 or 2) from a group of a specific number (m) of paragraphs (e.g., 5).

[0092] Specifically, first, n first paragraphs are paired with a specific number (m) and divided into multiple groups (610). Preferably, the specific number (m) may be 5, but is not limited thereto.

[0093] The first paragraph (pi, etc.) belonging to each group {e.g., the first group (611)} is input to the encoder (430a) in a form combined with an identifier (i=1, 2, 3, 4, 5) and a query (410) as described in Fig. 3, and the relevance between multiple paragraphs {e.g., [p1, ..., p5] of the first group (611)} and the query is evaluated for each group, and the sequence [h1, ..., h] of the result value (hi) is generated. i ](450){For example, in the case of the first group (611), [h1, ..., h5]} is expressed and inputted into the decoder (430b) to obtain the sorted sequence of the first paragraph index (identifier) ​​[i1', ..., i i '](460){For example, for the second group, generate [i1', ..., i5']}, where [i1', ..., i i '] can be printed in order from paragraphs with low relevance to paragraphs with high relevance, but is not limited to this.

[0094] The next decoder (430b) generates the first paragraph index (identifier) ​​sequence [i1',..., i i '](460), the first paragraph of the basic operating unit (r) is extracted and output as the second paragraph (p2)(623). The basic operating unit (r) means the order in which high relevance is output up to a predetermined rank, and can be 1 or 2 (i.e., 'top 1 in relevance' or 'from top 1 to top 2 in relevance'), but is not limited thereto. In the case of the present embodiment illustrated in Fig. 6, the second paragraph (p2)(623), which is the paragraph up to the top 1 in relevance, is output as an embodiment in which the basic operating unit (r) is '1' (up to top 1 in relevance), and an embodiment in which the basic operating unit (r) is '2' (from top 1 to top 2 in relevance) is illustrated in Fig. 8}.

[0095] Also, for the group consisting of the remaining n first paragraphs omitting specific cities, including the group indicated as the second group (612) in Fig. 6, the step of extracting paragraphs with high relevance is performed in the same manner as the first group (611) described above. Taking the second group (612) as an example, the paragraphs to which the second group (612) belongs, [p6, ..., p 10 ] to evaluate the relevance of the sequence [h6, ..., h 10 ] and input it into the decoder (430b) to obtain a sorted sequence of indices, [i6', ..., i 10 After generating '], the paragraphs are output in order from low to high relevance. After that, the decoder (430b) outputs [i6', ..., i 10 '] extracts the paragraph of the basic operating unit (r) and outputs it as the second paragraph (p7 in the case of Fig. 6).

[0096] Next, the second paragraphs output from each group {e.g., p2 (623a), p7 (623b), etc.} are paired again into a specific number (m) (e.g., 5) and divided into multiple groups (620), and the second paragraphs {e.g., p2 (623a), p7 (623b), etc.} included in each of the multiple groups (620) {e.g., the second group (621)} are input to an encoder (430a) to express a sequence, which is then input to a decoder (430b) to output a third paragraph (633) of a basic operating unit having a high degree of relevance up to a predetermined rank in the same manner as performed in the first group (610).

[0097] The same process as for the first and second paragraphs is repeated for the third paragraph (633), and the same process is repeated for the nth paragraph that is output as a result. This process is repeated until only one paragraph (643) is output.

[0098] The last paragraph printed (643) is extracted as the most relevant paragraph.

[0099] This method of extracting the single paragraph (p2) (643) most relevant to the query is called tournament sorting.

[0100] Figure 7 is a block diagram for extracting multiple paragraphs highly related to a query (410) by performing tournament sorting multiple times.

[0101] By performing tournament sorting once, only one paragraph with the highest relevance to the query can be extracted. However, according to the present invention, by performing tournament sorting multiple times, as shown in FIG. 7, it is also possible to extract multiple paragraphs with the highest relevance to the query (410).

[0102] For example, the tournament sorting described above is performed repeatedly, but after extracting the most relevant paragraph (p2) (743a) extracted from the first tournament sorting, the second tournament sorting is performed on the same n paragraphs, but the most relevant paragraph (p2) (743a) is randomly replaced with another paragraph (k1) (743b) to perform the second tournament sorting.

[0103] Through this, it is possible to extract highly relevant paragraphs from among the paragraphs other than the most relevant one (p2)(743a).

[0104] According to the present invention, the results of at least one relevance evaluation performed in at least one tournament alignment can be reused in the step of evaluating the relevance of any other one or more tournament alignments.

[0105] Specifically, if the most relevant paragraph (743a) is replaced with another paragraph (743b), the relevance evaluation results of other groups, excluding the group (740) containing the other paragraph (743b), may be identical to the results of the previously performed tournament sorting. Accordingly, it is desirable to reuse the results of the previously performed tournament sorting in the newly performed tournament sorting (this type of operation is called output caching).

[0106] This allows for efficient and fast output of results by reducing redundant computations and lowering the overall computational cost by reusing the saved results.

[0107] FIG. 8 is a block diagram for re-ranking paragraphs by performing multiple tournament alignments according to an embodiment of the present invention (when the basic operating unit (r) is '2' (from 1st to 2nd in terms of relevance)).

[0108] The embodiment illustrated in Fig. 8 differs only in that it illustrates an embodiment in which the basic operating unit (r) is 2. That is, it differs from the embodiment illustrated in Fig. 6 only in that, instead of extracting only the paragraph with the highest relevance and outputting it as the second paragraph, paragraphs from the first to the second highest relevance are extracted and outputted as the second paragraph (p2, p3) (823a, 823b) (otherwise, the configuration in which the referenced drawing symbols are the same as those in Fig. 6 is executed in the same manner as the embodiment illustrated in Fig. 6).

[0109] Below, the asymptotic complexity of listwise re-ranking according to the present invention is described. Specifically, since the first tournament sorting according to the present invention requires evaluating the relevance of all paragraphs, the complexity can be expressed as the following equation.

[0110]

[0111] According to the present invention, since n>>m, the complexity can be expressed as n, that is, O(n). Meanwhile, once the tournament tree is constructed, most of the values ​​of the existing tournament sort are reused when the tournament sort is performed again. Accordingly, only one path of the tournament sort needs to be recalculated. In other words, since most nodes and edges are cached, the values ​​of the one-time tournament sort are reused during k iterations of the tournament sort, and accordingly, only the cost of O(log2n) is asymptotically consumed in the repeated tournament sort. Therefore, the listwise re-ranking according to the present invention is ultimately O(n + k * log m n) incurs an asymmetric cost.

[0112] Figure 9 shows the results comparing the complexity of the present invention with other models. The present invention has the advantage of being efficient because it has a complexity that is significantly lower than the complexity of the pairwise re-ranking model (DuoT5) that compares each paragraph pairwise, i.e., O(n^2). In addition, the present invention has a complexity level almost equivalent to that of the pointwise models (MonoT5 and RankT5) that compare each paragraph one-to-one, and yet, as explained below, exhibits superior performance on average in computational performance and BEIR performance (performance in extracting paragraphs with high relevance).

[0113] Figure 10 shows the results of comparing the computational performance (FLOPs; Floating Point Operations per Second) (x-axis) of the present invention and the BEIR performance (y-axis) for evaluating paragraph relevance with other models. LISTT5 (r=1) and LISTT5 (r=2) represent the present invention, and it was confirmed that they exhibited superior computational performance and BEIR performance compared to the conventional pointwise models (MonoT5 and RankT5).

[0114] Figure 11 shows the results of comparing the present invention with pointwise models (MonoT5 and RankT5) at NDCG@10 based on the BEIR criterion. In the data set indicated by boxes in LISTT5-base, the present invention outperforms the prior art (MonoT5-3B or RankT5-3B) in both performance and model size.

[0115] Fig. 12 is a table showing the results of comparing the present invention with the prior art listwise re-ranking models (RankGPT, RankVicuna, RankZephyr) and pairwise re-ranking model (DuoT5) based on NDCG@10. The initial results refer to BM25 Top100, and the scores including RankGPT (GPT-3.5-turbo-0301) refer to the RankGPT paper. The present invention {LISTT5-3B (r=2)} shows high performance in most datasets, and in particular, it shows better performance than the listwise re-ranking models (RankGPT, RankVicuna, RankZephyr) and pairwise re-ranking model (DuoT5) in the BEIR subset.

[0116] Below, the effects of the present invention are described.

[0117] According to the present invention, a significantly lower level of complexity {O(log2n)} is required compared to the prior art, and since an efficient architecture such as FiD or T5-base rather than large-scale language models (LLMs) is adopted, operations can be processed efficiently.

[0118] According to the present invention, paragraphs are distinguished through identifiers rather than the positions of each input paragraph due to the characteristics of FiD, thereby enhancing robustness against the problem of intermediate information loss.

[0119] According to the present invention, the zero-shot performance in the BEIR benchmark is improved compared to the prior art by performing tournament sorting or generating output in the order of paragraphs with low relevance to paragraphs with high relevance (see Figs. 11 and 12).

[0120] Meanwhile, a re-ranking method for extracting paragraphs with high relevance to a query according to embodiments of the present invention can be implemented by the system described with reference to FIG. 1.

[0121] Systems according to embodiments of the present invention may be controlled, executed, learned, driven, etc. by a processor, and thus, the systems may perform at least one of the tasks of execution, learning, and driving by at least one processor. Furthermore, the systems may be stored in memory, and feature data according to the present invention may also be stored in memory.

[0122] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0123] Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0124] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present disclosure can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. In a system including at least one processor and a memory storing one or more instructions, The at least one processor executes the one or more instructions stored in the memory to perform a plurality of tournament sorts to perform re-ranking of a plurality of paragraphs, wherein the tournament sorts are: A step of dividing the first paragraphs included in the above paragraph into a plurality of groups by pairing them by a specific number, evaluating the relevance between the first paragraphs included in each group and the query, and extracting paragraphs among the first paragraphs that have a high relevance up to a predetermined rank and outputting them as second paragraphs; and A step of dividing the second paragraph into a plurality of groups by pairing a specific number of the second paragraphs, evaluating the relevance between the second paragraphs included in each group and the query, and extracting paragraphs among the second paragraphs that have a high relevance up to a predetermined rank and outputting them as a third paragraph; A system including:

2. In claim 1, A system wherein the first paragraph, which is identical to the third paragraph output in at least one of the above tournament alignments, is not used as a target for evaluating the relevance in any one or more of the above tournament alignments performed thereafter.

3. In claim 1, A system wherein the results of at least one relevance evaluation performed in at least one of the above tournament alignments are reused in a step of evaluating the relevance of any one or more of the above tournament alignments performed thereafter.

4. In claim 1, The step of evaluating the above relevance is a system that generates output in order of the relevance from low to high.

5. A system according to claim 1, wherein the relevance is evaluated through an encoder-decoder structure, and the encoder-decoder structure is a FiD or T5-base architecture.

6. A system according to claim 1, wherein the query or the paragraph is data stored in a database converted into a vector.

7. In claim 1, the specific number is 5, the system.

8. In claim 1, the predetermined ranking is the top 1 or top 2, the system.

9. A method for performing re-ranking of multiple paragraphs by performing multiple tournament sorts performed by at least one processor, A step of dividing the first paragraphs included in the above paragraph into a plurality of groups by pairing them by a specific number, evaluating the relevance between the first paragraphs included in each group and the query, and extracting paragraphs among the first paragraphs that have a high relevance up to a predetermined rank and outputting them as second paragraphs; and A step of dividing the second paragraph into a plurality of groups by pairing a specific number of the second paragraphs, evaluating the relevance between the second paragraphs included in each group and the query, and extracting paragraphs among the second paragraphs that have a high relevance up to a predetermined rank and outputting them as a third paragraph; A method comprising:

10. In claim 9, A method wherein the first paragraph, which is identical to the third paragraph output in at least one of the above tournament alignments, is not used as a target for evaluating the relevance in any one or more of the above tournament alignments performed thereafter.

11. In claim 9, A method wherein the results of at least one relevance evaluation performed in at least one of the above tournament alignments are reused in a step of evaluating the relevance of any one or more of the above tournament alignments performed thereafter.

12. In claim 9, The step of evaluating the above relevance is a system that generates output in order of the relevance from low to high.

13. A method according to claim 9, wherein the relevance is evaluated through an encoder-decoder structure, and the encoder-decoder structure is a FiD or T5-base architecture.

14. A method according to claim 9, wherein the query or the paragraph is data stored in a database converted into a vector.

15. A method according to claim 9, wherein the specific number is 5.

16. A method according to claim 9, wherein the predetermined ranking is the first or second highest ranking.

17. A program stored in a computer-readable recording medium that causes a computer to execute the method of any one of claims 9 to 16.

Images