Reordering method and device based on large language model, equipment and storage medium

Abstract

The invention provides a reordering method and device based on a large language model, equipment and a storage medium. The invention provides a reordering method based on a large language model. The reordering method comprises the following steps: acquiring a plurality of paragraphs to be ordered; sequentially inputting the plurality of paragraphs to be sequenced into a reordering language model by using a sliding window strategy according to windows, and for each window, enabling the reordering language model to generate an arrangement sequence of each paragraph according to the correlation of query data, the reordering language model is obtained by training according to a reference large language model and a NovelEval test set; after the arrangement sequence of the paragraphs of all the windows is obtained, the arrangement sequence of the paragraphs of all the windows is integrated, and a final arrangement result is obtained. According to the reordering method based on the large language model, the ordering efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval technology, and in particular to a reordering method, device, equipment and storage medium based on a large language model. Background Art

[0002] An information retrieval system is a programmed system for information collection, processing, storage and retrieval that is established based on specific information needs. Its main purpose is to provide information services to people. In existing information retrieval systems, manual supervision signals are usually relied on to sort a large number of documents. This sorting scheme not only requires a lot of human resources and is inefficient, but also has low accuracy when dealing with novel and dynamically changing information needs. Although the generative large language model (LLM) performs well in content generation tasks, its accuracy in directly sorting relevance is also low. Summary of the invention

[0003] The purpose of the present invention is to provide a reordering method, device, equipment and storage medium based on a large language model to solve the technical problems of low sorting efficiency and low accuracy in the prior art.

[0004] The technical solution of the present invention is as follows: a reordering method based on a large language model is provided, comprising:

[0005] Get multiple paragraphs to be sorted;

[0006] The plurality of paragraphs to be sorted are sequentially input into a re-ranking language model by using a sliding window strategy, and for each window, the re-ranking language model is used to generate a ranking order of each paragraph according to the relevance of the query data, wherein the re-ranking language model is trained based on a benchmark large language model and a NovelEval test set;

[0007] After obtaining the arrangement order of each paragraph of all windows, the arrangement order of each paragraph of all windows is integrated to obtain the final sorting result.

[0008] Furthermore, the step of using a sliding window strategy to sequentially input the multiple paragraphs to be sorted into the reordering language model in divided windows includes: setting the size and step size of the sliding window, and using the sliding window strategy to sequentially input the multiple paragraphs to be sorted into divided windows from back to front into the reordering language model.

[0009] Furthermore, after obtaining a plurality of paragraphs to be sorted, it also includes assigning unique identifiers to the plurality of paragraphs to be sorted; enabling the reordering language model to generate an arrangement order of each paragraph according to the relevance of the query data, including enabling the reordering language model to generate an arrangement order of each paragraph corresponding to the unique identifier according to the relevance of the query data.

[0010] Furthermore, the training step of the reordering language model includes:

[0011] A query data sample is randomly extracted from the MS MARCO training set, and multiple candidate paragraph samples are obtained according to the query data sample, so that the benchmark large language model generates an arrangement order of the multiple candidate paragraph samples. The arrangement order of the multiple candidate paragraph samples is used as a training target, and a natural language processing model is trained according to the MS MARCO training set and the NovelEval test set to obtain a reordering language model.

[0012] Furthermore, obtaining a plurality of candidate paragraph samples according to the query data sample includes: obtaining a plurality of candidate paragraph samples through BM25 algorithm and the query data sample retrieval.

[0013] Furthermore, the NovelEval test set includes a plurality of novel query data samples and a plurality of candidate paragraph samples corresponding to the novel query data samples and annotated with relevance.

[0014] Furthermore, the step of causing the benchmark large language model to generate the arrangement order of the plurality of candidate paragraph samples includes: using an instructional arrangement generation method to cause the benchmark large language model to generate the arrangement order of the plurality of candidate paragraph samples.

[0015] Another technical solution of the present invention is as follows: a reordering device based on a large language model is provided, comprising a paragraph acquisition module, a sorting module and a sorting result acquisition module;

[0016] The paragraph acquisition module is used to acquire a plurality of paragraphs to be sorted;

[0017] The sorting module is used to sequentially input the plurality of paragraphs to be sorted into a re-sorting language model in windows by using a sliding window strategy, and for each window, the re-sorting language model generates a sorting order of each paragraph according to the relevance of the query data, wherein the re-sorting language model is trained based on a benchmark large language model and a NovelEval test set;

[0018] The sorting result acquisition module is used to acquire the sorting order of each paragraph of all windows, and then integrate the sorting order of each paragraph of all windows to obtain the final sorting result.

[0019] Another technical solution of the present invention is as follows: an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the large language model-based reordering method as described in any of the above technical solutions is implemented.

[0020] Another technical solution of the present invention is as follows: a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the large language model-based reordering method as described in any of the above technical solutions is implemented.

[0021] The beneficial effects of the present invention are as follows: a plurality of paragraphs to be sorted are obtained; the plurality of paragraphs to be sorted are sequentially input into a reordering language model in windows using a sliding window strategy, and for each window, the reordering language model is used to generate an arrangement order of each paragraph according to the relevance of query data, wherein the reordering language model is trained based on a benchmark large language model and a NovelEval test set; after the arrangement order of each paragraph of all windows is obtained, the arrangement order of each paragraph of all windows is integrated to obtain a final sorting result; through the above technical solution, the efficiency and accuracy of sorting can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of a flow chart of a reordering method based on a large language model provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the structure of a reordering device based on a large language model provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In the description of this application, the words "first", "second", etc. are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order. In this specification, the terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0027] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] Figure 1 FIG. 1 is a flow chart of a large language model-based reordering method according to an embodiment of the present invention. It should be noted that if substantially the same results are achieved, the large language model-based reordering method of the present invention is not limited to the large language model. Figure 1 The process sequence shown is limited. Figure 1 As shown, the reordering method based on the large language model mainly includes the following steps:

[0029] S101, obtaining a plurality of paragraphs to be sorted;

[0030] S102, using a sliding window strategy, sequentially inputting the plurality of paragraphs to be sorted into a re-ranking language model in windows, and for each window, causing the re-ranking language model to generate a ranking order of each paragraph according to the relevance of the query data, wherein the re-ranking language model is trained based on a benchmark large language model and a NovelEval test set;

[0031] It should be noted that the above-mentioned multiple paragraphs to be sorted correspond to the query data (query), and the multiple paragraphs to be sorted corresponding to the query data can be obtained through a search engine.

[0032] In an optional embodiment, after obtaining the multiple paragraphs to be sorted, it also includes assigning unique identifiers to the multiple paragraphs to be sorted; allowing the reordering language model to generate an arrangement order for each paragraph according to the relevance of the query data, including allowing the reordering language model to generate an arrangement order for each paragraph corresponding to the unique identifier according to the relevance of the query data.

[0033] In a specific embodiment, without loss of generality, the unique identifier may be [1], [2], [3], etc., so that the re-ranking language model generates an arrangement order of the unique identifiers corresponding to each paragraph according to the relevance of the query data. For example, the unique identifier of the first paragraph to be sorted may be [1], the unique identifier of the second paragraph to be sorted may be [2], and the unique identifier of the third paragraph to be sorted may be [3]. So that the re-ranking language model generates an arrangement order of the unique identifiers corresponding to each paragraph according to the relevance of the query data, which may be [2], [3], [1].

[0034] In an embodiment of the present invention, after obtaining multiple paragraphs to be sorted, unique identifiers are assigned to the multiple paragraphs to be sorted, so that the re-ranking language model generates an arrangement order of each paragraph corresponding to the unique identifier according to the relevance of the query data, thereby making better use of the understanding and reasoning capabilities of the re-ranking language model and improving the accuracy of sorting.

[0035] In an optional embodiment, the step of using a sliding window strategy to sequentially input the multiple paragraphs to be sorted into the reordering language model in divided windows includes: setting the size and step size of the sliding window, and using the sliding window strategy to sequentially input the multiple paragraphs to be sorted into divided windows from back to front into the reordering language model.

[0036] In a specific embodiment, the window parameters are set, that is, the window size and step size are set, and a plurality of paragraphs to be sorted are divided into windows from back to front and input into the reordering language model in sequence using a sliding window strategy. Without loss of generality, if there are M paragraphs to be sorted, the window size is w, and the step size is s, the sliding window strategy is used to segment from back to front, first the sorted Mw-th paragraph to the M-th paragraph are input into the reordering language model for sorting, and then the window is slid according to the step size s, and the next w paragraphs are input into the reordering language model for sorting.

[0037] Since there may be token limitation problems when processing long documents, in an embodiment of the present invention, by setting the size and step size of the sliding window, and using the sliding window strategy, the multiple paragraphs to be sorted are input into the reordering language model from back to front in windows in sequence, which can improve the efficiency and accuracy of the reordering language model when processing long documents.

[0038] In an optional implementation, the step of training the reordering language model includes:

[0039] A query data sample is randomly extracted from the MS MARCO training set, and multiple candidate paragraph samples are obtained according to the query data sample, so that the benchmark large language model generates an arrangement order of the multiple candidate paragraph samples. The arrangement order of the multiple candidate paragraph samples is used as a training target, and a natural language processing model is trained according to the MS MARCO training set and the NovelEval test set to obtain a reordering language model.

[0040] In a specific embodiment, query data samples are randomly extracted from the MS MARCO training set, and the query data samples may be multiple, for example, 10,000. Multiple candidate paragraph samples are obtained based on the query data samples, and each query data sample corresponds to multiple candidate paragraph samples, for example, 20. The benchmark large language model generates an arrangement order of the multiple candidate paragraph samples for each query data sample. The arrangement order of the multiple candidate paragraph samples is used as the training target, and a natural language processing model is trained according to the MS MARCO training set and the NovelEval test set to obtain a reordered language model, wherein the natural language processing model may be DeBERTa-large or LLaMA-7B.

[0041] The embodiment of the present invention randomly extracts query data samples from the MS MARCO training set, obtains multiple candidate paragraph samples according to the query data samples, enables the benchmark large language model to generate the arrangement order of the multiple candidate paragraph samples, takes the arrangement order of the multiple candidate paragraph samples as the training target, trains the natural language processing model according to the MS MARCO training set and the NovelEval test set, and obtains the reordering language model, so that the reordering language model has a good accuracy rate, and at the same time, reduces the calculation cost and resource requirements, and improves the sorting efficiency and response speed.

[0042] In an optional implementation, obtaining a plurality of candidate paragraph samples according to the query data sample includes: obtaining a plurality of candidate paragraph samples by retrieving through a BM25 algorithm and the query data sample.

[0043] In the embodiment of the present invention, multiple candidate paragraph samples are retrieved through the BM25 algorithm and the query data sample, so that relatively accurate candidate paragraph samples can be obtained, thereby improving the accuracy of the retrieval result.

[0044] In an optional implementation, the NovelEval test set includes a plurality of novel query data samples and a plurality of candidate paragraph samples corresponding to the novel query data samples and annotated with relevance.

[0045] In a specific embodiment, without loss of generality, 21 novel query data (question) samples are collected, which have never appeared in the training data of the large language model (such as the MS MARCO training set). 20 candidate paragraphs of each novel query data sample are retrieved through a search engine and used as candidate paragraph samples. The relevance of the candidate paragraph samples is annotated, where the relevance may include 0, i.e., irrelevant, 1, i.e., partially relevant, and 2, i.e., relevant. Thus, a NovelEval test set, multiple novel query data samples in the NovelEval test set, and multiple candidate paragraph samples with annotated relevance corresponding to the novel query data samples can be obtained.

[0046] The embodiments of the present invention make the NovelEval test set include multiple novel query data samples and multiple candidate paragraph samples corresponding to the novel query data samples and annotated with relevance, so as to more objectively evaluate the ability of the re-ranking language model in processing unknown knowledge, solve the data contamination problem that may exist in the existing test set, improve the authenticity and credibility of the ranking, and improve the ranking accuracy of the re-ranking language model.

[0047] In an optional implementation, causing the benchmark large language model to generate the arrangement order of the plurality of candidate paragraph samples includes: using an instructional arrangement generation method to cause the benchmark large language model to generate the arrangement order of the plurality of candidate paragraph samples.

[0048] In a specific embodiment, an instructed arrangement generation method is used to enable a benchmark large language model to generate an arrangement order for multiple candidate paragraph samples. The benchmark large language model may be an AI large language model such as ChatGPT.

[0049] S103, after obtaining the arrangement order of each paragraph of all windows, the arrangement order of each paragraph of all windows is integrated to obtain a final sorting result.

[0050] The embodiment of the present invention ensures the continuity and consistency of the final sorting result by integrating the sorting results of each window, thereby improving the accuracy of the overall sorting.

[0051] The large language model-based reordering method provided in an embodiment of the present invention obtains a plurality of paragraphs to be sorted; the plurality of paragraphs to be sorted are sequentially input into the reordering language model in windows using a sliding window strategy; for each window, the reordering language model generates an arrangement order of each paragraph according to the relevance of query data, wherein the reordering language model is trained based on a benchmark large language model and a NovelEval test set; after obtaining the arrangement order of each paragraph of all windows, the arrangement order of each paragraph of all windows is integrated to obtain a final sorting result; through the above technical solution, the efficiency and accuracy of sorting can be improved.

[0052] The large language model-based reordering method provided in the embodiment of the present invention can be constructed based on artificial intelligence, and relevant data can be acquired and processed based on artificial intelligence technology to achieve unattended large language model-based reordering. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0053] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0054] Figure 2 is a schematic diagram of the structure of a reordering device based on a large language model according to an embodiment of the present invention. Figure 2 As shown, the large language model-based reordering device 20 includes a paragraph acquisition module 21, a sorting module 22, and a sorting result acquisition module 23;

[0055] The paragraph acquisition module 21 is used to acquire a plurality of paragraphs to be sorted;

[0056] The sorting module 22 is used to sequentially input the plurality of paragraphs to be sorted into a re-sorting language model in windows by using a sliding window strategy, and for each window, the re-sorting language model generates a sorting order of each paragraph according to the relevance of the query data, wherein the re-sorting language model is trained based on a benchmark large language model and a NovelEval test set;

[0057] The sorting result acquisition module 23 is used to acquire the sorting order of each paragraph of all windows, and then integrate the sorting order of each paragraph of all windows to obtain the final sorting result.

[0058] In an optional embodiment, the sorting module 22 uses a sliding window strategy to input the multiple paragraphs to be sorted into the reordering language model in divided windows in sequence, including: setting the size and step size of the sliding window, and using the sliding window strategy to input the multiple paragraphs to be sorted into the reordering language model in divided windows from back to front in sequence.

[0059] In a specific embodiment, the window parameters are set, that is, the window size and step size are set, and a plurality of paragraphs to be sorted are divided into windows from back to front and input into the reordering language model in sequence using a sliding window strategy. Without loss of generality, if there are M paragraphs to be sorted, the window size is w, and the step size is s, the sliding window strategy is used to segment from back to front, first the sorted Mw-th paragraph to the M-th paragraph are input into the reordering language model for sorting, and then the window is slid according to the step size s, and the next w paragraphs are input into the reordering language model for sorting.

[0060] Since there may be token limitation problems when processing long documents, in an embodiment of the present invention, by setting the size and step size of the sliding window, and using the sliding window strategy, the multiple paragraphs to be sorted are input into the reordering language model from back to front in windows in sequence, which can improve the efficiency and accuracy of the reordering language model when processing long documents.

[0061] In an optional embodiment, the large language model-based reordering device 20 also includes an identifier allocation module, which is used to allocate unique identifiers to the multiple paragraphs to be sorted after obtaining the multiple paragraphs to be sorted; the sorting module 22 enables the reordering language model to generate an arrangement order for each paragraph according to the relevance of the query data, including enabling the reordering language model to generate an arrangement order for each paragraph corresponding to the unique identifier according to the relevance of the query data.

[0062] In a specific embodiment, without loss of generality, the unique identifier may be [1], [2], [3], etc., so that the re-ranking language model generates an arrangement order of the unique identifiers corresponding to each paragraph according to the relevance of the query data. For example, the unique identifier of the first paragraph to be sorted may be [1], the unique identifier of the second paragraph to be sorted may be [2], and the unique identifier of the third paragraph to be sorted may be [3]. So that the re-ranking language model generates an arrangement order of the unique identifiers corresponding to each paragraph according to the relevance of the query data, which may be [2], [3], [1].

[0063] In an embodiment of the present invention, after obtaining multiple paragraphs to be sorted, unique identifiers are assigned to the multiple paragraphs to be sorted, so that the re-ranking language model generates an arrangement order of each paragraph corresponding to the unique identifier according to the relevance of the query data, thereby making better use of the understanding and reasoning capabilities of the re-ranking language model and improving the accuracy of sorting.

[0064] In an optional embodiment, the large language model-based reordering device 20 also includes a model training module, which is used to randomly extract query data samples from the MS MARCO training set, obtain multiple candidate paragraph samples based on the query data samples, enable the benchmark large language model to generate an arrangement order of the multiple candidate paragraph samples, take the arrangement order of the multiple candidate paragraph samples as a training target, train a natural language processing model according to the MS MARCO training set and the NovelEval test set, and obtain a reordering language model.

[0065] In a specific embodiment, query data samples are randomly extracted from the MS MARCO training set, and the query data samples may be multiple, for example, 10,000. Multiple candidate paragraph samples are obtained based on the query data samples, and each query data sample corresponds to multiple candidate paragraph samples, for example, 20. The benchmark large language model generates an arrangement order of the multiple candidate paragraph samples for each query data sample. The arrangement order of the multiple candidate paragraph samples is used as the training target, and a natural language processing model is trained according to the MS MARCO training set and the NovelEval test set to obtain a reordered language model, wherein the natural language processing model may be DeBERTa-large or LLaMA-7B.

[0066] The embodiment of the present invention randomly extracts query data samples from the MS MARCO training set, obtains multiple candidate paragraph samples according to the query data samples, enables the benchmark large language model to generate the arrangement order of the multiple candidate paragraph samples, takes the arrangement order of the multiple candidate paragraph samples as the training target, trains the natural language processing model according to the MS MARCO training set and the NovelEval test set, and obtains the reordering language model, so that the reordering language model has a good accuracy rate, and at the same time, reduces the calculation cost and resource requirements, and improves the sorting efficiency and response speed.

[0067] In an optional implementation, the sorting module 22 obtains a plurality of candidate paragraph samples according to the query data sample, including: obtaining a plurality of candidate paragraph samples through BM25 algorithm and the query data sample retrieval.

[0068] In the embodiment of the present invention, multiple candidate paragraph samples are retrieved through the BM25 algorithm and the query data sample, so that relatively accurate candidate paragraph samples can be obtained, thereby improving the accuracy of the retrieval result.

[0069] In an optional implementation, the NovelEval test set includes a plurality of novel query data samples and a plurality of candidate paragraph samples corresponding to the novel query data samples and annotated with relevance.

[0070] In a specific embodiment, without loss of generality, 21 novel query data (question) samples are collected, which have never appeared in the training data of the large language model (such as the MS MARCO training set). 20 candidate paragraphs of each novel query data sample are retrieved through a search engine and used as candidate paragraph samples. The relevance of the candidate paragraph samples is annotated, where the relevance may include 0, i.e., irrelevant, 1, i.e., partially relevant, and 2, i.e., relevant. Thus, a NovelEval test set, multiple novel query data samples in the NovelEval test set, and multiple candidate paragraph samples with annotated relevance corresponding to the novel query data samples can be obtained.

[0071] The embodiments of the present invention make the NovelEval test set include multiple novel query data samples and multiple candidate paragraph samples corresponding to the novel query data samples and annotated with relevance, so as to more objectively evaluate the ability of the re-ranking language model in processing unknown knowledge, solve the data contamination problem that may exist in the existing test set, improve the authenticity and credibility of the ranking, and improve the ranking accuracy of the re-ranking language model.

[0072] In an optional implementation, the sorting module 22 enables the benchmark large language model to generate the arrangement order of the plurality of candidate paragraph samples, including: utilizing an instructional arrangement generation method to enable the benchmark large language model to generate the arrangement order of the plurality of candidate paragraph samples.

[0073] In a specific embodiment, an instructed arrangement generation method is used to enable a benchmark large language model to generate an arrangement order for multiple candidate paragraph samples. The benchmark large language model may be an AI large language model such as ChatGPT.

[0074] The large language model-based reordering device provided in the embodiment of the present invention obtains a plurality of paragraphs to be sorted; the plurality of paragraphs to be sorted are sequentially input into the reordering language model in windows using a sliding window strategy; for each window, the reordering language model generates the arrangement order of each paragraph according to the relevance of query data, wherein the reordering language model is obtained by training based on a benchmark large language model and a NovelEval test set; after obtaining the arrangement order of each paragraph of all windows, the arrangement order of each paragraph of all windows is integrated to obtain a final sorting result; through the above technical solution, the efficiency and accuracy of sorting can be improved.

[0075] Figure 3 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 3 As shown, the electronic device 30 includes a processor 31 and a memory 32 that is communicatively connected to the processor 31 .

[0076] The memory 32 stores program instructions for implementing the large language model-based reordering method described in any one of the above embodiments.

[0077] The processor 31 is used to execute program instructions stored in the memory 32 to perform a reordering based on a large language model.

[0078] The processor 31 may also be referred to as a CPU (Central Processing Unit). The processor 31 may be an integrated circuit chip having signal processing capabilities. The processor 31 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0079] An embodiment of the present invention provides a storage medium, and the storage medium of the embodiment of the present invention stores program instructions that can implement all the above methods, and the storage medium can be non-volatile or volatile. Among them, the program instructions can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, tablets, etc.

[0080] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0081] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit. The above is only an implementation mode of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present invention.

[0082] The above description is only an implementation mode of the present invention. It should be pointed out that, for ordinary technicians in this field, improvements can be made without departing from the creative concept of the present invention, but these all belong to the protection scope of the present invention.

Claims

1. A reordering method based on a large language model, characterized in that: include: Get multiple paragraphs to be sorted; The plurality of paragraphs to be sorted are sequentially input into a re-ranking language model by using a sliding window strategy, and for each window, the re-ranking language model is used to generate a ranking order of each paragraph according to the relevance of the query data, wherein the re-ranking language model is trained based on a benchmark large language model and a NovelEval test set; After obtaining the arrangement order of each paragraph of all windows, the arrangement order of each paragraph of all windows is integrated to obtain the final sorting result.

2. The reordering method based on a large language model according to claim 1, characterized in that: The method of using a sliding window strategy to sequentially input the multiple paragraphs to be sorted into the reordering language model in windows includes: setting the size and step size of the sliding window, and using the sliding window strategy to sequentially input the multiple paragraphs to be sorted into the reordering language model in windows from back to front.

3. The reordering method based on a large language model according to claim 1, characterized in that: After obtaining the plurality of paragraphs to be sorted, the method further includes assigning unique identifiers to the plurality of paragraphs to be sorted; The reordering language model is enabled to generate the arrangement order of each paragraph according to the relevance of the query data, including enabling the reordering language model to generate the arrangement order of each paragraph corresponding to the unique identifier according to the relevance of the query data.

4. The reordering method based on a large language model according to claim 1, characterized in that: The training step of the reordering language model includes: A query data sample is randomly extracted from the MS MARCO training set, and multiple candidate paragraph samples are obtained according to the query data sample, so that the benchmark large language model generates an arrangement order of the multiple candidate paragraph samples. The arrangement order of the multiple candidate paragraph samples is used as a training target, and a natural language processing model is trained according to the MS MARCO training set and the NovelEval test set to obtain a reordering language model.

5. The reordering method based on a large language model according to claim 4, characterized in that: Acquiring a plurality of candidate paragraph samples according to the query data sample includes: obtaining a plurality of candidate paragraph samples through BM25 algorithm and the query data sample retrieval.

6. The reordering method based on a large language model according to claim 4, characterized in that: The NovelEval test set includes a plurality of novel query data samples and a plurality of candidate paragraph samples corresponding to the novel query data samples and annotated with relevance.

7. The reordering method based on a large language model according to claim 4, characterized in that: The step of causing the benchmark large language model to generate the arrangement order of the plurality of candidate paragraph samples comprises: using an instructional arrangement generation method to cause the benchmark large language model to generate the arrangement order of the plurality of candidate paragraph samples.

8. A reordering device based on a large language model, characterized in that: It includes a paragraph acquisition module, a sorting module and a sorting result acquisition module; The paragraph acquisition module is used to acquire a plurality of paragraphs to be sorted; The sorting module is used to sequentially input the plurality of paragraphs to be sorted into a re-sorting language model in windows by using a sliding window strategy, and for each window, the re-sorting language model generates a sorting order of each paragraph according to the relevance of the query data, wherein the re-sorting language model is trained based on a benchmark large language model and a NovelEval test set; The sorting result acquisition module is used to acquire the sorting order of each paragraph of all windows, and then integrate the sorting order of each paragraph of all windows to obtain the final sorting result.

9. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable by the processor, wherein: When the processor executes the computer program, the large language model-based reordering method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the large language model-based reordering method according to any one of claims 1 to 7 is implemented.

Images