Data pre-labeling method and device based on document reordering, equipment and storage medium

Abstract

The invention discloses a data pre-labeling method and device based on document reordering, equipment and a storage medium, and relates to the technical field of natural language processing, and the method comprises the following steps: setting a first to-be-ordered document in a current ordering list as an ordered document, and setting a next to-be-ordered document as a current to-be-ordered document; according to a sequence from back to front, selecting one sorted document which is not selected as a current sorted document to be compared; determining whether the current to-be-sorted document is higher than the correlation between the current to-be-compared sorted document and the target retrieval problem, if so, exchanging the current to-be-sorted document and the current to-be-compared sorted document, and skipping to the step of setting the first to-be-sorted document in the current sorting list as the sorted document; and if not, skipping to the step of setting the next to-be-sorted document as the current to-be-sorted document until all the to-be-sorted documents are sorted, and pre-labeling the sorted documents to obtain a document data set. Therefore, the data labeling efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a data pre-labeling method, device, equipment and storage medium based on document reordering. Background Art

[0002] Currently, there are four schemes for constructing re-ranking datasets:

[0003] First, we can build a re-ranking dataset by collecting user feedback. For example, we can leverage log systems to collect user search behavior data, including queries, clicked documents, and user feedback on the relevance of results. This can be achieved by adding a user feedback mechanism to the search results page, such as allowing users to mark a result as relevant. The effectiveness of this approach is highly dependent on user engagement and the quality of feedback, so the quality of the dataset is subject to uncertainty.

[0004] Second, constructing a re-ranking dataset through manual annotation: for example, organizing a dedicated team or outsourcing to a professional annotation company to annotate the relevance of queries and documents in a specific field. Although this method can ensure the high quality of the dataset, its high cost is a drawback that cannot be ignored.

[0005] Third, construct a reranking dataset through intelligent assisted annotation: This involves pre-annotating the data using an open-source document reranking model or one trained on a small number of samples, and then combining this with manual review and correction. This approach combines the advantages of machine automation and manual precision, aiming to reduce annotation costs while maintaining high dataset quality. However, the effectiveness of intelligent assisted annotation depends heavily on the quality of the reranking model used for pre-annotation. Currently, open-source reranking models are often general models trained on internet search data, and their performance on proprietary datasets for specific business scenarios is insufficient.

[0006] Fourth, using large models to construct re-ranking datasets: In recent years, with the significant advancements in the capabilities of large models for natural language processing tasks, some approaches have attempted to leverage these models for document re-ranking tasks. However, document re-ranking is not a strong capability of large models, and the performance has been less than satisfactory. Furthermore, directly feeding queries and large, lengthy document collections into large models not only results in longer inference times, but also poses greater challenges to model accuracy due to the complexity and length of the input information.

[0007] As can be seen from the above, how to improve the efficiency of data annotation in the data pre-annotation process based on document reordering is a problem that needs to be solved urgently. Summary of the Invention

[0008] In view of this, the purpose of the present invention is to provide a data pre-annotation method, apparatus, device, and storage medium based on document reordering, which can improve the efficiency of document reordering in the data pre-annotation process based on document reordering, thereby improving the efficiency of data annotation. The specific scheme is as follows:

[0009] In a first aspect, the present application provides a data pre-annotation method based on document reordering, comprising:

[0010] Generating a current sorting list based on a document set according to a preset random sorting rule, setting the first document to be sorted in the current sorting list as a sorted document, and then determining the next document to be sorted in the current sorting list as the current document to be sorted;

[0011] According to the order of the sorted documents from back to front, a sorted document in the current sorted list that has not been selected in the current round of document sorting is selected as the current sorted document to be compared;

[0012] Determine using a preset large language model whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question; if so, swap the positions of the current document to be sorted and the current sorted document to be compared in the current sorted list, and then jump back to the step of selecting a sorted document in the current sorted list that has not been selected in the current document sorting process as the current sorted document to be compared;

[0013] If not, then the document sorting process for the current document to be sorted is completed, and the process jumps back to the step of determining the next document to be sorted in the current sorting list as the current document to be sorted, until the document sorting process for all documents to be sorted in the current sorting list is completed, thereby obtaining the target re-sorted document list;

[0014] The sorted documents are pre-labeled based on the correlations between the sorted documents in the target re-sorted document list to obtain a labeled document dataset.

[0015] Optionally, before generating a current sorted list according to a preset random sorting rule and based on a document set, setting the first document to be sorted in the current sorted list as a sorted document, and then determining the next document to be sorted in the current sorted list as the current document to be sorted, the method further includes:

[0016] Constructing a preset instruction template based on preset constraints; the preset constraints include document length constraints, format difference constraints, and semantic relevance constraints;

[0017] Determine the input formats corresponding to the target retrieval question and the documents to be sorted, respectively, and then use a preset instruction template and determine a relevance judgment instruction based on the input format, the target retrieval question and the documents to be sorted, so that when the large language model receives the relevance judgment instruction, it judges the relevance between the target retrieval question and the documents to be sorted and obtains a relevance judgment result; the structural form of the relevance judgment result is a preset symbolic structure form; the relevance judgment result is the document in each of the documents to be sorted that is more relevant to the target retrieval question.

[0018] Optionally, generating a current sorting list according to a preset random sorting rule and based on the document set, and setting the first document to be sorted in the current sorting list as a sorted document, includes:

[0019] Randomly select a document to be sorted from the document collection, set the selected document as a sorted document, and set the position of the sorted document in the current sort list to the first position, to obtain a sort list to be processed;

[0020] All documents in the document collection except the sorted document at the first position are randomly sorted according to a preset random sorting rule to obtain a random sorting result, and then a current sorting list is determined based on the random sorting result and the to-be-processed sorting list; each element in the current sorting list includes each document and the corresponding order of each document.

[0021] Optionally, the step of using a preset large language model to determine whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question, and if so, swapping the positions of the current document to be sorted and the current sorted document to be compared in the current sorted list includes:

[0022] After the preset large language model receives the large language model call request, the preset large language model compares the relevance between the current document to be sorted and the target retrieval question with the relevance between the current sorted document to be compared and the target retrieval question, thereby obtaining a comparison result corresponding to the current sorted document to be compared; the large language model call request includes a retrieval paragraph corresponding to the target retrieval question;

[0023] Performing conversion processing on the comparison result to obtain a binary judgment signal, and judging whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question based on the binary judgment signal; if the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question, shifting the document position indicator value corresponding to the current sorted document to be compared right by one position;

[0024] Correspondingly, if not, the document sorting process for the current document to be sorted is completed, including:

[0025] If the correlation between the current document to be sorted and the target retrieval question is not higher than the correlation between the current sorted document to be compared and the target retrieval question, the operation of comparing the current document to be sorted with the sorted document is terminated, and the insertion position corresponding to the current document to be sorted is determined to be the right adjacent position of the current sorted document to be compared.

[0026] Optionally, swapping the positions of the current document to be sorted and the current sorted document to be compared in the current sort list includes:

[0027] Determining document position indicator values ​​corresponding to all documents in the current sorted list that are located after the insertion position corresponding to the insertion position order, and sequentially incrementing each determined document position indicator value to obtain corresponding new document position indicator values;

[0028] Based on each of the new document position indicator values, the corresponding document is shifted to obtain an insertion position vacancy, and the original document position indicator value corresponding to the current document to be sorted is modified to the document position indicator value corresponding to the insertion position vacancy.

[0029] Optionally, pre-labeling the ranked documents based on the correlations between the ranked documents in the target re-ranked document list to obtain a labeled document dataset includes:

[0030] Associating and storing the order of each document stored in the target reordered document list with the target retrieval question, to obtain annotation data including the target retrieval question, the document set in the target reordered document list, and the order of the documents in the target reordered document list;

[0031] Generating numerical annotation labels based on the order of each document in the target reordered document list; the numerical annotation labels are used to quantify the relevance strength between the document and the question;

[0032] Based on the labeled data and the numerical labeled labels, a labeled document dataset is determined, and a training model is trained using the labeled document dataset and based on weights corresponding to the numerical labeled labels, and then the obtained trained model is used to process the target retrieval problem.

[0033] In a second aspect, the present application provides a data pre-annotation device based on document reordering, comprising:

[0034] A document determination module is configured to generate a current sorting list based on a document set according to a preset random sorting rule, set the first document to be sorted in the current sorting list as a sorted document, and then determine the next document to be sorted in the current sorting list as the current document to be sorted;

[0035] A document selection module is used to select, in order from back to front of the sorted documents, a sorted document in the current sorted list that has not been selected in the current round of document sorting as the current sorted document to be compared;

[0036] a document position swapping module, configured to use a preset large language model to determine whether the relevance between the current document to be sorted and the target retrieval question is higher than the relevance between the current sorted document to be compared and the target retrieval question; if so, swapping the positions of the current document to be sorted and the current sorted document to be compared in the current sorted list, and then re-jumping to the step of selecting a sorted document in the current sorted list that has not been selected in the current document sorting process as the current sorted document to be compared;

[0037] a loop jump module, configured to, if no, complete the document sorting process for the current document to be sorted in this round and jump again to the step of determining the next document to be sorted in the current sorting list as the current document to be sorted, until the document sorting process for all documents to be sorted in the current sorting list is completed, thereby obtaining a target re-sorted document list;

[0038] The document pre-labeling module is used to pre-label the sorted documents based on the correlations between the sorted documents in the target re-sorted document list to obtain a labeled document dataset.

[0039] Optionally, the document position exchange module includes:

[0040] a comparison result determining unit configured to compare the relevance between the current document to be sorted and the target retrieval question with the relevance between the current sorted document to be compared and the target retrieval question after the preset large language model receives a large language model call request, thereby obtaining a comparison result corresponding to the current sorted document to be compared; the large language model call request includes a retrieval paragraph corresponding to the target retrieval question;

[0041] an indicator shifting unit, configured to convert the comparison result to obtain a binary judgment signal, and determine, based on the binary judgment signal, whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question; if the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question, shifting the document position indicator value corresponding to the current sorted document to be compared right by one position;

[0042] Correspondingly, the loop jump module is specifically used to terminate the operation of comparing the current document to be sorted with the sorted document if the correlation between the current document to be sorted and the target retrieval problem is not higher than the correlation between the current sorted document to be compared and the target retrieval problem, and determine the insertion position corresponding to the current document to be sorted as the right adjacent position of the current sorted document to be compared, and then jump to the step of determining the next document to be sorted in the current sorted list as the current document to be sorted.

[0043] In a third aspect, the present application provides an electronic device, comprising:

[0044] Memory, used to store computer programs;

[0045] A processor is configured to execute the computer program to implement the aforementioned data pre-labeling method based on document reordering.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned data pre-labeling method based on document reordering.

[0047] As can be seen from the above, before performing data pre-annotation based on document reordering, the present application needs to generate a current sort list based on a preset random sorting rule and based on a document set, and set the first document to be sorted in the current sort list as a sorted document, and then select the unsorted current document to be sorted from the current sort list; use the large language model to compare the current document to be sorted with the sorted documents in the current sort list that are located before the current document to be sorted from back to front, and obtain the corresponding comparison results, and then judge whether the correlation between the current document to be sorted and the target retrieval problem is higher than the correlation between the current sorted document and the target retrieval problem based on the comparison results. If higher, the current document to be sorted is selected as the sorted document. The order of the sorted documents is shifted back one position, and then the previous document of the current sorted document is set as the new current sorted document; a step is called to determine whether the relevance between the current document to be sorted and the target retrieval problem is higher than the relevance between the current sorted document and the target retrieval problem based on the comparison result, and then the insertion position order of the current document to be sorted is determined; a new current sorted list is determined based on the current document to be sorted and the insertion position order, and it is determined whether there is an unsorted document in the current sorted list. If so, the step is jumped to the step of selecting the unsorted current document to be sorted in the current sorted list to obtain the target re-sorted document list, so as to determine the annotation data based on the target re-sorted document list.

[0048] It can be seen that this application first needs to generate a current sorted list based on the document set according to the preset random sorting rules, and set the first document to be sorted in the current sorted list as the sorted document, and then select the current document to be sorted that has not been sorted from the current sorted list; then, use the large language model to compare the current document to be sorted with the sorted documents in the current sorted list that are located before the current document to be sorted from back to front, and obtain the corresponding comparison results, and then judge whether the correlation between the current document to be sorted and the target retrieval problem is higher than the correlation between the current sorted document and the target retrieval problem based on the comparison results. If higher, the order of the current sorted document will be moved back 1, and then set the previous document of the current sorted document as the new current sorted document; then, call the step of judging whether the relevance between the current document to be sorted and the target retrieval problem is higher than the relevance between the current sorted document and the target retrieval problem based on the comparison result, and then determine the insertion position order of the current document to be sorted; finally, determine the new current sorted list based on the current document to be sorted and the insertion position order, and judge whether there is an unsorted document in the current sorted list. If so, jump to the step of selecting the unsorted current document to be sorted in the current sorted list to obtain the target re-sorted document list, and determine the annotation data based on the target re-sorted document list. In this way, the efficiency of re-sorting documents is improved in the data pre-annotation process based on document re-sorting, thereby improving the efficiency of data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0050] Figure 1 This is a flow chart of a data pre-annotation method based on document reordering disclosed in this application;

[0051] Figure 2 This is a schematic structural diagram of a data pre-annotation device based on document reordering disclosed in this application;

[0052] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] Currently, there are four schemes for constructing re-ranking datasets: First, constructing a re-ranking dataset through user feedback collection, but since the effectiveness is highly dependent on user participation and feedback quality, there is a certain uncertainty in the quality of the dataset; second, constructing a re-ranking dataset through manual annotation, but its high cost is a disadvantage that cannot be ignored; third, constructing a re-ranking dataset through intelligent assisted annotation, but the performance is not good enough on proprietary datasets in specific business scenarios; fourth, constructing a re-ranking dataset through a large model, but when query questions and a large, long document collection are directly input into the large model, not only will each inference take a long time, but due to the complexity and length of the input information, the accuracy of the model will also face greater challenges. To this end, the present application provides a data pre-annotation method based on document re-ranking, which can improve the efficiency of re-ranking documents in the data pre-annotation process based on document re-ranking, thereby improving the efficiency of data annotation.

[0055] See also Figure 1 As shown, an embodiment of the present invention discloses a data pre-labeling method based on document reordering, comprising:

[0056] Step S11: Generate a current sorting list based on a document set according to a preset random sorting rule, set the first document to be sorted in the current sorting list as a sorted document, and then determine the next document to be sorted in the current sorting list as the current document to be sorted.

[0057] In this embodiment, before performing data pre-labeling based on document reordering, the embodiment of the present application chooses to use large model technology and traditional quick sorting algorithm to reorder and pre-label documents to obtain a document reordering dataset containing n samples: .in, Each sample in , The organizational form can be abstracted as ,in, represents the target retrieval problem, is a list of documents, and , The k documents in the search are initially ranked by the searcher and considered to be The k most relevant documents, each Represents a document.

[0058] It is worth mentioning that Is an array used to store the relative order of each document relative to the query The relevance label of Is an indicator that represents the documents in P according to the query The actual optimal ranking position after correlation evaluation. and , the embodiment of the present application can be obtained by the following process :

[0059] First, design a prompt to use the above instructions to guide the large language model to judge given any target retrieval question and two documents to be sorted and Which document has a higher relevance to the question, that is, to guide the large model to judge and 、 Which is more relevant.

[0060] In a specific embodiment, = "Kuanzhai Alley, District A", .in, = "Area A includes 12 urban districts and 2 economic functional zones. In the central urban area, famous attractions include Kuanzhai Alley."; ="Kuanzhai Alley is located near B Street. It consists of Kuanxiang, Zhaixiang, and Jingxiang, all arranged in parallel. These ancient courtyards are all decorated with indigo bricks and tiles, making it a relatively large ancient street. In the 1980s, Kuanzhai Alley was included in the "Chengdu Historical and Cultural City Protection Plan." In 2003, the Kuanzhai Alley neighborhood officially appeared in the world's lexicon. Kuanzhai Alley has been awarded several honorary titles. Its Chinese name is "Kuanzhai Alley"; its climate is a subtropical monsoon humid climate; its hours of operation are year-round; its admission price is free; its area is 479 mu (approximately 16 acres); its famous attractions include Kuanxiangzi, Jingxiangzi, Zhaixiangzi, and the Horse Tie Stone." ="The author of "Kuanxiangzi and Zhaixiang" told reporters that the book includes over 200 old photos of Kuanxiang and Zhaixiang before restoration, selected from over 2,000 photos he took in 2005. Interestingly, the author also provides essays for each image. The book will provide readers with a deeper understanding of Kuanxiang and Zhaixiang, and a deeper understanding of Chengdu culture. It is reported that the author is preparing to release a picture book showcasing the restored Kuanxiang and Zhaixiang. He hopes that more readers will experience the charm of Kuanxiang and Zhaixiang through the comparison of the before and after restoration."; ="Kuanxiangzi is a large-scale ancient street from the Qing Dynasty. Together with its narrow alleys, it epitomizes City B's past, a symbol etched deep in memory. As visitors gaze at the setting sun and the smoke from cooking fires, they experience long-lost scenes of urban life in the old city. Chinese name: Kuanxiangzi; History: Over 2,300 years old."; ="Kuanxiangzi is located in District C. It's lined with shops, primarily restaurants, but also beef and mutton shops, dried fruit shops, fruit shops, and supermarkets. Breakfast here is primarily wheat, with a wide variety available. Shaomai, mutton offal, and baked buns are the most popular. Kuanxiangzi in District C is an urban neighborhood with distinct ethnic characteristics and a high reputation. This popularity is primarily due to its vibrant culinary culture. Many of the city's most renowned time-honored brands and establishments are concentrated here, and local favorites like shumai, mutton offal, and baked buns are particularly popular. District C suffers from a significant temperature swing between day and night, so night markets aren't as popular. However, the entrance to Kuanxiangzi attracts crowds of diners on summer nights. During festivals, the area becomes bustling and unique."

[0061] Based on the above target retrieval question and document list, the instruction designed in the embodiment of the present application is: "Please complete a relevance comparison task. I will show you a question and two documents. Please judge which document is more relevant to the question and can better help readers understand and answer the question. Please output the serial number of the document with higher relevance without analyzing the reason. Now, the question is: , the content of document 1 is: , the content of document 2 is: Now, please start to judge the relevance of Document 1 and Document 2 to the question. "

[0062] Specifically, according to the preset random sorting rules and based on the document collection, the current sorting list is generated, and the first document to be sorted in the current sorting list is set as the sorted document. Before the next document to be sorted in the current sorting list is determined as the current document to be sorted, it can also include: constructing a preset instruction template based on preset constraints; the preset constraints include document length constraints, format difference constraints and semantic relevance constraints; determining the input formats corresponding to the target retrieval problem and the documents to be sorted respectively, and then using the preset instruction template and determining the relevance judgment instruction based on the input format, the target retrieval problem and the documents to be sorted, so that when the large language model receives the relevance judgment instruction, it judges the relevance between the target retrieval problem and the documents to be sorted and obtains a relevance judgment result; the structural form of the relevance judgment result is a preset symbol structure form; the relevance judgment result is the document in each document to be sorted that is more relevant to the target retrieval problem.

[0063] Furthermore, the embodiment of the present application needs to be initialized , as Each The initial sorting of At the same time, assuming The first element of is already a sorted subarray, that is, assuming Already with The most relevant documents. Then, the embodiment of the present application will The second document in Set as the document to be sorted, called , and its corresponding indicator , called .at this time, The value of Specifically, generating a current sorting list according to a preset random sorting rule and based on a document collection, and setting the first document to be sorted in the current sorting list as a sorted document, may include: randomly selecting a document to be sorted from the document collection, setting the selected document as a sorted document, and setting the position of the sorted document in the current sorting list as the first position to obtain a sorting list to be processed; randomly sorting all documents in the document collection except the sorted document in the first position according to the preset random sorting rule to obtain a random sorting result, and then determining the current sorting list based on the random sorting result and the sorting list to be processed; each element in the current sorting list includes each document and the corresponding order of each document.

[0064] Step S12: According to the order of the sorted documents from back to front, a sorted document in the current sorted list that has not been selected in the current round of document sorting is selected as the current sorted document to be compared.

[0065] It is worth mentioning that when determining the next document to be sorted in the current sorting list as the current document to be sorted, the embodiment of the present application needs to select a sorted document in the current sorting list that has not been selected in this round of document sorting process as the current sorted document to be compared in the order of the sorted documents from back to front, so as to compare the current sorted document to be compared with the current document to be sorted.

[0066] Step S13: Use the preset large language model to determine whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question. If so, swap the positions of the current document to be sorted and the current sorted document to be compared in the current sorting list, and then jump again to the step of selecting a sorted document in the current sorting list that has not been selected in this round of document sorting as the current sorted document to be compared.

[0067] In this embodiment, after determining the sorted document to be compared and the document to be sorted, the embodiment of the present application may Compare with the currently sorted documents from back to front. It is worth mentioning that the comparison method is to 、 and a document in the ordered interval Input the big model and use the above instructions to guide the big model to make correlation judgments.

[0068] It is worth mentioning that the embodiment of this application needs to 、 、 Fill in the contents of the command 、 、 The corresponding position is obtained to obtain a complete instruction to input the above instruction into the large model. For example, when inputting the "Tongyi Qianwen" model, the answer obtained is: "Document 2". In other words, the large model is used to judge Compare Then, if the large model determines and Correlation ratio and If the relevance is higher or equal, the indicator corresponding to the document Shift right one position to Make room, that is, Updated to . Again, Compare with the currently sorted documents in the document until the The position where it should be inserted so that when a suitable position is found, Insert it into the corresponding position and The value of is updated to

[0069] Specifically, using a preset large language model to determine whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question, and if so, swapping the positions of the current document to be sorted and the current sorted document to be compared in the current sorted list, can include: after the preset large language model receives a large language model call request, comparing the correlation between the current document to be sorted and the target retrieval question with the correlation between the current sorted document to be compared and the target retrieval question, and obtaining a comparison result corresponding to the current sorted document to be compared; the large language model call request includes a retrieval paragraph corresponding to the target retrieval question; converting the comparison result to obtain a binary judgment signal, and judging whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question based on the binary judgment signal; if the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question, shifting the document position indicator value corresponding to the current sorted document to be compared by one position to the right;

[0070] Correspondingly, if not, the document sorting process for the current document to be sorted is completed, which may include: if the correlation between the current document to be sorted and the target retrieval problem is not higher than the correlation between the current sorted document to be compared and the target retrieval problem, then terminating the operation of comparing the current document to be sorted with the sorted document, and determining the insertion position corresponding to the current document to be sorted as the adjacent position on the right side of the current sorted document to be compared.

[0071] It is worth mentioning that the embodiment of the present application needs to compare each document to be sorted in the document list with the currently sorted document to be compared, that is, each document is sequentially used as , then find its correct position in the sorted part and insert it.

[0072] In one embodiment, when the After sorting, set the current document to be sorted to , and the ordered interval corresponding to the current document to be sorted is [2,1], =3, then Compare the documents in the previous ordered interval from back to front, that is, 、 、 The content is filled into the instruction. If the result obtained by the large model is: The higher the correlation, the Shift right one position to Make room and The value of is updated to: Then, 、 、 The content is filled into the instruction, and then judged based on the above instructions and the large model. If the result is: The correlation is higher than At this time, it is not correct Moving is not right The value of is updated, and The corresponding value is still .at this time, The comparison with each sorted document has been completed and the appropriate insertion position has been found. At this time, The value of is updated to .

[0073] In addition, for , the embodiment of this application needs to 、 、 The content of the command is filled in, and then the model judges Compare More relevant and will Updated to ; Then, 、 、 The content is filled into the instruction, and then the model judges Compare More relevant, Updated to , and then 、 、 Fill in the content of the instruction, and then the large model judges Compare More relevant, no Perform the update operation and obtain The value of .

[0074] Finally, for , corresponding to The value of is 5, and 、 、 The content is filled into the instruction, and then the model judges Compare More relevant. Since in the existing ordered sequence [2,4,3,1], “1” points to Already currently with The least relevant content, while Correlation ratio is lower, then there is no need to compare with documents ranked higher, and finally, The value of .

[0075] Specifically, swapping the positions of the current document to be sorted and the current sorted document to be compared in the current sort list may include: determining the document position indicator values ​​corresponding to all documents in the current sort list that are located after the insertion position corresponding to the insertion position order, and incrementing the determined document position indicator values ​​in sequence to obtain corresponding new document position indicator values; shifting the corresponding documents based on the new document position indicator values ​​to obtain an insertion position vacancy, and modifying the original document position indicator value corresponding to the current document to be sorted to the document position indicator value corresponding to the insertion position vacancy.

[0076] Step S14: If not, complete this round of document sorting process for the current document to be sorted, and jump again to the step of determining the next document to be sorted in the current sorting list as the current document to be sorted, until the document sorting process of all documents to be sorted in the current sorting list has been completed, and the target re-sorted document list is obtained.

[0077] In this embodiment, when using a preset large language model to determine whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question, if the correlation between the current document to be sorted and the target retrieval question is not higher than the correlation between the current sorted document to be compared and the target retrieval question, it is necessary to swap the corresponding positions of the current document to be sorted and the current sorted document to be compared in the current sorting list, and then reselect the current sorted document to be compared to continue comparing the correlation between the current document to be sorted and the current sorted document to be compared with the target retrieval question, that is, jump again to the step of selecting a sorted document in the current sorting list that has not been selected in this round of document sorting as the current sorted document to be compared.

[0078] Step S15: pre-labeling the ranked documents based on the correlations between the ranked documents in the target re-ranked document list to obtain a labeled document dataset.

[0079] In this embodiment, after all documents in the document list have been compared for relevance, the array L becomes an array according to and An array sorted from largest to smallest in terms of relevance. Specifically, pre-labeling the sorted documents based on the relevance between the sorted documents in the target re-ranked document list to obtain a labeled document dataset can include: associating and storing the order of each document stored in the target re-ranked document list with the target retrieval question to obtain labeled data including the target retrieval question, the set of documents in the target re-ranked document list, and the order of the documents in the target re-ranked document list; generating numerical labeling tags based on the order corresponding to each document in the target re-ranked document list; the numerical labeling tags are used to quantify the strength of the relevance between the document and the question; determining a labeled document dataset based on the labeling data and the numerical labeling tags, training a to-be-trained model using the labeled document dataset and based on weights corresponding to the numerical labeling tags, and then processing the target retrieval question using the trained model.

[0080] It is worth mentioning that the embodiment of the present application uses the idea of ​​the "insertion sort" sorting algorithm to sort documents. In addition, in terms of time complexity, for the best case, that is, When the initial sorting is already the optimal sorting, the time complexity of each document insertion is , and in the worst case, that is When the initial sequence is completely the reverse order of the optimal sort, the time complexity is .because The initial order of the documents in comes from the result of the preliminary ordering by the retriever, which is usually not completely reversed from the optimal order. At the same time, in the general document re-ranking dataset, each Corresponding The number of documents k in is usually controllable, thus ensuring that the time of the above intelligent pre-labeling algorithm can be controlled within an acceptable range. In addition, since the insertion sort algorithm based on pairwise comparison is used, only two documents need to be input into the model for comparison each time. The correlation between the two documents can be judged without inputting all the documents into the model, which greatly alleviates the problem of large model performance being affected by long and complex input.

[0081] It can be seen that the embodiment of the present application first needs to generate a current sorted list based on the document set according to the preset random sorting rules, and set the first document to be sorted in the current sorted list as the sorted document, and then select the current document to be sorted that has not been sorted from the current sorted list; then, use the large language model to compare the current document to be sorted with the sorted documents in the current sorted list that are located before the current document to be sorted from back to front, and obtain the corresponding comparison results, and then judge whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document and the target retrieval question based on the comparison results. If higher, the order of the current sorted document is changed to the next one. Shift one position, and then set the previous document of the current sorted document as the new current sorted document; then, call the step of judging whether the relevance between the current document to be sorted and the target retrieval problem is higher than the relevance between the current sorted document and the target retrieval problem based on the comparison result, and then determine the insertion position order of the current document to be sorted; finally, determine the new current sorted list based on the current document to be sorted and the insertion position order, and judge whether there is an unsorted document in the current sorted list. If so, jump to the step of selecting the unsorted current document to be sorted in the current sorted list to obtain the target re-sorted document list, and determine the annotation data based on the target re-sorted document list. In this way, the efficiency of re-sorting documents is improved in the data pre-annotation process based on document re-sorting, thereby improving the efficiency of data annotation.

[0082] Accordingly, see Figure 2 As shown, the present application also provides a data pre-annotation device based on document reordering, comprising:

[0083] The document determination module 11 is configured to generate a current sorting list based on a document set according to a preset random sorting rule, set the first document to be sorted in the current sorting list as a sorted document, and then determine the next document to be sorted in the current sorting list as the current document to be sorted;

[0084] The document selection module 12 is configured to select, in order from back to front, a sorted document in the current sort list that has not been selected in the current document sorting process as the current sorted document to be compared;

[0085] The document position swapping module 13 is configured to use a preset large language model to determine whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question; if so, swap the positions of the current document to be sorted and the current sorted document to be compared in the current sorted list, and then jump back to the step of selecting a sorted document in the current sorted list that has not been selected in the current document sorting process as the current sorted document to be compared;

[0086] a loop jump module 14 configured to, if not, complete the document sorting process for the current document to be sorted in this round and re-jump to the step of determining the next document to be sorted in the current sorting list as the current document to be sorted, until the document sorting process for all documents to be sorted in the current sorting list is completed, thereby obtaining a target re-sorted document list;

[0087] The document pre-annotation module 15 is configured to pre-annotate the sorted documents based on the correlations between the sorted documents in the target re-sorted document list to obtain a marked document dataset.

[0088] As can be seen from the above, before performing data pre-annotation based on document re-ranking, the embodiment of the present application first needs to generate a current sorted list based on a preset random sorting rule and based on a document set, and set the first document to be sorted in the current sorted list as a sorted document, and then select the current document to be sorted that has not been sorted from the current sorted list; then, use the large language model to compare the current document to be sorted with each sorted document in the current sorted list that is located before the current document to be sorted from back to front, and obtain corresponding comparison results, and then judge whether the correlation between the current document to be sorted and the target retrieval problem is higher than the correlation between the current sorted document and the target retrieval problem based on the comparison results. If higher, the current document to be sorted will be compared with the sorted documents that are located before the current document to be sorted in the current sorted list. The order of the previously sorted document is shifted back one position, and then the previous document of the currently sorted document is set as the new currently sorted document; then, a step is called to determine whether the relevance between the current document to be sorted and the target retrieval problem is higher than the relevance between the currently sorted document and the target retrieval problem based on the comparison result, and then the insertion position order of the currently sorted document is determined; finally, a new currently sorted list is determined based on the currently sorted document and the insertion position order, and it is determined whether there is an unsorted document in the currently sorted list. If so, the step jumps to the step of selecting the unsorted currently sorted document in the currently sorted list to obtain the target re-sorted document list, and the annotation data is determined based on the target re-sorted document list. In this way, the efficiency of re-sorting documents is improved in the data pre-annotation process based on document re-sorting, thereby improving the efficiency of data annotation.

[0089] In some specific implementations, the data pre-labeling apparatus based on document reordering may further include:

[0090] An instruction template construction unit is used to construct a preset instruction template based on preset constraints; the preset constraints include document length constraints, format difference constraints, and semantic relevance constraints;

[0091] A judgment instruction acquisition unit is used to determine the input formats corresponding to the target retrieval question and the documents to be sorted, and then use a preset instruction template and determine the relevance judgment instruction based on the input format, the target retrieval question and the documents to be sorted, so that when the large language model receives the relevance judgment instruction, it judges the relevance between the target retrieval question and the documents to be sorted and obtains a relevance judgment result; the structural form of the relevance judgment result is a preset symbolic structure form; the relevance judgment result is the document in each of the documents to be sorted that is more relevant to the target retrieval question.

[0092] In some specific implementations, the document determination module 11 may specifically include:

[0093] A first sort list determination unit is configured to randomly select a document to be sorted from the document collection, set the selected document as a sorted document, and set the position of the sorted document in the current sort list as the first position, thereby obtaining a sort list to be processed;

[0094] The second sorting list determination unit is used to randomly sort all documents in the document collection except the sorted document at the first position according to a preset random sorting rule to obtain a random sorting result, and then determine a current sorting list based on the random sorting result and the sorting list to be processed; each element in the current sorting list includes each document and the corresponding order of each document.

[0095] In some specific implementations, the document position replacement module 13 may specifically include:

[0096] a comparison result determining unit configured to compare the relevance between the current document to be sorted and the target retrieval question with the relevance between the current sorted document to be compared and the target retrieval question after the preset large language model receives a large language model call request, thereby obtaining a comparison result corresponding to the current sorted document to be compared; the large language model call request includes a retrieval paragraph corresponding to the target retrieval question;

[0097] an indicator shifting unit, configured to convert the comparison result to obtain a binary judgment signal, and determine, based on the binary judgment signal, whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question; if the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question, shifting the document position indicator value corresponding to the current sorted document to be compared right by one position;

[0098] Accordingly, the loop jump module may specifically include:

[0099] An insertion position determining unit is configured to terminate the operation of comparing the current document to be sorted with the sorted document if the correlation between the current document to be sorted and the target retrieval question is not higher than the correlation between the current sorted document to be compared and the target retrieval question, and determine the insertion position corresponding to the current document to be sorted as the right adjacent position of the current sorted document to be compared, and then jump to the step of determining the next document to be sorted in the current sorted list as the current document to be sorted.

[0100] In some specific implementations, the document position replacement module 13 may specifically include:

[0101] an indicator value acquiring unit, configured to determine document position indicator values ​​corresponding to all documents located after the insertion position corresponding to the insertion position order in the current sorted list, and to increment each determined document position indicator value in sequence to obtain a corresponding new document position indicator value;

[0102] The indicator value modification unit is used to shift the corresponding document based on each new document position indicator value to obtain an insertion position space, and modify the original document position indicator value corresponding to the current document to be sorted to the document position indicator value corresponding to the insertion position space.

[0103] In some specific implementations, the document pre-annotation module 15 may specifically include:

[0104] a labeling data acquisition unit, configured to associate and store the order of each document stored in the target reordered document list with the target retrieval question, and obtain labeling data including the target retrieval question, the document set in the target reordered document list, and the order of the documents in the target reordered document list;

[0105] a label determination unit, configured to generate a numerical label based on the order of the documents in the target reordered document list; the numerical label is used to quantify the relevance between the document and the question;

[0106] A model training unit is used to determine a labeled document dataset based on the labeled data and the numerical labeled labels, to train a to-be-trained model using the labeled document dataset and based on the weights corresponding to the numerical labeled labels, and then to process the target retrieval problem using the obtained trained model.

[0107] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data pre-annotation method based on document reordering disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0108] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0109] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0110] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of implementing the data pre-annotation method based on document reordering performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program capable of implementing other specific tasks.

[0111] Furthermore, this application discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned document reordering-based data pre-annotation method. The specific steps of this method can be found in the corresponding contents disclosed in the aforementioned embodiments and will not be further described here.

[0112] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0113] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0115] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0116] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data pre-annotation method based on document reordering, characterized in that: include: Generating a current sorting list based on a document set according to a preset random sorting rule, setting the first document to be sorted in the current sorting list as a sorted document, and then determining the next document to be sorted in the current sorting list as the current document to be sorted; According to the order of the sorted documents from back to front, a sorted document in the current sorted list that has not been selected in the current round of document sorting is selected as the current sorted document to be compared; Determine using a preset large language model whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question; if so, swap the positions of the current document to be sorted and the current sorted document to be compared in the current sorted list, and then jump back to the step of selecting a sorted document in the current sorted list that has not been selected in the current document sorting process as the current sorted document to be compared; If not, then the document sorting process for the current document to be sorted is completed, and the process jumps back to the step of determining the next document to be sorted in the current sorting list as the current document to be sorted, until the document sorting process for all documents to be sorted in the current sorting list is completed, thereby obtaining the target re-sorted document list; The sorted documents are pre-labeled based on the correlations between the sorted documents in the target re-sorted document list to obtain a labeled document dataset.

2. The data pre-annotation method based on document reordering according to claim 1 is characterized in that: The method further includes: generating a current sorted list based on a document set according to a preset random sorting rule, setting the first document to be sorted in the current sorted list as a sorted document, and determining the next document to be sorted in the current sorted list as the current document to be sorted. Constructing a preset instruction template based on preset constraints; the preset constraints include document length constraints, format difference constraints, and semantic relevance constraints; Determine the input formats corresponding to the target retrieval question and the documents to be sorted, respectively, and then use a preset instruction template and determine a relevance judgment instruction based on the input format, the target retrieval question and the documents to be sorted, so that when the large language model receives the relevance judgment instruction, it judges the relevance between the target retrieval question and the documents to be sorted and obtains a relevance judgment result; the structural form of the relevance judgment result is a preset symbolic structure form; the relevance judgment result is the document in each of the documents to be sorted that is more relevant to the target retrieval question.

3. The data pre-annotation method based on document reordering according to claim 1 is characterized in that: The step of generating a current sorting list based on a document set according to a preset random sorting rule, and setting the first document to be sorted in the current sorting list as a sorted document, includes: Randomly select a document to be sorted from the document collection, set the selected document as a sorted document, and set the position of the sorted document in the current sort list to the first position, to obtain a sort list to be processed; All documents in the document collection except the sorted document at the first position are randomly sorted according to a preset random sorting rule to obtain a random sorting result, and then a current sorting list is determined based on the random sorting result and the to-be-processed sorting list; each element in the current sorting list includes each document and the corresponding order of each document.

4. The data pre-annotation method based on document reordering according to claim 1 is characterized in that: The method of using a preset large language model to determine whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question, and if so, swapping the positions of the current document to be sorted and the current sorted document to be compared in the current sorted list, includes: After the preset large language model receives the large language model call request, the preset large language model compares the relevance between the current document to be sorted and the target retrieval question with the relevance between the current sorted document to be compared and the target retrieval question, thereby obtaining a comparison result corresponding to the current sorted document to be compared; the large language model call request includes a retrieval paragraph corresponding to the target retrieval question; Performing conversion processing on the comparison result to obtain a binary judgment signal, and judging whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question based on the binary judgment signal; if the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question, shifting the document position indicator value corresponding to the current sorted document to be compared right by one position; Correspondingly, if not, the document sorting process for the current document to be sorted is completed, including: If the correlation between the current document to be sorted and the target retrieval question is not higher than the correlation between the current sorted document to be compared and the target retrieval question, the operation of comparing the current document to be sorted with the sorted document is terminated, and the insertion position corresponding to the current document to be sorted is determined to be the right adjacent position of the current sorted document to be compared.

5. The data pre-annotation method based on document reordering according to claim 1 is characterized in that: The step of swapping the positions of the current document to be sorted and the current sorted document to be compared in the current sort list includes: Determining document position indicator values ​​corresponding to all documents in the current sorted list that are located after the insertion position corresponding to the insertion position order, and sequentially incrementing each determined document position indicator value to obtain corresponding new document position indicator values; Based on each of the new document position indicator values, the corresponding document is shifted to obtain an insertion position vacancy, and the original document position indicator value corresponding to the current document to be sorted is modified to the document position indicator value corresponding to the insertion position vacancy.

6. The data pre-labeling method based on document reordering according to any one of claims 1 to 5, characterized in that: The pre-labeling of the sorted documents based on the correlations between the sorted documents in the target re-sorted document list to obtain a labeled document dataset includes: Associating and storing the order of each document stored in the target reordered document list with the target retrieval question, to obtain annotation data including the target retrieval question, the document set in the target reordered document list, and the order of the documents in the target reordered document list; Generating numerical annotation labels based on the order of each document in the target reordered document list; the numerical annotation labels are used to quantify the relevance strength between the document and the question; Based on the labeled data and the numerical labeled labels, a labeled document dataset is determined, and a training model is trained using the labeled document dataset and based on weights corresponding to the numerical labeled labels, and then the obtained trained model is used to process the target retrieval problem.

7. A data pre-labeling device based on document reordering, characterized in that: include: A document determination module is configured to generate a current sorting list based on a document set according to a preset random sorting rule, set the first document to be sorted in the current sorting list as a sorted document, and then determine the next document to be sorted in the current sorting list as the current document to be sorted; A document selection module is used to select, in order from back to front of the sorted documents, a sorted document in the current sorted list that has not been selected in the current round of document sorting as the current sorted document to be compared; a document position swapping module, configured to use a preset large language model to determine whether the relevance between the current document to be sorted and the target retrieval question is higher than the relevance between the current sorted document to be compared and the target retrieval question; if so, swapping the positions of the current document to be sorted and the current sorted document to be compared in the current sorted list, and then re-jumping to the step of selecting a sorted document in the current sorted list that has not been selected in the current document sorting process as the current sorted document to be compared; a loop jump module, configured to, if no, complete the document sorting process for the current document to be sorted in this round and jump again to the step of determining the next document to be sorted in the current sorting list as the current document to be sorted, until the document sorting process for all documents to be sorted in the current sorting list is completed, thereby obtaining a target re-sorted document list; The document pre-labeling module is used to pre-label the sorted documents based on the correlations between the sorted documents in the target re-sorted document list to obtain a labeled document dataset.

8. The data pre-labeling device based on document reordering according to claim 7, characterized in that: The document position exchange module includes: a comparison result determining unit configured to compare the relevance between the current document to be sorted and the target retrieval question with the relevance between the current sorted document to be compared and the target retrieval question after the preset large language model receives a large language model call request, thereby obtaining a comparison result corresponding to the current sorted document to be compared; the large language model call request includes a retrieval paragraph corresponding to the target retrieval question; an indicator shifting unit, configured to convert the comparison result to obtain a binary judgment signal, and determine, based on the binary judgment signal, whether the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question; if the correlation between the current document to be sorted and the target retrieval question is higher than the correlation between the current sorted document to be compared and the target retrieval question, shifting the document position indicator value corresponding to the current sorted document to be compared right by one position; Correspondingly, the loop jump module is specifically used to terminate the operation of comparing the current document to be sorted with the sorted document if the correlation between the current document to be sorted and the target retrieval problem is not higher than the correlation between the current sorted document to be compared and the target retrieval problem, and determine the insertion position corresponding to the current document to be sorted as the right adjacent position of the current sorted document to be compared, and then jump to the step of determining the next document to be sorted in the current sorted list as the current document to be sorted.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the data pre-labeling method based on document reordering according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the data pre-labeling method based on document reordering according to any one of claims 1 to 6 is implemented.

Images