Document duplicate checking method, device and equipment based on large language model and storage medium

Abstract

The embodiment of the invention relates to the technical field of document duplicate checking, and discloses a document duplicate checking method and device based on a large language model, equipment and a storage medium. The document duplicate checking method based on the large language model can comprise the steps of obtaining a document to be subjected to duplicate checking and a document feature library corresponding to a plurality of historical documents; performing at least one semantic summary on the to-be-duplicated document based on the large language model to obtain a target summary document; performing feature extraction on the target summary document to obtain target document features; and determining a target recall document based on the target document features and a document feature library. Based on the technical scheme of the invention, the technical problem that the accuracy of recalled historical documents is relatively low can be solved. Meanwhile, as the target document features are obtained by performing feature extraction on the target summary document, compared with feature extraction of the document to be subjected to duplicate checking before semantic summary, the calculation amount is smaller, and the document duplicate checking speed can be increased.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of document duplication detection, and specifically to a document duplication detection method, apparatus, device and storage medium based on a large language model. Background Art

[0002] Currently, when checking for duplicate documents, a feature extraction model is used to extract semantic features from a large number of historical documents, generating a large number of historical semantic features. Next, a document feature library is established based on these historical semantic features. Next, the same semantic feature extraction is performed on the document to be checked for duplicates, generating semantic features corresponding to the document. Finally, the semantic document feature library is searched based on the semantic features corresponding to the document to be checked for duplicates, and historical documents with a high degree of similarity are retrieved.

[0003] However, in the above-mentioned document duplication checking method, when semantic features are extracted from a large number of historical documents obtained through a feature extraction model, a large number of historical documents need to be processed by word segmentation, sentence segmentation, division, etc. Therefore, due to the limitation of the feature extraction model on the length of the input document, the semantic features obtained by extracting semantic features from a large number of historical documents are relatively local and not comprehensive enough, resulting in a relatively low accuracy rate of historical documents recalled based on the semantic features. Summary of the Invention

[0004] In view of the above problems, an embodiment of the present invention provides a document duplicate checking method based on a large language model, which is used to solve the problem that the accuracy of historical documents recalled by the document duplicate checking and recall method in the prior art is relatively low.

[0005] A first aspect of an embodiment of the present application provides a document duplication checking method using a large language model, the method comprising: obtaining a document feature library corresponding to a document to be checked for duplicates and a plurality of historical documents; wherein the historical document comprises a plurality of historical chapters, and the document feature library comprises historical chapter features corresponding to each historical chapter; performing at least one semantic summary of the document to be checked for duplicates based on the large language model to obtain a target summary document; performing feature extraction on the target summary document to obtain target document features; and determining a target recall document based on the target document features and the document feature library.

[0006] In some embodiments, a semantic summary is performed on the document to be checked for duplicates at least once based on a large language model to obtain a target summary document corresponding to the document to be processed, including: dividing the document to be checked for duplicates by chapters to obtain multiple chapters to be checked; a semantic summary is performed on each chapter to be checked at least once based on a large language model to obtain a target chapter summary corresponding to each chapter to be checked; wherein, the target summary document includes a target chapter summary corresponding to each chapter to be checked; the target document features include chapter features corresponding to each chapter to be checked; and feature extraction is performed on the target summary document to obtain target document features, including: feature extraction is performed on each target chapter summary to obtain chapter features to be checked.

[0007] In some embodiments, a semantic summary is performed on each to-be-checked chapter at least once based on a large language model to obtain a target chapter summary corresponding to each to-be-checked chapter, including: a semantic summary is performed on each to-be-checked chapter based on a large language model to obtain an initial chapter summary corresponding to each to-be-checked chapter; a semantic summary is performed on the initial chapter summary based on a large language model to obtain a target chapter summary corresponding to each to-be-checked chapter.

[0008] In some embodiments, a semantic summary is performed on each to-be-checked chapter based on a large language model to obtain an initial chapter summary corresponding to each to-be-checked chapter, including: sentence segmentation and encoding of each to-be-checked chapter based on an encoder in the large language model to obtain a plurality of to-be-checked sentence features; feature fusion of each to-be-checked sentence features based on a decoder in the large language model to obtain a fused feature of the to-be-checked sentence; data conversion processing is performed on the fused feature of each to-be-checked sentence to obtain a semantic summary of the to-be-checked sentence; semantic aggregation is performed on the semantic summaries of the plurality of to-be-checked sentence summaries to obtain an initial chapter summary corresponding to each to-be-checked chapter.

[0009] In some embodiments, the target document features include multiple chapter features to be checked, and the target recall document is determined based on the target document features and the document feature library, including: retrieving multiple corresponding historical chapter features in the document feature library based on each chapter feature to be checked, obtaining at least one target recall feature corresponding to each chapter feature to be checked, and the chapter similarity between the chapter feature to be checked and each target recall feature; determining the document similarity between each historical document and the document to be checked for duplicates based on the chapter similarity between the chapter feature to be checked and each target recall feature; and determining the target recall document from multiple historical documents based on the document similarity between each historical document and the document to be checked for duplicates.

[0010] In some embodiments, based on the chapter similarity between the chapter feature to be checked and each target recall feature, the document similarity between each historical document and the document to be checked for duplicates is determined, including: based on the chapter similarity between the chapter feature to be checked and each target recall feature, determining the similarity between the historical chapter feature of each historical document and the corresponding chapter feature to be checked; determining the weight parameter corresponding to each chapter feature to be checked; based on the weight parameter corresponding to each chapter feature to be checked, weighting the similarity between the historical chapter feature of each historical document and the corresponding chapter feature to be checked to obtain the document similarity between each historical document and the document to be checked for duplicates.

[0011] In some embodiments, obtaining a document feature library corresponding to multiple historical documents includes: obtaining multiple historical documents; dividing each historical document into chapters to obtain multiple historical chapters; performing at least one semantic summary on each historical chapter based on a large language model to obtain a historical chapter summary corresponding to each historical chapter; performing feature extraction on the historical chapter summary corresponding to each historical chapter to obtain historical chapter features corresponding to each historical chapter; and constructing a document feature library based on the historical chapter features corresponding to each historical chapter.

[0012] The second aspect of the embodiment of the present application provides a document duplication checking device based on a large language model, which includes: an acquisition module, used to obtain a document feature library corresponding to a document to be checked for duplicates and multiple historical documents; wherein the historical document includes multiple historical chapters, and the document feature library includes historical chapter features corresponding to each historical chapter; a semantic summarization module, used to perform at least one semantic summary on the document to be checked for duplicates based on the large language model, and obtain a target summary document corresponding to the document to be checked for duplicates; a feature extraction module, used to perform feature extraction on the target summary document, and obtain target document features; a determination module, used to determine the target recall document based on the target document features and the document feature library.

[0013] A third aspect of the embodiments of the present application provides an electronic device, the electronic device comprising:

[0014] processor;

[0015] a memory for storing processor-executable instructions;

[0016] A processor is used to read executable instructions from a memory and execute the instructions to implement the document duplication checking method based on a large language model of the first aspect mentioned above.

[0017] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium storing a computer program for executing the document duplication checking method based on a large language model of the first aspect.

[0018] In the document duplication checking method of the large language model provided in the embodiment of the present application, since the large language model has powerful language processing capabilities, excellent reasoning and generalization capabilities, and efficient calculation and optimization capabilities, the document summary document can be determined more accurately by performing at least one semantic summary on the document to be checked for duplicates based on the large language model. Then, feature extraction is performed on the target summary document, and target document features with relatively comprehensive semantics can be obtained. The document feature library is retrieved based on the target document features, and historical documents with high similarity can be accurately recalled from the document feature library, solving the technical problem of relatively low accuracy of recalled historical documents. At the same time, since the target document features are obtained by feature extraction of the target summary document, the amount of calculation is less than the feature extraction of the document to be checked for duplicates before the semantic summary, which is conducive to improving the speed of document duplication checking.

[0019] The above description is only an overview of the technical solutions of the embodiments of the present invention. In order to more clearly understand the technical means of the embodiments of the present invention, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0021] Figure 1 A flowchart of a document duplication checking method based on a large language model provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of another method for checking duplicate documents based on a large language model provided in an embodiment of the present application;

[0023] Figure 3 A flowchart of another method for checking duplicate documents based on a large language model provided in an embodiment of the present application;

[0024] Figure 4 A flowchart of another method for checking duplicate documents based on a large language model provided in an embodiment of the present application;

[0025] Figure 5 A flowchart of another method for checking duplicate documents based on a large language model provided in an embodiment of the present application;

[0026] Figure 6 A flowchart of another method for checking duplicate documents based on a large language model provided in an embodiment of the present application;

[0027] Figure 7A schematic diagram of the structure of a document duplication checking device based on a large language model provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of the structure of another document duplication checking device based on a large language model provided in an embodiment of the present application;

[0029] Figure 9 A schematic diagram of the structure of another document duplication checking device based on a large language model provided in an embodiment of the present application;

[0030] Figure 10 A schematic diagram of the structure of another document duplication checking device based on a large language model provided in an embodiment of the present application;

[0031] Figure 11 A schematic diagram of the structure of another document duplication checking device based on a large language model provided in an embodiment of the present application;

[0032] Figure 12 A schematic diagram of the structure of another document duplication checking device based on a large language model provided in an embodiment of the present application;

[0033] Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of this application, the singular expressions "a", "an", "above", "the", and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise.

[0035] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0036] The terms "first" and "second" in the following embodiments of this application are used for descriptive purposes only and should not be understood as implying or suggesting relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0037] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0038] In the related art, document duplication checking can be performed by word string retrieval. The document duplication checking method based on word string retrieval includes: obtaining a large number of historical documents, performing word segmentation and part-of-speech classification on the large number of historical documents, and obtaining a large number of word strings. Then, a semantic vocabulary is established based on the large number of word strings. Then, the same analysis and part-of-speech classification are performed on the document to be checked for duplicates to obtain the word strings corresponding to the document to be checked for duplicates. Finally, the semantic vocabulary is retrieved based on the word strings corresponding to the document to be checked for duplicates, and historical documents with high similarity are recalled. Since the document duplication checking method based on word string retrieval is based on the word string retrieval of the document, the retrieval speed is very fast and the resources occupied are relatively small. However, since the document duplication checking method based on word string retrieval belongs to string matching, it is impossible to retrieve words with the same semantics, and therefore the retrieval accuracy is very low.

[0039] In order to improve the retrieval accuracy, a method for checking for duplicate documents by means of feature retrieval is proposed. The document duplication checking method based on feature retrieval comprises: extracting semantic features from a large number of historical documents through a semantic feature extraction model to obtain a large number of historical semantic features. Then, a document feature library is established based on the large number of historical semantic features. Then, feature extraction is performed on the document to be checked for duplicates through the same semantic feature extraction model to obtain semantic features corresponding to the document to be checked for duplicates. Finally, the semantic document feature library is retrieved based on the semantic features corresponding to the document to be checked for duplicates, and historical documents with high similarity are recalled. Since the document duplication checking method based on feature retrieval needs to perform word segmentation, sentence segmentation, division, and other processing on a large number of historical documents when extracting semantic features from a large number of historical documents through a semantic feature extraction model, the semantic features obtained by performing semantic feature extraction on a large number of historical documents are relatively local and not comprehensive enough due to the limitation of the feature extraction model on the length of the input document. Therefore, the retrieval accuracy of the document duplication checking method based on feature retrieval is not high enough.

[0040] Based on the above technical problems, an embodiment of the present invention provides a document duplicate checking method based on a large language model. The method performs at least one semantic summary on the document to be checked for duplicates based on the large language model to obtain a target summary document corresponding to the document to be checked for duplicates. Feature extraction is then performed on the target summary document to obtain the target document feature corresponding to the document to be checked for duplicates. Finally, a document feature library is retrieved based on the target document feature, and a target recall document is determined from the document feature library. Since the large language model has powerful language processing capabilities, excellent reasoning and generalization capabilities, and efficient computing and optimization capabilities, the method performs at least one semantic summary on the document to be checked for duplicates based on the large language model, and can more accurately determine the document summary document. Then, feature extraction is performed on the target summary document, and the target document feature with relatively comprehensive semantics can be obtained. The document feature library is retrieved based on the target document feature, and historical documents with high similarity can be accurately recalled from the document feature library, solving the technical problem of relatively low accuracy of recalled historical documents.

[0041] Figure 1 A flowchart of a document duplication checking method based on a large language model is provided in an embodiment of the present application, such as Figure 1 As shown, the document duplication checking method using a large language model can be applied in electronic devices and may include the following steps 101 to 104.

[0042] Step 101: Obtain a document feature library corresponding to the document to be checked for duplicates and multiple historical documents.

[0043] The historical document may include multiple historical chapters, and the document feature library may include historical chapter features corresponding to each historical chapter. In some examples, the document feature library may be a database constructed based on a large number of historical chapter features.

[0044] This application does not limit the number of historical documents in the plurality of historical documents. The embodiment of the present disclosure is illustratively described by taking the case where the number of historical documents is greater than 10,000.

[0045] The number of historical chapters in a historical document is related to the format of the historical document. If the format of the historical document indicates that the historical document consists of an overview, content, methods, results, and conclusions, then the historical document includes a historical overview chapter, a historical content chapter, a historical methods chapter, a historical results chapter, and a historical conclusion chapter. Correspondingly, the document feature library may include historical overview features corresponding to each historical overview chapter, historical content features corresponding to each historical content chapter, historical method features corresponding to each historical methods chapter, historical chapter features corresponding to each historical results chapter, and historical conclusion features corresponding to each historical conclusion chapter.

[0046] For another example, if the format of a historical document indicates that the document consists of a technical field, a technical background, and technical content, then the historical document includes a historical technical field section, a historical background technical section, and a historical technical content section. Correspondingly, the document feature library may include historical technical field features corresponding to the historical technical field section, historical technical background features corresponding to the historical technical background section, and historical technical content features corresponding to the historical technical content section.

[0047] The electronic device may pre-build a document feature library and call upon the library when performing a duplicate check on a document to be checked for duplicates. For example, the electronic device may pre-build a document feature library based on multiple historical documents. In some examples, the electronic device may first divide each historical document into chapters to obtain multiple historical chapters, then determine historical chapter features corresponding to each historical chapter, and finally, build a document feature library based on the historical chapter features corresponding to each historical chapter.

[0048] For example, the electronic device may first perform a semantic summary on each historical chapter, and then perform feature extraction on the obtained historical chapter summary to obtain historical chapter features corresponding to each historical chapter.

[0049] Step 102: Perform at least one semantic summary on the document to be checked for duplicates based on the large language model to obtain a target summary document corresponding to the document to be checked for duplicates.

[0050] For example, the large language model can be a trained neural network model with a semantic summarization function. In some examples, the large language model is input with a document to be checked for duplicates, and the large language model can output a target summary document corresponding to the document to be checked for duplicates.

[0051] Exemplarily, the number of semantic summaries corresponding to at least one semantic summary can be any integer greater than or equal to 1, and is related to the structure of the large language model and the number of words in each chapter to be checked in the duplicate document to be checked. For a large speech model with a fixed structure, if the number of words in the chapter to be checked is more, the number of semantic summaries corresponding to the at least one semantic summary is more. For example, if the number of words in the chapter to be checked is 5000 words, the number of semantic summaries corresponding to the at least one semantic summary can be 2 times, and if the number of words in the document segment is 8000 words, the number of semantic summaries corresponding to the at least one semantic summary can be 3 times. The embodiment of the present application does not limit the number of semantic summaries corresponding to at least one semantic summary. The embodiment of the present application is exemplarily described with the number of semantic summaries corresponding to at least one semantic summary being 2 times.

[0052] Exemplarily, the target summary document can be a document with a fixed word count that is much smaller than the word count of the document to be checked for duplicates. In some examples, the word count of the document to be checked for duplicates can be hundreds of thousands, and the fixed word count can be a few thousand. For example, the word count of the document to be checked for duplicates is 200,000 or 300,000, and the fixed word count can be 2,000 or 3,000 accordingly. The embodiment of the present application does not limit the word count of the target summary document. The embodiment of the present application takes the example of the target summary document having a fixed word count of 2,000 as an example for illustrative explanation.

[0053] The electronic device can call the large speech model and perform at least one semantic summary on the document to be checked for duplicates through the large language model to obtain a target summary document corresponding to the document to be checked for duplicates.

[0054] Step 103: extract features from the target summary document to obtain target document features corresponding to the document to be checked for duplicates.

[0055] Taking the case where the document to be checked for duplicates includes multiple chapters to be checked as an example, the target document features may include chapter features corresponding to each of the chapters to be checked.

[0056] The electronic device can perform feature extraction on the target summary document based on the feature extraction model to obtain target document features corresponding to the document to be checked for duplicates.

[0057] The feature extraction model can be any trained neural network model with feature extraction capabilities. In some examples, the feature extraction model can be part of a trained large language model. In other examples, the feature extraction model can be a neural network model that is trained separately from the large language model. The embodiments of the present application do not limit the type of feature extraction model. The embodiments of the present application are illustrative using an example in which the feature extraction model is a neural network model that is trained separately from the large language model.

[0058] The feature extraction model can be any one of a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a gated recurrent unit (GRU), a residual network (ResNet), a VGGNet (Visual Geometry Group Network), etc. The embodiment of the present application takes the CNN as an example for exemplary description of the feature extraction model.

[0059] Exemplarily, the feature extraction model may include multiple encoding layers and multiple decoding layers. Extracting features from a target summary document based on the feature extraction model to obtain target document features corresponding to the document to be checked for duplicates may include: encoding and embedding the target summary document based on the multiple encoding layers in the feature extraction model to obtain summary document features; and performing comprehensive data fusion on the summary document features based on the multiple decoding layers in the feature extraction model to obtain target document features. The multiple decoding layers may be multiple transformer layers.

[0060] Step 104: Determine the target recall document based on the target document features and the document feature library.

[0061] Exemplarily, the target recall document may be the TopK historical documents that are most similar to the document to be checked for duplicates in terms of overall semantics. In some examples, the target recall document may include historical documents whose number is greater than or equal to 0. For example, the target recall document may include 1 historical document. For another example, the target recall document may include 10 historical documents. The embodiment of the present application does not limit the number of historical documents included in the target recall document. The embodiment of the present application is illustrative for example, taking the case where the target recall document includes 10 historical documents.

[0062] Exemplarily, the electronic device may retrieve a document feature library based on the target document feature, obtain the recalled document feature with the highest similarity to the target document feature, and the similarity between the target document feature and the recalled document feature, and determine the target recalled document from multiple historical documents based on the similarity between the target document feature and the recalled document feature.

[0063] In the document duplication checking method of the large language model provided in the embodiment of the present application, since the large language model has powerful language processing capabilities, excellent reasoning and generalization capabilities, and efficient calculation and optimization capabilities, the document summary document can be determined more accurately by performing at least one semantic summary on the document to be checked for duplicates based on the large language model. Then, feature extraction is performed on the target summary document, and target document features with relatively comprehensive semantics can be obtained. The document feature library is retrieved based on the target document features, and historical documents with high similarity can be accurately recalled from the document feature library, solving the technical problem of relatively low accuracy of recalled historical documents. At the same time, since the target document features are obtained by feature extraction of the target summary document, the amount of calculation is less than the feature extraction of the document to be checked for duplicates before the semantic summary, which is conducive to improving the speed of document duplication checking.

[0064] like Figure 2 As shown in the above Figure 1 On the basis of the illustrated embodiment, step 102 performs at least one semantic summary on the document to be checked for duplicates based on the large language model to obtain a target summary document corresponding to the document to be checked for duplicates, which may include the following steps 1021 and 1022.

[0065] Step 1021, divide the document to be checked for duplicates into chapters to obtain multiple chapters to be checked.

[0066] The quantity of chapters and sections to be checked in a plurality of chapters and sections to be checked can be any integer greater than or equal to 1, and is relevant to the form of duplicate document to be checked.In some examples, the quantity of chapters and sections to be checked in a plurality of chapters and sections to be checked is identical with the number of chapters and sections that the format indication of duplicate document to be checked comprises.For example, if the format indication of duplicate document to be checked comprises overview, content, method, result and conclusion, totally 5 chapters and sections, then a plurality of chapters and sections to be checked comprise overview chapters and sections to be checked, content chapters and sections to be checked, method chapters and sections to be checked, result chapters and sections to be checked and conclusion chapters and sections to be checked, totally 5 chapters and sections to be checked.For another example, if the format indication of duplicate document to be checked comprises technical field, technical background and technical content, totally 3 chapters and sections, then a plurality of chapters and sections to be checked can comprise technical field chapters and sections to be checked, technical background chapters and sections to be checked and technical content chapters and sections to be checked, totally 3 chapters and sections to be checked.

[0067] The electronic device can divide the document to be checked for duplicates by extracting the chapter titles of the document to be checked for duplicates. In some examples, the electronic device can parse the document to be checked for duplicates based on a document parser, extract the chapter titles of the document to be checked for duplicates, and thus obtain multiple chapters to be checked. Here, the document parser can be an application with a document parsing function, and by inputting the document to be checked for duplicates into the document parser, the document parser can automatically parse out multiple chapters to be checked corresponding to the document to be checked for duplicates. For example, if the document to be checked for duplicates includes an overview, content, method, results, and conclusion, then after the document parser parses the document to be checked for duplicates, it can automatically parse out the overview chapter, content chapter, method chapter, result chapter, and conclusion chapter.

[0068] Step 1022 : Perform at least one semantic summary on each to-be-searched chapter based on the large language model to obtain a target chapter summary corresponding to each to-be-searched chapter.

[0069] The target summary document includes target chapter summaries corresponding to each chapter to be checked.

[0070] The target chapter summary can be a text of a preset number of words obtained after performing at least one semantic summary of the chapter to be checked. In some examples, the preset number of words may be smaller than the number of words in the chapter to be checked. For example, the preset number of words may be 512. For another example, the preset number of words may be 1024. The embodiment of the present application does not limit the size of the preset number of words. The embodiment of the present application uses the preset number of words of 512 as an example for illustrative description.

[0071] For example, if the chapters to be checked include the overview chapter, content chapter, method chapter, results chapter, and conclusion chapter, the target chapter summary can include the summary of the overview chapter, content chapter, method chapter, results chapter, and conclusion chapter.

[0072] For example, if the chapters to be checked include the technical field chapter to be checked, the technical background chapter to be checked, and the technical content chapter to be checked, the target chapter summary may include the technical field chapter summary, the technical background chapter summary, and the technical content chapter summary.

[0073] In some embodiments of the present application, the target document features may include the to-be-checked chapter features corresponding to each to-be-checked chapter. Step 103 extracts features from the target summary document to obtain the target document features, which may include: extracting features from each target chapter summary to obtain the to-be-checked chapter features.

[0074] Taking the target chapter summary as an example, which includes the overview chapter summary to be checked, the content chapter summary to be checked, the method chapter summary to be checked, the results chapter summary to be checked, and the conclusion chapter summary to be checked, the chapter features to be checked may include the overview chapter features to be checked, the content chapter features to be checked, the method chapter features to be checked, the results chapter features to be checked, and the conclusion chapter features to be checked.

[0075] Taking the example that the target chapter summary may include the chapter summary of the technical field to be checked, the chapter summary of the technical background to be checked, and the chapter summary of the technical content to be checked, the chapter features to be checked may include the chapter features of the technical field to be checked, the chapter features of the technical background to be checked, and the chapter features of the technical content to be checked.

[0076] The chapter features to be checked may be features of a preset dimension, and the size of the preset dimension is related to the speed of feature fusion during feature extraction. In some examples, the preset dimension may be 100. In other examples, the preset dimension may be 128. The embodiments of this application do not limit the size of the preset dimension. The embodiments of this application use a preset dimension of 128 as an example for illustrative purposes.

[0077] For example, the electronic device may extract features from the target chapter summary based on the feature extraction model to obtain features of the chapter to be checked. The electronic device may extract features from the target chapter summary based on the feature extraction model to obtain features of the chapter to be checked, which may include: encoding, embedding, and data fusion processing of the target chapter summaries corresponding to each chapter to be checked based on the feature extraction model to obtain features of the chapter to be checked corresponding to each chapter to be checked, and using the features of the chapter to be checked corresponding to each chapter to be checked as the target document features.

[0078] The document duplication checking method based on a large language model provided in the embodiments of the present application can divide the document to be checked by chapter, thereby obtaining multiple chapters to be checked, because the number of words in the document to be checked may be very large. Since the number of words in the divided chapters to be checked is relatively small, it meets the word count requirement of the large language model for the input document. Therefore, based on the large language model, at least one semantic summary is performed on each chapter to be checked, and the target chapter summary corresponding to each chapter to be checked can be accurately obtained. Feature extraction is then performed on the target chapter summary, which can obtain semantically more comprehensive features of the chapter to be checked.

[0079] Take the case where the number of semantic summaries corresponding to at least one semantic summary is 2 as an example. Figure 3 As shown in the above Figure 2 On the basis of the illustrated embodiment, step 1022 performs at least one semantic summary on each to-be-checked chapter based on the large language model to obtain a target chapter summary corresponding to each to-be-checked chapter, which may include the following steps 301 and 302 .

[0080] Step 301 : semantically summarize each chapter to be checked based on the large language model to obtain an initial chapter summary corresponding to each chapter to be checked.

[0081] For example, the large language model may include an encoder and a decoder. The encoder may include multiple encoding layers. The decoder may include multiple decoding layers. The encoder may be used to segment and encode the input chapter to be searched, thereby obtaining features of the chapter to be searched. The decoder may be used to perform feature fusion on the features of the chapter to be searched.

[0082] In some examples of the present application, step 301 may include: based on the encoder in the large language model, sentence segmentation and encoding of each to-be-checked chapter to obtain multiple to-be-checked sentence features; based on the decoder in the large language model, feature fusion of each to-be-checked sentence feature to obtain a to-be-checked sentence fusion feature corresponding to each to-be-checked sentence feature; performing data conversion processing on each to-be-checked sentence fusion feature to obtain a semantic summary of the to-be-checked sentence corresponding to the to-be-checked sentence fusion feature; performing semantic aggregation on multiple to-be-checked sentence semantic summaries corresponding to each to-be-checked chapter to obtain an initial chapter summary corresponding to each to-be-checked chapter.

[0083] Exemplarily, the encoder based on the large language model can sequentially separate each chapter to be checked into sentences, and encode the multiple sentence blocks corresponding to each chapter to be checked, and obtain multiple sentence features to be checked corresponding to each chapter to be checked. In some examples, take the case where multiple chapters to be checked include an overview chapter, and the overview chapter includes N (where N is an integer greater than or equal to 1) sentence blocks as an example. The electronic device can first separate the overview chapter into sentences based on the encoder in the large language model, and then encode the obtained N sentence blocks to obtain N sentence features to be checked corresponding to the overview chapter. Among them, the implementation method of encoding the obtained N sentence blocks may include tokenizing the obtained N sentence blocks through a token module.

[0084] For example, if multiple to-be-searched chapters include an overview chapter, and the overview chapter includes N sentence blocks, the electronic device can perform feature fusion on each of the N to-be-searched sentence features corresponding to the overview chapter based on the decoder in the large language model to obtain N to-be-searched sentence fusion features corresponding to the N to-be-searched sentence features.

[0085] In some examples, the dimension of the fused features of the query sentence may be less than or equal to the dimension of the features of the query sentence. For example, in a large language model, the decoder includes multiple Transformer layers. The electronic device can sequentially fuse N query sentence features using multiple Transformer layers to obtain N fused features corresponding to the N query sentence features.

[0086] The semantic summary of the sentence to be checked can be a summary text with fewer words than the number of words in the sentence block, obtained by semantically summarizing the sentence blocks obtained after sentence segmentation. Data conversion processing is performed on the fused features of each sentence to be checked to obtain a semantic summary corresponding to each fused feature of the sentence to be checked. This can include obtaining an index between words and feature values, converting the fused features of each sentence to be checked into text based on the index, and obtaining a semantic summary corresponding to each fused feature of the sentence to be checked. In some examples, the index between words and feature values ​​can be pre-established and stored in a database for easy access when running a large language model.

[0087] The initial chapter summary can be a summary text obtained after the first semantic summary of the chapter to be checked is performed based on the large language model. Semantic aggregation is performed on the semantic summaries of the multiple sentences to be checked corresponding to each chapter to be checked to obtain the initial chapter summary corresponding to each chapter to be checked. This may include: splicing the multiple semantic summaries of the sentences to be checked corresponding to each chapter to be checked to obtain the initial chapter summary corresponding to each chapter to be checked. In some examples, the order of the semantic summaries of the sentences to be checked can be determined, and then the semantic summaries of the sentences to be checked can be sorted according to the order, and then the sorted semantic summaries of the sentences to be checked can be finished and spliced ​​to obtain the initial chapter summary corresponding to each chapter to be checked.

[0088] Since the decoder in the large language model has the function of generating high-quality results, the decoder in the large language model is used to fuse the features of multiple sentences to be checked corresponding to each chapter to be checked, so as to obtain high-quality fused features of the sentences to be checked; and then data conversion processing is performed on the fused features of multiple sentences to be checked corresponding to each chapter to be checked, so as to obtain high-quality semantic summaries of the sentences to be checked; and then, the quality of the initial chapter summaries corresponding to each chapter to be checked obtained based on the high-quality semantic summaries of the sentences to be checked is also high.

[0089] Step 302 : semantically summarize the initial chapter summary based on the large language model to obtain a target chapter summary corresponding to each to-be-searched chapter.

[0090] Exemplarily, the implementation method of semantically summarizing the initial chapter summary based on the large language model to obtain the target chapter summary corresponding to each chapter to be checked can be similar to the implementation method of semantically summarizing each chapter to be checked based on the large language model to obtain the initial chapter summary corresponding to each chapter to be checked, and the embodiments of the present application will not be repeated here.

[0091] The document duplication checking method based on a large language model provided in an embodiment of the present application performs semantic summaries on multiple sections to be checked based on the large language model to obtain initial section summaries corresponding to each section to be checked. Furthermore, semantic summaries are performed on the initial section summaries based on the large language model to obtain target section summaries corresponding to each section to be checked. In this way, a target section summary of a preset number of words can be obtained, facilitating feature extraction based on the target section summary of the preset number of words.

[0092] In some embodiments of the present application, Figure 4 As shown in the above Figure 1 On the basis of the illustrated embodiment, step 104 determines the target recall document based on the target document features and the document feature library, and may include the following steps 1041 to 1043 .

[0093] Step 1041 , based on each to-be-checked chapter feature, a plurality of corresponding historical chapter features in the document feature library are retrieved to obtain at least one target recall feature corresponding to each to-be-checked chapter feature, and a chapter similarity between the to-be-checked chapter feature and each target recall feature.

[0094] The at least one target recall feature corresponding to the to-be-searched chapter feature may be one or more historical chapter features that have the highest similarity to the to-be-searched chapter feature among the multiple historical chapter features corresponding to the document feature library. In some examples, the one or more historical chapter features may correspond to M historical chapter features, where M is a positive integer greater than or equal to 1. The embodiment of this application does not limit the size of M.

[0095] Chapter similarity can be the similarity between chapter features. In some examples, the chapter similarity between the to-be-queried chapter feature and each target recall feature can be calculated using Euclidean distance or cosine similarity. This embodiment of the present application does not limit the method for calculating chapter similarity. This embodiment of the present application uses cosine similarity as an example for illustrative purposes.

[0096] For example, if the to-be-checked chapter features include an overview chapter feature, and the document feature library includes multiple historical overview chapter features, the electronic device can retrieve multiple historical overview chapter features based on the to-be-checked overview chapter features, obtain similarities between the to-be-checked overview chapter features and each of the historical overview chapter features, and use the similarities between the to-be-checked overview chapter features and each of the historical overview chapter features as the overview chapter similarity. Simultaneously, at least one historical overview chapter feature with the highest similarity to the to-be-checked overview chapter feature is used as at least one target recall feature.

[0097] For example, if the to-be-checked section features include the to-be-checked technical field section features, and the document feature library includes multiple technical field section features, the electronic device can retrieve multiple historical technical field section features based on the to-be-checked technical field section features, obtain the similarity between the to-be-checked technical field section features and the respective historical technical field section features, and use the similarity between the to-be-checked technical field section features and the respective historical technical field section features as the technical field section similarity. Simultaneously, at least one historical technical field section feature having the highest similarity to the to-be-checked technical field section feature is used as at least one target recall feature.

[0098] Step 1042 : Based on the chapter similarity between the chapter feature to be checked and each target recall feature, determine the document similarity between each historical document and the document to be checked for duplicates.

[0099] The document similarity can be the overall similarity between documents. The document similarity between each historical document and the document to be checked for duplicates can include (historical document ID1, document similarity 1), (historical document ID2, document similarity 2) ... (historical document IDN, document similarity N).

[0100] In some examples, document similarity can be obtained by weighting the similarities between corresponding sections of a document. For example, the document similarity between each historical document and the document to be checked for duplicates can be obtained by weighting the similarities between the features of each historical section of the historical document and the corresponding section to be checked.

[0101] Step 1043 : Based on the document similarity between each historical document and the document to be checked for duplicates, a target recall document is determined from the multiple historical documents.

[0102] Taking the document similarities between each historical document and the document to be checked for duplicates as (historical document ID 1, document similarity 1), (historical document ID 2, document similarity 2), ... (historical document ID N, document similarity N) as an example, the electronic device can sort (historical document ID 1, document similarity 1), (historical document ID 2, document similarity 2), ... (historical document ID N, document similarity N) in descending order according to the paragraph similarity field, with the historical document at the top having the highest document similarity. The first K elements of the sorted list are selected, and the K historical documents in these K elements are the top K documents that are most semantically similar to the document to be checked for duplicates.

[0103] The document duplication checking method based on a large language model provided in the embodiment of the present application is relatively accurate and comprehensive in the features of each section to be checked. Therefore, based on the features of each section to be checked, a plurality of corresponding historical section features in the document feature library are retrieved, and at least one target recall feature corresponding to each section feature to be checked, as well as the section similarity between the section feature to be checked and each target recall feature can be obtained relatively accurately. Based on the section similarity between the section feature to be checked and each target recall feature, the document similarity between each historical document and the document to be checked for duplicates can be determined relatively accurately. Furthermore, the document similarity between each historical document and the document to be checked for duplicates can relatively accurately determine the target recall document from multiple historical documents.

[0104] like Figure 5 As shown in the above Figure 4 Based on the illustrated embodiment, step 1042 determines the document similarity between each historical document and the document to be checked for duplicates based on the chapter similarity between the chapter feature to be checked and each target recall feature, and may include the following steps 501 to 503.

[0105] Step 501 : Based on the chapter similarity between the chapter feature to be checked and each target recall feature, the similarity between the historical chapter feature of each historical document and the corresponding chapter feature to be checked is determined.

[0106] Exemplarily, when the electronic device retrieves the corresponding multiple historical chapter features in the document feature library based on each of the to-be-checked chapter features, it can simultaneously record the historical document ID to which the target recall feature belongs, thereby determining the similarity between the historical chapter features of each historical document and the corresponding to-be-checked chapter features based on the obtained chapter similarity between the to-be-checked chapter features and each target recall feature, as well as the historical document ID to which the target recall feature belongs.

[0107] For example, the chapter features to be checked include the chapter features of the technical field to be checked, the chapter features of the technical background to be checked, and the chapter features of the technical content to be checked, and the document feature library includes multiple historical technical field chapter features, multiple historical technical background chapter features, and multiple historical technical content chapter features. If the target recall feature corresponding to the chapter features of the technical content to be checked belongs to the historical document ID1, then the similarity between the chapter features of the technical field in the historical document ID1 and the chapter features of the technical field to be checked (denoted as S1), the similarity between the chapter features of the technical background in the historical document ID1 and the chapter features of the technical background to be checked (denoted as S2), and the similarity between the chapter features of the technical content in the historical document ID1 and the chapter features of the technical content to be checked (denoted as S3) can be determined.

[0108] Step 502: Determine the weight parameters corresponding to the features of each chapter to be checked.

[0109] Exemplarily, step 501 may include: determining weight parameters corresponding to the features of each to-be-checked section in the to-be-checked duplicate document according to the importance of each to-be-checked section. In some instances, if the features of the to-be-checked section include features of the technical field section to be checked, features of the technical background section to be checked, and features of the technical content section to be checked, and the importance of the to-be-checked section corresponding to each to-be-checked section feature is, from largest to smallest, technical content, technical field, and technical background, then the weight parameters corresponding to the features of the technical field section to be checked, the features of the technical background section to be checked, and the features of the technical content section to be checked may be determined to be W1, W2, and W3, respectively. In some examples, W1, W2, and W3 may be 60%, 30%, and 10%, respectively.

[0110] Step 503 : Based on the weight parameters corresponding to the features of each to-be-checked chapter, weighted processing is performed on the similarities between the historical chapter features of each historical document and the corresponding features of the to-be-checked chapter to obtain the document similarity between each historical document and the to-be-checked duplicate document.

[0111] Taking the historical document ID1 as an example, the document similarity Stotal between the historical document ID1 and the document to be checked for duplicates can be found in the following formula (1).

[0112] Stotal=W1*S1+W2*S2+W3*S3 (1).

[0113] The document duplication checking method based on a large language model provided in the embodiments of the present application determines the similarity between the historical section features of each historical document and the features of the corresponding section to be checked, and then weights these similarities to obtain the target recalled document. This not only increases the flexibility of the document duplication checking process and results, but also ensures that the target recalled document better meets the needs and understanding of business personnel.

[0114] like Figure 6 As shown in the above Figure 1 Based on the illustrated embodiment, step 101 of obtaining a document feature library corresponding to a plurality of historical documents may include the following steps 1011 to 1014 .

[0115] Step 1011, obtaining multiple historical documents.

[0116] Exemplarily, the plurality of historical documents may be a large number of historical documents. In some examples, the plurality of historical documents may be a number greater than 10,000 historical documents. For example, the plurality of historical documents may include 100,000 historical documents.

[0117] Step 1012: Divide each historical document into chapters to obtain multiple historical chapters.

[0118] For example, the implementation of step 1012 is as follows: Figure 2 The implementation method of step 1021 in the illustrated embodiment is similar and will not be repeated here in the embodiment of the present application.

[0119] Step 1013 , performing at least one semantic summary on each historical chapter based on the large language model to obtain a historical chapter summary corresponding to each historical chapter.

[0120] For example, the implementation of step 1013 is as follows: Figure 2 The implementation method of step 1022 in the illustrated embodiment is similar and will not be repeated here in the embodiment of the present application.

[0121] Step 1014 , performing feature extraction on the historical chapter summary corresponding to each historical chapter to obtain the historical chapter features corresponding to each historical chapter.

[0122] The method of extracting features from the historical chapter summaries corresponding to each historical chapter to obtain the historical chapter features corresponding to each historical chapter is similar to the method of extracting features from the summaries of each target chapter to obtain the features of the chapters to be checked, and the embodiments of this application will not be repeated here.

[0123] Step 1015: construct a document feature library based on the historical chapter features corresponding to each historical chapter.

[0124] The electronic device may store the historical chapter features corresponding to each historical chapter in a database to obtain a document feature library.

[0125] The document duplication checking method based on a large language model provided in the embodiments of the present application divides each historical document into chapters to obtain multiple historical chapters, and performs at least one semantic summary on each historical chapter based on the large language model to obtain a historical chapter summary corresponding to each historical chapter. Because the historical chapter summary corresponding to each historical chapter can meet the input requirements of the feature extraction model, when feature extraction is performed on the historical chapter summary corresponding to each historical chapter, a relatively complete historical chapter feature can be obtained. Furthermore, based on the historical chapter features corresponding to each historical chapter, a relatively comprehensive semantic feature of the chapter to be checked can be constructed.

[0126] On the basis of the above embodiment, the embodiment of the present application also provides a document duplicate checking device based on a large language model. Figure 7 As shown, the document duplicate checking device 70 based on a large language model may include an acquisition module 701 , a semantic summarization module 702 , a feature extraction module 703 and a determination module 704 .

[0127] The acquisition module 701 is used to obtain a document feature library corresponding to the document to be checked for duplicates and multiple historical documents.

[0128] The historical document includes multiple historical chapters, and the document feature library includes historical chapter features corresponding to each historical chapter;

[0129] The semantic summarization module 702 is used to perform at least one semantic summary on the document to be checked for duplicates based on the large language model to obtain a target summary document corresponding to the document to be checked for duplicates.

[0130] Feature extraction module 703, used to extract features from the target summary document to obtain target document features;

[0131] The determination module 704 is configured to determine the target recall document based on the target document features and the document feature library.

[0132] like Figure 8 As shown in the above Figure 7 Based on the illustrated embodiment, the semantic summarizing module 702 may include a first dividing unit 7021 and a first semantic summarizing unit 7022 .

[0133] The first division unit 7021 is used to divide the document to be checked for duplicates into chapters to obtain multiple chapters to be checked;

[0134] The first semantic summarizing unit 7022 is configured to perform at least one semantic summary on each to-be-checked chapter based on the large language model to obtain a target chapter summary corresponding to each to-be-checked chapter.

[0135] The feature extraction module 703 is specifically used to extract features from each target chapter summary to obtain features of the chapter to be checked.

[0136] like Figure 9 As shown, in Figure 8 Based on the illustrated embodiment, the first semantic summarizing unit 7022 may include a first sub-semantic summarizing unit 901 and a second sub-semantic summarizing unit 902 .

[0137] The first sub-semantic summarizing unit 901 is used to perform semantic summarization on each to-be-searched chapter based on the large language model to obtain an initial chapter summary corresponding to each to-be-searched chapter;

[0138] The second sub-semantic summarization unit 902 is used to perform semantic summarization on the initial chapter summary based on the large language model to obtain a target chapter summary corresponding to each chapter to be checked.

[0139] In some embodiments of the present invention, the first sub-semantic summary unit 901 is specifically used to segment and encode each chapter to be checked based on the encoder in the large language model to obtain multiple features of the sentences to be checked; perform feature fusion on the features of each sentence to be checked based on the decoder in the large language model to obtain fused features of the sentences to be checked; perform data conversion processing on the fused features of each sentence to be checked to obtain a semantic summary of the sentences to be checked; perform semantic aggregation on the multiple semantic summaries of the sentences to be checked to obtain an initial chapter summary corresponding to each chapter to be checked.

[0140] like Figure 10 As shown, based on the embodiment shown in above 7, the target document features include multiple chapter features to be checked, and the determination module 704 may include a retrieval unit 7041, a document similarity determination unit 7042 and a target recalled document determination unit 7043.

[0141] A retrieval unit 7041 is configured to retrieve a plurality of corresponding historical chapter features in a document feature library based on each chapter feature to be searched, and obtain at least one target recall feature corresponding to each chapter feature to be searched, as well as a chapter similarity between the chapter feature to be searched and each target recall feature;

[0142] A document similarity determination unit 7042 is configured to determine the document similarity between each historical document and the document to be checked for duplicates based on the section similarity between the section feature to be checked and each target recall feature;

[0143] The target recall document determination unit 7043 is configured to determine a target recall document from a plurality of historical documents based on the document similarity between each historical document and the document to be checked for duplicates.

[0144] like Figure 11 As shown in the above Figure 10 Based on the illustrated embodiment, the document similarity determination unit 7042 may include a first determination subunit 1101 , a second determination subunit 1102 and a weighted processing subunit 1103 .

[0145] The first determining subunit 1101 is configured to determine the similarity between the historical chapter features of each historical document and the corresponding chapter features to be checked based on the chapter similarity between the chapter features to be checked and each target recall feature;

[0146] The second determining subunit 1102 is used to determine the weight parameters corresponding to the features of each chapter to be checked;

[0147] The weighted processing subunit 1103 is used to perform weighted processing on the similarity between the historical chapter features of each historical document and the corresponding chapter features to be checked based on the weight parameters corresponding to each chapter feature to be checked, so as to obtain the document similarity between each historical document and the document to be checked for duplicates.

[0148] like Figure 12 As shown in the above Figure 11 Based on the illustrated embodiment, the acquisition module 701 may include an acquisition unit 7011 , a second division unit 7012 , a second semantic summarization unit 7013 , a feature extraction unit 7014 and a construction unit 7015 .

[0149] The acquisition unit 7011 is used to acquire multiple historical documents.

[0150] The second division unit 7012 is used to divide each historical document into chapters to obtain multiple historical chapters.

[0151] The second semantic summarizing unit 7013 is configured to perform at least one semantic summary on each historical chapter based on the large language model to obtain a historical chapter summary corresponding to each historical chapter.

[0152] The feature extraction unit 7014 is used to extract features from the historical chapter summaries corresponding to each historical chapter to obtain historical chapter features corresponding to each historical chapter.

[0153] The construction unit 7015 is used to construct a document feature library based on the historical chapter features corresponding to each historical chapter.

[0154] Regarding the document duplication checking device based on a large speech model in the above embodiment, the specific manner in which each module performs operations and the corresponding beneficial effects have been described in detail in the corresponding embodiment part of the aforementioned audio signal simulation method part. Please refer to the corresponding execution operation manner and beneficial technical effects of the above exemplary method part, and no further details will be given here.

[0155] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 13 As shown, the electronic device 130 includes one or more processors 1301 and a memory 1302 .

[0156] The processor 1301 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 130 to perform desired functions.

[0157] The memory 1302 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1701 may execute the program instructions to implement the document duplicate checking method based on the large language model and / or other desired functions of the various embodiments of the present application described above.

[0158] In one example, the electronic device 130 may further include an input device 1303 and an output device 1304 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0159] Of course, to simplify, Figure 13 Only some of the components related to the present application in the electronic device 130 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 130 may further include any other appropriate components according to specific application scenarios.

[0160] In addition to the above-mentioned methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the document duplication detection method based on a large language model in various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.

[0161] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0162] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the document duplication detection method based on a large language model of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.

[0163] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or component comprising electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0164] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be considered as essential to each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0165] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A document duplication checking method based on a large language model, characterized in that: in, The method comprises: Obtaining a document feature library corresponding to a document to be checked for duplicates and a plurality of historical documents; the historical documents include a plurality of historical chapters, and the document feature library includes historical chapter features corresponding to each of the historical chapters; Performing at least one semantic summary on the document to be checked for duplicates based on the large language model to obtain a target summary document corresponding to the document to be checked for duplicates; Performing feature extraction on the target summary document to obtain target document features; Based on the target document features and the document feature library, a target recall document is determined.

2. The method according to claim 1, characterized in that The step of performing at least one semantic summary on the document to be checked for duplicates based on the large language model to obtain a target summary document corresponding to the document to be processed includes: Dividing the document to be checked for duplicates by chapters to obtain a plurality of chapters to be checked; Performing at least one semantic summary on each of the to-be-checked chapters based on the large language model to obtain a target chapter summary corresponding to each of the to-be-checked chapters; wherein the target summary document includes a target chapter summary corresponding to each of the to-be-checked chapters; The target document features include the to-be-checked chapter features corresponding to each of the to-be-checked chapters; and extracting features from the target summary document to obtain the target document features includes: extracting features from each of the target chapter summaries to obtain the to-be-checked chapter features.

3. The method according to claim 2, characterized in that The step of performing at least one semantic summary on each of the to-be-checked chapters based on the large language model to obtain a target chapter summary corresponding to each of the to-be-checked chapters includes: Performing semantic summaries on each of the to-be-searched chapters based on the large language model to obtain initial chapter summaries corresponding to each of the to-be-searched chapters; The initial chapter summaries are semantically summarized based on the large language model to obtain target chapter summaries corresponding to the chapters to be checked.

4. The method according to claim 3, characterized in that The semantic summarizing of each of the to-be-checked chapters based on the large language model to obtain an initial chapter summary corresponding to each of the to-be-checked chapters includes: Sentence segmentation and encoding of each of the to-be-checked chapters are performed based on the encoder in the large language model to obtain a plurality of to-be-checked sentence features; Performing feature fusion on each of the to-be-checked sentence features based on the decoder in the large language model to obtain a fused feature of the to-be-checked sentence; Performing data conversion processing on the fusion features of each sentence to be checked to obtain a semantic summary of the sentence to be checked; Semantic aggregation is performed on the semantic summaries of the multiple sentences to be checked to obtain an initial chapter summary corresponding to each of the chapters to be checked.

5. The method according to any one of claims 1 to 4, characterized in that The target document features include multiple to-be-checked chapter features, and determining the target recall document based on the target document features and the document feature library includes: Retrieving a plurality of corresponding historical chapter features in the document feature library based on each of the to-be-checked chapter features, obtaining at least one target recall feature corresponding to each to-be-checked chapter feature, and a chapter similarity between the to-be-checked chapter feature and each of the target recall features; Determining the document similarity between each of the historical documents and the document to be checked for duplicates based on the chapter similarity between the chapter feature to be checked and each of the target recall features; Based on the document similarity between each of the historical documents and the document to be checked for duplicates, a target recalled document is determined from the multiple historical documents.

6. The method according to claim 5, characterized in that The determining of the document similarity between each of the historical documents and the document to be checked for duplicates based on the section similarity between the section feature to be checked and each of the target recall features includes: Determining the similarity between the historical chapter feature of each of the historical documents and the corresponding chapter feature to be checked based on the chapter similarity between the chapter feature to be checked and each of the target recall features; Determine the weight parameters corresponding to the features of each of the sections to be checked; Based on the weight parameters corresponding to the features of each section to be checked, the similarities between the historical section features of each historical document and the corresponding features of the section to be checked are weighted to obtain the document similarity between each historical document and the document to be checked for duplicates.

7. The method according to any one of claims 1 to 4, characterized in that Obtaining document feature libraries corresponding to multiple historical documents includes: Obtaining the plurality of historical documents; Dividing each of the historical documents into chapters to obtain the multiple historical chapters; Performing at least one semantic summary on each of the historical chapters based on the large language model to obtain a historical chapter summary corresponding to each of the historical chapters; Performing feature extraction on the historical chapter summaries corresponding to the historical chapters to obtain the historical chapter features corresponding to the historical chapters; The document feature library is constructed based on the historical chapter features corresponding to each of the historical chapters.

8. A document duplication checking device based on a large language model, characterized in that: The device comprises: An acquisition module is used to acquire a document feature library corresponding to a document to be checked for duplicates and a plurality of historical documents; wherein the historical document includes a plurality of historical chapters, and the document feature library includes historical chapter features corresponding to each of the historical chapters; A semantic summarization module, configured to perform at least one semantic summary on the document to be checked for duplicates based on the large language model to obtain a target summary document corresponding to the document to be checked for duplicates; A feature extraction module is used to extract features from the target summary document to obtain target document features; The determination module is used to determine the target recall document based on the target document feature and the document feature library.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the document duplication checking method based on a large language model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program is used to execute the document duplication checking method based on a large language model as described in any one of claims 1 to 7.

Images