Templated document management method and device

Abstract

The invention discloses a template document management method and device. The method comprises the following steps: determining a third templated document with the highest similarity with a second templated document to be stored in a plurality of first templated documents in a document library, and determining the first complexity of the second templated document and the second complexity of the third templated document; dividing the second templated document and the third templated document into a plurality of first character blocks and a plurality of second character blocks according to the first complexity and the second complexity; performing repeated character block checking on the plurality of first character blocks and the plurality of second character blocks by utilizing the target Bloom filter, and determining whether the second templated document and the third templated document are repeated or not according to a checking result; and if not, storing the second templated document into the document library. According to the method and the device, the technical problems that a duplicate checking result is inaccurate and the storage efficiency of a document library is influenced due to the fact that a related document duplicate checking technology does not fully consider the difference of document contents are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a template document management method and device. BACKGROUND

[0002] In the AI era, enterprises usually need to build a knowledge base containing a large amount of domain knowledge to support AI model training and digital transformation. The content in the knowledge base is usually composed of template documents (such as device manuals, operation documents, industry reports, technical solutions, etc.) in Word and PDF formats. These documents usually follow a fixed template format, and the template part of the content is highly similar, while the non-template content of each document has great differences. Therefore, when a new document needs to be stored, it needs to be checked for duplication to optimize the storage space of the knowledge base and reduce the redundancy of training data.

[0003] Traditional duplicate checking methods can generate fingerprints based on the hash values of documents and calculate the similarity of documents through Hamming distance, but they often do not fully consider the actual differences in document content, especially when facing multiple documents under the same template, they cannot effectively judge the differences between these documents, resulting in inaccurate duplicate checking results, and misjudging as duplicate documents, affecting the storage efficiency and retrieval efficiency of the document library.

[0004] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0005] The embodiments of the present application provide a template document management method and device to at least solve the technical problem that related document duplicate checking technology does not fully consider the differences in document content, resulting in inaccurate duplicate checking results and affecting the storage efficiency of the document library.

[0006] According to an aspect of an embodiment of the present application, a template document management method is provided, comprising: determining a third template document with the highest first similarity to a second template document to be stored in a document library from a plurality of first template documents in the document library, and determining a first complexity of the second template document and a second complexity of the third template document, respectively; dividing the second template document and the third template document into a first number of first character blocks and a second number of second character blocks according to the first complexity and the second complexity; checking the first number of first character blocks and the second number of second character blocks for duplicate character blocks using a target Bloom filter, and determining whether the second template document and the third template document are duplicate according to the checking result; and if the second template document and the third template document are not duplicate, storing the second template document in the document library.

[0007] Optionally, determining the third template document in the plurality of first template documents in the document library having the highest first similarity with the second template document to be stored, comprises: determining a first hash value of each of the plurality of first template documents in the document library and a second hash value of the second template document using a preset hash algorithm; determining a distance between each of the first hash value and the second hash value using a preset similarity algorithm, and taking the distance as the first similarity between the second template document and the corresponding first template document; and determining the third template document in the plurality of first template documents in the document library having the highest first similarity with the second template document.

[0008] Optionally, the first complexity and the second complexity each comprise a content complexity and a structure complexity, wherein determining the first complexity of the second template document and the second complexity of the third template document respectively comprises: determining, for any one of the second template document and the third template document, a total number of vocabularies, a number of non-repeated vocabularies, a total number of sentences, a length of each of all sentences, a total number of titles, a total number of images, and a total number of tables; determining a ratio of the number of non-repeated vocabularies to the total number of vocabularies to obtain a vocabulary diversity, determining a ratio of a sum of the length of each of all sentences to the total number of sentences to obtain an average sentence length, and performing weighted summation on the vocabulary diversity and the average sentence length to obtain a corresponding content complexity; and performing weighted summation on the total number of titles, the total number of images, and the total number of tables to obtain a corresponding structure complexity.

[0009] Optionally, dividing the second template document and the third template document into a first number of first character blocks and a second number of second character blocks according to the first complexity and the second complexity comprises: determining a division character block size according to the first complexity and the second complexity according to the following formula: , wherein, , content complexity and the structure complexity of the second template document, , content complexity and the structure complexity of the third template document, and a represents an adjustment factor; and performing block processing on the second template document and the third template document according to the division character block size to obtain the first number of first character blocks and the second number of second character blocks.

[0010] Optionally, before the repeated character block checking of the first number of first character blocks and the second number of second character blocks by the target Bloom filter, the method further comprises: determining an average of the content complexity of the second templated document and the content complexity of the third templated document to obtain a content complexity factor, and determining an average of the structure complexity of the second templated document and the structure complexity of the third templated document to obtain a structure complexity factor; determining a third number of hash functions in the target Bloom filter according to the content complexity factor and a preset false positive rate according to the following formula: , wherein, represents the content complexity factor, and p represents the false positive rate; determining a fourth number of arrays in the target Bloom filter according to a sum of the first number and the second number, the structure complexity factor and the preset false positive rate according to the following formula: , wherein, represents the structure complexity factor, p represents the false positive rate, and n represents the sum of the first number and the second number.

[0011] Optionally, the repeated character block checking of the first number of first character blocks and the second number of second character blocks by the target Bloom filter comprises: for each character block in the first number of first character blocks and the second number of second character blocks, processing the character block by the third number of hash functions in the target Bloom filter to obtain a third number of hash values, and mapping the third number of hash values to target positions in the fourth number of arrays respectively to mark the target positions; determining a fifth number of target positions in the fourth number of arrays that are simultaneously marked by the first number of first character blocks and the second number of second character blocks to obtain the fifth number of repeated character blocks in the second templated document and the third templated document.

[0012] Optionally, determining whether the second templated document and the third templated document are repeated according to the checking result comprises: calculating a ratio of the fifth number to the first number to obtain a second similarity of the second templated document and the third templated document; in a case where the second similarity is not lower than a first preset threshold, determining that the second templated document and the third templated document are repeated; and in a case where the second similarity is lower than the first preset threshold, determining that the second templated document and the third templated document are not repeated.

[0013] According to another aspect of the embodiments of this application, a templated document management device is also provided, comprising: a determining module, configured to determine a third templated document among a plurality of first templated documents in a document library that has the highest first similarity to a second templated document to be added to the library, and to determine a first complexity of the second templated document and a second complexity of the third templated document respectively; a segmentation module, configured to divide the second templated document and the third templated document into a first number of first character blocks and a second number of second character blocks respectively according to the first complexity and the second complexity; a duplicate checking module, configured to perform duplicate character block checks on the first number of first character blocks and the second number of second character blocks using a target Bloom filter, and to determine whether the second templated document and the third templated document are duplicates based on the check results; and an inbound management module, configured to store the second templated document in the document library if the second templated document and the third templated document are not duplicates.

[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, the computer program product comprising: a computer program, wherein the computer program, when executed by a processor, implements the above-described templated document management method.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described templated document management method through the computer program.

[0016] In this embodiment, during the new document entry and deduplication process, the system first identifies the third templated document with the highest first similarity among multiple first templated documents in the document library and the second templated document to be entered into the library. Then, it determines the first complexity of the second templated document and the second complexity of the third templated document. Based on the first and second complexities, the second and third templated documents are divided into a first number of first character blocks and a second number of second character blocks, respectively. A target Bloom filter is used to check for duplicate character blocks in the first number of first character blocks and the second number of second character blocks, and the system determines whether the second and third templated documents are duplicates based on the check results. If the second and third templated documents are not duplicates, the second templated document is stored in the document library. This achieves the technical effect of accurately identifying the differences between templated documents, especially in parts with highly similar templates in content and structure, while distinguishing the differences in their non-template content. This optimizes the knowledge base storage space and improves the accuracy of document entry, ensuring efficient management and maintenance of the knowledge base. Furthermore, it solves the technical problem that related document deduplication technologies do not fully consider the differences in document content, leading to inaccurate deduplication results and affecting the storage efficiency of the document library. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a flowchart illustrating an optional templated document management method according to an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the structure of an optional templated document management device according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the hardware structure of a computer terminal for implementing a templated document management method, according to an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] To better understand the embodiments of this application, the following is a translation and explanation of some nouns or terms that appear in the description of the embodiments of this application:

[0024] SimHash algorithm: It is an algorithm used to detect text similarity. Its main idea is to reduce the dimensionality by mapping high-dimensional feature vectors to low-dimensional feature vectors, and then determine the similarity of texts by calculating the Hamming distance between the two vectors.

[0025] A Bloom filter is a highly efficient space-efficient algorithm widely used in data processing, especially when dealing with large datasets. It allows testing whether an element is part of a set, but may have a certain false positive rate. Bloom filters are well-suited for solving set membership problems, especially when the set is very large, as they can significantly reduce memory usage.

[0026] Example 1

[0027] According to an embodiment of this application, a templated document management method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] Figure 1 This is a flowchart illustrating a templated document management method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0029] Step S102: Determine the third templated document with the highest first similarity to the second templated document to be added to the document library from among multiple first templated documents, and determine the first complexity of the second templated document and the second complexity of the third templated document respectively.

[0030] Step S104: Divide the second templated document and the third templated document into a first number of first character blocks and a second number of second character blocks respectively, based on the first complexity and the second complexity.

[0031] Step S106: Use a target Bloom filter to check for duplicate character blocks in a first number of first character blocks and a second number of second character blocks, and determine whether the second templated document and the third templated document are duplicates based on the check results.

[0032] Step S108: If the second templated document and the third templated document are not duplicated, store the second templated document in the document library.

[0033] Based on the scheme defined in steps S102 to S108 above, it can be seen that in this application example, a two-layer detection mechanism and a fine-grained adaptive Bloom filter algorithm are adopted. First, the most similar document pairs are quickly located. Then, the parameters of the Bloom filter are dynamically adjusted according to the complexity of the document pairs to perform fine-grained character block segmentation and duplicate checking. This effectively avoids misjudgment of template content and improves the accuracy of duplicate checking. It achieves the goal of quickly and accurately determining the eligibility of new documents for inclusion in the database, and significantly improves the construction efficiency and storage quality of the knowledge base.

[0034] The following section explains each step of the templated document management method in conjunction with a specific implementation process.

[0035] First, in the technical solution provided in step S102 above, the system first determines the third templated document with the highest first similarity to the second templated document to be added to the document library from among multiple first templated documents.

[0036] Specifically, the system can determine the third templated document by following these steps:

[0037] Step 1: Use a preset hash algorithm to determine the first hash value of each of the multiple first templated documents in the document library, and the second hash value of the second templated document.

[0038] Step 2: Use a preset similarity algorithm to determine the distance between each first hash value and the second hash value, and use the distance as the first similarity between the second templated document and the corresponding first templated document.

[0039] Step 3: Identify the third templated document with the highest first similarity among multiple first templated documents in the document library and the second templated document to be added to the library.

[0040] For example, the system can use the SimHash algorithm to map all feature words in multiple first-templated documents in the document library and the second-templated documents to be added to the library to a fixed-length hash value. Then, for multiple document pairs composed of multiple first-templated documents and second-templated documents, the system calculates the number of different characters at corresponding positions in the two equal-length hash strings within each document pair to obtain the first similarity between the corresponding document pairs. Subsequently, the first-templated document with the highest first similarity is taken as the document most similar to the second-templated document, i.e., the third-templated document.

[0041] In addition to the above-mentioned implementation schemes, based on the basic concept of the present invention, those skilled in the art can also use other technical solutions to determine the hash values ​​and similarities between multiple first templated documents and second templated documents to be entered into the database. For example, methods such as MinHash, Locality Sensitive Hashing, and Perceptual Hashing can be used to calculate the hash values ​​of documents, and methods such as Cosine Similarity, Jaccard Index, and Deep Learning Approaches can be used to calculate the similarity between documents. Any modifications made by those skilled in the art to the above-mentioned implementation schemes should also be within the protection scope of the present invention.

[0042] It should be noted that if the first similarity between multiple first-templated documents in the document library and the second-templated document to be added to the library is all below the preset threshold, it means that the second-templated document has significant differences from the existing templated documents in the library as a whole. That is, the second-templated document is non-duplicate content, so it can be added to the document library and will not be excluded due to misjudgment as duplicate, thus ensuring the comprehensiveness and timeliness of the knowledge base.

[0043] Furthermore, regarding the issue of high repetition and substantial differences in content among documents under the same template, although the initial screening can determine whether the first templated document in the document library is significantly dissimilar to the second templated document to be added, a single rough screening is insufficient to accurately determine the repetition of templated documents with similar overall structures but different details, thus posing a risk of misjudgment.

[0044] Therefore, this application embodiment will further conduct a secondary similarity search to more meticulously analyze whether the second templated document to be added to the database and the most similar third templated document are duplicate documents, thereby avoiding valuable updates or new information from being mistakenly regarded as duplicates and thus missed.

[0045] Specifically, the system can first determine the first complexity of the second templated document and the second complexity of the third templated document, where both the first and second complexities include content complexity and structural complexity.

[0046] Optionally, for either the second or third templated document, the system can determine its complexity using the following method:

[0047] Step 1: Determine the total number of words, unique words, total number of sentences, length of each sentence, total number of headings, total number of images, and total number of tables in the templated document. The number of unique words refers to the total number of all unique words (i.e., non-repeating words) appearing in the templated document.

[0048] Step 2: Determine the ratio of the number of unique words to the total number of words to obtain the vocabulary diversity; determine the ratio of the sum of the lengths of all sentences to the total number of sentences to obtain the average sentence length; and perform a weighted summation of the vocabulary diversity and the average sentence length to obtain the corresponding content complexity.

[0049] The expression for the above lexical diversity can be written as:

[0050]

[0051] In the formula, V represents the number of unique words in the templated document, and N represents the total number of words in the templated document. Lexical diversity is also included. A higher lexical diversity indicates that the templated document uses more diverse vocabulary, resulting in richer content and more varied language expression; conversely, a lower lexical diversity indicates less vocabulary. A lower average sentence length may indicate that the templated document content is relatively simple or highly repetitive. The expression for average sentence length can be written as:

[0052]

[0053] In the formula, M represents the total number of sentences in the templated document. This represents the length of the i-th sentence within the templated document. The average sentence length is also included. It is often used to assess the complexity of templated documents. Longer average sentence length often means more complex sentence structure and potentially more professional arguments, while shorter sentences may indicate that the text is more accessible or concise.

[0054] Therefore, based on the above lexical diversity and average sentence length The content complexity of a templated document can be obtained using the following formula:

[0055]

[0056] In the formula, Indicates the complexity of the content. Weighting coefficients representing lexical diversity. The weighting coefficient represents the average sentence length.

[0057] Step 3: Calculate the weighted sum of the total number of titles, images, and tables to obtain the corresponding structural complexity. The specific calculation formula is as follows:

[0058]

[0059] In the formula, Indicates structural complexity. This indicates the number of headings in the templated document. This indicates the number of tables in the templated document. This indicates the number of images in the templated document. , and These represent the respective weighting coefficients for the number of titles, tables, and images mentioned above.

[0060] Therefore, this application embodiment comprehensively considers the lexical diversity, sentence structure complexity, and the number of non-text elements such as titles, images, and tables in the document to determine the content complexity and structural complexity, thereby comprehensively evaluating the information density and structural complexity of the document.

[0061] Next, in the technical solution provided in step S104 above, the system can divide the second templated document and the third templated document into a first number of first character blocks and a second number of second character blocks respectively according to the first complexity and the second complexity.

[0062] Optionally, the system can divide the second templated document and the third templated document according to the following steps:

[0063] Step 1: Based on the first and second time complexity, determine the size of the character block according to the following formula:

[0064]

[0065] In the formula, Indicates the size of the character block. These represent the content complexity and structural complexity of the second templated document, respectively. These represent the content complexity and structural complexity of the third-generation templated document, respectively. This represents an adjustment factor used to balance the precision of the segmentation with computational efficiency. Proper settings can ensure that subtle differences in documents can be identified without excessive consumption of computing resources due to overly fine segmentation.

[0066] It should be noted that, if If the value is less than 1, then in this embodiment of the application, the final size of the segmented character blocks can be determined according to certain mapping rules to ensure optimal fine-grained segmentation of the document. For example:

[0067]

[0068] In the formula, This indicates the final size of the partitioned character blocks after mapping. The above mapping rules are only illustrative examples, and the embodiments of this application do not impose specific limitations on them.

[0069] Step 2: Divide the second and third templated documents into blocks according to the size of the character blocks, to obtain a first number of first character blocks and a second number of second character blocks.

[0070] Then, in the technical solution provided in step S106 above, the system can use a target Bloom filter to perform duplicate character block checks on a first number of first character blocks and a second number of second character blocks, and determine whether the second templated document and the third templated document are duplicates based on the check results.

[0071] In order to quickly and thoroughly analyze the repetition between a first number of first character blocks and a second number of second character blocks, this application proposes to use a Bloom filter to check for duplicate character blocks between the first number of first character blocks and the second number of second character blocks. This is because, as a probabilistic data structure, the Bloom filter can quickly determine whether a document has duplicates while consuming less memory. Therefore, the Bloom filter is very suitable for fast existence queries and similarity estimation in massive amounts of data.

[0072] However, the performance of Bloom filters is affected by their parameters, including array size and the number of hash functions. This is because complex documents may require larger bit arrays and more hash functions to reduce false positive rates and improve plagiarism detection accuracy. Therefore, by dynamically adjusting parameters based on complexity, the false positive rate can be kept within an acceptable range.

[0073] Specifically, the system can dynamically adjust the array size and the number of hash functions of the target Bloom filter based on the first and second complexities of the second and third templated documents, as follows:

[0074] Step 1: Determine the average content complexity of the second and third templated documents to obtain the content complexity factor. Then, determine the average structural complexity of the second and third templated documents to obtain the structural complexity factor. The specific formula is shown below:

[0075]

[0076]

[0077] In the formula, This represents the content complexity factor. This represents the structural complexity factor.

[0078] Step 2: Based on the content complexity factor and the preset false positive rate, determine the third number of hash functions within the target Bloom filter according to the following formula:

[0079]

[0080] In the formula, denoted by the third number of hash functions within the target Bloom filter, and p represents the false positive rate.

[0081] Step 3: Determine the fourth quantity of the inner array of the target Bloom filter based on the sum of the first and second quantities, the structural complexity factor, and the preset false positive rate.

[0082]

[0083] In the formula, represents the fourth number in the array within the target Bloom filter, p represents the false positive rate, and n represents the sum of the first and second numbers.

[0084] Furthermore, the system will use a target Bloom filter to check for duplicate character blocks in a first number of first character blocks and a second number of second character blocks as follows:

[0085] Step 1: For each character block within the first number of first character blocks and the second number of second character blocks, process the character block using the third number of hash functions in the target Bloom filter to obtain the third number of hash values, and map the third number of hash values ​​to the target positions in the fourth number of arrays to mark the target positions.

[0086] Step 2: Determine the fifth number of target positions within the fourth number of arrays that are simultaneously marked by the first number of first character blocks and the second number of second character blocks, thus obtaining the fifth number of repeated character blocks in the second and third templated documents.

[0087] This can be understood as follows: for each character block, the character block is converted into a fixed-length hash value through a third set of independent hash functions within the target Bloom filter. The hash value is then mapped to a specific position in a fourth set of arrays (the bit array of the Bloom filter). Each array position corresponding to a hash value is marked, indicating that the character block at that position exists in at least one document. Once the hash values ​​of all character blocks have been mapped and marked in the bit arrays, the system checks which target positions have been marked by character blocks from both the second and third templated documents. These simultaneously marked positions indicate the existence of duplicate character blocks in the two documents; their number is the fifth set, reflecting the degree of overlap in the actual content of the second and third templated documents after template similarity has been removed.

[0088] Furthermore, based on the inspection results, the system can determine whether the second and third templated documents are duplicates using the following method:

[0089] First, calculate the ratio of the fifth quantity to the first quantity to obtain the second similarity between the second and third templated documents;

[0090] If the second similarity is not lower than the first preset threshold, it is determined that the second templated document and the third templated document are duplicates;

[0091] If the second similarity is lower than the first preset threshold, it is determined that the second templated document and the third templated document are not duplicates.

[0092] Finally, in the technical solution provided in step S108 above, if the second templated document does not overlap with the third templated document, it indicates that the second templated document contains a sufficient amount of original or different content and can be considered a document with independent value. In this case, the system should allow new documents to be added to the database to enrich the content of the document database and ensure the comprehensiveness and up-to-dateness of the knowledge base. At the same time, the system can build a content index for the second templated document within the document database. This index can be constructed based on the document's first hash value, keywords, or other features to avoid repeatedly calculating the document hash value in future queries and improve the speed of deduplication when adding new documents to the database.

[0093] If the second templated document is identical to the third templated document, it indicates that the second templated document highly overlaps with the documents already in the document library in terms of content, lacking new information or independent value. To avoid redundant information in the knowledge base and save storage space, the system should refuse to store the second templated document in the document library.

[0094] It should be noted that if a second templated document is similar to an existing document in the document library, but contains updated or revised information, the system can be designed to merge this updated information into the existing document, rather than adding it as a new, independent document. This helps maintain the timeliness and accuracy of information in the knowledge base and avoids version inconsistencies.

[0095] In summary, the templated document management method provided in this application has the following technical advantages compared to existing methods:

[0096] (1) Existing methods often misjudge templated documents due to the high repetition of template content, mistaking non-duplicate documents for duplicates. However, the embodiments of this application, through dual-layer detection and optimized Bloom filter algorithm, can accurately identify subtle differences on the basis of quickly locating potentially duplicate documents. This is suitable for deduplication of templated documents and effectively avoids the situation of misjudging documents as duplicates simply because of template similarity.

[0097] (2) Existing methods typically use fixed algorithm parameters when checking for duplicates, which may not be suitable for all types of documents. The embodiments of this application dynamically adjust parameters according to the content complexity and structural complexity of the templated document to be entered into the database, such as the size of the character block, the array size of the Bloom filter, and the number of hash functions, to ensure the best deduplication effect, reduce resource waste, and improve the universality and flexibility of deduplication checking.

[0098] (3) By dynamically adjusting the parameters of the Bloom filter, the embodiments of this application can effectively reduce memory consumption and computing costs while ensuring the accuracy of plagiarism detection. This optimization is particularly important for the management and operation of large knowledge bases when processing documents with complex structures or rich content.

[0099] (4) By accurately identifying duplicates and differences between documents, this solution helps optimize document retrieval and storage efficiency. Documents stored in the knowledge base will be more unique, avoiding resource waste caused by duplicate storage, while improving the speed and accuracy of document retrieval.

[0100] Example 2

[0101] According to an embodiment of this application, a templated document management device for implementing the templated document management method in Embodiment 1 is also provided, such as... Figure 2 As shown, the templated document management device includes at least: a determination module 22, a segmentation module 24, a duplicate check module 26, and an inbound management module 28, wherein:

[0102] The determination module 22 is used to determine the third templated document with the highest first similarity to the second templated document to be added to the document library among multiple first templated documents, and to determine the first complexity of the second templated document and the second complexity of the third templated document respectively.

[0103] The segmentation module 24 is used to divide the second templated document and the third templated document into a first number of first character blocks and a second number of second character blocks respectively according to the first complexity and the second complexity;

[0104] The duplicate checking module 26 is used to perform duplicate character block checks on a first number of first character blocks and a second number of second character blocks using a target Bloom filter, and to determine whether the second templated document and the third templated document are duplicates based on the check results.

[0105] The inbound management module 28 is used to analyze the duplication detection results between the second templated document and the third templated document. If the second templated document and the third templated document are not duplicated, the second templated document is stored in the document library.

[0106] The following section describes the functions of each module of the templated document management device in conjunction with the specific implementation process.

[0107] Specifically, the determining module 22 can determine the third templated document with the highest first similarity among multiple first templated documents in the document library and the second templated document to be added to the library.

[0108] Specifically, module 22 can determine the third templated document by following these steps:

[0109] Step 1: Use a preset hash algorithm to determine the first hash value of each of the multiple first templated documents in the document library, and the second hash value of the second templated document.

[0110] Step 2: Use a preset similarity algorithm to determine the distance between each first hash value and the second hash value, and use the distance as the first similarity between the second templated document and the corresponding first templated document.

[0111] Step 3: Identify the third templated document with the highest first similarity among multiple first templated documents in the document library and the second templated document to be added to the library.

[0112] It should be noted that if the first similarity between multiple first-templated documents in the document library and the second-templated document to be added to the library is all below the preset threshold, it means that the second-templated document has significant differences from the existing templated documents in the library as a whole. That is, the second-templated document is non-duplicate content, so it can be added to the document library and will not be excluded due to misjudgment as duplicate, thus ensuring the comprehensiveness and timeliness of the knowledge base.

[0113] Furthermore, regarding the issue of high repetition and substantial differences in content among documents under the same template, although the initial screening can determine whether the first templated document in the document library is significantly dissimilar to the second templated document to be added, a single rough screening is insufficient to accurately determine the repetition of templated documents with similar overall structures but different details, thus posing a risk of misjudgment.

[0114] Therefore, this application embodiment will further conduct a secondary similarity search to more meticulously analyze whether the second templated document to be added to the database and the most similar third templated document are duplicate documents, thereby avoiding valuable updates or new information from being mistakenly regarded as duplicates and thus missed.

[0115] Therefore, the determining module 22 can determine the first complexity of the second templated document and the second complexity of the third templated document, respectively, wherein the first complexity and the second complexity both include: content complexity and structural complexity.

[0116] Optionally, for either the second or third templated document, the system can determine its complexity using the following method:

[0117] Step 1: Determine the total number of words, the number of unique words, the total number of sentences, the length of each sentence, the total number of headings, the total number of images, and the total number of tables in the templated document.

[0118] Step 2: Determine the ratio of the number of unique words to the total number of words to obtain the vocabulary diversity; determine the ratio of the sum of the lengths of all sentences to the total number of sentences to obtain the average sentence length; and perform a weighted summation of the vocabulary diversity and the average sentence length to obtain the corresponding content complexity.

[0119] Step 3: Perform a weighted sum of the total number of titles, images, and tables to obtain the corresponding structural complexity.

[0120] Therefore, this application embodiment comprehensively considers the lexical diversity, sentence structure complexity, and the number of non-text elements such as titles, images, and tables in the document to determine the content complexity and structural complexity, thereby comprehensively evaluating the information density and structural complexity of the document.

[0121] Next, the segmentation module 24 can divide the second templated document and the third templated document into a first number of first character blocks and a second number of second character blocks respectively, based on the first complexity and the second complexity.

[0122] Optionally, the segmentation module 24 can divide the second templated document and the third templated document according to the following steps:

[0123] Step 1: Determine the size of the character block based on the first and second complexity.

[0124] Step 2: Divide the second and third templated documents into blocks according to the size of the character blocks, to obtain a first number of first character blocks and a second number of second character blocks.

[0125] It should be noted that if the value of the segmented character block size calculated according to the above method is less than 1, then the embodiments of this application can determine the final segmented character block size according to a certain mapping rule to ensure optimal fine-grained segmentation of the document.

[0126] Then, the duplicate checking module 26 can use the target Bloom filter to check for duplicate character blocks in the first number of first character blocks and the second number of second character blocks, and determine whether the second templated document and the third templated document are duplicates based on the check results.

[0127] To quickly and thoroughly analyze the duplication between a first number of first character blocks and a second number of second character blocks, embodiments of this application propose using a Bloom filter to check for duplicate character blocks between the first number of first character blocks and the second number of second character blocks. This is because, as a probabilistic data structure, a Bloom filter can quickly determine whether a document is duplicated while consuming less memory. Therefore, a Bloom filter is very suitable for fast existence queries and similarity estimation in massive amounts of data.

[0128] However, the performance of Bloom filters is affected by their parameters, including array size and the number of hash functions. This is because complex documents may require larger bit arrays and more hash functions to reduce false positive rates and improve plagiarism detection accuracy. Therefore, by dynamically adjusting parameters based on complexity, the false positive rate can be kept within an acceptable range.

[0129] Specifically, the duplicate checking module 26 can dynamically adjust the array size and the number of hash functions of the target Bloom filter based on the first and second complexities of the second and third templated documents. The specific implementation steps are as follows:

[0130] Step 1: Determine the average content complexity of the second and third templated documents to obtain the content complexity factor, and determine the average structural complexity of the second and third templated documents to obtain the structural complexity factor.

[0131] Step 2: Based on the content complexity factor and the preset false positive rate, determine the third number of hash functions within the target Bloom filter according to the following formula.

[0132] Step 3: Determine the fourth number of the inner array of the target Bloom filter based on the sum of the first and second numbers, the structural complexity factor, and the preset false positive rate.

[0133] Furthermore, the duplicate checking module 26 will use a target Bloom filter to check for duplicate character blocks in a first number of first character blocks and a second number of second character blocks as follows:

[0134] Step 1: For each character block within the first number of first character blocks and the second number of second character blocks, process the character block using the third number of hash functions in the target Bloom filter to obtain the third number of hash values, and map the third number of hash values ​​to the target positions in the fourth number of arrays to mark the target positions.

[0135] Step 2: Determine the fifth number of target positions within the fourth number of arrays that are simultaneously marked by the first number of first character blocks and the second number of second character blocks, thus obtaining the fifth number of repeated character blocks in the second and third templated documents.

[0136] Furthermore, the duplicate checking module 26 can also determine whether the second templated document and the third templated document are duplicates based on the checking results, using the following method:

[0137] First, calculate the ratio of the fifth quantity to the first quantity to obtain the second similarity between the second and third templated documents;

[0138] If the second similarity is not lower than the first preset threshold, it is determined that the second templated document and the third templated document are duplicates;

[0139] If the second similarity is lower than the first preset threshold, it is determined that the second templated document and the third templated document are not duplicates.

[0140] Finally, if the second templated document does not overlap with the third templated document, it indicates that the second templated document contains sufficient original or different content and can be considered a document with independent value. In this case, the inbound management module 28 should allow the new document to be added to the document library to enrich its content and ensure the comprehensiveness and up-to-dateness of the knowledge base. Simultaneously, the inbound management module 28 can create a content index for the second templated document within the document library. This index can be built based on the document's first hash value, keywords, or other characteristics, avoiding repeated calculations of document hash values ​​in future queries and improving the speed of deduplication for subsequent new document additions.

[0141] If the second templated document is identical to the third templated document, it indicates that the second templated document highly overlaps with the documents already stored in the document library in terms of content, lacking new information or independent value. To avoid redundant information in the knowledge base and save storage space, the storage management module 28 should refuse to store the second templated document in the document library.

[0142] It should be noted that each module in the templated document management device in this application embodiment corresponds one-to-one with each implementation step of the templated document management method in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.

[0143] Example 3

[0144] According to an embodiment of this application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, it implements the templated document management method in embodiment 1.

[0145] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the templated document management method in Embodiment 1 by running the computer program.

[0146] According to an embodiment of this application, a processor is also provided for running a computer program, wherein the computer program executes the templated document management method in embodiment 1 during runtime.

[0147] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the templated document management method of Embodiment 1 through the computer program.

[0148] Specifically, the computer program executes the following steps during runtime: It identifies the third templated document among multiple first templated documents in the document library that has the highest first similarity to the second templated document to be added to the library, and determines the first complexity of the second templated document and the second complexity of the third templated document; based on the first and second complexities, it divides the second and third templated documents into a first number of first character blocks and a second number of second character blocks, respectively; it uses a target Bloom filter to perform duplicate character block checks on the first number of first character blocks and the second number of second character blocks, and determines whether the second templated document and the third templated document are duplicates based on the check results; if the second templated document and the third templated document are not duplicates, it stores the second templated document in the document library.

[0149] As an alternative implementation, the above-mentioned electronic device may exist in the form of a mobile terminal, a computer terminal, or a similar computing device. Figure 3 A hardware structure block diagram of a computer terminal for implementing a templated document management method is shown. Figure 3 As shown, the computer terminal 30 may include one or more processors 302 (shown as 302a, 302b, ..., 302n in the figure) 302 (processor 302 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 304 for storing data, and a transmission device 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 3 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, computer terminal 30 may also include... Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown.

[0150] It should be noted that the aforementioned one or more processors 302 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 30. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0151] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the templated document management method in this embodiment. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, thereby implementing the aforementioned application vulnerability detection method. The memory 304 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include memory remotely located relative to the processor 302, and these remote memories can be connected to the computer terminal 30 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0152] The transmission device 306 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 30. In one example, the transmission device 306 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 306 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0153] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 30.

[0154] The sequence numbers of the above embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0155] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0159] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0160] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A templated document management method, characterized in that, include: Identify the third templated document with the highest first similarity among multiple first templated documents in the document library and the second templated document to be added to the library, and determine the first complexity of the second templated document and the second complexity of the third templated document respectively; Based on the first complexity and the second complexity, the second templated document and the third templated document are respectively divided into a first number of first character blocks and a second number of second character blocks; The first number of first character blocks and the second number of second character blocks are checked for duplicate character blocks using a target Bloom filter, and the second templated document is determined to be duplicated with the third templated document based on the check results. If the second templated document is not the same as the third templated document, the second templated document is stored in the document library.

2. The method according to claim 1, characterized in that, Identify the third templated document with the highest first similarity among multiple first templated documents in the document library and the second templated document to be added to the library, including: A preset hash algorithm is used to determine the first hash value of each of the multiple first templated documents in the document library, and the second hash value of the second templated document. The distance between each of the first hash value and the second hash value is determined using a preset similarity algorithm, and the distance is used as the first similarity between the second templated document and the corresponding first templated document; Identify the third templated document that has the highest first similarity to the second templated document among multiple first templated documents in the document library.

3. The method according to claim 1, characterized in that, Both the first complexity and the second complexity include: content complexity and structural complexity, wherein determining the first complexity of the second templated document and the second complexity of the third templated document respectively includes: For any one of the second and third templated documents, determine the total number of words, the number of unique words, the total number of sentences, the length of each sentence, the total number of titles, the total number of images, and the total number of tables in the templated document; The ratio of the number of unique words to the total number of words is determined to obtain the vocabulary diversity. The ratio of the sum of the lengths of all sentences to the total number of sentences is determined to obtain the average sentence length. The vocabulary diversity and the average sentence length are then weighted and summed to obtain the corresponding content complexity. The total number of titles, images, and tables are weighted and summed to obtain the corresponding structural complexity.

4. The method according to claim 3, characterized in that, Based on the first complexity and the second complexity, the second templated document and the third templated document are respectively divided into a first number of first character blocks and a second number of second character blocks, including: Based on the first complexity and the second complexity, the size of the character block is determined according to the following formula. In the formula, These represent the content complexity and structural complexity of the second templated document, respectively. These represent the content complexity and structural complexity of the third templated document, respectively. Indicates the adjustment factor; The second templated document and the third templated document are divided into blocks according to the specified character block size to obtain the first number of first character blocks and the second number of second character blocks.

5. The method according to claim 3, characterized in that, Before performing duplicate character block checks on the first number of first character blocks and the second number of second character blocks using a targeted Bloom filter, the method further includes: The content complexity factor is obtained by determining the average of the content complexity of the second templated document and the content complexity of the third templated document, and the structural complexity factor is obtained by determining the average of the structural complexity of the second templated document and the structural complexity of the third templated document. Based on the content complexity factor and the preset false positive rate, the third number of hash functions within the target Bloom filter is determined according to the following formula: In the formula, p represents the content complexity factor, and p represents the false positive rate; The fourth quantity of the inner array of the target Bloom filter is determined based on the sum of the first and second quantities, the structural complexity factor, and the preset false positive rate. In the formula, Let p represent the structural complexity factor, p represent the false positive rate, and n represent the sum of the first quantity and the second quantity.

6. The method according to claim 1, characterized in that, The duplicate character block check is performed on the first number of first character blocks and the second number of second character blocks using a targeted Bloom filter, including: For each character block within the first number of first character blocks and the second number of second character blocks, the character block is processed using the third number of hash functions within the target Bloom filter to obtain the third number of hash values. The third number of hash values ​​are then mapped to target positions within the fourth number of arrays, and the target positions are marked. Determine the fifth number of target positions within the fourth number of arrays that are simultaneously marked by the first number of first character blocks and the second number of second character blocks, and obtain the fifth number of repeated character blocks in the second templated document and the third templated document.

7. The method according to claim 1, characterized in that, Determining whether the second templated document and the third templated document are duplicates based on the inspection results includes: Calculate the ratio of the fifth quantity to the first quantity to obtain the second similarity between the second templated document and the third templated document; If the second similarity is not lower than the first preset threshold, it is determined that the second templated document and the third templated document are duplicates. If the second similarity is lower than the first preset threshold, it is determined that the second templated document and the third templated document do not overlap.

8. A templated document management device, characterized in that, include: The determination module is used to determine the third templated document with the highest first similarity among multiple first templated documents in the document library and the second templated document to be added to the library, and to determine the first complexity of the second templated document and the second complexity of the third templated document respectively. The segmentation module is used to divide the second templated document and the third templated document into a first number of first character blocks and a second number of second character blocks respectively, based on the first complexity and the second complexity. The duplicate checking module is used to perform duplicate character block checks on the first number of first character blocks and the second number of second character blocks using a target Bloom filter, and determine whether the second templated document and the third templated document are duplicates based on the check results; The inbound management module is used to store the second templated document into the document library if the second templated document is not a duplicate of the third templated document.

9. A computer program product, characterized in that, include: A computer program, wherein when executed by a processor, the computer program implements the templated document management method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the templated document management method according to any one of claims 1 to 7 through the computer program.

Images