Attribute weighted fusion-based multi-source heterogeneous document deduplication method and system, equipment and medium
By extracting and weighting key document attributes to perform deduplication of multi-source heterogeneous documents, the accuracy and robustness issues in existing technologies under information-deficient scenarios are solved, achieving high-precision document deduplication and adapting to the application needs of various fields.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN XINGHAN DIGITAL INTELLIGENCE TECH CO LTD
- Filing Date
- 2025-12-29
- Publication Date
- 2026-05-01
AI Technical Summary
Existing methods for deduplicating multi-source heterogeneous documents lack accuracy and robustness in scenarios with missing information, and are unable to effectively overcome the interference of differences in the surface of documents, resulting in low accuracy and robustness in duplicate detection.
By extracting a set of key attributes from the document content and assigning a weight value to each key attribute, and performing duplicate judgment based on the key attribute set and mapping table, combined with semantic vector similarity retrieval, high-precision document deduplication is achieved.
It improves the accuracy and robustness of document deduplication, reduces the dependence on information integrity, adapts to different vertical fields, and has efficient versatility and generalization capabilities.
Smart Images
Figure CN121960432A_ABST
Abstract
Description
A method, system, device, and media for deduplication of multi-source heterogeneous documents based on attribute-weighted fusion. Technical Field
[0001] This invention relates to the field of information retrieval technology, and in particular to a method and system for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion, an electronic device, and a computer-readable storage medium. Background Technology
[0002] In the internet and big data environment, information about the same entity or event is often published from different data sources, resulting in a large number of documents with duplicate content but different presentations on the internet. Many application platforms (such as search engines and data analysis platforms) require data cleaning; therefore, deduplication of multi-source heterogeneous documents is necessary. Existing commonly used document deduplication methods mainly include fingerprint-based methods, text similarity-based methods, and rule-matching methods. Fingerprint-based methods (such as the common SimHash and MD5 algorithms) generate a unique fingerprint (usually a hash value) based on the document content. If two multi-source heterogeneous documents have the same fingerprint, they are considered duplicates. Their advantage is fast computation speed, but their disadvantage is that they are extremely sensitive to minor changes to the document (such as adding or removing punctuation, adjusting word order, etc.), easily leading to false positives and false negatives. Text similarity-based deduplication methods use the bag-of-words model combined with the TF-IDF algorithm (Term Frequency-Inverse Document). Frequency (word frequency-inverse document frequency) algorithm, or using a deep learning model (such as the BERT model) to generate text vectors for two documents and then determining whether they are duplicates by calculating the cosine similarity between the two text vectors. This method can capture some semantic information, but it is difficult to deal with situations where the core content is the same but the narrative style and structural layout are very different, and the computational cost is large. Rule-based deduplication methods, on the other hand, use manually defined matching rules for key fields (such as ID, title, etc.) to determine whether two documents are duplicates. This method is simple and effective in specific domains, but the cost of writing and maintaining the rules is high, it lacks flexibility, and it cannot adapt to new document sources or formats.
[0003] In summary, existing fingerprint-based and rule-matching-based deduplication methods treat documents as a single text object, lacking the identification and utilization of key attributes with varying discriminative power within the document. This results in low accuracy and robustness in vertical domain deduplication tasks. Existing fingerprint-based methods struggle to effectively remove noise unrelated to core content (such as advertising text, copyright information, formatting errors, and synonym substitutions), which significantly impacts the accuracy and robustness of duplicate detection. Furthermore, existing rule-matching-based methods exhibit poor accuracy in scenarios with missing information. When documents from certain data sources are missing key fields (e.g., lacking an ID field), rule-matching-based methods fail, and even overall text comparison suffers from reduced accuracy due to the reduced information content. Therefore, effectively overcoming the interference of surface-level document differences and achieving high-precision, robust document deduplication in multi-source, heterogeneous, and information-missing scenarios is a critical issue that urgently needs to be addressed. Summary of the Invention
[0004] This invention provides a method and system for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion, as well as an electronic device and a computer-readable storage medium. It can perform high-precision and robust document deduplication in multi-source, heterogeneous, and information-missing scenarios, and has strong versatility and generalization ability.
[0005] According to one aspect of the present invention, a method for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion is provided, comprising the following:
[0006] Receive the document to be deduplicated and extract the set of key attributes from the document content.
[0007] A mapping table between key attributes and weights is pre-defined, with each key attribute corresponding to a weight value. The magnitude of the weight value intuitively reflects the discriminative power of the corresponding key attribute in deduplication decision-making.
[0008] Based on the key attribute set and mapping table, duplicate judgment is performed on the document to be deduplicated and the existing documents in the document database. If a duplicate document is determined, the document to be deduplicated is deleted; if no duplicate document is determined, the document to be deduplicated is stored in the document database.
[0009] Furthermore, the process of performing duplicate judgment on the document to be deduplicated and the existing documents in the document database based on the key attribute set and mapping table includes the following:
[0010] The document to be deduplicated is compared one by one with all existing documents in the document database. During the comparison process, for each pair of documents, the common non-empty key attributes of the two documents are traversed, and the values of each common non-empty key attribute are matched.
[0011] The weight values corresponding to all key attributes that successfully match the value are summed up to obtain the total repetition score.
[0012] The duplication score is compared with a preset threshold. If the duplication score is greater than or equal to the preset threshold, the two documents are determined to be duplicates; otherwise, the two documents are determined not to be duplicates.
[0013] Furthermore, the following is included before performing the repetitiveness check step:
[0014] Extract the main identifier text of the document to be deduplicated, and convert the main identifier text into a semantic vector. Based on the semantic vector, perform similar document retrieval in the document database, sort them from high to low similarity, and select multiple similar documents with similarity higher than a preset threshold to generate a candidate deduplicated document set.
[0015] Furthermore, if no similar documents are found during the retrieval process, the documents to be deduplicated are directly stored in the document database.
[0016] Furthermore, when comparing the duplication score with a preset threshold, a confidence interval is introduced. If the duplication score is greater than or equal to the preset threshold but within the confidence interval, it is submitted for manual review; if the duplication score is greater than or equal to the preset threshold but exceeds the confidence interval, the two documents are determined to be duplicates.
[0017] Furthermore, for news documents, key attributes include the subject of the event, the time of occurrence, and the core location; for academic papers, key attributes include DOI, title, list of authors, and year of publication; and for product documents, key attributes include product model, brand, manufacturer, and key specifications.
[0018] In addition, the present invention also provides a multi-source heterogeneous document deduplication system based on attribute-weighted fusion, comprising:
[0019] The key attribute extraction module is used to receive the document to be deduplicated and extract the set of key attributes from the document content.
[0020] The mapping table preset module is used to preset the mapping table between key attributes and weights. Each key attribute corresponds to a weight value. The magnitude of the weight value intuitively reflects the discriminative power of the corresponding key attribute in the deduplication decision.
[0021] The deduplication decision module is used to determine the duplication of the document to be deduplicated and the existing documents in the document database based on the key attribute set and mapping table. If a duplicate document is determined, the document to be deduplicated is deleted; if no duplicate document is determined, the document to be deduplicated is stored in the document database.
[0022] Furthermore, it also includes:
[0023] The similar document retrieval module is used to extract the main identifier text of the document to be deduplicated, convert the main identifier text into a semantic vector, perform similar document retrieval in the document database based on the semantic vector, sort the documents from high to low similarity, and generate a candidate deduplicated document set by selecting multiple similar documents with similarity higher than a preset threshold.
[0024] In addition, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the steps of the method described above by calling the computer program stored in the memory.
[0025] In addition, the present invention provides a computer-readable storage medium for storing a computer program for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion, wherein the computer program executes the steps of the method described above when running on a computer.
[0026] This invention presents a multi-source heterogeneous document deduplication method based on attribute-weighted fusion. It extracts a set of key attributes from the document content, where each key attribute represents a core aspect of the document. Different weight values are assigned to different key attributes, directly reflecting their discriminative power in deduplication decisions. The method then uses the key attribute set and mapping table to determine duplicate content between two documents. This focuses on and weights the core attributes of the documents, transforming the deduplication problem from text similarity comparison to a weighted fusion decision based on the discriminative power of key attributes. It ignores non-critical differences between documents (such as formatting and wording), resulting in more accurate identification of duplicate content and significantly improving the accuracy and robustness of deduplication judgment. Furthermore, the weighting mechanism allows for reliable judgments based on existing key attributes even when some key attributes are missing or contain errors, reducing the stringent dependence on information integrity. It achieves high-precision duplicate content identification even in scenarios with missing information. In addition, by configuring different key attribute sets and mapping tables, it can be quickly and flexibly adapted to various vertical fields, greatly improving its versatility and generalization ability.
[0027] In addition, the multi-source heterogeneous document deduplication system based on attribute weighted fusion of the present invention also has the above-mentioned advantages.
[0028] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the figures. Attached Figure Description
[0029] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:
[0030] Figure 1 is a flowchart illustrating the multi-source heterogeneous document deduplication method based on attribute-weighted fusion according to a preferred embodiment of this application;
[0031] Figure 2 is a schematic diagram of the sub-process of step S3 in Figure 1;
[0032] Figure 3 is another flowchart of the multi-source heterogeneous document deduplication method based on attribute weighted fusion according to a preferred embodiment of this application;
[0033] Figure 4 is a schematic diagram of the module structure of a multi-source heterogeneous document deduplication system based on attribute weighted fusion according to another embodiment of this application;
[0034] Figure 5 is a schematic diagram of another module structure of a multi-source heterogeneous document deduplication system based on attribute weighted fusion according to another embodiment of this application. Detailed Implementation
[0035] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0036] Referring to Figure 1, a preferred embodiment of this application provides a method for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion, including the following:
[0037] Step S1: Receive the document to be deduplicated and extract the set of key attributes from the document content.
[0038] Understandably, the documents to be deduplicated are multi-source heterogeneous documents. Multi-source means that the data comes from different sources, such as websites, databases, file servers, etc. Heterogeneous means that the data formats and types are inconsistent, including structured, semi-structured and unstructured data.
[0039] Specifically, in step S1, after receiving the document to be deduplicated, an information extraction technique is used to extract a predefined set of key attributes from the document. In this set, each key attribute represents a core aspect of the document. Furthermore, the information extraction technique extracts structured key attribute-value pairs from the document's text content, facilitating accurate document matching in the subsequent process. The information extraction technique includes an integrated named entity recognition model and predefined rule templates, all of which are existing technologies and will not be elaborated upon here.
[0040] Optionally, different document types correspond to different key attributes. For news documents, key attributes include the subject of the event, the time of occurrence, and the core location; for academic papers, key attributes include DOI, title, author list, and publication year; and for product documents, key attributes include product model, brand, manufacturer, and key specifications. This invention sets different key attributes for different document types, which helps improve the accuracy of deduplication for different document types.
[0041] Step S2: Preset a mapping table between key attributes and weights. Each key attribute corresponds to a weight value. The magnitude of the weight value intuitively reflects the discriminative power of the corresponding key attribute in deduplication decision-making.
[0042] Understandably, in step S2, a mapping table between key attributes and weights is pre-defined in the document database. This mapping table assigns a weight value to each key attribute. The magnitude of the weight value directly reflects the discriminative power (i.e., importance) of the corresponding key attribute in the deduplication decision. The stronger the discriminative power of a key attribute (such as a unique ID), the larger its corresponding weight value; conversely, the weaker the discriminative power of a key attribute (such as descriptive text), the smaller its corresponding weight value. For example, for news documents, the main body of the event has the highest weight; for academic papers, the DOI has the highest weight, and if there is no DOI, the title and author list are given high weights; for product documents, the product model has the highest weight. The document database includes a vector index library, an attribute database, and a weight database. The vector index library mainly stores the index of document vectors for fast semantic vector similarity retrieval; the attribute database mainly stores the set of key attributes corresponding to existing documents, serving as the basis for subsequent duplicate judgments; and the weight database stores pre-configured attribute field weights.
[0043] Understandably, this invention, by pre-setting a mapping table between key attributes and weights, assigns different weight values to different key attributes, reflecting the discriminative power of different key attributes in subsequent deduplication decisions, and providing an accurate data foundation for subsequent document deduplication.
[0044] Furthermore, in other embodiments of the present invention, the mapping table can be dynamically updated. For example, the system can automatically learn the discriminative power of each key attribute based on historical deduplicated data using a machine learning model, and dynamically adjust its weight values. Additionally, different mapping tables can be preset for different vertical domains.
[0045] Step S3: Based on the key attribute set and mapping table, perform duplicate judgment on the document to be deduplicated and the existing documents in the document database. If duplicate documents are found, delete the document to be deduplicated. If no duplicate documents are found, store the document to be deduplicated in the document database.
[0046] Understandably, based on a set of key attributes and a pre-defined mapping table, the document to be deduplicated is compared with existing documents in the document database for duplicate identification. A deduplication decision is then made based on the result; for example, if duplicate documents are identified, the document to be deduplicated is deleted; if no duplicate documents are identified, it means the document to be deduplicated is a new document, and it is stored in the document database. The document database includes an attribute database, which stores the key attribute sets corresponding to all existing documents. Each key attribute set is associated with a corresponding existing document. Furthermore, after duplicate document filtering, the multi-source, heterogeneous documents to be deduplicated form a unique set of documents in the document database, providing accurate and reliable clean data for upper-layer applications (such as search engines and data analysis platforms). This data can be widely applied in big data preprocessing, information aggregation, knowledge base construction, and search engine optimization.
[0047] As shown in Figure 2, in step S3, the process of performing duplicate judgment on the document to be deduplicated and the existing documents in the document database based on the key attribute set and mapping table includes the following:
[0048] Step S31: Compare the document to be deduplicated with all existing documents in the document database one by one. During the comparison process, for each pair of documents, iterate through the common non-empty key attributes of the two documents and perform value matching for each common non-empty key attribute.
[0049] Step S32: Sum the weight values of all key attributes that successfully match the value to get the total repetition score.
[0050] Step S33: Compare the duplication evaluation score with a preset threshold. If the duplication evaluation score is greater than or equal to the preset threshold, the two documents are determined to be duplicates; otherwise, the two documents are determined not to be duplicates.
[0051] Specifically, the document to be deduplicated is compared one by one with all existing documents in the document database. During the comparison, for each pair of documents (consisting of the document to be deduplicated and an existing document in the document database), the common non-empty key attributes of the two documents are traversed, and value matching is performed on each common non-empty key attribute. The weight values corresponding to all key attributes with successful value matching are accumulated as the total duplication evaluation score S. The duplication evaluation score S is compared with a preset threshold T. If the duplication evaluation score is greater than or equal to the preset threshold T, i.e., S≥T, the two documents are determined to be duplicates; otherwise, the two documents are determined to be non-duplicates. If the duplication evaluation score S of the document to be deduplicated and all existing documents in the document database is less than the preset threshold T, the document to be deduplicated is determined to be a new document and is stored in the document database.
[0052] Optionally, attribute value matching can be achieved through exact matching, edit distance matching, or by using semantic similarity calculations based on pre-trained models (such as the BERT model) as the matching criterion, combined with weights. These matching methods are all existing technologies and will not be elaborated upon here.
[0053] Optionally, when comparing the duplication evaluation score with the preset threshold, the concept of a confidence interval is introduced. If the duplication evaluation score is greater than or equal to the preset threshold but within the confidence interval, it is submitted for manual review to improve the accuracy of handling complex edge cases. Only when the duplication evaluation score is greater than or equal to the preset threshold and exceeds the confidence interval are the two documents determined to be duplicates, which helps to improve the accuracy of handling complex edge cases.
[0054] Optionally, as shown in Figure 3, the multi-source heterogeneous document deduplication method based on attribute-weighted fusion further includes the following before performing the duplicate judgment step:
[0055] Step S23: Extract the main identifier text of the document to be deduplicated, and convert the main identifier text into a semantic vector; based on the semantic vector, perform similar document retrieval in the document database, sort them from high to low similarity, and generate a candidate deduplicated document set by taking multiple similar documents with similarity higher than a preset threshold.
[0056] Specifically, after receiving the document to be deduplicated, it is parsed to extract its main identifier text (e.g., title, topic). Then, an embedding model is used to convert the main identifier text into a high-dimensional semantic vector. Based on the semantic vector, a similar document retrieval is performed in the document database. The document database includes a pre-defined document vector library, which stores the semantic vectors of all existing documents. The semantic vector corresponding to the document to be deduplicated is used as the query vector, and an approximate nearest neighbor search is performed in the document vector library to quickly retrieve the K most similar documents to the main identifier text of the document to be deduplicated, forming a candidate deduplication set. Subsequent duplication judgment only requires comparing the document to be deduplicated with the K similar documents in the candidate deduplication set, greatly reducing the computational load and burden of subsequent value matching and improving efficiency. Furthermore, if no similar document is found during the similar document retrieval process in the document vector library, the document to be deduplicated is directly stored in the document database. The embedding model is existing technology and will not be described in detail here.
[0057] Understandably, before performing duplicate judgment, this invention performs a coarse screening through rapid semantic similarity, limiting the computationally expensive fine attribute comparison to a very small candidate set, thus achieving efficient pre-filtering optimization. Through a two-stage screening strategy of semantic coarse screening and precise attribute comparison, it greatly improves the processing efficiency of massive data while ensuring high accuracy.
[0058] Understandably, the attribute-weighted fusion-based multi-source heterogeneous document deduplication method of this invention extracts a set of key attributes from the document content. Each key attribute in the set represents a core aspect of the document, and corresponding weight values are assigned to different key attributes. The magnitude of the weight value intuitively reflects the discriminative power of the key attribute in the deduplication decision. Then, based on the key attribute set and mapping table, two documents are judged for duplication. This makes the duplication judgment process focus on and weight the core attributes of the document, transforming the deduplication problem from text similarity comparison to a weighted fusion decision based on the discriminative power of key attributes. It ignores non-critical differences between documents (such as format, wording, etc.), making the identification of duplicate content more accurate and effectively improving the accuracy and robustness of duplication judgment. Furthermore, the weight mechanism allows for reliable judgment based on existing key attributes even when some key attributes are missing or have errors, reducing the stringent dependence on information integrity. Even in scenarios with missing information, it can still achieve high-precision duplicate content identification. In addition, by configuring different key attribute sets and mapping tables, it can be quickly and flexibly adapted to various vertical fields, greatly improving its versatility and generalization ability.
[0059] In addition, as shown in Figure 4, another embodiment of the present invention also provides a multi-source heterogeneous document deduplication system based on attribute-weighted fusion, preferably employing the multi-source heterogeneous document deduplication method based on attribute-weighted fusion as described above, including:
[0060] The key attribute extraction module is used to receive the document to be deduplicated and extract the set of key attributes from the document content.
[0061] The mapping table preset module is used to preset the mapping table between key attributes and weights. Each key attribute corresponds to a weight value. The magnitude of the weight value intuitively reflects the discriminative power of the corresponding key attribute in the deduplication decision.
[0062] The deduplication decision module is used to perform duplicate judgment on the document to be deduplicated and the existing documents in the document database based on the key attribute set and mapping table. If a duplicate document is determined, the document to be deduplicated will not be stored in the document database. If no duplicate document is determined, the document to be deduplicated will be stored in the document database.
[0063] Understandably, the attribute-weighted fusion-based multi-source heterogeneous document deduplication system in this embodiment extracts a set of key attributes from the document content. Each key attribute in the set represents a core aspect of the document, and different key attributes are assigned corresponding weight values. The magnitude of the weight value intuitively reflects the discriminative power of the key attribute in the deduplication decision. Then, based on the key attribute set and mapping table, two documents are judged for duplication. This makes the duplication judgment process focus on and weight the core attributes of the document, transforming the deduplication problem from text similarity comparison to a weighted fusion decision based on the discriminative power of key attributes. It ignores non-critical differences between documents (such as format, wording, etc.), making the identification of duplicate content in documents more accurate and effectively improving the accuracy and robustness of duplication judgment. Furthermore, the weight mechanism allows for reliable judgment based on existing key attributes even when some key attributes are missing or have errors, reducing the stringent dependence on information integrity. Even in scenarios with missing information, it can still achieve high-precision duplicate content identification. In addition, by configuring different key attribute sets and mapping tables, it can be quickly and flexibly adapted to various vertical fields, greatly improving its versatility and generalization ability.
[0064] In addition, as shown in Figure 5, the multi-source heterogeneous document deduplication system based on attribute-weighted fusion also includes:
[0065] The similar document retrieval module is used to extract the main identifier text of the document to be deduplicated, convert the main identifier text into a semantic vector, perform similar document retrieval in the document database based on the semantic vector, sort the similarity from high to low, and generate a candidate deduplicated document set by selecting multiple similar documents with similarity higher than a preset threshold.
[0066] It is understood that each step in this system embodiment corresponds to each step in the above method embodiment. Therefore, the specific working process and principle of each module will not be repeated here. Please refer to the steps in the above method embodiment.
[0067] In addition, another embodiment of the present invention provides an electronic device including a processor and a memory, wherein the memory stores a computer program, and the processor executes the steps of the method described above by calling the computer program stored in the memory.
[0068] In addition, another embodiment of the present invention provides a computer-readable storage medium for storing a computer program for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion, wherein the computer program executes the steps of the method described above when running on a computer.
[0069] Common computer-readable storage media include: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with perforated patterns, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), flash erasable programmable read-only memory (FLASH-EPROM), any other memory chips or cartridges, or any other media readable by a computer. Instructions may further be transmitted or received by a transmission medium. The term transmission medium can include any tangible or intangible medium used to store, encode, or carry instructions for execution by a machine, and includes digital or analog communication signals or intangible media that facilitate communication of such instructions. Transmission media include coaxial cables, copper wires, and optical fibers, which contain conductors for transmitting a bus of computer data signals.
[0070] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.
[0071] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.
[0072] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.
[0073] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.
[0074] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0075] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion, characterized in that, It includes the following: receiving documents to be deduplicated, extracting a set of key attributes from the document content; pre-setting a mapping table between key attributes and weights, where each key attribute corresponds to a weight value, and the magnitude of the weight value intuitively reflects the discriminative power of the corresponding key attribute in the deduplication decision; Based on the key attribute set and mapping table, duplicate judgment is performed on the document to be deduplicated and the existing documents in the document database. If a duplicate document is determined, the document to be deduplicated is deleted; if no duplicate document is determined, the document to be deduplicated is stored in the document database.
2. The method for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion as described in claim 1, characterized in that, The process of determining the duplication of the document to be deduplicated and the existing documents in the document database based on the key attribute set and mapping table includes the following: comparing the document to be deduplicated with all existing documents in the document database one by one; during the comparison process, for each pair of documents, traversing the common and non-empty key attributes of the two documents, and performing value matching for each common and non-empty key attribute; accumulating the weight values corresponding to all key attributes with successful value matching as the total duplication evaluation score; comparing the duplication evaluation score with a preset threshold; if the duplication evaluation score is greater than or equal to the preset threshold, the two documents are determined to be duplicates; otherwise, the two documents are determined not to be duplicates.
3. The method for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion as described in claim 1, characterized in that, Before performing the duplicate identification step, the following steps are also included: extracting the main identifier text of the document to be deduplicated, converting the main identifier text into a semantic vector, performing similar document retrieval in the document database based on the semantic vector, sorting the similarity from high to low, and generating a candidate deduplicated document set by taking multiple similar documents with similarity higher than a preset threshold.
4. The method for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion as described in claim 3, characterized in that, If no similar documents are found during the search process, the documents to be deduplicated are directly stored in the document database.
5. The method for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion as described in claim 2, characterized in that, When comparing the duplication score with a preset threshold, a confidence interval is introduced. If the duplication score is greater than or equal to the preset threshold but within the confidence interval, it is submitted for manual review; if the duplication score is greater than or equal to the preset threshold but exceeds the confidence interval, the two documents are determined to be duplicates.
6. The method for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion as described in claim 1, characterized in that, For news documents, key attributes include the subject of the event, the time of occurrence, and the key location; for academic papers, key attributes include DOI, title, list of authors, and year of publication; for product documents, key attributes include product model, brand, manufacturer, and key specifications.
7. A multi-source heterogeneous document deduplication system based on attribute-weighted fusion, characterized in that, include: The key attribute extraction module is used to receive the document to be deduplicated and extract the set of key attributes from the document content. The mapping table preset module is used to preset the mapping table between key attributes and weights. Each key attribute corresponds to a weight value. The magnitude of the weight value intuitively reflects the discriminative power of the corresponding key attribute in the deduplication decision. The deduplication decision module is used to determine the duplication of the document to be deduplicated and the existing documents in the document database based on the key attribute set and mapping table. If a duplicate document is determined, the document to be deduplicated is deleted; if no duplicate document is determined, the document to be deduplicated is stored in the document database.
8. The multi-source heterogeneous document deduplication system based on attribute-weighted fusion as described in claim 7, characterized in that, Also includes: The similar document retrieval module is used to extract the main identifier text of the document to be deduplicated, convert the main identifier text into a semantic vector, perform similar document retrieval in the document database based on the semantic vector, sort the documents from high to low similarity, and generate a candidate deduplicated document set by selecting multiple similar documents with similarity higher than a preset threshold.
9. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the steps of the method as described in any one of claims 1 to 6 by calling the computer program stored in the memory.
10. A computer-readable storage medium for storing a computer program for deduplicating multi-source heterogeneous documents based on attribute-weighted fusion, characterized in that, The computer program, when run on a computer, performs the steps of the method as described in any one of claims 1 to 6.