Document metadata fusion method and system based on dual-rail decoupling and support transfer

Abstract

The application discloses a literature metadata fusion method and system based on double-track decoupling and support degree transmission, and belongs to the technical field of data management and knowledge fusion. The application comprises the following steps: aligning literature metadata from multiple sources based on DOI to form a multi-source candidate text pool; calculating character-level similarity and semantic-level similarity of source text pairs in parallel, and extracting the difference between the two-track similarities as a decoupling feature; automatically determining the data difference into four types of consistent and reliable, transcription error, semantic rewriting or completely different according to the two-track similarities and the difference; and triggering differentiated support transmission strategies for different types. The application can effectively utilize the cognitive bias between models in the double-track architecture, identify and repair the OCR recognition errors and special symbol garbled codes that are difficult to detect by single-track models in multi-source metadata fusion, break the limitations of static priority, and significantly improve the physical purity and adaptive fault tolerance of massive multi-source academic metadata fusion.

Description

Technical Field

[0001] This invention relates to the field of document metadata fusion and data governance technology, specifically to a document metadata fusion method and system based on dual-track decoupling and support transfer. Background Technology

[0002] With the explosive growth of global academic big data and scientific and technological information, the amount of literature data from various publishers, preprint platforms, and academic search engines (such as Web of Science, Scopus, PubMed, arXiv, etc.) is increasing exponentially. Building high-quality academic knowledge graphs and underlying search engines inevitably requires the merging and unification of multi-source, heterogeneous literature metadata. However, due to the significant diversity in data collection methods, OCR (Optical Character Recognition) technology levels, manual input standards, and underlying character encoding among various data sources, the metadata of the same document often exhibits serious conflicts and differences across different data sources. How to achieve efficient and accurate integration of cross-platform, multi-source literature metadata has become a major technical challenge in the field of data governance.

[0003] In practice, metadata, as the carrier describing the core attributes of documents (such as title, abstract, author, keywords, etc.), directly determines the success or failure of upper-level data mining and retrieval analysis based on its accuracy and purity. Currently, most technical solutions for multi-source document metadata fusion still have limitations. Early data fusion platforms mainly relied on crude rules such as "static priority" or "predefined source authority" (e.g., rigidly stipulating that "journal website data always covers preprint data"), but this method completely ignores the possibility of local field parsing failures or transcription garbled characters in high-quality data sources, making it difficult to cope with complex and ever-changing real-world data environments.

[0004] With the development of artificial intelligence and natural language processing technologies, current metadata fusion mainly adopts text similarity-based calculation methods: First, multi-source data are aligned to form a candidate pool using unique document identifiers (such as DOI); then, for conflicting metadata fields, either traditional string matching algorithms (such as Jaccard coefficient and edit distance) are used to calculate literal similarity, or high-dimensional vectors are extracted and semantic similarity is calculated using deep learning pre-trained models (such as BERT and its variants); finally, a similarity threshold is set, and the final metadata to be included in the database is selected by combining traditional majority voting or weighted average mechanisms.

[0005] However, existing methods for multi-source metadata fusion still have significant shortcomings. Specifically:

[0006] (1) Single-dimensional similarity has a “physical error blind spot”: Existing technologies usually use literal matching or semantic matching in an either-or manner. Pure literal matching cannot identify synonym substitution or legitimate rewriting; while pure semantic models cannot distinguish transcription errors that are “semantically consistent but physically corrupted” when processing document metadata (for example, the number “1” in the title is misidentified as the letter “l” by OCR, or mathematical symbols become HTML garbled text under different encodings). Semantic models will determine that text containing garbled text has a very high similarity to clean text, which makes the system very likely to mistake “dirty data” for valid data.

[0007] (2) Static priority fixation and partial failure: In cross-source data fusion, there is an over-reliance on the preset authority level of the data source. When a high-priority data source experiences partial field truncation, failure to parse special characters, or manual spelling errors, the existing system lacks the ability to dynamically penetrate the priority adaptively, resulting in high-quality but low-priority data sources not being reasonably adopted.

[0008] (3) Traditional majority voting mechanisms lack error detection and targeted repair capabilities: When faced with multi-source text conflicts, existing voting fusion mechanisms often only perform simple summation or averaging of votes. When there is obvious data corruption, the system cannot diagnose the type of conflict (whether it is consistent and reliable, semantic rewriting or transcriptional corruption), nor can it establish asymmetric "vote absorption and transfer" between conflicting nodes, resulting in a lack of logical depth in the fusion process, which easily leads to deadlock or random selection.

[0009] (4) Fragmentation of rules and high maintenance costs in cross-field fusion: When processing different types of metadata fields (such as the title summary of long texts, authors or keywords with multiple values), existing technologies often require the design of complex rule templates or independent cleaning and mapping logic for specific fields. This results in a bloated and unified system architecture, high computational overhead when processing large-scale real-time data, and poor scalability and universality.

[0010] (5) Low purity and consistency of the final output data format: Due to the lack of "decoupling computation" of the differences between the deep semantics and the surface physical literal of the text, the existing tools cannot maximize the preservation of the data format purity while ensuring multi-source consensus when processing massive heterogeneous data, resulting in a large number of micro errors and noise characters remaining in the final standardized metadata.

[0011] In summary, existing technologies have failed to fully utilize the cognitive bias between deep learning models and traditional algorithms to infer the type of data quality problems, nor have they been able to customize differentiated support delivery strategies for different error types in fusion decision-making. Summary of the Invention

[0012] To address the shortcomings of existing technologies, the present invention aims to propose a document metadata fusion method and system based on dual-track decoupling and support transfer.

[0013] The objective of this invention is achieved through the following technical solution: a document metadata fusion system based on dual-track decoupling and support transfer, comprising:

[0014] The data preprocessing and alignment module is used to align metadata from multiple sources based on unique identifiers, preprocess metadata from different fields, construct a multi-source candidate text pool for the same document, and count the number of times each exact text appears in different data sources and record it as direct support.

[0015] The dual-track feature decoupling calculation module is used to adaptively select a character-level similarity calculation method based on the field type for texts of the same field from any two sources in the multi-source candidate text pool, and to calculate character-level similarity in parallel. and semantic similarity The dual-track similarity is calculated, and the difference between the dual-track similarities is extracted. As a feature decoupling variable;

[0016] The error type diagnosis module is used to receive the dual-track similarity and the difference between the dual-track similarity, and based on a preset threshold, to determine the data conflict between multi-source text pairs into four error types, namely consistent and reliable, transcription error, semantic rewriting and completely different, respectively, and to represent different levels of the degree of data conflict between text pairs through different error types;

[0017] The differentiated fusion decision module is used to perform differentiated directed transmission of support between multi-source text pairs based on the determined error type, and output the optimal standardized metadata in combination with the format purity assessment.

[0018] Furthermore, the data preprocessing and alignment module includes: aggregating the same document records from multiple data sources (including but not limited to Web of Science, arXiv, PubMed, OpenAlex, Semantic Scholar, etc.) based on the document's digital object identifier; cleaning the original text of each field (title, abstract, author, keywords, etc.), including removing HTML tags, escape characters, invisible characters, and uniformly converting them to lowercase and standardizing whitespace; expanding the cleaned text by source to form a multi-source candidate text set for each field; and counting the number of times each exact text appears in different data sources, and recording the number of occurrences as direct support.

[0019] Furthermore, in the dual-track feature decoupling calculation module, the adaptive selection of the character-level similarity calculation method based on the field type specifically includes:

[0020] For the title and abstract fields, character-level similarity is calculated using normalized edit distance (Levenshtein distance), and the formula is as follows:

[0021] ;

[0022] in, For string and Edit distance between The length of the string;

[0023] For short text or multi-valued fields such as Author and Keywords, character-level similarity is calculated using Jaccard similarity based on lexical units. First, the text is segmented into a set of lexical units. and The calculation formula is:

[0024] .

[0025] Furthermore, in the dual-track feature decoupling calculation module, the semantic-level similarity is calculated by extracting high-dimensional embedding vectors of the text using a pre-trained deep learning language model and calculating cosine similarity. The deep learning language model includes Sentence-BERT and its variants.

[0026] Furthermore, in the error type diagnosis module, the step of determining data conflicts between multi-source text pairs into different error types based on a preset threshold specifically involves:

[0027] When the semantic similarity is not less than the first threshold and the character similarity is not less than the second threshold, it is judged as Type 1 (consistent and trustworthy), which means that different data sources are normal representations of the same content;

[0028] When the semantic similarity is not less than the first threshold and the difference is not less than the third threshold, it is judged as type two (transcription / OCR error), which means that the semantics are consistent but the physical format is destroyed, such as OCR recognition error or garbled text;

[0029] When the semantic similarity is not less than the fourth threshold and less than the first threshold, and the character similarity is less than the fifth threshold, it is judged as type three (semantic rewriting), which means that the abbreviation or rewritten version is manually written from different data sources;

[0030] When the semantic similarity is less than the fourth threshold and the character similarity is less than the fifth threshold, it is judged as type four (completely different), indicating that the data source database is incorrect or contains completely irrelevant content;

[0031] Wherein, the first threshold is greater than the fourth threshold, and the second threshold is greater than the fifth threshold; and the above thresholds are all preset by historical data statistics or domain experience, and can be dynamically adjusted; since different fields use different character-level similarity calculation methods, the thresholds can be configured according to the field type, but a unified threshold is used in this embodiment for simplification.

[0032] Furthermore, the first threshold value ranges from 0.8 to 0.9, the second threshold value ranges from 0.7 to 0.8, the third threshold value ranges from 0.25 to 0.35, the fourth threshold value ranges from 0.45 to 0.55, and the fifth threshold value ranges from 0.35 to 0.45.

[0033] Furthermore, in the differentiated fusion decision module, a differentiated support delivery strategy is executed based on the error type, including:

[0034] For text pairs classified as type one, a two-way similarity support transfer is performed, i.e., source Sources of increased total support direct support ,source Sources of increased total support direct support ;

[0035] For text pairs determined to be of type two, an asymmetric one-way vote absorption mechanism is triggered, including: evaluating the format purity of the two sources, selecting the source with higher purity as the absorber, and unilaterally transferring the direct support of the source with lower purity to the absorber. The total support of the absorber is its own direct support plus the direct support of the source with lower purity, while the direct support of the source with lower purity remains unchanged.

[0036] For text pairs classified as type 3 and type 4, the transmission of similarity support is blocked, only their respective direct support is retained, and the process reverts to a downgraded voting mode based on the static priority and completeness of the data source.

[0037] Furthermore, the format purity P is obtained by weighted summation of character purity and length reasonableness. Character purity is the proportion of normal alphanumeric characters to the total number of characters, and length reasonableness is the smaller value of the ratio of the total number of characters to the expected length of the field and 1. The sum of the weight coefficients of the two items is 1.

[0038] The specific formula for calculating P is as follows:

[0039] ;

[0040] in, The number of normal alphanumeric characters in the text. Total number of characters To preset the expected length based on the field type, and These are the weighting coefficients, and .

[0041] Furthermore, after passing the support scores for all text pairs, the normalized score for each source is calculated: the total support score for each source is divided by the difference between the total number of sources containing data in that field and 1, and this difference is used as the normalized score for that source; the calculation formula is as follows:

[0042] ;

[0043] in, For source Total support This represents the total number of data sources for this field; when When the score is set to NULL, the source with the highest score is selected as the final data source for this field. If multiple sources have the same score, they are selected according to the preset field priority order (such as WoS > arXiv > PubMed, etc.).

[0044] This invention also provides a document metadata fusion method based on dual-track decoupling and support transfer, comprising the following steps:

[0045] Obtain metadata from multiple sources, align them based on unique identifiers, preprocess the metadata of different fields, construct a multi-source candidate text pool for the same document, and count the number of times each exact text appears in different data sources as direct support.

[0046] For any two texts of the same field from any two sources in the multi-source candidate text pool, the character-level similarity calculation method is adaptively selected according to the field type, and the dual-track similarity including character-level similarity and semantic-level similarity is calculated in parallel, and the difference between the dual-track similarity is extracted as a feature decoupling variable.

[0047] Based on the dual-track similarity and difference, and according to the preset threshold, the data conflict between multi-source text pairs is determined into four error types, which represent consistent credibility, transcription error, semantic rewriting and completely different, respectively. Different error types represent different levels of the degree of data conflict between text pairs.

[0048] Based on the identified error type, the support between multi-source text pairs is differentially propagated in a directed manner, and combined with the format purity assessment, the optimal standardized metadata is output.

[0049] The beneficial effects of this invention are as follows:

[0050] (1) Overcoming the physical error blind spot of pure semantic models: This invention pioneers the "syntactic-semantic dual-track decoupling" architecture, which calculates the cognitive bias (difference) between models. This invention successfully enables the system to identify OCR errors (such as "1" and "l") and failures in parsing special symbols. Compared to the problem of dirty data being easily imported into the database when using existing large models such as SBERT, this invention significantly improves the physical purity of the fused data.

[0051] (2) Breaking the static priority rigidity and improving the fault tolerance rate: This invention proposes an asymmetric one-way vote absorption mechanism, which can dynamically penetrate the preset source priority when a transcription error is diagnosed. Even if the data source has low authority, as long as its format is clean, it can legally absorb the votes of the authoritative data source and win, ensuring the system's adaptive ability to dynamic data quality.

[0052] (3) Field-adaptive character similarity calculation: This invention uses edit distance for titles and abstracts to capture subtle differences at the character level, and Jaccard similarity for authors and keywords to handle multi-valued entities and order insensitivity issues, thereby more accurately measuring text differences at the character level and improving the accuracy of subsequent diagnosis.

[0053] (4) Unified process reduces system complexity: The present invention adopts a unified fusion process for all fields, eliminating the need to design complex rule templates for different types of fields such as title, abstract, author, and keywords. This greatly simplifies the system architecture, reduces maintenance costs, and ensures the consistency of the fusion logic for each field.

[0054] (5) Dynamic support transfer enhances fusion robustness: Through error type-driven directed vote transfer, the system can intelligently identify the nature of data conflicts, strengthen consensus when consistent and reliable, perform targeted repair when transcription errors occur, and handle semantic rewriting or complete errors with care, thereby significantly improving the accuracy and robustness of the fusion results. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the module composition of a document metadata fusion system based on dual-track feature decoupling and dynamic support transfer according to an embodiment of the present invention.

[0057] Figure 2This is a logic block diagram of the dual-track similarity calculation and difference extraction in the dual-track feature decoupling calculation module of this invention.

[0058] Figure 3 This is a flowchart of character-level similarity calculation for different field types in the dual-track feature decoupling calculation module of this invention.

[0059] Figure 4 This is a schematic diagram of the overall architecture and execution logic of the differentiated fusion decision module in an embodiment of the present invention;

[0060] Figure 5 This is a schematic diagram of the dynamic support transfer strategy executed for three different conflict types in the differentiated fusion decision module of this invention. Detailed Implementation

[0061] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0062] like Figure 1 As shown, this embodiment of the invention provides a document metadata fusion system based on dual-track decoupling and support transfer, comprising:

[0063] The data preprocessing and alignment module is used to align metadata from multiple sources based on unique identifiers, preprocess metadata from different fields, construct a multi-source candidate text pool for the same document, and count the number of times each exact text appears in different data sources as direct support.

[0064] The dual-track feature decoupling calculation module is used to adaptively select a character-level similarity calculation method based on the field type for texts of the same field from any two sources in the multi-source candidate text pool, and to calculate character-level similarity in parallel. and semantic similarity The dual-track similarity is calculated, and the difference between the dual-track similarities is extracted. As a feature decoupling variable.

[0065] The error type diagnosis module is used to receive the dual-track similarity and difference, and based on a preset threshold, to determine the data conflict between multi-source text pairs into four error types, namely consistent and reliable, transcription error, semantic rewriting and completely different, respectively, and to represent different levels of the degree of data conflict between text pairs through different error types.

[0066] The differentiated fusion decision module is used to perform differentiated directed transmission of support between multi-source text pairs based on the determined error type, and output the optimal standardized metadata in combination with the format purity assessment.

[0067] In a preferred embodiment, the data preprocessing and alignment module is responsible for standardizing and unifying the metadata of raw documents from multiple sources, forming a multi-source candidate text pool for each document. Specifically:

[0068] Alignment based on unique identifiers: Documents from multiple common data sources are aggregated using the Digital Object Identifier (DOI) as the unique identifier. For records without a DOI, fuzzy matching can be performed using a combination of title, author, and year; however, this embodiment uses the DOI as the primary key.

[0069] For example, the same document from nine common data sources was selected, including WoS, arXiv, OA, SemanticScholar, NSTL, PubMed, OpenAlex, OpenCitations, and Sci-Hub.

[0070] Field expansion and cleaning: Each field (title, abstract, author, keywords, etc.) is processed separately.

[0071] For example, taking the title as an example, the title text of each source is collected into the list title_long. The cleaning operation includes:

[0072] Remove HTML tags and escape characters (such as...) , );

[0073] Convert full-width characters to half-width characters;

[0074] Convert all fields to lowercase (for case-insensitive fields).

[0075] Remove leading and trailing spaces; replace multiple consecutive spaces with a single space.

[0076] Remove invisible control characters (such as \u200b).

[0077] For the author field, note that the original format may contain various delimiters (such as ";", ",", "and", etc.). During cleaning, these should be uniformly converted to the standard delimiter ";" for subsequent word segmentation.

[0078] Constructing direct support: The cleaned text is precisely matched and counted. The number of times each unique text appears is the direct support D for that text. For example, if "A study on deep learning" appears in WoS, arXiv, and PubMed, its direct support D = 3 (in actual calculation, the D value for each source is the total number of times the text appears minus 1, because it is not included in the support count; however, the frequency of the text is recorded uniformly before transmission, and will be adjusted during subsequent support calculations).

[0079] As a preferred embodiment, such as Figure 2 As shown, the dual-track feature decoupling calculation module is used to calculate the dual-track similarity and difference for each pair of sources (i,j) in the candidate text pool. Since the computational complexity is O(N²), where N is the number of sources (usually...), the calculation is efficient. Therefore, the computational cost is controllable. For characteristic orbitals A and B, specifically including:

[0080] 2.1. Feature Track A: Character-level Similarity

[0081] like Figure 3 As shown, different calculation methods are selected based on the field type:

[0082] For the title and abstract: normalized edit distance (Levenshtein distance) is used. Edit distance measures the minimum number of single-character edits required to transform one string into another. Let the length of string A be... The length of string B is Edit distance is Then the normalized similarity is:

[0083] ;

[0084] The value ranges from 0 to 1, where 1 indicates that the characters are exactly the same and 0 indicates that they are completely different. Edit distance is sensitive to character-level insertions, deletions, and replacements, making it suitable for catching OCR errors (such as replacing "l" with "1").

[0085] For authors and keywords: Jaccard similarity based on lexical units is used. First, the text is segmented into words.

[0086] Author field: The list of author names is obtained by splitting the names by a standard delimiter (such as ";"). Each name is treated as a word (spaces and punctuation in the names can be further cleaned up, for example, "Smith, J." becomes "smith j".

[0087] Keyword field: A list of keywords is obtained by separating them with commas or semicolons, and each keyword is treated as a word element (also processed by lowercase conversion, removal of spaces, etc.).

[0088] After obtaining the word sets T(A) and T(B), calculate the Jaccard similarity:

[0089] .

[0090] 2.2. Feature Track B: Semantic Similarity

[0091] A Sentence-BERT model (such as all-mpnet-base-v2) fine-tuned on an academic literature corpus is used to generate 768-dimensional embedding vectors for the texts. For texts A and B, the vectors are obtained by inputting them into the model. and Calculate the cosine similarity:

[0092] ;

[0093] To improve efficiency, vectors for all texts can be pre-calculated and cached, allowing for direct similarity calculation during online fusion. Semantic similarity is insensitive to synonyms and word order changes, and can capture consistency in deeper meanings.

[0094] 2.3 Feature decoupling difference :

[0095] ;

[0096] in, The value range is between [-1, 1]. In practice... This often suggests physical format corruption (such as OCR errors or garbled text) because the semantics are the same but the literal differences are significant.

[0097] In a preferred embodiment, the error type diagnosis module is used to receive dual-track similarity data and perform four-quadrant diagnosis by matching it with a preset threshold library, specifically:

[0098] When the semantic similarity is not less than the first threshold and the character similarity is not less than the second threshold, it is determined to be type I, which means that different data sources are normal representations of the same content.

[0099] When the semantic similarity is not less than the first threshold and the difference is not less than the third threshold, it is judged as type II, which means that the semantics are consistent but the physical format is broken.

[0100] When the semantic similarity is not less than the fourth threshold and less than the first threshold, and the character similarity is less than the fifth threshold, it is judged as Type III, which means that the abbreviation or rewritten version is manually written from different data sources.

[0101] When the semantic similarity is less than the fourth threshold and the character similarity is less than the fifth threshold, it is classified as type IV, indicating that the data source database is incorrect or contains completely irrelevant content.

[0102] Wherein, the first threshold is greater than the fourth threshold, and the second threshold is greater than the fifth threshold.

[0103] For example, as shown in Table 1, this embodiment sets the following typical values: , , , , For each pair of sources, determine the type. This involves considering the different fields... The numerical distribution may vary slightly depending on the calculation method, but through experimental calibration, the above thresholds are applicable to most cases and can also be configured separately for each field.

[0104] Table 1: Error Type Diagnosis Four-Quadrant Table .

[0105] If a pair of sources simultaneously satisfies multiple conditions, the first condition that is satisfied is selected according to the priority of type I→II→III→IV.

[0106] As a preferred embodiment, such as Figure 4 As shown, the differentiated fusion decision module performs support pass-through based on the type of each source pair and ultimately selects the optimal source, specifically including:

[0107] 4.1. Format Purity Assessment

[0108] In Type II, it is necessary to determine which source of text is "cleaner," and the purity index P is defined as follows:

[0109] ;

[0110] in: The number of normal alphanumeric characters (counted by the regular expression [a-zA-Z0-9]); Total number of characters; The desired length can be set as follows: for example, the title can be 50 characters, the abstract can be 200 characters, the author list can be 50 characters, and the keyword list can be 100 characters. and These are the weighting coefficients, and In this embodiment, , .

[0111] Among them, a higher P-value indicates a cleaner text. When the P-values ​​of two sources differ significantly (e.g., ...), the text is cleaner. The source with high purity was identified as the absorber.

[0112] 4.2. Support Transfer Algorithm

[0113] Initialize the total support for each source (Direct support). Traverse all source pairs. :

[0114] like Figure 5 As shown, if the type is I, then bidirectional passing is performed:

[0115] ;

[0116] Indicate source Sources of increased total support direct support ,source Sources of increased total support direct support .

[0117] If the type is II, and the purity is Then it is a one-way transmission:

[0118] ;

[0119] Conversely if ,but absorb ;

[0120] Finally, the total support for the absorber increases the direct support for the low-purity source, while the total support for the low-purity source remains unchanged.

[0121] If the type is III or IV, no transmission is performed, and only the direct support of each type is retained.

[0122] In this case, the same source pair is processed only once to avoid duplicate accumulation.

[0123] 4.3. Score Calculation and Final Selection

[0124] After all passes, calculate the normalized score for each source:

[0125] ;

[0126] Where N is the total number of data sources for this field. If N=1, (This indicates that a consensus cannot be reached to determine the outcome.)

[0127] Ultimately, the system selects the source with the highest score as the final data source for that field. If there are multiple sources with the highest score, the first one is selected according to the field's preset priority order (e.g., title priority: WoS > arXiv > OA > Semantic Scholar > NSTL > PubMed > OpenAlex > OpenCitations > SciHub), and the metadata of that source is output as standardized metadata.

[0128] Preferably, this system employs differentiated design for character-level similarity calculation based on the text characteristics of different fields:

[0129] Title and Abstract: Character-level similarity is calculated using normalized edit distance to keenly detect errors in individual characters (such as OCR errors and encoding issues), which is crucial for the physical cleanliness of long texts.

[0130] For example, the edit distance between the titles "Deep Learning Models" and "Deep Learning Mode1s" (where the last letter 'l' is misidentified as the number 1) is 1, and the normalized similarity is approximately 0.95, which is still relatively high. However, the difference in semantic similarity (close to 1) is significant. A small edit distance may not be enough to trigger Type II, but if there are many errors (such as multiple character errors), the edit distance will decrease significantly, resulting in a larger edit distance. This triggers the error correction mechanism.

[0131] Authors and keywords: Character-level similarity is calculated using lexical-based Jaccard similarity to better handle lexical order changes and the inclusion relationship between abbreviations and full names.

[0132] For example, the author lists "Smith, J.; Lee, K." and "Smith, John; Lee, Kevin" may have low Jaccard similarity (because J vs John, K vs Kevin), but high semantic similarity (because SBERT can understand the equivalence between abbreviations and full names), thus the difference... If the value is large, triggering type II, after purity evaluation, the full name version (usually longer and more complete) will be selected as the absorber, and the full name version will be output in the end. This achieves automatic selection of the author field without the need to design complex rules separately.

[0133] This invention also provides a document metadata fusion method based on dual-track decoupling and support transfer, comprising the following steps:

[0134] Obtain metadata from multiple sources, align them based on unique identifiers, preprocess the metadata of different fields, construct a multi-source candidate text pool for the same document, and count the number of times each exact text appears in different data sources as direct support.

[0135] For any two texts of the same field from any two sources in the multi-source candidate text pool, a character-level similarity calculation method is adaptively selected according to the field type. A dual-track similarity, which includes character-level similarity and semantic-level similarity, is calculated in parallel, and the difference between the dual-track similarities is extracted as a feature decoupling variable.

[0136] Based on the dual-track similarity and difference, and according to the preset threshold, the data conflict between multi-source text pairs is determined into four error types, which represent consistent credibility, transcription error, semantic rewriting and completely different, respectively. Different error types represent different levels of the degree of data conflict between text pairs.

[0137] Based on the identified error type, the support between multi-source text pairs is differentially propagated in a directed manner, and combined with the format purity assessment, the optimal standardized metadata is output.

[0138] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A document metadata fusion system based on dual-track decoupling and support transfer, characterized in that, include: The data preprocessing and alignment module is used to align metadata from multiple sources based on unique identifiers, preprocess metadata from different fields, construct a multi-source candidate text pool for the same document, and count the number of times each exact text appears in different data sources and record it as direct support. The dual-track feature decoupling calculation module is used to adaptively select a character-level similarity calculation method based on the field type for the same field text from any two sources in the multi-source candidate text pool, calculate the dual-track similarity including character-level similarity and semantic-level similarity in parallel, and extract the difference between the dual-track similarities as a feature decoupling variable. The error type diagnosis module is used to receive the dual-track similarity and the difference between the dual-track similarity, and based on a preset threshold, to determine the data conflict between multi-source text pairs as different error types, and to represent different levels of the degree of data conflict between text pairs through different error types; The differentiated fusion decision module is used to perform differentiated directed transmission of support between text pairs from different sources based on the determined error type, and output the optimal standardized metadata in combination with the format purity assessment.

2. The system according to claim 1, characterized in that, The data preprocessing and alignment module includes: aggregating the same document records from multiple data sources based on the document's digital object identifier; cleaning the original text of each field, including removing HTML tags, escape characters, invisible characters, and uniformly converting them to lowercase and standardizing whitespace; expanding the cleaned text according to its source to form a multi-source candidate text set for each field; and counting the number of times each exact text appears in different data sources, and recording the number of occurrences as direct support.

3. The system according to claim 1, characterized in that, The dual-track feature decoupling calculation module adaptively selects the character-level similarity calculation method according to the field type, specifically including: For the title and abstract fields, normalized edit distance is used to calculate character-level similarity; For the author and keyword fields, character-level similarity is calculated using lexical-based Jaccard similarity.

4. The system according to claim 1, characterized in that, In the dual-track feature decoupling calculation module, the semantic-level similarity is calculated by extracting high-dimensional embedding vectors of the text using a pre-trained deep learning language model and calculating cosine similarity. The deep learning language model includes Sentence-BERT and its variants.

5. The system according to claim 1, characterized in that, In the error type diagnosis module, the step of determining data conflicts between multi-source text pairs into different error types based on preset thresholds specifically involves: When the semantic similarity is not less than the first threshold and the character similarity is not less than the second threshold, it is judged as type one, which means that different data sources are normal representations of the same content. When the semantic similarity is not less than the first threshold and the difference is not less than the third threshold, it is judged as type two, which means that the semantics are consistent but the physical format is broken; When the semantic similarity is not less than the fourth threshold and less than the first threshold, and the character similarity is less than the fifth threshold, it is judged as type three, which means that the abbreviation or rewritten version is manually written from different data sources. When the semantic similarity is less than the fourth threshold and the character similarity is less than the fifth threshold, it is judged as type four, indicating that the data source database is incorrect or contains completely irrelevant content; Wherein, the first threshold is greater than the fourth threshold, and the second threshold is greater than the fifth threshold.

6. The system according to claim 5, characterized in that, The first threshold ranges from 0.8 to 0.9, the second threshold ranges from 0.7 to 0.8, the third threshold ranges from 0.25 to 0.35, the fourth threshold ranges from 0.45 to 0.55, and the fifth threshold ranges from 0.35 to 0.

45.

7. The system according to claim 5, characterized in that, The differentiated fusion decision module executes a differentiated support delivery strategy based on the error type, including: For text pairs classified as type 1, a two-way similarity support transfer is performed, meaning the total support of source i is increased by the direct support of source j. The total support of source j increases the direct support of source i. ; For text pairs determined to be of type two, an asymmetric one-way vote absorption mechanism is triggered, including: evaluating the format purity of the two sources, selecting the source with higher purity as the absorber, and unilaterally transferring the direct support of the source with lower purity to the absorber. The total support of the absorber is its own direct support plus the direct support of the source with lower purity, while the direct support of the source with lower purity remains unchanged. For text pairs classified as type 3 and type 4, the transmission of similarity support is blocked, and only their respective direct support is retained.

8. The system according to claim 7, characterized in that, The format purity is obtained by weighted summation of character purity and length reasonableness. Character purity is the proportion of normal alphanumeric characters to the total number of characters, and length reasonableness is the smaller of the ratio of the total number of characters to the expected length of the field and 1. The sum of the weight coefficients of the two is 1.

9. The system according to claim 8, characterized in that, After the support of all text pairs is transferred, the normalized score of each source is calculated. Specifically, the total support of each source is divided by the difference between the total number of sources with data in the field and 1, and the result is the normalized score of that source. When there is only one source with data in the field, the normalized score is set to null. Finally, the source with the highest score is selected as the final data source for the field. If there are multiple sources with the same score, they are selected according to the preset field priority order.

10. A document metadata fusion method based on dual-track decoupling and support transfer, characterized in that, Includes the following steps: Obtain metadata from multiple sources, align it based on unique identifiers, preprocess the metadata of different fields, construct a multi-source candidate text pool for the same document, count the number of times each exact text appears in different data sources and record it as direct support. For any two texts of the same field from any two sources in the multi-source candidate text pool, the character-level similarity calculation method is adaptively selected according to the field type, and the dual-track similarity including character-level similarity and semantic-level similarity is calculated in parallel, and the difference between the dual-track similarity is extracted as a feature decoupling variable. Based on the dual-track similarity and difference, and according to the preset threshold, the data conflict between multi-source text pairs is determined into four error types, which represent consistent credibility, transcription error, semantic rewriting and completely different, respectively. Different error types represent different levels of the degree of data conflict between text pairs. Based on the identified error type, the support between multi-source text pairs is differentially propagated in a directed manner, and combined with the format purity assessment, the optimal standardized metadata is output.

Images