A metadata-based multi-source heterogeneous data integration method and system

Abstract

This invention discloses a method and system for integrating multi-source heterogeneous data based on metadata, relating to the field of data management technology. The method accesses data sources through an interface protocol called by a metadata acquisition engine, standardizes the extracted field metadata, and aggregates it to form a metadata information table. The syntactic-structure coupling degree Ygo is calculated. ij Perform a syntactic structure consistency assessment, and if the syntactic structure is compatible, perform semantic parsing to convert field names and description text into semantic feature vectors, and couple them with the syntactic construction degree Ygo. ij Perform fitting calculation of semantic coupling mapping index Yyo ij The data is then matrix-integrated to form a semantic mapping matrix and written into a time-series database. The decision index Jcj is then integrated through data integration. i A fusion confidence assessment is performed. This method can identify the structural consistency and semantic association mapping of multi-source data based on the metadata level, forming a unified data access interface table and recording the mapping index.

Description

Technical Field

[0001] This invention relates to the field of data management technology, specifically to a method and system for integrating multi-source heterogeneous data based on metadata. Background Technology

[0002] With the widespread construction of information systems, various business platforms have accumulated a large amount of data resources with diverse structures over long-term operation, including relational databases, file storage systems, and real-time streaming systems. Driven by the needs of cross-system business collaboration, data sharing, and data reuse, data integration has become a crucial aspect of information system construction. However, different data sources differ in data models, field naming conventions, unit dimension expressions, and hierarchical organizational methods, resulting in typical multi-source heterogeneous data characteristics that directly affect cross-system data alignment and unified access. To address the issues of difference identification and alignment during the integration process of multi-source heterogeneous data, the industry has gradually introduced metadata as a fundamental information carrier describing data structure, attributes, semantics, and version status. Data fusion driven by metadata has become a commonly adopted technical approach.

[0003] While metadata-based multi-source heterogeneous data integration methods can describe the structural attributes of data, in practice they often rely on manually established rules to create field mapping relationships, lacking automated analysis tools that address syntactic structural features and semantic descriptions. Existing metadata systems tend to focus on field definitions and classifications, while paying insufficient attention to information such as field hierarchical paths, differences in units of measurement, linguistic expressions of field meanings, and field version change trends. This is particularly true when data sources come from different industries, vendors, or time versions, where the lack of a unified structured description and interrelationships in metadata content makes it difficult to accurately determine the correspondence between fields across systems, affecting the coherence of data integration and the sufficiency of judgment criteria. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for integrating multi-source heterogeneous data based on metadata, thus solving the problems mentioned in the background technology.

[0005] To achieve the above objectives, the present invention provides a method for integrating multi-source heterogeneous data based on metadata, comprising the following steps:

[0006] S1. Access the data source by calling the interface protocol through the metadata collection engine, extract and identify field metadata, perform standardization processing, and summarize to form a metadata information table.

[0007] S2. Automatically analyze and deconstruct the metadata information tables of each data source and extract syntactic feature elements. Perform structural consistency analysis on the data source fields based on the syntactic feature elements, and evaluate the syntactic structure consistency based on the analysis results.

[0008] S3. When the syntax structure consistency assessment is that the syntax structure is compatible, extract the semantic feature elements from the metadata information table, process them, extract the semantic feature vectors to form a semantic vector set, and fit it with the structure consistency analysis results to perform semantic association analysis on the data source fields.

[0009] S4. Extract the version change rate Δv from the metadata information table, fit it with the semantic association analysis results to perform fusion confidence analysis, and evaluate the fusion confidence based on the analysis results.

[0010] Preferably, S1 includes S11 and S12;

[0011] S11. Establish access channels with various data sources through the metadata collection engine, call the corresponding interface protocol to access the data source, and automatically select the access method according to the data source type.

[0012] The data sources include database-type data sources, file-type data sources, and streaming data sources;

[0013] The metadata acquisition engine automatically reads information tables and field definitions from database-type data sources using the JDBC database driver;

[0014] The metadata acquisition engine parses the file header definition and data description block of file-type data sources through the file header parsing interface;

[0015] The metadata acquisition engine parses the real-time data transmission protocol header of the streaming data source through a message subscription interface;

[0016] S12. After completing the data source access, the metadata attribute parsing program is automatically executed to extract and identify field metadata from the data source. During the extraction process, the field path parsing algorithm is automatically executed to complete the logical path and perform hierarchical mapping for fields with different structures.

[0017] Preferably, S1 further includes S13;

[0018] S13. Call the standardization processing engine to standardize the field metadata and summarize it to form a metadata information table.

[0019] The standardization process includes naming standardization, unit conversion standardization, hierarchical structure correction, and version indexing.

[0020] The naming standardization is used to unify the character set for field naming and remove special symbols to generate unified field identifiers.

[0021] Unit conversion standardization is used to convert different units of measurement into standard dimensional form based on the UMDL dimensional matrix.

[0022] Hierarchical structure correction is used to correct the hierarchical numbering of fields to ensure that the hierarchical depth of fields from different sources remains consistent.

[0023] Version indexing is used to create a time-based index table for the rate of change of a field version;

[0024] The metadata information table includes field name F, field description text D, unit dimension code value U, field hierarchy structure L, and field version change rate Δv.

[0025] Preferably, S2 includes S21 and S22;

[0026] S21. Call the metadata parsing engine to automatically analyze and deconstruct the metadata information tables of each data source, extract the syntax feature elements including the field hierarchy structure L and the unit dimension encoding value U, and represent the hierarchy path of each field in the form of a node chain according to the UMDL structure identifier rules during the extraction process.

[0027] S22. Perform structural consistency analysis on any two data source fields based on syntactic feature elements, and construct the syntactic construction coupling degree Ygo. ij This indicates the degree of matching between different field structures in terms of morphology, position, and combination rules, reflecting the degree of matching between field i and field j, specifically: In the formula, Nc(L) i L j ) represents the number of identical syntax nodes Nc and Nt(L) that field i and field j have in the field hierarchy. i L j The sum of the hierarchical nodes of fields i and j in the field hierarchy is Nt, U. i and U j These represent the unit dimension codes for fields i and j, respectively.

[0028] Preferably, S2 further includes S23;

[0029] S23. Extract several syntax construction coupling degrees Ygo defined by the industry standard library. ijAccording to the industry standards for information technology data management, data structure elements are considered structurally compatible when the hierarchical matching rate reaches 80% or higher; if it is lower than 50%, structural reconstruction should be performed. Using the percentile method, the Q50th percentile value is set as the syntax structure reconstruction threshold YC, and the Q80th percentile value is set as the syntax structure compatibility threshold YJ. This is then combined with the real-time acquired syntax construction coupling degree Ygo. ij A syntactic consistency assessment is conducted, and the specific assessment scheme is as follows;

[0030] When the syntactic construction coupling degree Ygo ij When the threshold YC for syntax structure reconstruction is reached, it indicates that the syntax structure is incompatible. At this time, the UMDL reconstruction process is triggered to re-parse the field hierarchy structure, update the node tree structure, and input the updated data into S1 for iterative analysis.

[0031] When the syntactic structure reconstruction threshold YC ≤ syntactic construction coupling Ygo ij When the value is less than or equal to the syntax structure compatibility threshold YJ, it indicates that there are local differences in the syntax structure. At this time, the structure correction technique is called to adjust the local nodes, and the units are converted and then synchronously updated to the UMDL mapping table.

[0032] When the syntactic construction coupling degree Ygo ij When the syntax structure compatibility threshold YJ is reached, it indicates that the syntax structure is compatible. At this time, the structure pass status is recorded, and semantic mapping analysis is triggered.

[0033] Preferably, S3 includes S31;

[0034] S31. When the grammatical structure consistency assessment is grammatically compatible, extract the semantic feature elements from the metadata information table of each data source. The semantic feature elements include the field name F and the field description text D. Call the semantic parsing engine to perform natural language segmentation and semantic embedding processing to transform unstructured language information into a measurable semantic expression. Then, extract semantic feature vectors based on the text comparison algorithm of edit distance and the word vector model to form a semantic vector set.

[0035] The semantic vector set includes a semantic vector set F for field names. i ∈{v(F1), v(F2), ..., v(F n )} and the set of semantic vectors describing the fields D i ∈{v(D1), v(D2), ..., v(D n )}.

[0036] Preferably, S3 further includes S32 and S33;

[0037] S32, Coupling the semantic vector set with the syntactic construction degree Ygo ijThe algorithm performs fitting and calls the semantic similarity calculation engine to perform semantic association analysis on fields from different data sources, and calculates the semantic coupling mapping index Yyo. ij This indicates the degree of semantic correspondence in the mapping space, reflecting the comprehensive correlation between the meaning and syntax of fields i and j, specifically: In the formula, Sim represents the similarity function, F i and F j D represents the field name of the i-th segment and the j-th segment in the field, respectively. i and D j Sim(F) represents the semantic description of the i-th segment and the j-th segment in the field, respectively. i F j Sim(D) represents the semantic similarity of the naming language between field i and field j. i D j ) represents the semantic similarity between the annotation text of field i and field j, and ln represents the logarithmic function;

[0038] S33. The semantic coupling mapping exponent Yyo of all field pairs (i, j) ij Perform matrix integration to form a semantic mapping matrix Y, Y=[Yyo ij ] n×n Where n is the number of fields involved in semantic matching, the semantic mapping matrix Y is then standardized and denoised to generate a structured semantic mapping table, which is written to the time series database and a version number record is generated.

[0039] Preferably, S4 includes S41;

[0040] S41. Extract the version change rate Δv from the metadata information table and map it to the semantic coupling index Yyo. ij The data integration decision index Jcj was constructed by performing a fitting test, analyzing the fusion confidence of target and candidate fields from different data sources, and then constructing the data integration decision index. i This is used to measure the optimal semantic mapping relationship of target field i among multiple candidate fields, specifically: In the formula, max represents the maximum value selection sign, e represents the exponential function, and Δv j This indicates the semantic definition update rate of field j.

[0041] Preferably, S4 further includes S42;

[0042] S42. Extract several historical data integration decision indices Jcj defined by industry standard libraries. iFurthermore, in accordance with the semantic matching industry standard, entities are considered semantically equivalent when the semantic similarity score is greater than 0.85. The Q85th percentile value is set as the semantic fusion confidence threshold YZ using the percentile method, and then integrated with the real-time acquired data to form the decision index Jcj. i A fusion confidence assessment was conducted, and the specific assessment plan is as follows;

[0043] When the data integration decision index Jcj i When the semantic fusion confidence threshold YZ is less than the semantic fusion confidence threshold, it indicates that the semantic mapping is unreliable. At this time, the semantic model reconstruction process is triggered, the current field meta-information is recorded in real time, and iterative analysis is performed through S1.

[0044] When the data integration decision index Jcj i When the semantic fusion confidence threshold YZ is greater than or equal to the threshold value, it indicates that the semantic mapping is reliable. At this time, the current mapping relationship is locked, field i and target field j are written into the unified access interface table, and the mapping index is recorded.

[0045] A multi-source heterogeneous data integration system based on metadata includes an information acquisition module, a metadata structure analysis module, a semantic mapping module, and a fusion decision module;

[0046] The information collection module is used to access the data source through the interface protocol of the metadata collection engine, extract and identify field metadata, perform standardization processing, and summarize to form a metadata information table.

[0047] The metadata structure analysis module is used to automatically analyze and deconstruct the metadata information tables of each data source and extract syntactic feature elements. Based on the syntactic feature elements, it performs structural consistency analysis on the data source fields and evaluates the syntactic structure consistency based on the analysis results.

[0048] The semantic mapping module is used to extract semantic feature elements from the metadata information table when the syntactic structure consistency assessment is syntactic structure compatible, process them to extract semantic feature vectors to form a semantic vector set, and fit them with the structural consistency analysis results to perform semantic association analysis on the data source fields.

[0049] The fusion decision module is used to extract the field version change rate Δv from the metadata information table, fit it with the semantic association analysis results to perform fusion confidence analysis, and evaluate the fusion confidence based on the analysis results.

[0050] This invention provides a method and system for integrating multi-source heterogeneous data based on metadata. It has the following beneficial effects:

[0051] (1) This method utilizes a metadata acquisition engine to automatically select the access method based on the data source type. It performs structured and unstructured field metadata extraction on data sources of database type, file type, and streaming type. Through a standardization processing engine, it completes naming standardization, unit conversion standardization, hierarchical structure correction, and version indexing, so that data fields from different sources can be expressed in a unified field identifier, unified physical or logical unit encoding, and unified hierarchical structure. By forming a metadata information table, subsequent analysis does not need to concern itself with the differences in the internal storage format of the data source, ensuring that subsequent structural matching and semantic matching have a standard reference object for comparison, so that the data has comparability, parsability, and traceability before entering the fusion analysis.

[0052] (2) After achieving consistency in the basic metadata representation, this method constructs the syntactic construction coupling degree Ygo by extracting the hierarchical structure L of the fields and the unit dimension encoding value U to form syntactic feature elements. ij Utilizing the coupling degree of syntactic construction Ygo ij This study quantitatively describes the structural mapping relationships between different fields in terms of hierarchical position, number of structural nodes, and dimensional values. It also distinguishes between three states—structural incompatibility, local differences, and structural compatibility—using industry percentile criteria, providing clear judgment criteria for structural comparison. Under the premise of structural compatibility, natural language parsing and semantic embedding are further performed on field names F and description text D to generate a set of semantic vectors. Finally, a semantic coupling mapping index Yyo is constructed based on a semantic similarity calculation engine. ij This process ensures that the meaning and association of fields are determined by both structural and semantic relationships, rather than by manual interpretation. It makes the logical basis for field associations clearer, independent of empirical judgments or manual rules.

[0053] (3) After obtaining the semantic mapping relationship, this method incorporates the field version change rate Δv into the analysis and links it with the semantic coupling mapping index Yyo. ij To construct the data integration decision index Jcj through fitting. i Decision-making index Jcj through data integration i The semantic stability of candidate fields is measured, ensuring that the final matching of fields depends not only on semantic similarity but also on whether the field maintains consistent definition over a long period. The decision index Jcj is determined based on industry semantic fusion thresholds. i The system determines whether a field's semantic interpretation and structural path are consistently consistent. If so, the field pair is written to the unified access interface table, and the corresponding mapping index is recorded. This achieves a traceable one-to-one or one-to-many mapping relationship between fields across data sources. This approach allows data integration to go beyond mere format matching, forming a unified cross-source data representation system that is maintainable long-term, dynamically traceable, and expandable. Attached Figure Description

[0054] Figure 1 This is a schematic diagram illustrating the steps of a multi-source heterogeneous data integration method based on metadata according to the present invention;

[0055] Figure 2 This is a schematic diagram of the process of a multi-source heterogeneous data integration system based on metadata according to the present invention;

[0056] Figure 3 This is a logical block diagram of the steps of a multi-source heterogeneous data integration method based on metadata according to the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Example 1

[0059] Please see Figure 1 This invention provides a method for integrating multi-source heterogeneous data based on metadata. To achieve the above objectives, this invention employs the following technical solution, comprising the following steps:

[0060] S1. Access the data source by calling the interface protocol through the metadata collection engine, extract and identify field metadata, perform standardization processing, and summarize to form a metadata information table.

[0061] S2. Automatically analyze and deconstruct the metadata information tables of each data source and extract syntactic feature elements. Perform structural consistency analysis on the data source fields based on the syntactic feature elements, and evaluate the syntactic structure consistency based on the analysis results.

[0062] S3. When the syntax structure consistency assessment is that the syntax structure is compatible, extract the semantic feature elements from the metadata information table, process them, extract the semantic feature vectors to form a semantic vector set, and fit it with the structure consistency analysis results to perform semantic association analysis on the data source fields.

[0063] S4. Extract the version change rate Δv from the metadata information table, fit it with the semantic association analysis results to perform fusion confidence analysis, and evaluate the fusion confidence based on the analysis results.

[0064] In this embodiment, a metadata acquisition engine accesses different types of data sources and calls corresponding interface protocols to obtain field metadata information. After standardization, this information is aggregated into a unified-format metadata information table. Subsequently, the metadata information table undergoes structural parsing and syntactic structure recognition to extract comparable syntactic feature elements. Based on differences in structural hierarchy and unit units, structural consistency analysis is performed on the data source fields, and syntactic structural consistency is evaluated based on the analysis results, providing a structural foundation for semantic association. Under the premise of structural compatibility, field names and descriptive text are further extracted to form a set of semantic feature vectors. These vectors are then fitted with the structural consistency analysis results to perform semantic association analysis on the data source fields, achieving semantic-level field mapping relationship analysis. By introducing the field version change rate Δv as a dynamic factor, the rate of field definition evolution over time is incorporated into the fusion confidence judgment process. This is then fitted with the semantic association analysis results to perform fusion confidence analysis, establishing a comprehensive decision-making mechanism oriented towards semantic mapping stability. The fusion confidence analysis not only considers semantic similarity but also assesses the stability of field definitions in actual business operations, helping to select fields that maintain long-term semantic consistency as the final mapping objects. The entire process implements a comprehensive analysis and judgment mechanism across three dimensions: field structure, field semantics, and field timeliness. It achieves orderly connections between the four stages of field mapping: filtering, fitting, confirmation, and calibration, providing a data foundation for the construction of a unified access interface and forming a complete pre-processing system for heterogeneous data integration. Starting from field metadata, this method avoids typical problems such as ambiguous names, structural mismatches, semantic deviations, and version mismatches through a joint judgment mechanism based on "structure, semantics, and version." The introduction of the fusion confidence analysis stage solves the problem of unstable mapping relationships caused by frequent changes in field definitions, avoiding the phenomenon of repeated adjustments to the structure mapping table during long-term business operations. This makes the data integration relationship more stable, traceable, and iteratively maintainable, facilitating the continuous operation of subsequent data fusion, interface calls, and semantic unification.

[0065] Example 2

[0066] Please refer to Figure 3 Specifically: S1 includes S11 and S12;

[0067] S11. Establish access channels with various data sources through the metadata collection engine, call the corresponding interface protocol to access the data source, and automatically select the access method according to the data source type.

[0068] The data sources include database-type data sources, file-type data sources, and streaming data sources;

[0069] The metadata acquisition engine automatically reads information tables and field definitions from database-type data sources using the JDBC database driver;

[0070] The metadata acquisition engine parses the file header definition and data description block of file-type data sources through the file header parsing interface;

[0071] The metadata acquisition engine parses the real-time data transmission protocol header of the streaming data source through a message subscription interface;

[0072] S12. After completing the data source access, the metadata attribute parsing program is automatically executed to extract and identify field metadata from the data source. During the extraction process, the field path parsing algorithm is automatically executed to complete the logical path and perform hierarchical mapping for fields with different structures.

[0073] S1 also includes S13;

[0074] S13. Call the standardization processing engine to standardize the field metadata and summarize it to form a metadata information table.

[0075] The standardization process includes naming standardization, unit conversion standardization, hierarchical structure correction, and version indexing.

[0076] The naming standardization is used to unify the character set for field naming and remove special symbols to generate unified field identifiers.

[0077] Unit conversion standardization is used to convert different units of measurement into standard dimensional form based on the UMDL dimensional matrix.

[0078] Hierarchical structure correction is used to correct the hierarchical numbering of fields to ensure that the hierarchical depth of fields from different sources remains consistent.

[0079] Version indexing is used to create a time-based index table for the rate of change of a field version;

[0080] The metadata information table includes field name F, field description text D, unit dimension code value U, field hierarchy structure L, and field version change rate Δv;

[0081] Field name F is used to identify the naming information of the data item;

[0082] The field description text D is used to reflect the semantic description of the field;

[0083] The unit dimension code value U is used to identify the physical and logical dimensions corresponding to the field;

[0084] The field hierarchy structure L is used to describe the hierarchical relationship of fields in the original system;

[0085] The field version change rate Δv is used to record the current version information and change status of the field.

[0086] In this embodiment, an access channel is established based on the data source type through a metadata acquisition engine. The engine calls the database driver JDBC, file header parsing interface, and streaming data transmission protocol interface to extract structured field information from database-type data sources, file-type data sources, and streaming data sources. Based on this, a field path parsing algorithm is executed to complete the logical path of fields and perform hierarchical mapping, ensuring that fields with different structures have a consistent path positioning basis. Subsequently, a standardization processing engine performs naming conventions, unit conversions, structural hierarchy corrections, and version index construction on the extracted field metadata. Each field is uniformly encoded into five elements: field name F, field description text D, unit dimension code value U, field hierarchical structure L, and field version change rate Δv. This ultimately forms a metadata information table with a unified format, complete structure, and quantifiable traceability. This implementation achieves synchronous unification of heterogeneous data source fields in terms of syntax structure, unit expression, and temporal evolution characteristics. It provides a unified data expression entry point for subsequent data structure alignment, semantic mapping, and fusion confidence assessment, avoiding the docking errors and data processing interruptions caused by inconsistent field naming, incompatible unit systems, or missing field structures in traditional solutions.

[0087] Example 3

[0088] Please refer to Figure 3 Specifically: S2 includes S21 and S22;

[0089] S21. Call the metadata parsing engine to automatically analyze and deconstruct the metadata information tables of each data source, extract the syntax feature elements including the field hierarchy structure L and the unit dimension encoding value U, and represent the hierarchy path of each field in the form of a node chain according to the UMDL structure identifier rules during the extraction process.

[0090] S22. Perform structural consistency analysis on any two data source fields based on syntactic feature elements, and construct the syntactic construction coupling degree Ygo. ij This indicates the degree of matching between different field structures in terms of morphology, position, and combination rules, reflecting the degree of matching between field i and field j, specifically: In the formula, Nc(L) i L j ) represents the number of identical syntax nodes Nc and Nt(L) that field i and field j have in the field hierarchy. i L j The sum of the hierarchical nodes of fields i and j in the field hierarchy is Nt, U. i and U j These represent the unit dimension codes for fields i and j, respectively. This is used to quantify the proportion of structural similarity between field i and field j, reflecting the matching degree between field i and field j in terms of hierarchical structure. The absolute value that reflects the difference between units.

[0091] S2 also includes S23;

[0092] S23. Extract several syntax construction coupling degrees Ygo defined by the industry standard library. ij According to the industry standards for information technology data management, data structure elements are considered structurally compatible when the hierarchical matching rate reaches 80% or higher; if it is lower than 50%, structural reconstruction should be performed. Using the percentile method, the Q50th percentile value is set as the syntax structure reconstruction threshold YC, and the Q80th percentile value is set as the syntax structure compatibility threshold YJ. This is then combined with the real-time acquired syntax construction coupling degree Ygo. ij A syntactic consistency assessment is conducted, and the specific assessment scheme is as follows;

[0093] When the syntactic construction coupling degree Ygo ij When the threshold YC for syntax structure reconstruction is reached, it indicates that the syntax structure is incompatible. At this time, the UMDL reconstruction process is triggered to re-parse the field hierarchy structure, update the node tree structure, and input the updated data into S1 for iterative analysis.

[0094] When the syntactic structure reconstruction threshold YC ≤ syntactic construction coupling Ygo ij When the value is less than or equal to the syntax structure compatibility threshold YJ, it indicates that there are local differences in the syntax structure. At this time, the structure correction technique is called to adjust the local nodes, and the units are converted and then synchronously updated to the UMDL mapping table.

[0095] When the syntactic construction coupling degree Ygo ij When the syntax structure compatibility threshold YJ is reached, it indicates that the syntax structure is compatible. At this time, the structure pass status is recorded, and semantic mapping analysis is triggered.

[0096] In this embodiment, S21 calls the metadata parsing engine to extract syntactic feature elements including the field hierarchy structure L and the unit dimension encoding value U, and converts the field structure into a node chain representation according to the UMDL structure identifier rules; S22 calculates the syntactic configuration coupling degree Ygo between any two fields based on the syntactic feature elements. ij This formula reflects the structural similarity of two sets A and B at the levels of morphology, position, and combination rules. Its mathematical foundation stems from the Jaccard similarity coefficient and structural similarity function in set theory and similarity analysis. In set theory, the similarity between two sets A and B can be expressed as... The ratio of the common part to the total part is used to characterize the degree of overlap between two sets. In structured data analysis, to measure the degree of matching between different structures, the ratio of the number of matching nodes to the total number of nodes is usually used. Extending this to field hierarchy structures, we can obtain... In physical measurement systems, to prevent excessive dimensional differences from distorting similarity calculations, a decay term is often introduced to account for the differences between variables. The function takes the value 1 as Δ approaches 0, indicating that the dimensions are completely identical. As Δ increases, f(Δ) gradually decreases, indicating that the comparability between structures decreases. Extending the unit difference to f(Δ) yields... By multiplying the syntactic structural similarity by a dimensional difference compensation factor, joint coupling of structural and unit constraints is achieved, yielding the syntactic construction coupling degree Ygo. ij . In this formula It is a dimensionless ratio. The difference is encoded as a unit dimension, and its dimension itself has been standardized to a pure number form. Therefore, the formula as a whole is a dimensionless operation, and there is no dimension influence. S23, combining the structural compatibility judgment criteria in the information technology data management industry standard, uses the percentile method to set the syntactic structure reconstruction threshold YC and the syntactic structure compatibility threshold YJ for the Q50 and Q80 percentile values ​​respectively, to achieve a quantitative and graded judgment of field structure relationships. When the syntactic configuration coupling degree Ygo... ij When the syntax structure reconstruction threshold YC is less than the threshold value, the structure reconstruction process is triggered and the reconstruction result is sent back to the metadata collection stage for iteration; when the syntax structure reconstruction threshold YC is less than or equal to the syntax construction coupling degree Ygo, the process is initiated. ij When the syntax structure compatibility threshold YJ is less than or equal to the grammar structure compatibility threshold, local node correction and unit conversion operations are performed; when the syntax construction coupling degree Ygo is less than or equal to the grammar structure compatibility threshold, local node correction and unit conversion operations are performed. ij When the syntax structure compatibility threshold YJ is reached, the system is directly determined to be structurally compatible and proceeds to the semantic mapping stage. This method enables streamlined and quantifiable control over the field structure matching process, reduces data integration failures caused by inconsistent field structures, ensures the uniformity of structural standards before data integration, and lays a clear structural mapping foundation for semantic layer processing.

[0097] Example 4

[0098] Please refer to Figure 3 Specifically: S3 includes S31;

[0099] S31. When the grammatical structure consistency assessment is grammatically compatible, extract the semantic feature elements from the metadata information table of each data source. The semantic feature elements include the field name F and the field description text D. Call the semantic parsing engine to perform natural language segmentation and semantic embedding processing to transform unstructured language information into a measurable semantic expression. Then, extract semantic feature vectors based on the text comparison algorithm of edit distance and the word vector model to form a semantic vector set.

[0100] The semantic vector set includes a semantic vector set F for field names. i ∈{v(F1), v(F2), ..., v(F n )} and the set of semantic vectors describing the fields D i ∈{v(D1), v(D2), ..., v(D n )}.

[0101] S3 further includes S32 and S33;

[0102] S32, Coupling the semantic vector set with the syntactic construction degree Ygo ij The algorithm performs fitting and calls the semantic similarity calculation engine to perform semantic association analysis on fields from different data sources, and calculates the semantic coupling mapping index Yyo. ij This indicates the degree of semantic correspondence in the mapping space, reflecting the comprehensive correlation between the meaning and syntax of fields i and j, specifically: In the formula, Sim represents the similarity function, F i and F j D represents the field name of the i-th segment and the j-th segment in the field, respectively. i and D j Sim(F) represents the semantic description of the i-th segment and the j-th segment in the field, respectively. i F j Sim(D) represents the semantic similarity of the naming language between field i and field j. i D j The expression ) represents the semantic similarity between the annotation text of field i and field j, and ln represents the logarithmic function. This represents the semantic-level two-channel average similarity, used to balance the semantic contributions of names and descriptions. This represents a semantic enhancement factor, used to enhance the semantic score of fields with high structural similarity.

[0103] S33. The semantic coupling mapping exponent Yyo of all field pairs (i, j) ij Perform matrix integration to form a semantic mapping matrix Y, Y=[Yyo ij ] n×nWhere n is the number of fields involved in semantic matching, the semantic mapping matrix Y is then standardized and denoised to generate a structured semantic mapping table, which is written to the time series database and a version number record is generated.

[0104] In this embodiment, S31, when the syntax structure consistency evaluation result is that the syntax structure is compatible, the field name F and field description text D are extracted from the metadata information tables of each data source. A semantic parsing engine is then used to perform word segmentation and semantic embedding processing, transforming the original natural language description into a measurable set of semantic feature vectors VF and VD. S32, the semantic vector set is coupled with the grammatical construction degree Ygo. ij The fitting process is performed, and the semantic coupling mapping index Yyo is calculated using a semantic similarity calculation engine. ij This formula reflects the semantic correspondence between field names and descriptions, and incorporates both semantic and structural features into the association analysis. It originates from cosine similarity in Natural Language Processing (NLP) and the theories of information entropy and signal amplification; in text semantic comparison, cosine similarity... This formula measures the directional consistency of two vectors. In information retrieval and semantic matching, it measures the similarity between two texts in the semantic space. The formula uses the field name F... i and F j Field description semantics D i and D j Treating them as semantic vector pairs, the semantic proximity of the name layer and description layer is calculated separately. Since the name and description contribute differently to the semantic information, an arithmetic mean is taken to achieve dual-channel balance. This corresponds to the equal-weighted channel averaging method in signal processing, forming... In information theory, ln(1+x) is a logarithmic growth function used to describe the gradual increase of gain with input. Using a logarithmic function instead of a nonlinear product can prevent semantic score distortion when structural similarity is too high, which conforms to the asymptotic saturation value characteristic in semantic matching. Since semantics and syntax belong to orthogonal but interactive feature spaces, the two terms are multiplied to form the semantic coupling mapping index Yyo. ij This formula, Sim(F) i F j ) and Sim(D i D j Both are dimensionless similarity functions, and the syntactic construction coupling degree Ygo ij Since the coupling degree is dimensionless, this formula is not affected by dimensions. S33 maps the semantic coupling of all field pairs to the exponent Yyo. ijA matrix-based integration process is performed to form a semantic mapping matrix Y. This matrix is ​​then standardized and denoised to generate a structured semantic mapping table, which records version information. This implementation transforms the semantic relationships between fields across data sources from readable descriptions into comparable vector representations. The semantic association analysis process acquires quantifiable, traceable, and reusable attributes, laying a clear, stable, and scalable semantic mapping foundation for subsequent data fusion, field matching decisions, and the construction of a unified access interface.

[0105] Example 5

[0106] Please refer to Figure 3 Specifically: S4 includes S41;

[0107] S41. Extract the version change rate Δv from the metadata information table and map it to the semantic coupling index Yyo. ij The data integration decision index Jcj was constructed by performing a fitting test, analyzing the fusion confidence of target and candidate fields from different data sources, and then constructing the data integration decision index. i This is used to measure the optimal semantic mapping relationship of target field i among multiple candidate fields, specifically: In the formula, max represents the maximum value selection sign, e represents the exponential function, and Δv j This indicates the semantic definition update rate of field j.

[0108] S4 also includes S42;

[0109] S42. Extract several historical data integration decision indices Jcj defined by industry standard libraries. i Furthermore, in accordance with the semantic matching industry standard, entities are considered semantically equivalent when the semantic similarity score is greater than 0.85. The Q85th percentile value is set as the semantic fusion confidence threshold YZ using the percentile method, and then integrated with the real-time acquired data to form the decision index Jcj. i A fusion confidence assessment was conducted, and the specific assessment plan is as follows;

[0110] When the data integration decision index Jcj i When the semantic fusion confidence threshold YZ is less than the semantic fusion confidence threshold, it indicates that the semantic mapping is unreliable. At this time, the semantic model reconstruction process is triggered, the current field meta-information is recorded in real time, and iterative analysis is performed through S1.

[0111] When the data integration decision index Jcj i When the semantic fusion confidence threshold YZ is greater than or equal to the threshold value, it indicates that the semantic mapping is reliable. At this time, the current mapping relationship is locked, field i and target field j are written into the unified access interface table, and the mapping index is recorded.

[0112] In this embodiment, S41 extracts the field version change rate Δv from the metadata information table and maps it to the semantic coupling index Yyo.ij Perform fitting and construct the data integration decision index Jcj i This ensures that the field mapping result depends not only on the semantic proximity but also on the stability of the field's semantic definition in actual business operations. This formula originates from the combination of the exponential decay model and the maximum likelihood decision-making concept; in mathematical physics, the phenomenon of a quantity decreasing with time or rate of change is often expressed in exponential form e. -kt The description, in its general form, is A(t) = A0 × e -kt Where A(t) represents the residual value of a physical quantity at time t, A0 is the initial value, and k is the decay constant. This formula draws on this principle, treating the field version change rate Δv as the decay rate in the sense of a time variable, reflecting the stability of the semantic definition with version iteration. It becomes a decay factor for semantic stability; the maximum likelihood principle is used to select the one with the highest probability from multiple candidate results, mathematically expressed as... In this formula, through The data integration decision index Jcj is selected from among multiple candidate fields based on the highest confidence level. i Corresponding to the optimal semantic mapping relationship, the exponential decay model is combined with the maximum likelihood principle to form the data integration decision index Jcj. i . In this formula, The input to the exponential function must be dimensionless, -Δv j Yyo is the dimensionless value after time standardization, representing the semantic coupling mapping exponent. ij Since the value is dimensionless, the formula is unaffected by dimensions. S42 calls upon historical integration records and semantic matching industry standards, setting the Q85th percentile value as the semantic fusion confidence threshold YZ using the percentile method, and then calculating the real-time data integration decision index Jcj. i To determine, when the data integration decision index Jcj... i When the value falls below the semantic fusion confidence threshold YZ, the field is considered to have a risk of semantic drift or meaning deviation, triggering semantic model reconstruction and re-entering the metadata collection and analysis process to ensure the continuity and consistency of field relationships; when the data integration decision index Jcj i When the semantic fusion confidence threshold YZ is greater than or equal to the threshold value, it indicates that the target field and candidate fields maintain a stable correspondence at both the semantic and version levels. In this state, the field pairs are written into the unified access interface table and the mapping index is recorded. Through the above processing, the final cross-source field mapping relationship has a clear source basis, semantic interpretation path, and timeliness constraints, forming a traceable, updatable, and maintainable unified field access structure.

[0113] Example 6

[0114] Please refer to Figure 2A multi-source heterogeneous data integration system based on metadata includes an information acquisition module, a metadata structure analysis module, a semantic mapping module, and a fusion decision module;

[0115] The information collection module is used to access the data source through the interface protocol of the metadata collection engine, extract and identify field metadata, perform standardization processing, and summarize to form a metadata information table.

[0116] The metadata structure analysis module is used to automatically analyze and deconstruct the metadata information tables of each data source and extract syntactic feature elements. Based on the syntactic feature elements, it performs structural consistency analysis on the data source fields and evaluates the syntactic structure consistency based on the analysis results.

[0117] The semantic mapping module is used to extract semantic feature elements from the metadata information table when the syntactic structure consistency assessment is syntactic structure compatible, process them to extract semantic feature vectors to form a semantic vector set, and fit them with the structural consistency analysis results to perform semantic association analysis on the data source fields.

[0118] The fusion decision module is used to extract the field version change rate Δv from the metadata information table, fit it with the semantic association analysis results to perform fusion confidence analysis, and evaluate the fusion confidence based on the analysis results.

[0119] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for integrating multi-source heterogeneous data based on metadata, characterized in that: Includes the following steps: S1. Access the data source by calling the interface protocol through the metadata collection engine, extract and identify field metadata, perform standardization processing, and summarize to form a metadata information table. S2. Automatically analyze and deconstruct the metadata information tables of each data source and extract syntactic feature elements. Perform structural consistency analysis on the data source fields based on the syntactic feature elements, and evaluate the syntactic structure consistency based on the analysis results. S3. When the syntax structure consistency assessment is that the syntax structure is compatible, extract the semantic feature elements from the metadata information table, process them, extract the semantic feature vectors to form a semantic vector set, and fit it with the structure consistency analysis results to perform semantic association analysis on the data source fields. S4. Extract the version change rate Δv of the fields in the metadata information table, fit it with the semantic association analysis results to perform fusion confidence analysis, and evaluate the fusion confidence based on the analysis results. S2 includes S21 and S22; S21. Call the metadata parsing engine to automatically analyze and deconstruct the metadata information tables of each data source, extract the syntax feature elements including the field hierarchy structure L and the unit dimension encoding value U, and represent the hierarchy path of each field in the form of a node chain according to the UMDL structure identifier rules during the extraction process. S22. Perform structural consistency analysis on any two data source fields based on syntactic feature elements, and construct the syntactic construction coupling degree Ygo. ij This indicates the degree of matching between different field structures in terms of morphology, position, and combination rules, reflecting the degree of matching between field i and field j, specifically: In the formula, Nc(L) i L j ) represents the number of identical syntax nodes Nc and Nt(L) that field i and field j have in the field hierarchy. i L j The sum of the hierarchical nodes of fields i and j in the field hierarchy is Nt, U. i and U j These represent the unit dimension codes for fields i and j, respectively. S3 further includes S32 and S33; S32, Coupling the semantic vector set with the syntactic construction degree Ygo ij The algorithm performs fitting and calls the semantic similarity calculation engine to perform semantic association analysis on fields from different data sources, and calculates the semantic coupling mapping index Yyo. ij This indicates the degree of semantic correspondence in the mapping space, reflecting the comprehensive correlation between the meaning and syntax of fields i and j, specifically: In the formula, Sim represents the similarity function, F i and F j D represents the field name of the i-th segment and the j-th segment in the field, respectively. i and D j Sim(F) represents the semantic description of the i-th segment and the j-th segment in the field, respectively. i F j Sim(D) represents the semantic similarity between field i and field j in their naming languages. i D j ) represents the semantic similarity between the annotation text of field i and field j, and ln represents the logarithmic function; S33. The semantic coupling mapping exponent Yyo of all field pairs (i, j) ij Perform matrix integration to form a semantic mapping matrix Y, Y=[Yyo ij ] n×n Where n is the number of fields involved in semantic matching, the semantic mapping matrix Y is then standardized and denoised to generate a structured semantic mapping table, which is written to the time series database and a version number record is generated.

2. The method for integrating multi-source heterogeneous data based on metadata according to claim 1, characterized in that: S1 includes S11 and S12; S11. Establish access channels with various data sources through the metadata collection engine, call the corresponding interface protocol to access the data source, and automatically select the access method according to the data source type. The data sources include database-type data sources, file-type data sources, and streaming data sources; The metadata acquisition engine automatically reads information tables and field definitions from database-type data sources using the JDBC database driver. The metadata acquisition engine parses the file header definition and data description block of file-type data sources through the file header parsing interface; The metadata acquisition engine parses the real-time data transmission protocol header of the streaming data source through a message subscription interface; S12. After completing the data source access, the metadata attribute parsing program is automatically executed to extract and identify field metadata from the data source. During the extraction process, the field path parsing algorithm is automatically executed to complete the logical path and perform hierarchical mapping for fields with different structures.

3. The method for integrating multi-source heterogeneous data based on metadata according to claim 2, characterized in that: S1 also includes S13; S13. Call the standardization processing engine to standardize the field metadata and summarize it to form a metadata information table. The standardization process includes naming standardization, unit conversion standardization, hierarchical structure correction, and version indexing. The naming standardization is used to unify the character set for field naming and remove special symbols to generate unified field identifiers. Unit conversion standardization is used to convert different units of measurement into standard dimensional form based on the UMDL dimensional matrix. Hierarchical structure correction is used to correct the hierarchical numbering of fields to ensure that the hierarchical depth of fields from different sources remains consistent. Version indexing is used to create a time-based index table for the rate of change of a field version; The metadata information table includes field name F, field description text D, unit dimension code value U, field hierarchy structure L, and field version change rate Δv.

4. The method for integrating multi-source heterogeneous data based on metadata according to claim 3, characterized in that: S2 also includes S23; S23. Extract several syntax construction coupling degrees Ygo defined by the industry standard library. ij According to the industry standards for information technology data management, data structure elements are considered structurally compatible when the hierarchical matching rate reaches 80% or higher; if it is lower than 50%, structural reconstruction should be performed. Using the percentile method, the Q50th percentile value is set as the syntax structure reconstruction threshold YC, and the Q80th percentile value is set as the syntax structure compatibility threshold YJ. This is then combined with the real-time acquired syntax construction coupling degree Ygo. ij A syntactic consistency assessment is conducted, and the specific assessment scheme is as follows; When the syntactic construction coupling degree Ygo ij When the threshold YC for syntax structure reconstruction is reached, it indicates that the syntax structure is incompatible. At this time, the UMDL reconstruction process is triggered to re-parse the field hierarchy structure, update the node tree structure, and input the updated data into S1 for iterative analysis. When the syntactic structure reconstruction threshold YC ≤ syntactic construction coupling Ygo ij When the value is less than or equal to the syntax structure compatibility threshold YJ, it indicates that there are local differences in the syntax structure. At this time, the structure correction technique is called to adjust the local nodes, and the units are converted and then synchronously updated to the UMDL mapping table. When the syntactic construction coupling degree Ygo ij When the syntax structure compatibility threshold YJ is reached, it indicates that the syntax structure is compatible. At this time, the structure pass status is recorded, and semantic mapping analysis is triggered.

5. The method for integrating multi-source heterogeneous data based on metadata according to claim 4, characterized in that: S3 includes S31; S31. When the grammatical structure consistency assessment is grammatically compatible, extract the semantic feature elements from the metadata information table of each data source. The semantic feature elements include the field name F and the field description text D. Call the semantic parsing engine to perform natural language segmentation and semantic embedding processing to transform unstructured language information into a measurable semantic expression. Then, extract semantic feature vectors based on the text comparison algorithm of edit distance and the word vector model to form a semantic vector set. The semantic vector set includes a semantic vector set F for field names. i ∈{v(F1), v(F2), ..., v(F n )} and the set of semantic vectors describing the fields D i ∈{v(D1), v(D2), ..., v(D n )}.

6. The method for integrating multi-source heterogeneous data based on metadata according to claim 5, characterized in that: S4 includes S41; S41. Extract the version change rate Δv from the metadata information table and map it to the semantic coupling index Yyo. ij The data integration decision index Jcj was constructed by performing a fitting test, analyzing the fusion confidence of target and candidate fields from different data sources, and then constructing the data integration decision index. i This is used to measure the optimal semantic mapping relationship of target field i among multiple candidate fields, specifically: In the formula, max represents the maximum value selection sign, e represents the exponential function, and Δv j This indicates the semantic definition update rate of field j.

7. The method for integrating multi-source heterogeneous data based on metadata according to claim 6, characterized in that: S4 also includes S42; S42. Extract several historical data integration decision indices Jcj defined by industry standard libraries. i Furthermore, in accordance with the semantic matching industry standard, entities are considered semantically equivalent when the semantic similarity score is greater than 0.

85. The Q85th percentile value is set as the semantic fusion confidence threshold YZ using the percentile method, and then integrated with the real-time acquired data to form the decision index Jcj. i A fusion confidence assessment was conducted, and the specific assessment plan is as follows; When the data integration decision index Jcj i When the semantic fusion confidence threshold YZ is less than the semantic fusion confidence threshold, it indicates that the semantic mapping is unreliable. At this time, the semantic model reconstruction process is triggered, the current field meta-information is recorded in real time, and iterative analysis is performed through S1. When the data integration decision index Jcj i When the semantic fusion confidence threshold YZ is greater than or equal to the threshold value, it indicates that the semantic mapping is reliable. At this time, the current mapping relationship is locked, field i and target field j are written into the unified access interface table, and the mapping index is recorded.

8. A metadata-based multi-source heterogeneous data integration system, comprising the metadata-based multi-source heterogeneous data integration method according to any one of claims 1-7, characterized in that: It includes an information acquisition module, a metadata structure analysis module, a semantic mapping module, and a fusion decision module; The information collection module is used to access the data source through the interface protocol of the metadata collection engine, extract and identify field metadata, perform standardization processing, and summarize to form a metadata information table. The metadata structure analysis module is used to automatically analyze and deconstruct the metadata information tables of each data source and extract syntactic feature elements. Based on the syntactic feature elements, it performs structural consistency analysis on the data source fields and evaluates the syntactic structure consistency based on the analysis results. The semantic mapping module is used to extract semantic feature elements from the metadata information table when the syntactic structure consistency assessment is syntactic structure compatible, process them to extract semantic feature vectors to form a semantic vector set, and fit them with the structural consistency analysis results to perform semantic association analysis on the data source fields. The fusion decision module is used to extract the field version change rate Δv from the metadata information table, fit it with the semantic association analysis results to perform fusion confidence analysis, and evaluate the fusion confidence based on the analysis results.

Images