A method and storage medium for integrating multi-source heterogeneous data based on dynamic metadata modeling

Abstract

This invention discloses a method and storage medium for integrating multi-source heterogeneous data based on dynamic metadata modeling. It collects structured, semi-structured, and unstructured data from multi-source heterogeneous data, extracts the format, structure, and semantic information of the data to form a metadata set, providing the original basis for subsequent resolution of naming conflicts and data type differences. By comprehensively capturing data features, it provides input for dynamic metadata modeling. Furthermore, through the dynamic metadata model, the metadata set is uniformly described to obtain unified descriptive information. Further, based on dynamic mapping and dimensionality reduction optimization using neighborhood graphs, high-dimensional data is mapped to a low-dimensional space while preserving the global structure. Finally, based on the dimensionality-reduced data, a fused database supporting cross-source queries is generated, eliminating data silos and providing a standardized view.

Description

Technical Field

[0001] This invention relates to the field of multi-source heterogeneous data processing, and in particular to a method and storage medium for integrating multi-source heterogeneous data based on dynamic metadata modeling. Background Technology

[0002] Enterprises are currently facing an explosive growth in data sources (such as CRM, ERP, IoT devices, and social media), but this data is scattered across different systems, forming "data silos." For example, retail companies may need to integrate POS systems (structured), user behavior logs (semi-structured), and social media comments (unstructured) to analyze user preferences. However, the integration of multi-source heterogeneous data faces the following technical challenges:

[0003] Traditional methods rely on static metadata models, which struggle to dynamically adapt to changes in data sources (such as new fields or updated formats). For example, when "customer ID" is named "custID" or "customer_id" in different systems, the static model cannot adapt to standardized naming rules, leading to naming conflicts and semantic ambiguity. Furthermore, differences in data types (such as date formats "YYYY-MM-DD" versus "MM / DD / YYYY") and structural differences (such as order data scattered across multiple tables) require manual intervention for mapping, which is inefficient and prone to introducing errors.

[0004] On the other hand, in the process of integrating multi-source heterogeneous data, dimensionality reduction before integration is a powerful means to improve data analysis and integration. However, existing techniques often use Euclidean distance to directly calculate data similarity and rely on linear dimensionality reduction methods such as multidimensional scaling (MDS). However, multi-source heterogeneous data often exhibits a nonlinear manifold structure (such as a Swiss roll distribution), and its true correlation needs to be described by geodesic distance (the shortest path along the manifold). Euclidean distance can incorrectly compress or stretch local structures, leading to distortion of data relationships in the mapped low-dimensional space. For example, time-series event streams in user behavior logs may lose local clustering features due to linear dimensionality reduction, affecting the accuracy of subsequent analysis. Summary of the Invention

[0005] The purpose of this invention is to provide a method for integrating multi-source heterogeneous data based on dynamic metadata modeling, which solves the above-mentioned technical problems pointed out in the prior art.

[0006] This invention provides a method for integrating multi-source heterogeneous data based on dynamic metadata modeling, comprising the following steps:

[0007] Collect and acquire multi-source heterogeneous data; collect metadata from the multi-source heterogeneous data to obtain a metadata set;

[0008] The metadata set is uniformly described based on the dynamic metadata model to obtain unified description field information;

[0009] Based on the unified description field information, the multi-source heterogeneous data is mapped to obtain mapped data;

[0010] The mapped data is integrated to obtain a fused database.

[0011] Preferably, the metadata set includes the data format, structure, and semantic information of multi-source heterogeneous data;

[0012] The multi-source heterogeneous data includes structured data, semi-structured data, and unstructured data.

[0013] Preferably, the step of mapping the multi-source heterogeneous data based on the unified description field information to obtain mapped data specifically includes the following steps:

[0014] A neighborhood graph is constructed based on the Euclidean distance between every two unified description fields; each node in the neighborhood graph represents a multi-source heterogeneous data; the line connecting every two nodes represents the Euclidean distance between two unified description fields.

[0015] Based on the neighborhood graph and the artificial bee colony algorithm, calculate the shortest distance matrix between every two nodes in the graph;

[0016] Based on the shortest distance matrix, a dual centering process is performed using multidimensional scaling analysis to obtain the low-dimensional mapped coordinates of each node.

[0017] The low-dimensional mapping coordinates are optimized based on the shortest distance matrix to obtain the mapping data for each node.

[0018] Preferably, the step of calculating the shortest distance matrix between every two nodes in the neighborhood graph using the artificial bee colony algorithm includes the following steps:

[0019] Obtain data conflict information during the unified description process of the metadata set;

[0020] Based on the data conflict information and the Euclidean distance from each node to its k nearest neighbor nodes in the neighborhood graph, the optimization factor of the path segment between each pair of adjacent nodes is obtained.

[0021] Traverse each of the nodes, taking each node as the starting point and any node other than the current node as the ending point, and combine them to obtain multiple initial paths from the starting point to the ending point; there are no duplicate nodes in the initial paths;

[0022] Obtain the edge distance of each initial path; calculate the path fitness based on the edge distance and the optimization factor;

[0023] Traverse each of the initial paths; replace path segments based on the optimization factors of each path segment in the current initial path and the optimization factors of the path segments between the endpoints of the current initial path and each neighboring node of the endpoint to obtain the second path; and replace the second path based on the new fitness of the second path and the fitness of the original initial path to obtain the third path.

[0024] The shortest distance matrix is ​​obtained by replacing and optimizing the fitness of the third path according to the distribution of the tuning factors.

[0025] Preferably, the step of determining and replacing the third path based on the new fitness of the second path and the fitness of the original initial path includes the following steps:

[0026] The new fitness is calculated based on the second path;

[0027] When the new fitness is greater than the path fitness, the current second path replaces the current initial path to obtain the third path.

[0028] Preferably, the shortest distance matrix is ​​obtained by replacing and optimizing the fitness of the third path according to the distribution of the tuning factor, and includes the following steps:

[0029] The fitness of all third paths is calculated, and a high-potential path is selected by roulette wheel probability allocation. Based on the optimization factor distribution of the high-potential path, the optimized third path is obtained by replacing the path segment with the highest optimization factor with the path segment with the lowest optimization factor among the path segments from each node to each node's neighboring nodes.

[0030] Determine whether the tuning factor of each path segment in the optimized third path is less than a preset tuning factor threshold; if so, determine the optimized third path as the target path.

[0031] The shortest distance matrix is ​​obtained based on all the target paths.

[0032] Preferably, the step of analyzing the path segment between each pair of adjacent nodes based on the data conflict information and the Euclidean distance from each node to its k nearest neighbors in the neighborhood graph includes the following steps:

[0033] The data conflict information is subjected to conflict quantification detection processing to obtain the conflict quantification score of each data conflict information; the metadata conflict factor is calculated based on the conflict quantification score;

[0034] The neighborhood fluctuation factor is obtained by analyzing the Euclidean distances from each node to its k nearest Euclidean neighbors in the neighborhood graph.

[0035] The tuning factor α is obtained by comprehensively calculating based on the neighborhood fluctuation factor and the metadata conflict factor. uv .

[0036] Preferred methods for integrating multi-source heterogeneous data based on dynamic metadata modeling also include:

[0037] The neighborhood fluctuation factor V of two nodes based on a path segment formed by two adjacent nodes. u V v Calculate the average neighborhood volatility factor

[0038] Based on the aforementioned metadata conflict factor C uv The average value of the neighborhood fluctuation factor is summed after weighing the balance parameter λ to obtain the tuning factor α. uv ;

[0039] The optimization factor α uv The calculation method is as follows:

[0040]

[0041] In the formula, α uv C is the optimization factor for the path segment between node u and node v; uv V is the metadata conflict factor between node u and node v; u V is the neighborhood fluctuation factor of node u; v Let λ be the neighborhood fluctuation factor of node v; λ is the equilibrium parameter.

[0042] Preferably, the step of analyzing the neighborhood fluctuation factor based on the Euclidean distances from each node in the neighborhood graph to its k nearest Euclidean neighbors includes the following steps:

[0043] Obtain the neighboring nodes corresponding to each node in the neighborhood graph; construct a distance set based on the Euclidean distance between the node and the neighboring nodes;

[0044] Sort each element in the distance set in ascending order of Euclidean distance to obtain a distance sequence set; select the first k elements of the distance sequence set as the baseline distance data; calculate the median distance (MedianD) corresponding to the current node based on the baseline distance data. u and interquartile range (IQRD) u ;

[0045] MedianD is based on the median distance of all nodes in the current neighborhood graph.u The global distance median GMadianD is obtained by statistical analysis.

[0046] MedianD based on the distance median of the current node u and interquartile range (IQRD) u The neighborhood fluctuation factor is calculated by taking the global distance median GMadianD.

[0047] In another aspect, the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for integrating multi-source heterogeneous data based on dynamic metadata modeling.

[0048] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0049] Analysis of the multi-source heterogeneous data integration method and storage medium based on dynamic metadata modeling provided by the present invention reveals that the method includes: acquiring multi-source heterogeneous data; collecting metadata from the multi-source heterogeneous data to obtain a metadata set; uniformly describing the metadata set based on a dynamic metadata model to obtain unified description field information; mapping the multi-source heterogeneous data based on the unified description field information to obtain mapped data; and integrating the mapped data to obtain a fused database.

[0050] In practical applications, the above integration method collects structured data (such as order tables), semi-structured data (such as JSON logs), and unstructured data (such as images and videos) from multi-source heterogeneous systems (such as CRM, ERP, IoT devices, and social media). It extracts the data's format (such as date format "YYYY-MM-DD"), structure (such as single-table or multi-table joins), and semantic information (such as the business meaning of fields) to form a metadata set. This provides the original basis for resolving naming conflicts (such as "custID" vs. "customer_id") and data type differences (such as string and integer conversion).

[0051] By comprehensively capturing data features, input is provided for dynamic metadata modeling. Furthermore, through the dynamic metadata model, the metadata set is uniformly described to obtain unified description information (i.e., unified description field information). Based on the unified description field information, the multi-source heterogeneous data is mapped to obtain mapped data. In specific operations, based on the dynamic mapping and dimensionality reduction optimization of the neighborhood graph, high-dimensional data is mapped to a low-dimensional space while preserving the global structure. Finally, the dimensionality-reduced data is integrated to generate a fusion database that supports cross-source queries, eliminating data silos and providing standardized views (such as unified date formats and semantically consistent table structures). Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the main process of a multi-source heterogeneous data integration method based on dynamic metadata modeling.

[0053] Figure 2 This is a flowchart of the process topology for obtaining mapped data in a multi-source heterogeneous data integration method based on dynamic metadata modeling.

[0054] Figure 3 This is a schematic diagram of a neighborhood graph simulation in a multi-source heterogeneous data integration method based on dynamic metadata modeling.

[0055] Figure 4 This is a schematic diagram simulating the shortest distance path in a multi-source heterogeneous data integration method based on dynamic metadata modeling.

[0056] Figure 5 This is a schematic diagram of the Swiss roll dataset simulation in a multi-source heterogeneous data integration method based on dynamic metadata modeling. Detailed Implementation

[0057] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0059] Example 1

[0060] like Figure 1 As shown, Embodiment 1 of the present invention provides a method for integrating multi-source heterogeneous data based on dynamic metadata modeling, including the following steps:

[0061] Step S10: Collect and acquire multi-source heterogeneous data; collect metadata from the multi-source heterogeneous data to obtain a metadata set; the metadata set includes the data format, structure, and semantic information of the multi-source heterogeneous data; the multi-source heterogeneous data includes structured data (e.g., order data in a database table), semi-structured data (e.g., server logs, XML or JSON format data), and unstructured data (e.g., documents, images, videos, social media data, etc.);

[0062] Step S20: Based on the dynamic metadata model, the metadata set is uniformly described to obtain unified description field information;

[0063] It should be noted that the unified description field information in the above embodiments of this application refers to a standardized and consistent description of each data element (e.g., field or attribute) in multi-source heterogeneous data under the dynamic metadata model framework. This description typically includes, but is not limited to, the following aspects:

[0064] Data field name: Defines the name of the data element, making it easy to identify and reference;

[0065] Type: Specifies the type of the data element, such as integer, floating-point, string, date / time, etc.

[0066] Constraint rules: These define the conditions or restrictions that data elements must follow, such as length limits, value ranges, and whether null values ​​are allowed.

[0067] Semantic tags: Add semantic information to data elements to help understand their meaning and purpose; for example, a field may be labeled as "customer ID" or "order date";

[0068] Specifically, a unified description method can be implemented. For example, for identical or similar data items from different data sources, a standard naming rule can be established through a dynamic metadata model. For instance, a customer identifier might be "customer_id" in some systems and "custID" in others. In this case, a standard name can be selected using the dynamic metadata model, ensuring that all relevant data uses this name. On the other hand, different data sources may represent the same data type differently (such as date formats). A common data type mapping scheme can be defined using the dynamic metadata model to convert data types from all sources into their corresponding types in the unified model. Furthermore, consistent constraint rules can be defined for each data element, including but not limited to length limits, value ranges, and whether null values ​​are allowed, which helps ensure the consistency of data quality. Additionally, semantic tags can be added to each data element to help understand its meaning and purpose; these can be based on industry standards or internally agreed-upon terminology.

[0069] In addition, consistency issues and conflicts can exist in metadata collections, including:

[0070] Naming conflicts: For example, the same entity may have different names in different systems;

[0071] Data type differences: The same information may be stored as different types of data (such as dates) in different systems;

[0072] Structural differences: The data is organized differently. For example, in one system, order information is contained in a single table, while in another system it is distributed across multiple related tables.

[0073] Semantic differences: Even if the names and types are the same, different systems may interpret the data differently;

[0074] Based on this, a clear, easy-to-understand, and easy-to-use naming convention is established through a dynamic metadata model; the mapping relationship between different types is defined to ensure cross-system compatibility; a standard schema for describing data structures is provided, including field names, types, constraints, and other information; and a metadata dictionary containing all data elements and their attributes is created for reference.

[0075] After establishing the aforementioned naming conventions, mapping relationships, standard patterns for describing data structures, and metadata dictionaries, each metadata set is mapped through a dynamic metadata model to obtain unified descriptive field information. Specifically, each data source is analyzed in detail to understand its data structure, type, semantics, and other characteristics. Based on the analysis results, specific conversion rules from the original data to the unified model are designed. ETL tools or custom scripts are used to implement these rules, converting the original data into a unified format. Testing is conducted before practical application to check whether the converted data meets expectations, and the conversion logic is adjusted based on feedback.

[0076] Through the above embodiments of this application, valuable information is extracted from multi-source heterogeneous data using a dynamic metadata model and integrated into a unified data model, thereby supporting higher-level data analysis and business decision-making.

[0077] Step S30: Map the multi-source heterogeneous data based on the unified description field information to obtain mapped data;

[0078] It should be noted that the above-described embodiments of this application convert the metadata (unified description field information) that has been uniformly described into a structured representation (mapped data) in a low-dimensional space through a dynamic mapping algorithm for subsequent integration; its core is to solve the problem of inaccurate high-dimensional space mapping caused by structural differences and semantic conflicts of multi-source heterogeneous data through neighborhood graph modeling, shortest path optimization and dimensionality reduction technology (see the subsequent implementation process for specific solutions);

[0079] Step S40: Integrate the mapping data to obtain a fused database.

[0080] It should be noted that the above embodiments of this application integrate the optimized mapping data (low-dimensional representation) into a consistent database according to a unified model, supporting cross-source querying and analysis. Specifically, this is achieved through data alignment (i.e., resolving consistency issues), conflict resolution, structural fusion, and storage optimization. Data alignment is based on the similarity of low-dimensional coordinates (e.g., Euclidean distance < 0.5), mapping fields from different sources to a unified model (e.g., grouping "custID" and "customer_id" into "customer ID"), and following the rules defined by the dynamic meta-model (e.g., unifying the date format to ISO). 8601); resolving conflicts involves defining the authority of data sources (e.g., "sales revenue" in system A takes precedence over system B); structural integration involves merging disparate table structures (e.g., order tables split across systems A and B) into a unified schema and linking different tables through foreign keys (e.g., linking customer tables with order tables); storage optimization involves creating indexes for frequently queried fields (e.g., "order date") and partitioning by time or business unit to improve query efficiency; based on the above data integration, a fused database is obtained, which uniformly manages multi-source heterogeneous data, providing a standardized and consistent data view by eliminating data silos and resolving differences in data format / structure / semantics, supporting efficient cross-system querying and analysis;

[0081] The embodiments described above collect structured data (such as order tables), semi-structured data (such as JSON logs), and unstructured data (such as images and videos) from multi-source heterogeneous systems (such as CRM, ERP, IoT devices, and social media). They extract the data's format (such as date format "YYYY-MM-DD"), structure (such as single-table or multi-table joins), and semantic information (such as field business meanings) to form a metadata set. This provides the initial basis for resolving naming conflicts (such as "custID" vs. "customer_id") and data type differences (such as string and integer conversion). By comprehensively capturing data characteristics, it provides input for dynamic metadata modeling. Furthermore, through a dynamic metadata model, the metadata set is uniformly described to obtain unified descriptive information. Further, based on dynamic mapping and dimensionality reduction optimization using a neighborhood graph, high-dimensional data is mapped to a low-dimensional space while preserving the global structure. Finally, based on the dimensionality-reduced data, a fused database supporting cross-source queries is generated, eliminating data silos and providing standardized views (such as unified date formats and semantically consistent table structures).

[0082] Multi-source heterogeneous data (such as structured, semi-structured, and unstructured data) have high dimensionality and complex structure, making direct processing computationally expensive and susceptible to noise interference; traditional dimensionality reduction methods (such as MDS) rely on Euclidean distance and cannot capture the manifold structure of the data (such as the curvature distribution of SwissRoll).

[0083] Specifically, such as Figure 2As shown, in step S30, the multi-source heterogeneous data is mapped based on the unified description field information to obtain mapped data, specifically including the following steps:

[0084] Step S31: Construct a neighborhood graph based on the Euclidean distance between every two unified description field information; each node in the neighborhood graph represents a multi-source heterogeneous data; the line connecting every two nodes represents the Euclidean distance between two unified description field information;

[0085] It should be noted that the neighborhood graph in the above embodiments of this application is constructed by calculating the Euclidean distance between every two uniform description field information, and then selecting the k uniform description field information with the closest Euclidean distance for each uniform description field information to jointly construct the neighborhood graph (e.g., Figure 3 (as shown);

[0086] Step S32: Calculate the shortest distance matrix between every two nodes in the graph based on the neighborhood graph and the artificial bee colony algorithm;

[0087] To explain, in the above embodiments of this application, the shortest distance path between every two nodes is first obtained, and then the shortest distance matrix is ​​constructed from all the shortest distance paths; for example... Figure 4 As shown, the shortest distance path is the shortest path distance among multiple path choices, such as... Figure 4 The thick black line in the middle represents the shortest path from node 1 to node 3.

[0088] Step S33: Apply multidimensional scaling analysis based on the shortest distance matrix to perform double centering processing to obtain the low-dimensional mapped coordinates of each node;

[0089] It should be noted that the above embodiments of this application use MDS (Multidimensional Scaling Analysis) to double-center the calculated shortest distance matrix to obtain the initial coordinates of each data node in the low-dimensional space; the original high-dimensional distance relationship is transformed into the low-dimensional space, thereby simplifying the data and improving the efficiency of subsequent analysis.

[0090] Step S34: Optimize the low-dimensional mapping coordinates based on the shortest distance matrix to obtain the mapping data for each node.

[0091] It should be noted that in step S33 of the above embodiment of this application, the shortest distance matrix is ​​double-centered by multidimensional scaling analysis (MDS) to obtain the initial coordinates of each data node in the low-dimensional space. However, since multi-source heterogeneous data may contain complex nonlinear relationships and local structures, MDS is difficult to completely retain such local and global intrinsic structural information during the dimensionality reduction process, resulting in certain errors during mapping.

[0092] Specifically, this refers to the curved surfaces, manifold structures, and local dense clustering relationships that data exhibits in high-dimensional space. These relationships often need to be accurately described using geodesic distance, rather than simple Euclidean distance. At the same time, the dimensionality reduction method of MDS is mainly based on Euclidean distance and global optimization. When faced with the nonlinear intrinsic structure and local detailed relationships of data, it is easy to ignore or distort this information, resulting in errors in the local or global structure restoration after dimensionality reduction.

[0093] Taking the SwissRoll dataset as an example, for instance... Figure 5 As shown, the SwissRoll dataset is a classic example of nonlinear manifold data, where the data points exhibit a three-dimensional “curled” structure formed by bending a two-dimensional plane (like a piece of paper).

[0094] In the original three-dimensional space, the data points are distributed along a curled two-dimensional manifold. Two data points may appear close in terms of Euclidean linear distance, but their true distance may be much greater if the actual path along the manifold (surface) is considered. When using MDS to reduce SwissRoll data to two dimensions, MDS attempts to preserve the linear Euclidean distance between all point pairs. However, since the inherent structure of SwissRoll is non-linear, the true relationships of the data are calculated along the geodesic distances of the manifold. Directly using Euclidean distance can lead to incomplete reflection of local information (such as the relative positions of points within the manifold), resulting in local structural distortion. That is, the local adjacency relationships of data points on the original manifold (along the curves of the manifold) cannot be captured by simple linear Euclidean distance. This can lead to the phenomenon that the distances between local point groups are too close or too far in the dimensionality-reduced two-dimensional result, which is inconsistent with their true distribution on the original manifold. At the same time, when the entire rolled structure is planarized, MDS may fail to fully restore the global topology of the original manifold, causing the distance relationships in some regions to be stretched or compressed, failing to faithfully reflect the inherent Euclidean geometry of the data.

[0095] Based on this, the embodiments of this application optimize the low-dimensional coordinates using the shortest distance matrix in step S34. The shortest distance matrix can more accurately capture the real relationship of data in high-dimensional space. Through this optimization method based on global and local structural information, the accuracy of low-dimensional mapping is improved, thereby achieving higher quality data integration and the construction of a fusion database.

[0096] The optimization in S34 above specifically uses the shortest distance matrix as a basis to minimize the difference between the low-dimensional Euclidean distance and the true geodesic distance (i.e., the shortest path distance) through iterative optimization (such as gradient descent), thereby correcting the initial MDS result and ensuring that the low-dimensional mapping not only maintains the rationality of the data distribution globally, but also accurately reflects the nonlinear structure of the high-dimensional data in local details, thereby achieving higher quality data integration and fusion. The above-mentioned optimization methods such as gradient descent are common knowledge to those skilled in the art, and will not be elaborated upon in this application.

[0097] The embodiments described above calculate the Euclidean distance between all uniformly described field information, select the k nearest neighbors of each node, and construct a neighborhood graph. This neighborhood graph preserves the local proximity relationships of data points, providing local structural information for subsequent dimensionality reduction. Simultaneously, the Euclidean distance between nodes in the neighborhood graph reflects the similarity between fields (such as naming, type, and semantic differences), providing a foundation for subsequent path calculation. Furthermore, based on the neighborhood graph, a shortest path algorithm (such as Dijkstra's algorithm) is used to calculate the geodesic distance (i.e., the path distance along the manifold) between any two nodes. The geodesic distance accurately describes the true relationship of data in a high-dimensional manifold, solving the problem that Euclidean distance cannot capture nonlinear structures, as seen in datasets like the SwissRoll dataset. The curled structure is preserved through geodesic distance, avoiding the "straight-line distance" bias of Euclidean distance. In this embodiment, multidimensional scaling analysis (MDS) is used to doubly center the shortest distance matrix, mapping the high-dimensional distance relationship to the low-dimensional space. MDS optimizes the objective function to approximately preserve the relative distance relationship of high-dimensional data to the low-dimensional space, providing low-dimensional initial coordinates for subsequent optimization steps and reducing computational complexity. Furthermore, the difference between the low-dimensional Euclidean distance and the high-dimensional geodesic distance is minimized through iterative optimization (such as gradient descent), correcting the error of MDS. The shortest distance matrix (the actual geodesic distance) is used as a reference to correct the local structural distortion caused by global optimization of MDS, ensuring that the low-dimensional mapping preserves both the rationality of the global distribution and the local manifold structure.

[0098] Specifically, in step S32, the shortest distance matrix between every two nodes in the graph is calculated based on the neighborhood graph combined with the artificial bee colony algorithm, including the following steps:

[0099] Step S321: Obtain data conflict information during the unified description process of the metadata set;

[0100] Step S322: Based on the data conflict information and the Euclidean distance from each node to the nearest k neighboring nodes in the neighborhood graph, analyze and obtain the optimization factor of the path segment between each pair of adjacent nodes;

[0101] It should be noted that the above-mentioned optimization factor is derived from the statistical data and deviation analysis generated during the unified description, metadata collection and mapping process of multi-source heterogeneous data in stages S10–S31. Typically, these analysis results reflect issues such as data format differences, mapping errors, and data migration fluctuations. Therefore, this factor can be regarded as a quantitative description of "data inconsistency" or "data migration risk".

[0102] In the subsequent implementation of the solution, the total cost of the path (or the total cost of the path used in the fitness calculation) is calculated by accumulating the cost of the path segments between each adjacent node. The attributes of the node itself (such as data quality and mapping accuracy) will be reflected in the edges connecting that node. Therefore, by setting corresponding optimization factors for each edge (i.e., path segment), the cumulative cost caused by data transmission or mapping errors can be corrected more intuitively. If a node has a problem, the optimization factors of the edges related to that node will be increased accordingly, thereby affecting the overall evaluation of the entire path.

[0103] Step S323: Traverse each of the nodes, taking each node as the starting point and any node other than the current node as the ending point, and combine them to obtain multiple initial paths from the starting point to the ending point; there are no duplicate nodes in the initial paths;

[0104] Step S324: Obtain the edge distance of each initial path (the edge distance is the sum of the Euclidean distances between each two adjacent nodes in the initial path); calculate the path fitness based on the edge distance and the tuning factor;

[0105] The above path fitness is calculated as follows:

[0106]

[0107] In the formula, Fitness(Path) represents the path fitness of the initial path; d uv Let α be the Euclidean distance between node u and node v (obtained through the summation formula, i.e., the edge distance of the entire initial path); uv d is the optimization factor for shortening the path between node u and node v; uv ·(1+α uv ) represents the cost of the shortest path segment between node u and node v;

[0108] Step S325: Traverse each of the initial paths; perform path segment replacement based on the optimization factors of each path segment in the current initial path and the optimization factors of the path segments between the endpoints of the current initial path and each neighboring node of the endpoint, to obtain a second path; calculate a new fitness based on the second path; replace the current initial path with the current second path when the new fitness is greater than the path fitness (if the new fitness is less than or equal to the path fitness, retain the current initial path and determine it as the third path, and filter out the second path obtained by the replacement), to obtain the third path;

[0109] It should be noted that in the above embodiments of this application, by traversing all the current initial paths, and then during the processing of each initial path, the optimization factors of each path segment in the current initial path (the optimization factors of the path segments between every two nodes are calculated in step S322) are selected to obtain the optimization factors to be processed that are higher than the average of the optimization factors of each path segment in the current initial path. At the same time, each neighboring node (the node adjacent to the endpoint) in the current initial path from the endpoint is selected to obtain the path segment with the lowest optimization factor (at this time, the neighboring node corresponding to the path segment with the lowest optimization factor among the multiple path segments from the endpoint to the endpoint is selected), and the neighboring node corresponding to the lowest optimization factor among the neighboring nodes from the endpoint to the endpoint is replaced on the path segment corresponding to the optimization factor to be processed. For example:

[0110] If the initial path A→B→D contains path segments A→B and B→D, and the optimization factor for each path segment is a. AB =0.7, a BD =0.4; Randomly select an edge with a tuning factor higher than the average (e.g., A→B), and extract the top k neighbor nodes (e.g., C, E) from the neighborhood of the target edge's endpoint (e.g., B); Calculate the tuning factor of the candidate edge (e.g., a BC =0.3, a BE =0.5); Select the candidate edge with the lowest tuning factor (e.g., B→C); Concatenate the new path: A→B→C→D (ensure no duplicate nodes); Then, calculate the new fitness of the second path, for example, 0.1398, or the fitness of the original initial path is 0.1316. Then replace the current initial path with the current second path (i.e., replace the original path with the new path after replacing the path segment).

[0111] Step S326: Based on the fitness of the third path, the shortest distance matrix is ​​obtained by replacing and optimizing according to the distribution of the tuning factor.

[0112] It should be noted that the above-described embodiments of this application obtain optimization factors by analyzing data conflict information, and then use optimization factors to optimize the shortest distance matrix to prevent data distribution changes caused by differences in metadata structure or unstable distance calculation caused by inconsistent metadata, thereby improving the accuracy of the shortest distance matrix analysis and making the subsequent establishment of the fusion database more robust.

[0113] Specifically, in step S326, the shortest distance matrix is ​​established after replacement optimization based on the fitness of the third path according to the distribution of the tuning factor, including the following operation steps:

[0114] Step S3261: Calculate the fitness of all third paths (the current initial path is the final third path obtained from S321-S325 above), and select high-potential paths by roulette wheel probability allocation; based on the optimization factor distribution in the high-potential paths, the optimized third path is obtained by replacing the path segments with high optimization factors with the path segments with the lowest optimization factors among the path segments from each node to each node's neighboring nodes.

[0115] It should be noted that, in the above embodiments of this application, the fitness of all third paths is first calculated (e.g., fitness of path 1 is 0.2, fitness of path 2 is 0.3, and fitness of path 3 is 0.5); a high-potential path (e.g., path 3A→C→D) is selected by roulette probability allocation; further, the optimization factor distribution in the high-potential path is calculated (e.g., A→C (α=0.3), C→D (α=0.2)); the path segment with the highest optimization factor (i.e., A→C (α=0.3)) is selected, and an edge with a lower optimization factor (e.g., A→E, α=0.1) is selected from the neighborhood of node A; thus, a new path A→E→C→D (i.e., the optimized third path) is generated; the fitness of the new path A→E→C→D is calculated again, and when the fitness is higher than that of the original path (the original path is the high-potential path 3A→C→D mentioned above), the original path is replaced to obtain the optimized third path A→E→C→D.

[0116] Step S3262: Determine whether the tuning factor of each path segment in the optimized third path is less than the preset tuning factor threshold; if so, determine the optimized third path as the target path (the target path is the shortest distance path in the explanation of step S32).

[0117] Step S3263: Build the shortest distance matrix based on all the target paths.

[0118] It should be noted that the technical solutions adopted in the above embodiments of this application, the artificial bee colony algorithm is usually used for optimization problems, including path finding. However, directly using the artificial bee colony algorithm to find the shortest path distance between every two nodes in the neighborhood graph will result in inaccurate optimization results due to data distribution changes caused by differences in metadata structure or unstable distance calculation caused by metadata inconsistency. Therefore, the above embodiments of this application analyze the conflict types (such as naming conflicts, data type differences, structural differences or semantic differences) in the metadata unification process of each data source (i.e., the above-mentioned nodes) to obtain an optimization factor (for details on the analysis and acquisition of the optimization factor, please refer to the subsequent operation steps S3221-S3223), and apply the optimization factor to the optimization process of the artificial bee colony algorithm to make the optimization results of the artificial bee colony algorithm more accurate, thereby obtaining a more accurate shortest path distance.

[0119] Specifically, in step S322, based on the data conflict information and the Euclidean distance from each node to its k nearest neighboring nodes in the neighborhood graph, the optimization factor of the path segment between each pair of adjacent nodes is obtained through analysis, including the following steps:

[0120] Step S3221: Perform conflict quantification detection processing on the data conflict information to obtain the conflict quantification score of each data conflict information; calculate the metadata conflict factor based on the conflict quantification score;

[0121] Specifically, the aforementioned data conflict quantification detection and processing refers to: for naming conflicts, using Natural Language Processing (NLP) tools to calculate the semantic similarity of field names; for type conflicts, comparing field data types and conversion rule bases; for structural conflicts, analyzing the table relationships and nesting levels of the data model; and for semantic conflicts, verifying the consistency of field meanings in conjunction with a business terminology table. Specifically, for naming conflicts, the semantic distance of name differences is used (e.g., completely unrelated = 1, partially matched = 0.5, synonym = 0.2), for example, custID and customer_id are semantically similar → score 0.3; for type conflicts, the complexity of type conversion (e.g., string...) is used... Integer = 0.5, Date format conversion = 0.8, No automatic conversion at all = 1); For example, string to integer conversion requires format cleaning → score 0.6; For structural conflicts, based on the difficulty of data model reorganization (e.g., single table → multi-table join = 0.7, nested structure expansion = 0.9); For example, order information split from a single table into multiple tables → score 0.8; For semantic conflicts, based on the degree of business impact (e.g., key indicator ambiguity = 1, minor field ambiguity = 0.3); For example, sales revenue including tax and excluding tax affects financial calculations → score 0.9.

[0122] Step S3222: Analyze the Euclidean distances of each node in the neighborhood graph to the k nearest neighboring nodes to obtain the neighborhood fluctuation factor;

[0123] It should be noted that the above-described embodiment of this application first calculates the Euclidean distance between each node in the neighborhood graph and each of its neighboring nodes. Then, it sorts the Euclidean distances from low to high and selects the top k target neighboring nodes. Finally, based on the Euclidean distance between the current node and the target neighboring nodes, it calculates the standard deviation and uses the ratio of the current standard deviation to the mean of the global standard deviations calculated for all nodes to obtain the neighborhood fluctuation factor. Specifically, for example, if node u has 3 nearest neighbors in its neighborhood graph with Euclidean distances of 1.2, 1.5, and 1.8, the standard deviation of node u is calculated to be approximately 0.245. Further, the mean of the calculated standard deviations of all nodes in the current neighborhood graph (i.e., the aforementioned mean of the global standard deviations) is calculated to be 0.2. The ratio of the current node u to the mean of the standard deviations is used as the neighborhood fluctuation factor, which is approximately 1.225.

[0124] Step S3223: Calculate the tuning factor α based on the neighborhood fluctuation factor and the metadata conflict factor. uv ;

[0125] Specifically, the neighborhood fluctuation factor V of two nodes in a path segment formed by two adjacent nodes. u V v Calculate the average neighborhood volatility factor

[0126] Based on the aforementioned metadata conflict factor C uv The average value of the neighborhood fluctuation factor is summed after weighing the balance parameter λ to obtain the tuning factor α. uv ;

[0127] The optimization factor α uv The calculation method is as follows:

[0128]

[0129] In the formula, α uv C is the optimization factor for the path segment between node u and node v; uv V is the metadata conflict factor between node u and node v; u V is the neighborhood fluctuation factor of node u; v Let λ be the neighborhood fluctuation factor of node v; λ is the equilibrium parameter.

[0130] It should be noted that in the above embodiments of this application, α uv C represents the optimization factor for the path segment from node u to node v;uv This represents the metadata conflict factor between node u and node v; this factor reflects the degree of inconsistency and conflict between the two nodes in the process of unified metadata description or data mapping; the higher the inconsistency, the higher the conflict factor C. uv The larger the value, the more serious the data matching or transformation problem on this side; V u With V v Let represent the neighborhood fluctuation factors of node u and node v, respectively. The neighborhood fluctuation factor describes the fluctuation or instability of the data within the neighborhood of a node. For example, during data migration or mapping, if the data quality, format, or semantics of a node's region differ significantly, its fluctuation factor will be higher. The average of the two factors is then used. It can comprehensively reflect the overall fluctuation of the path segment connecting these two nodes; λ is a balance parameter with a value range of [0, 1]; by adjusting λ, the metadata conflict factor C can be controlled. uv Neighborhood volatility factor The relative importance of λ in the final tuning factors; when the value of λ is large (close to 1), the role of the metadata conflict factor is amplified, and the algorithm pays more attention to the data inconsistency problem; when the value of λ is small (close to 0), more attention is paid to the impact of neighborhood data fluctuations.

[0131] Studies have shown that traditional methods for quantifying neighborhood data volatility directly use the ratio of the data's standard deviation to the mean of the global data's standard deviation. However, standard deviation is sensitive to outliers, and directly using the ratio of the standard deviation to the mean of the global standard deviation to describe the neighborhood volatility factor can lead to distortion of the volatility factor when noise or outliers are present in the data. Based on this, this application proposes a robustness adjustment parameter in step S3222 above to optimize the calculation of the neighborhood volatility factor. Specifically, a more robust statistical method is used instead of directly using the standard deviation to reduce the impact of outliers (noise or outliers) on the results, and global statistical information is used to correct local volatility, thereby obtaining a more accurate neighborhood volatility factor. For details of the specific scheme, please refer to the following steps S32221-S32224.

[0132] Specifically, in step S3222, the neighborhood fluctuation factor is obtained by analyzing the Euclidean distances of each node in the neighborhood graph to its k nearest Euclidean neighbors, including the following steps:

[0133] Step S32221: Obtain the neighboring nodes corresponding to each node in the neighborhood graph; construct a distance set based on the Euclidean distance between the node and the neighboring nodes;

[0134] Step S32222: Sort each element in the distance set in ascending order of Euclidean distance to obtain a distance sequence set; select the first k elements of the distance sequence set as the baseline distance data; calculate the median distance (MedianD) corresponding to the current node based on the baseline distance data. u and interquartile range (IQRD) u ;

[0135] It should be noted that in the above embodiments of this application, the first k elements in the selected distance sequence set are used as the reference distances. These reference distances are used as the reference data for calculating the neighborhood fluctuation factor, filtering out the noise influence of excessively large distances, so that the subsequent analysis and calculation are more focused on more robust data.

[0136] For example, after the filtering in the above embodiments of this application, the distances of the first 5 neighbors of node u (i.e., the baseline distance data) are: D u =[2.1,3.5,4.0,4.2,15.0]; After calculation and analysis, the median distance MedianD of node u is obtained. u =4.0, interquartile range (IQRD) u =Q3-Q1=4.2-3.5=0.7; In the above embodiments of this application, the median distance is used to replace the data mean in the traditional neighborhood data fluctuation analysis process, reducing the impact of outliers (such as 15.0) on the central trend; and the interquartile range is used to measure the degree of data dispersion, which is more robust to outliers than the standard deviation.

[0137] Step S32223: Based on the median distance MedianD of all nodes in the current neighborhood graph u The global distance median GMadianD (i.e., the median of the global distance median) is obtained by performing statistical analysis.

[0138] For example, if the local medians of the three nodes are 3.0, 4.0, and 5.0, then GMDianD is 4.0. This global statistic reflects the overall data distribution characteristics and avoids the absolute value of local fluctuation factors being affected by the overall scale of the dataset.

[0139] Step S32224: Based on the median distance of the current node MedianD u and interquartile range (IQRD) u The neighborhood fluctuation factor is calculated using the global distance median GMadianD.

[0140] The neighborhood fluctuation factor is calculated as follows:

[0141]

[0142] In the formula, δ is the weight that balances the local and global effects;

[0143] The technical solutions adopted in the above embodiments of this application are illustrated by examples:

[0144] Assuming the neighborhood graph contains 3 nodes, its neighborhood distance data is shown in the table below:

[0145]

[0146] Global statistics were calculated, with the local median set being 4.0, 1.5, and 5.3, resulting in a global median value of 4.0; and the local IQR set being 0.7, 0.6, and 0.4, resulting in a global IQR value of 0.6.

[0147] Furthermore, preliminary calculations of local fluctuations (δ = 0.5) were performed.

[0148] Node u: 0.7 / 4.0 + 0.5*(4.0 / 4.0) = 0.175 + 0.5 = 0.675

[0149] Node v: 0.6 / 1.5 + 0.5*(1.5 / 4.0) ≈ 0.4 + 0.1875 = 0.5875

[0150] Node w: 0.4 / 5.3 + 0.5*(5.3 / 4.0) ≈ 0.075 + 0.6625 = 0.7375

[0151] In summary, the technical solutions adopted in the embodiments of this application above avoid interference from outliers (such as 15.0, 20.0) in fluctuation calculations by using the median and IQR instead of the mean and standard deviation. For example, the neighborhood distance of node u includes 15.0, but MedianD... u =4.0 can still reflect the typical distance; furthermore, the influence of the overall distribution difference of the dataset is eliminated by the global statistic GMadianD. For example, the local median of node v, 1.5, is much smaller than GMadianD = 4.0, but its fluctuation factor is corrected by the global proportional term in the formula.

[0152] Example 2

[0153] On the other hand, this second embodiment, based on the multi-source heterogeneous data integration method based on dynamic metadata modeling provided in the first embodiment of the invention, also provides a storage medium (hereinafter referred to as the storage medium), in which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the above-mentioned multi-source heterogeneous data integration method based on dynamic metadata modeling.

[0154] In summary, this invention presents a method and storage medium for integrating multi-source heterogeneous data based on dynamic metadata modeling. It collects structured data (database tables), semi-structured data (JSON / XML logs), and unstructured data (images, videos) through multiple source interfaces (such as APIs, ETL tools, and file system scans). Then, it extracts metadata from the raw data, capturing data format (e.g., date format YYYY-MM-DD), structure (table relationships), and semantic (field business meaning) information to form a metadata set. This comprehensively captures data characteristics, providing input for subsequent dynamic metadata modeling. The metadata set quantifies data heterogeneity, providing a foundation for conflict detection (naming, type, structure, semantics).

[0155] Furthermore, by standardizing field names (e.g., mapping custID to customer ID), defining common data types (e.g., unifying dates to ISO 8601), constraint rules (e.g., NOT NULL constraints), and semantic tags (e.g., "customer ID"), a unified description of the metadata set is established, creating a metadata dictionary that records field names, types, semantics, and other attributes, thereby ensuring that the resulting database supports cross-system compatibility. Through dynamic model definition of standard naming rules (e.g., semantic similarity matching), design of common mapping schemes (e.g., string-to-integer cleansing rules), merging of multiple tables into a unified schema (e.g., foreign key association of the order table with the customer table), and verification of field meaning consistency based on a business terminology table, data silos are eliminated, a standardized metadata view is provided, and dynamic rules adapt to differences in multi-source data, ensuring the accuracy of subsequent mapping and integration.

[0156] Furthermore, since traditional dimensionality reduction methods (such as MDS) rely on Euclidean distance and cannot capture the nonlinear manifold structure of high-dimensional data, this embodiment calculates the Euclidean distance between unified description fields, selects the top k nearest neighbors of each node to construct a neighborhood graph, and then uses Dijkstra's algorithm to calculate the geodesic distance (the distance along the true path of the manifold) between any two nodes in the neighborhood graph. Then, by optimizing the tuning factor, and then performing conflict quantification (S3221) such as naming conflict (semantic similarity score), type conflict (conversion complexity score), structural conflict (reorganization difficulty score), and semantic conflict (business impact score), a neighborhood fluctuation factor (S3222) is obtained: calculated based on the median and interquartile range (IQR) of the node neighborhood distance to reduce the impact of outliers. Furthermore, through multidimensional scaling analysis (MDS) and optimization (S33-S34), the high-dimensional geodesic distance is mapped to low-dimensional initial coordinates, solving the problem of insufficient modeling of nonlinear manifold structures by traditional methods. The robustness of the shortest path matrix is ​​improved by dynamically adjusting the path cost through the tuning factor.

[0157] Ultimately, by integrating the mapped data, a fused database that supports cross-source queries is generated, eliminating data silos, providing standardized views, and thus supporting efficient data analysis.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; those skilled in the art can modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for integrating multi-source heterogeneous data based on dynamic metadata modeling, characterized in that, The following steps are included: Collect multi-source heterogeneous data; collect metadata from the multi-source heterogeneous data to obtain a metadata set; the metadata set includes the data format, structure, and semantic information of the multi-source heterogeneous data; the multi-source heterogeneous data includes structured data, semi-structured data, and unstructured data; the unstructured data includes documents, images, videos, and social media data; The metadata set is uniformly described based on the dynamic metadata model to obtain unified description field information; Based on the unified description field information, the multi-source heterogeneous data is mapped to obtain mapped data; The mapped data is then integrated to obtain a fused database; The process of mapping the multi-source heterogeneous data based on the unified description field information to obtain mapped data specifically includes the following steps: A neighborhood graph is constructed based on the Euclidean distance between every two unified description fields; each node in the neighborhood graph represents a multi-source heterogeneous data; the line connecting every two nodes represents the Euclidean distance between two unified description fields. Based on the neighborhood graph and the artificial bee colony algorithm, calculate the shortest distance matrix between every two nodes in the graph; Based on the shortest distance matrix, a dual centering process is performed using multidimensional scaling analysis to obtain the low-dimensional mapped coordinates of each node. The low-dimensional mapping coordinates are optimized based on the shortest distance matrix to obtain the mapping data for each node; The calculation of the shortest distance matrix between every two nodes in the neighborhood graph based on the artificial bee colony algorithm includes the following steps: Obtain data conflict information during the unified description process of the metadata set; Based on the data conflict information and the Euclidean distance from each node to its k nearest neighbors in the neighborhood graph, the optimization factor of the path segment between each pair of adjacent nodes is obtained. Each node is traversed, and starting from a given node and ending at any node other than the current node, multiple initial paths from the starting point to the ending point are obtained. No duplicate nodes exist in these initial paths. The edge distances of each initial path are obtained. The path fitness is calculated based on the edge distances and the optimization factor. Each initial path is then traversed. Based on the optimization factors of each path segment in the current initial path and the optimization factors of the path segments between each neighboring node of the current initial path, the path segments are replaced to obtain the second path; based on the new fitness of the second path and the fitness of the original initial path, the third path is obtained by making a judgment and replacement. The shortest distance matrix is ​​obtained by replacing and optimizing the fitness of the third path according to the distribution of the tuning factors.

2. The method for integrating multi-source heterogeneous data based on dynamic metadata modeling according to claim 1, characterized in that, The process of determining and replacing the third path based on the new fitness of the second path with the fitness of the original initial path includes the following steps: The new fitness is calculated based on the second path; When the new fitness is greater than the path fitness, the current second path replaces the current initial path to obtain the third path.

3. The method for integrating multi-source heterogeneous data based on dynamic metadata modeling according to claim 2, characterized in that, The shortest distance matrix is ​​obtained by replacing and optimizing the fitness based on the third path according to the distribution of the tuning factor, and includes the following steps: The fitness of all third paths is calculated, and a high-potential path is selected by roulette wheel probability allocation. Based on the optimization factor distribution of the high-potential path, the optimized third path is obtained by replacing the path segment with the highest optimization factor with the path segment with the lowest optimization factor among the path segments from each node to each node's neighboring nodes. Determine whether the tuning factor of each path segment in the optimized third path is less than the preset tuning factor threshold. If so, then the optimized third path is determined as the target path; The shortest distance matrix is ​​obtained based on all the target paths.

4. The method for integrating multi-source heterogeneous data based on dynamic metadata modeling according to claim 3, characterized in that, The process of analyzing and obtaining the optimization factor for the path segment between each pair of adjacent nodes based on the data conflict information and the Euclidean distance from each node to its k nearest neighbors in the neighborhood graph includes the following steps: The data conflict information is subjected to conflict quantification detection processing to obtain the conflict quantification score of each data conflict information; the metadata conflict factor is calculated based on the conflict quantification score; The neighborhood fluctuation factor is obtained by analyzing the Euclidean distances from each node to its k nearest Euclidean neighbors in the neighborhood graph. The tuning factor is obtained by comprehensively calculating the neighborhood fluctuation factor and the metadata conflict factor. .

5. The method for integrating multi-source heterogeneous data based on dynamic metadata modeling according to claim 4, characterized in that, Also includes: Neighborhood fluctuation factor of two nodes based on a path segment formed by two adjacent nodes , Calculate the average neighborhood volatility factor ; Based on the aforementioned metadata conflict factor The average value of the neighborhood fluctuation factor is obtained through the balance parameter. The optimization factor is obtained by summing the results after weighing the factors. ; The tuning factor The calculation method is as follows: ； In the formula, The optimization factor is the path segment between node u and node v; The metadata conflict factor between node u and node v; Let be the neighborhood fluctuation factor of node u; Let v be the neighborhood fluctuation factor of node v; These are the balancing parameters.

6. The method for integrating multi-source heterogeneous data based on dynamic metadata modeling according to claim 5, characterized in that, The neighborhood fluctuation factor is obtained by analyzing the Euclidean distances from each node to its k nearest Euclidean neighbors in the neighborhood graph, and includes the following steps: Obtain the neighboring nodes corresponding to each node in the neighborhood graph; construct a distance set based on the Euclidean distance between the node and the neighboring nodes; Sort each element in the distance set in ascending order of Euclidean distance to obtain a distance sequence set; select the first k elements of the distance sequence set as the baseline distance data; calculate the median distance corresponding to the current node based on the baseline distance data. and interquartile range (IQR) ; Median is based on the median distance of all nodes in the current neighborhood graph. The global distance median GMDian was obtained through statistical analysis. ; Median based on the distance from the current node and interquartile range (IQR) and the global distance median GMDian The neighborhood fluctuation factor is calculated.

7. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-source heterogeneous data integration method based on dynamic metadata modeling as described in any one of claims 1-6.

Images