Multi-source heterogeneous data integration method based on dynamic metadata modeling and storage medium

Abstract

The invention discloses a multi-source heterogeneous data integration method based on dynamic metadata modeling and a storage medium, and the method comprises the steps: collecting structured data, semi-structured data and unstructured data from multi-source heterogeneous data, extracting the format, structure and semantic information of the data, forming a metadata set, and storing the metadata set in a database; an original basis is provided for subsequently solving naming conflicts and data type differences; data features are comprehensively captured, and input is provided for dynamic metadata modeling; further, through a dynamic metadata model, uniformly describing the metadata set to obtain uniform description information; further, based on dynamic mapping and dimension reduction optimization of a neighborhood graph, high-dimensional data are mapped to a low-dimensional space, and a global structure is reserved; and finally, integrating and generating a fusion database supporting cross-source query based on the data subjected to dimension reduction, eliminating data islands, and providing a standardized view.

Description

Technical Field

[0001] The present invention relates to the field of multi-source heterogeneous data processing, and in particular to a multi-source heterogeneous data integration method and storage medium based on dynamic metadata modeling. Background Art

[0002] Enterprises today face an explosion of data sources (e.g., CRM, ERP, IoT devices, social media), but this data is scattered across disparate systems, creating "data silos." For example, a retail company might need to integrate point-of-sale (POS) systems (structured), user behavior logs (semi-structured), and social media comments (unstructured) to analyze user preferences. However, integrating heterogeneous data from multiple sources faces the following technical drawbacks:

[0003] Traditional methods rely on static metadata models, making it difficult to adapt to dynamic changes in data sources (such as added fields and format updates). For example, when "customer ID" is named "custID" or "customer_id" in different systems, the static model cannot adapt to standardized naming rules, leading to naming conflicts and semantic ambiguity. In addition, differences in data types (such as the date format "YYYY-MM-DD" versus "MM / DD / YYYY") and structural differences (such as order data scattered across multiple tables) require manual intervention in mapping, which is inefficient and prone to introducing errors.

[0004] On the other hand, in the process of integrating multi-source heterogeneous data, integrating multi-source heterogeneous data after dimensionality reduction is a powerful means to improve data analysis and integration. However, existing technologies often use Euclidean distance to directly calculate data similarity and rely on linear dimensionality reduction methods such as multidimensional scaling (MDS). However, multi-source heterogeneous data often present nonlinear manifold structures (such as Swiss roll distribution), and their true associations need to be described by geodesic distance (the shortest path along the manifold). However, Euclidean distance will incorrectly compress or stretch local structures, resulting in distortion of data relationships in the low-dimensional space after mapping. For example, the time series event stream in the user behavior log may lose local clustering features due to linear dimensionality reduction, affecting the accuracy of subsequent analysis. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-source heterogeneous data integration method based on dynamic metadata modeling, which solves the above-mentioned technical problems pointed out in the prior art.

[0006] The present invention provides a multi-source heterogeneous data integration method based on dynamic metadata modeling, comprising the following steps:

[0007] Collecting and acquiring multi-source heterogeneous data; collecting metadata for the multi-source heterogeneous data to obtain a metadata set;

[0008] Performing a unified description of the metadata set based on a dynamic metadata model to obtain unified description field information;

[0009] Performing mapping processing on the multi-source heterogeneous data based on the unified description field information to obtain mapping data;

[0010] Integration is performed based on the mapping data to obtain a fusion database.

[0011] Preferably, the metadata set includes data format, structure and semantic information of multi-source heterogeneous data;

[0012] The multi-source heterogeneous data includes structured data, semi-structured data and unstructured data.

[0013] Preferably, the mapping process is performed on the multi-source heterogeneous data based on the unified description field information to obtain the mapping data, specifically comprising the following steps:

[0014] A neighborhood graph is constructed based on the Euclidean distance between each two unified description field information; each node in the neighborhood graph represents a multi-source heterogeneous data; and the line between each two nodes represents the Euclidean distance between the two unified description field information;

[0015] Calculate the shortest distance matrix between every two nodes in the graph based on the neighborhood graph combined with the artificial bee colony algorithm;

[0016] Applying multidimensional scaling analysis to perform dual centering processing based on the shortest distance matrix to obtain low-dimensional mapping coordinates of each node;

[0017] The low-dimensional mapping coordinates are optimized based on the shortest distance matrix to obtain mapping data for each node.

[0018] Preferably, the calculation of the shortest distance matrix between every two nodes in the graph based on the neighborhood graph combined with the artificial bee colony algorithm includes the following steps:

[0019] Obtaining data conflict information during the unified description process of the metadata set;

[0020] Analyze the data conflict information and the Euclidean distances from each node to the nearest k neighboring nodes in the neighborhood graph to obtain a tuning factor for the path segment between each two adjacent nodes;

[0021] Traversing each of the nodes, taking the node as the starting point and any node other than the current node as the end point, and combining them to obtain multiple initial paths from the starting point to the end point; there are no repeated nodes in the initial paths;

[0022] Obtaining the edge distance of each initial path; calculating the path fitness based on the edge distance and the tuning factor;

[0023] Traversing each of the initial paths; replacing the path segments based on the tuning factors of each path segment in the current initial path and the tuning factors of the path segments between the end point and each neighboring node of the end point of the current initial path to obtain a second path; and determining and replacing the second path based on the new fitness of the second path and the fitness of the original initial path to obtain a third path;

[0024] The fitness of the third path is replaced and optimized according to the distribution of the tuning factors to establish a shortest distance matrix.

[0025] Preferably, the determining and replacing the new fitness of the second path with the fitness of the original initial path to obtain the third path comprises the following steps:

[0026] Calculating based on the second path to obtain a new fitness;

[0027] When the new fitness is greater than the path fitness, the current second path replaces the current initial path to obtain a third path.

[0028] Preferably, the fitness based on the third path is replaced and optimized according to the distribution of the tuning factors to establish the shortest distance matrix, which includes the following steps:

[0029] Counting the fitness of all third paths, and selecting a high-potential path using a roulette wheel probability distribution method; obtaining an optimized third path by replacing the path segments with high tuning factors with the path segments with the lowest tuning factors among the path segments from each node to the neighboring nodes of each node based on the distribution of tuning factors in the high-potential paths;

[0030] Determining whether a tuning factor of each path segment in the optimized third path is less than a preset tuning factor threshold; if so, determining that the optimized third path is the target path;

[0031] A shortest distance matrix is ​​established based on all the target paths.

[0032] Preferably, the step of analyzing the data conflict information in combination with the Euclidean distances from each node to the nearest k neighboring nodes in the neighborhood graph to obtain the tuning factor of the path segment between each two adjacent nodes comprises the following steps:

[0033] Performing conflict quantification detection processing on the data conflict information to obtain conflict quantification scores for each piece of data conflict information; and calculating metadata conflict factors based on the conflict quantification scores;

[0034] A neighborhood fluctuation factor is obtained by analyzing the Euclidean distances of each node in the neighborhood graph to the first k neighboring nodes with the closest Euclidean distances;

[0035] A tuning factor α is obtained by comprehensive calculation based on the neighborhood fluctuation factor and the metadata conflict factor. uv .

[0036] Preferably, the multi-source heterogeneous data integration method based on dynamic metadata modeling also includes:

[0037] The neighborhood fluctuation factor V of two nodes based on the path segment composed of two adjacent nodes u 、V v Calculate the average value of neighborhood fluctuation factor

[0038] Based on the metadata conflict factor C uv The average values ​​of the neighborhood fluctuation factors are weighed by the balance parameter λ and then summed to obtain the tuning factor α uv ;

[0039] The tuning factor α uv The calculation method is:

[0040]

[0041] Where, α uv is the tuning factor of the path segment between node u and node v; C uv V is the metadata conflict factor between node u and node v; u is the neighborhood fluctuation factor of node u; V v is the neighborhood fluctuation factor of node v; λ is the balance parameter.

[0042] Preferably, the neighborhood fluctuation factor is obtained by analyzing the Euclidean distance from each node to the first k neighboring nodes with the closest Euclidean distance in the neighborhood graph, comprising the following steps:

[0043] Obtaining neighboring nodes corresponding to each node in the neighborhood graph; constructing a distance set based on the Euclidean distance between the node and the neighboring node;

[0044] Sort the elements in the distance set from low to high according to the Euclidean distance to obtain a distance sequence set; select the first k elements of the distance sequence set as the benchmark distance data; calculate the median distance corresponding to the current node based on the benchmark distance data u and interquartile range IQRD u ;

[0045] Based on the median distance of all nodes in the current neighborhood graph, MedianDu Perform statistics to obtain the global median distance GMedianD;

[0046] Based on the median distance of the current node MedianD u and interquartile range IQRD u And the global distance median GMedianD are used to calculate the neighborhood fluctuation factor.

[0047] In another aspect, the present invention further provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned multi-source heterogeneous data integration method based on dynamic metadata modeling.

[0048] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0049] From the analysis of the above-mentioned multi-source heterogeneous data integration method and storage medium based on dynamic metadata modeling provided by the present invention, it can be seen that the method includes: collecting and acquiring multi-source heterogeneous data; collecting metadata on the multi-source heterogeneous data to obtain a metadata set; uniformly describing the metadata set based on a dynamic metadata model to obtain unified description field information; mapping the multi-source heterogeneous data based on the unified description field information to obtain mapping data; and integrating based on the mapping data to obtain a fusion database.

[0050] In specific applications, the above integration method collects structured data (such as order tables), semi-structured data (such as JSON logs), and unstructured data (such as images and videos) from multi-source heterogeneous systems (such as CRM, ERP, IoT devices, and social media). It extracts the data's format (such as the date format "YYYY-MM-DD"), structure (such as single-table or multi-table associations), and semantic information (such as the business meaning of fields) to form a metadata collection. This provides the original basis for subsequently resolving naming conflicts (such as "custID" vs. "customer_id") and data type differences (such as string and integer conversion).

[0051] By comprehensively capturing data features, input is provided for dynamic metadata modeling; further, through the dynamic metadata model, the metadata collection is uniformly described to obtain unified description information (i.e., unified description field information); based on the unified description field information, the multi-source heterogeneous data is mapped to obtain mapped data. In specific operations, based on the dynamic mapping and dimensionality reduction optimization of the neighborhood graph, high-dimensional data is mapped to a low-dimensional space to retain the global structure; finally, based on the dimensionality-reduced data, integration is performed to generate a fusion database that supports cross-source queries, eliminating data silos and providing standardized views (such as unified date formats and semantically consistent table structures). BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of the main process of a multi-source heterogeneous data integration method based on dynamic metadata modeling;

[0053] Figure 2 A topological diagram of the solution process for obtaining mapping data in a multi-source heterogeneous data integration method based on dynamic metadata modeling;

[0054] Figure 3 A schematic diagram of neighborhood graph simulation in a multi-source heterogeneous data integration method based on dynamic metadata modeling;

[0055] Figure 4 A schematic diagram of the shortest distance path simulation in a multi-source heterogeneous data integration method based on dynamic metadata modeling;

[0056] Figure 5 Schematic diagram of a Swiss roll dataset simulation in a multi-source heterogeneous data integration method based on dynamic metadata modeling. DETAILED DESCRIPTION

[0057] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] The present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0059] Example 1

[0060] like Figure 1 As shown, the first embodiment of the present invention provides a multi-source heterogeneous data integration method based on dynamic metadata modeling, including the following steps:

[0061] Step S10: Acquire multi-source heterogeneous data; perform metadata acquisition on the multi-source heterogeneous data to obtain a metadata set; the metadata set includes data format, structure, and semantic information of the multi-source heterogeneous data; the multi-source heterogeneous data includes structured data (such as order data in a database table), semi-structured data (such as server logs, data in XML or JSON format), and unstructured data (such as documents, pictures, videos, social media data, etc.);

[0062] Step S20: performing a unified description on the metadata set based on a dynamic metadata model to obtain unified description field information;

[0063] It should be noted that the unified description field information in the above embodiments of the present application refers to a standardized and consistent description of each data element (such as a field or attribute) in multi-source heterogeneous data under the dynamic metadata model framework. This description generally includes but is not limited to the following aspects:

[0064] Data field name: defines the name of the data element for easy identification and reference;

[0065] Type: Indicates the type of data element, such as integer, floating point, string, date and time, etc.

[0066] Constraint rules: specify some conditions or restrictions that data elements must follow, such as length limit, value range, whether null values ​​are allowed, etc.

[0067] Semantic labeling: attaching semantic information to data elements to help understand their meaning and purpose; for example, a field might be labeled "Customer ID" or "Order Date";

[0068] Specific unified description methods, for example, for the same or similar data items from different data sources, a standard naming rule is established through the dynamic metadata model. For example, the customer ID may be "customer_id" in some systems and "custID" in other systems. In this case, the dynamic metadata model is used to select a standard name and ensure that all related data use this name. On the other hand, different data sources may have different representations for the same data type (such as date format). The dynamic metadata model is used to define a common data type mapping scheme to convert data types from all sources into corresponding types in the unified model. Furthermore, consistent constraint rules are defined for each data element, including but not limited to length restrictions, value ranges, whether null values ​​are allowed, etc., to help ensure consistency in data quality. Furthermore, semantic labels are added to each data element to help understand its meaning and purpose. These can be based on industry standards or terms agreed upon within the enterprise.

[0069] Additionally, there may be consistency issues and conflicts within metadata collections, including:

[0070] Naming conflicts: For example, the same entity may have different names in different systems;

[0071] Data type differences: The same information may be stored as different types of data (such as dates) in different systems;

[0072] Structural differences: Data is organized differently, for example, order information is contained in a single table in one system, while in another system it is distributed across multiple related tables;

[0073] Semantic differences: Even if the name and type are the same, different systems may interpret the data differently;

[0074] Based on this, a set of clear, easy-to-understand and easy-to-use naming conventions is established through the dynamic metadata model; the mapping relationship between different types is defined to ensure cross-system compatibility; a standard model for describing data structures, including field names, types, constraints and other information, is created; and a metadata dictionary containing all data elements and their attributes is created as a reference.

[0075] After developing the aforementioned naming conventions, mapping relationships, standard schemas for describing data structures, and metadata dictionaries, each metadata set is mapped through a dynamic metadata model to obtain unified descriptive field information. Specifically, each data source is analyzed in detail to understand its data structure, type, semantics, and other characteristics. Based on the analysis results, specific conversion rules from raw data to the unified model are designed. ETL tools or custom scripts are used to implement the aforementioned rules, converting raw data into a unified format. Before actual application, tests are conducted to check whether the converted data meets expectations, and the conversion logic is adjusted based on feedback.

[0076] Through the above-mentioned embodiments of the present application, a dynamic metadata model is used to extract valuable information from multi-source heterogeneous data and integrate it into a unified data model, thereby supporting higher-level data analysis and business decision-making.

[0077] Step S30: mapping the multi-source heterogeneous data based on the unified description field information to obtain mapping data;

[0078] It should be noted that the above-mentioned embodiment of the present application converts the metadata (uniform description field information) that has undergone unified description into a structured representation (mapping data) in a low-dimensional space through a dynamic mapping algorithm for subsequent integration; its core is to solve the problem of inaccurate high-dimensional space mapping caused by structural differences, semantic conflicts, etc. of multi-source heterogeneous data through neighborhood graph modeling, shortest path optimization and dimensionality reduction technology (for details, please refer to the subsequent implementation process);

[0079] Step S40: Integrate based on the mapping data to obtain a fusion database.

[0080] It should be noted that the above embodiment of the present application integrates the optimized mapping data (low-dimensional representation) into a consistent database according to a unified model to support cross-source query and analysis; specifically, through data alignment (i.e., solving consistency problems), conflict resolution, structural fusion and storage optimization processing, wherein data alignment is based on the similarity of low-dimensional coordinates (such as Euclidean distance <0.5), mapping fields from different sources to a unified model (such as classifying "custID" and "customer_id" as "customer ID"), according to the rules defined by the dynamic metamodel (such as the date format is unified to ISO 8601); conflict resolution defines the authority of data sources (e.g., "sales" in system A takes precedence over "sales" in system B); structural fusion combines dispersed table structures (e.g., an order table split between systems A and B) into a unified schema and links different tables via foreign keys (e.g., a customer table is linked to an order table); storage optimization creates indexes for frequently queried fields (e.g., "order date") and partitions them by time or business unit to improve query efficiency; based on the above data integration, a fused database is created to uniformly manage multi-source heterogeneous data. By eliminating data silos and resolving differences in data format, structure, and semantics, it provides a standardized and consistent data view, supporting efficient cross-system query and analysis.

[0081] The above-mentioned embodiments of the present application collect structured data (such as order tables), semi-structured data (such as JSON logs) and unstructured data (such as pictures and videos) from multi-source heterogeneous systems (such as CRM, ERP, IoT devices, and social media), extract the data format (such as the date format "YYYY-MM-DD"), structure (such as single table or multi-table association) and semantic information (such as the business meaning of the field), and form a metadata set to provide the original basis for the subsequent resolution of naming conflicts (such as "custID" vs. "customer_id") and data type differences (such as string and integer conversion); by comprehensively capturing data features, it provides input for dynamic metadata modeling; further, through the dynamic metadata model, the metadata set is uniformly described to obtain unified description information; further, based on the dynamic mapping and dimensionality reduction optimization of the neighborhood graph, high-dimensional data is mapped to a low-dimensional space to retain the global structure; finally, based on the dimensionality-reduced data, it is integrated to generate a fusion database that supports cross-source queries, eliminates data silos, and provides standardized views (such as unified date formats and semantically consistent table structures).

[0082] Multi-source heterogeneous data (such as structured, semi-structured, and unstructured) has high dimensions and complex structures. Direct processing has high computational costs and is susceptible to noise interference. Traditional dimensionality reduction methods (such as MDS) rely on Euclidean distance and cannot capture the manifold structure of the data (such as the curl distribution of SwissRoll).

[0083] Specifically, if Figure 2As shown, in step S30, mapping processing is performed on the multi-source heterogeneous data based on the unified description field information to obtain mapping data, which specifically includes the following steps:

[0084] Step S31: constructing a neighborhood graph based on the Euclidean distance between each two unified description field information; each node in the neighborhood graph represents a multi-source heterogeneous data; and the line between each two nodes represents the Euclidean distance between the two unified description field information;

[0085] It should be noted that the neighborhood graph in the above embodiment of the present application is constructed by calculating the Euclidean distance between each two unified description field information, and then selecting the first k unified description field information with the closest Euclidean distance to each unified description field information, and jointly constructing the neighborhood graph (such as Figure 3 shown);

[0086] Step S32: Calculating the shortest distance matrix between every two nodes in the graph based on the neighborhood graph combined with the artificial bee colony algorithm;

[0087] Explanation: In the above embodiment of the present application, the shortest distance path between each two nodes is first obtained, and then the shortest distance matrix is ​​constructed by all the shortest distance paths; Figure 4 As shown, the shortest distance path is the shortest path distance among multiple path options, that is, Figure 4 The thick black line in the middle is the shortest distance path from node 1 to node 3.

[0088] Step S33: applying multidimensional scaling analysis to perform dual centralization based on the shortest distance matrix to obtain low-dimensional mapping coordinates of each node;

[0089] It should be noted that the above-mentioned embodiment of the present application performs doubly centralization on the calculated shortest distance matrix through MDS (multidimensional scaling analysis) to obtain the initial coordinates of each data node in the low-dimensional space; the original high-dimensional distance relationship is converted into the low-dimensional space to achieve data simplification and improve the efficiency of subsequent analysis.

[0090] Step S34: Optimizing the low-dimensional mapping coordinates based on the shortest distance matrix to obtain mapping data for each node.

[0091] It should be noted that in the above embodiment of the present application, in step S33, the shortest distance matrix is ​​double-centered by multidimensional scaling (MDS), thereby obtaining the initial coordinates of each data node in the low-dimensional space. However, since multi-source heterogeneous data may contain complex nonlinear relationships and local structures, it is difficult for MDS to fully retain such local and global intrinsic structural information during the dimensionality reduction process, resulting in certain errors in the mapping.

[0092] Specifically, it refers to the surface, manifold structure, and local dense clustering relationships presented by data in high-dimensional space. These relationships often need to be accurately described using geodesic distance rather than simple Euclidean distance. At the same time, the dimensionality reduction method of MDS is mainly based on Euclidean distance and global optimization. When faced with the nonlinear intrinsic structure and local detailed relationships of the data, it is easy to ignore or distort this information, resulting in errors in the local or global structure restoration after dimensionality reduction.

[0093] Let’s take the SwissRoll dataset as an example to illustrate: Figure 5 As shown in Figure 2, the SwissRoll dataset is a classic nonlinear manifold data set, whose data points present a three-dimensional "curled" structure formed by bending a two-dimensional plane (like a piece of paper).

[0094] In the original three-dimensional space, the data points are distributed along a curled two-dimensional manifold. Two data points may be close in terms of Euclidean distance, but if the actual path along the manifold (surface) is considered, the actual distance between them may be much larger. When using MDS to reduce the SwissRoll data to two dimensions, MDS attempts to preserve the straight-line Euclidean distance between all pairs of points. Due to the nonlinear inherent structure of SwissRoll, the true relationship of the data is the geodesic distance calculated along the manifold. Directly using Euclidean distance will result in local information (such as the relative position of the points within the manifold) not being fully reflected, resulting in local structural distortion. That is, the local neighboring relationship of data points on the original manifold (along the curve of the manifold) cannot be captured by simple straight-line Euclidean distance. As a result, the two-dimensional result after dimensionality reduction may show that the distances between local point groups are too close or too far, which is inconsistent with their true distribution on the original manifold. At the same time, when the entire curled structure is flattened, MDS may fail to fully restore the global topology of the original manifold, resulting in the distance relationship in some areas being stretched or compressed, and failing to faithfully reflect the intrinsic Euclidean geometric structure of the data.

[0095] Based on this, the embodiment of the present application optimizes the low-dimensional coordinates by using the shortest distance matrix in step S34. The shortest distance matrix can more accurately capture the true relationship of data in high-dimensional space. Through this optimization method based on global and local structural information, the accuracy of low-dimensional mapping is improved, thereby achieving higher-quality data integration and the construction of a fusion database.

[0096] The optimization in the above S34 is specifically based on the shortest distance matrix, and minimizes the difference between the low-dimensional Euclidean distance and the true geodesic distance (i.e., the shortest path distance) through iterative optimization (such as gradient descent and other methods), and then corrects the initial MDS result to ensure that the low-dimensional mapping not only maintains the rationality of the data distribution globally, but also accurately reflects the nonlinear structure of the high-dimensional data in local details, thereby achieving higher quality data integration and fusion. The above-mentioned optimization methods such as gradient descent are common knowledge to those skilled in the art and will not be repeated in this application.

[0097] The above embodiment of the present application calculates the Euclidean distance between all unified description field information, selects the first k nearest neighbor nodes of each node, constructs a neighborhood graph, and uses the neighborhood graph to retain the local proximity relationship of data points to provide local structural information for subsequent dimensionality reduction; at the same time, the Euclidean distance between nodes in the neighborhood graph reflects the similarity between fields (such as naming, type, semantic difference), providing a basis for subsequent path calculation; further, based on the neighborhood graph, the geodesic distance (i.e., the path distance along the manifold) between any two nodes is calculated by the shortest path algorithm (such as the Dijkstra algorithm); the geodesic distance accurately describes the true relationship of data in the high-dimensional manifold, solving the problem that the Euclidean distance cannot capture nonlinear structures, for example, the SwissRoll dataset The curled structure is preserved by the geodesic distance, avoiding the "straight-line distance" deviation of the Euclidean distance; the embodiment of the present application uses multidimensional scaling analysis (MDS) to perform doubly centralization on the shortest distance matrix to map the high-dimensional distance relationship to the low-dimensional space; MDS approximately preserves the relative distance relationship of high-dimensional data to the low-dimensional space by optimizing the objective function, provides low-dimensional initial coordinates for subsequent optimization steps, and reduces computational complexity; further, the difference between the low-dimensional Euclidean distance and the high-dimensional geodesic distance is minimized through iterative optimization (such as gradient descent), the error of MDS is corrected, and the shortest distance matrix (true geodesic distance) is used as a reference to correct the local structural distortion of MDS caused by global optimization, ensuring that the low-dimensional mapping retains both the rationality of the global distribution and the local manifold structure.

[0098] Specifically, in step S32, the shortest distance matrix between every two nodes in the graph is calculated based on the neighborhood graph in combination with the artificial bee colony algorithm, including the following steps:

[0099] Step S321: obtaining data conflict information during the unified description process of the metadata set;

[0100] Step S322: Analyze the data conflict information and the Euclidean distances from each node to the nearest k neighboring nodes in the neighborhood graph to obtain a tuning factor for the path segment between each two adjacent nodes;

[0101] It should be noted that the above tuning factors are derived from the statistical data and deviation analysis generated during the unified description, metadata collection, and mapping of multi-source heterogeneous data in stages S10–S31. These analysis results typically reflect issues such as data format differences, mapping errors, and data migration fluctuations. Therefore, this factor can be considered a quantitative description of "data inconsistency" or "data migration risk."

[0102] During the subsequent implementation of the solution, the total cost of the path (or the total cost of the path used in the fitness calculation) is calculated by adding the costs of each path segment between adjacent nodes, while the properties of the node itself (such as data quality and mapping accuracy) are reflected in the edges connecting the nodes. Therefore, by setting a corresponding tuning factor for each edge (i.e., path segment), the cost accumulation caused by data transmission or mapping errors can be more intuitively corrected. If there is a problem with a node, the tuning factor of the edge related to the node will be increased accordingly, thereby affecting the comprehensive evaluation of the entire path.

[0103] Step S323: traverse each of the nodes, taking the node as the starting point and any node other than the current node as the end point, and combine them to obtain multiple initial paths from the starting point to the end point; there is no repeated node in the initial path;

[0104] Step S324: Obtaining the edge distance of each initial path (the edge distance is the sum of the Euclidean distances between each two adjacent nodes in the initial path); calculating the path fitness based on the edge distance and the tuning factor;

[0105] The calculation method of the above path fitness is:

[0106]

[0107] Where, Fitness(Path) is the path fitness of the initial path; d uv is the Euclidean distance between node u and node v (through the summation formula, the edge distance of the entire initial path is obtained); α uv is the optimization factor for the shortest path between node u and node v; d uv ·(1+α uv ) is the shortest path segment cost between node u and node v;

[0108] Step S325: traverse each of the initial paths; replace path segments based on the tuning factors of each path segment in the current initial path and the tuning factors of the path segments between the end point and each neighboring node of the end point of the current initial path to obtain a second path; calculate a new fitness based on the second path; and replace the current initial path with the current second path when the new fitness is greater than the path fitness (if the new fitness is less than or equal to the path fitness, retain the current initial path and determine it as the third path, and filter out the second path obtained by the current replacement), to obtain a third path;

[0109] It should be noted that in the above embodiment of the present application, by traversing all current initial paths, and then in the process of processing each initial path, the tuning factors of each path segment in the current initial path (the tuning factors of the path segments between each two nodes are calculated in step S322) are selected to obtain a tuning factor to be processed whose tuning factor is higher than the average value of the tuning factors of each path segment in the current initial path, and at the same time, each neighboring node from the end point to the end point in the current initial path (the node adjacent to the end point) is selected to obtain a path segment with the lowest tuning factor (that is, the neighboring node corresponding to the path segment with the lowest tuning factor among the multiple path segments from the end point to the end point is selected), and the neighboring node corresponding to the lowest tuning factor among the neighboring nodes from the end point to the end point is replaced with the path segment corresponding to the above-mentioned tuning factor to be processed. For example:

[0110] For example, the initial path A→B→D contains path segments A→B and B→D, where the tuning factor of each path segment is a AB =0.7, a BD = 0.4; randomly select an edge with a tuning factor higher than the average (such as A→B), extract the first k neighbor nodes (such as C, E) from the neighborhood of the target edge endpoint (such as B); calculate the tuning factor of the candidate edge (such as a BC =0.3, a BE =0.5); select the candidate edge with the lowest tuning factor (such as B→C); concatenate the new path: A→B→C→D (make sure there are no duplicate nodes); then calculate the new fitness of the second path. For example, if it is 0.1398, and the path fitness of the original initial path is 0.1316, then the current second path will replace the current initial path (i.e., the new path after the replacement path segment replaces the original path).

[0111] Step S326: performing replacement optimization based on the fitness of the third path according to the distribution of the tuning factors to establish a shortest distance matrix;

[0112] It should be noted that the above-mentioned embodiment of the present application obtains a tuning factor by analyzing data conflict information, and then uses the tuning factor to optimize the shortest distance matrix to prevent the problem of data distribution changes caused by structural differences in metadata or unstable distance calculation caused by inconsistent metadata, thereby improving the analysis accuracy of the shortest distance matrix, thereby making the subsequent establishment of a fusion database more robust.

[0113] Specifically, in step S326, the shortest distance matrix is ​​established after performing replacement optimization based on the fitness of the third path according to the distribution of the tuning factors, including the following operation steps:

[0114] Step S3261: Count the fitness of all third paths (the current initial path is the final third path obtained in S321-S325 above), and select a high-potential path using a roulette wheel probability distribution method; based on the distribution of tuning factors in the high-potential paths, replace the path segments with high tuning factors with the path segments with the lowest tuning factors among the path segments from each node to the neighboring nodes of each node, to obtain an optimized third path;

[0115] It should be noted that the above-mentioned embodiment of the present application first counts the fitness of all third paths (for example, the fitness of path 1 is 0.2, the fitness of path 2 is 0.3, and the fitness of path 3 is 0.5); a high-potential path (such as path 3A→C→D) is selected by roulette probability allocation; further, by counting the tuning factor distribution in the high-potential path (such as A→C (α=0.3), C→D (α=0.2)); the path segment with the highest tuning factor (i.e., A→C (α=0.3)) is selected, and an edge with a lower tuning factor (such as A→E, α=0.1) is selected from the neighborhood of node A; thereby generating a new path A→E→C→D (i.e., the optimized third path); the fitness of the new path A→E→C→D is calculated again, and when the fitness is higher than the fitness of the original path (the original path is the above-mentioned high-potential path 3A→C→D), the original path is replaced to obtain the optimized third path A→E→C→D.

[0116] Step S3262: Determine whether the tuning factor of each path segment in the optimized third path is less than a preset tuning factor threshold; if so, determine that the optimized third path is the target path (the target path is the shortest distance path in the explanation of step S32);

[0117] Step S3263: Establishing a shortest distance matrix based on all the target paths.

[0118] It should be noted that the technical solution adopted in the above-mentioned embodiment of the present application, the artificial bee colony algorithm is generally used for optimization problems, including path finding. However, directly using the artificial bee colony algorithm to find the shortest path distance between each two nodes in the neighborhood graph will cause data distribution changes due to structural differences in metadata or unstable distance calculation due to inconsistent metadata, resulting in the optimization processing results using the artificial bee colony algorithm not being accurate enough. Therefore, the above-mentioned embodiment of the present application analyzes the conflict types (such as naming conflicts, data type differences, structural differences or semantic differences) in the metadata unification process in each data source (i.e., the above-mentioned nodes) to obtain a tuning factor (the analysis and acquisition of the tuning factor, see the subsequent operation steps S3221-step S3223 for details), and uses the tuning factor to apply to the optimization processing process of the artificial bee colony algorithm, so that the optimization processing results of the artificial bee colony algorithm are more accurate, thereby obtaining a more accurate shortest path distance.

[0119] Specifically, in step S322, based on the data conflict information and the Euclidean distances from each node to the nearest k neighboring nodes in the neighborhood graph, an analysis is performed to obtain a tuning factor for the path segment between each two adjacent nodes, including the following steps:

[0120] Step S3221: performing conflict quantization detection processing on the data conflict information to obtain conflict quantization scores for each piece of data conflict information; and calculating metadata conflict factors based on the conflict quantization scores;

[0121] Specifically, the above-mentioned conflict quantitative detection and processing of data conflict information refers to the use of natural language processing (NLP) tools to calculate the semantic similarity of field names for naming conflicts; for type conflicts, compare field data types and conversion rule libraries; for structural conflicts, analyze the table relationships and nested levels of the data model; for semantic conflicts, verify the consistency of field meanings in combination with the business term table; specifically, for naming conflicts, based on the semantic distance of the name differences (such as completely unrelated = 1, partial match = 0.5, synonym = 0.2), for example, custID and customer_id are semantically similar → score 0.3; for type conflicts, based on the type conversion complexity (such as string Integer = 0.5, Date format conversion = 0.8, No automatic conversion possible = 1); for example, string to integer conversion requires format cleaning → score 0.6; for structural conflicts, based on the difficulty of data model reorganization (such as single table → multi-table association = 0.7, nested structure expansion = 0.9); for example, order information is split from a single table to multiple tables → score 0.8; for semantic conflicts, based on the degree of business impact (such as key indicator ambiguity = 1, secondary field ambiguity = 0.3), for example, whether sales include or exclude tax affects financial calculations → score 0.9.

[0122] Step S3222: Analyze the Euclidean distances from each node in the neighborhood graph to the first k neighboring nodes with the closest Euclidean distances to obtain a neighborhood fluctuation factor;

[0123] It should be noted that the above embodiment of the present application first counts the Euclidean distance between each node in the neighborhood graph and each neighboring node, then sorts the Euclidean distances from low to high, selects the first k target neighborhood nodes, and finally, calculates the standard deviation based on the Euclidean distance between the current node and the target neighborhood node, and then uses the ratio of the current standard deviation to the global standard deviation mean calculated for all nodes to calculate the neighborhood fluctuation factor; specifically, for example, there are 3 nearest neighbors in the neighborhood graph of node u, and the Euclidean distances are 1.2, 1.5, and 1.8 respectively; the standard deviation of node u is calculated to be approximately 0.245; further, the mean of the standard deviations calculated for all nodes in the current neighborhood graph (that is, the above-mentioned global standard deviation mean) is 0.2; the ratio of the current node u to the mean of the standard deviation is used as the neighborhood fluctuation factor, that is, approximately 1.225;

[0124] Step S3223: Perform comprehensive calculation based on the neighborhood fluctuation factor and the metadata conflict factor to obtain the tuning factor α uv ;

[0125] Specifically, the neighborhood fluctuation factor V of two nodes based on the path segment composed of two adjacent nodes is u 、V v Calculate the average value of neighborhood fluctuation factor

[0126] Based on the metadata conflict factor C uv The average values ​​of the neighborhood fluctuation factors are weighed by the balance parameter λ and then summed to obtain the tuning factor α uv ;

[0127] The tuning factor α uv The calculation method is:

[0128]

[0129] Where, α uv is the tuning factor of the path segment between node u and node v; C uv V is the metadata conflict factor between node u and node v; u is the neighborhood fluctuation factor of node u; V v is the neighborhood fluctuation factor of node v; λ is the balance parameter.

[0130] It should be noted that, in the above embodiments of the present application, α uv represents the optimization factor of the path segment from node u to node v; Cuv Represents the metadata conflict factor between node u and node v; this factor reflects the inconsistency and conflict degree between the two nodes in the metadata unified description or data mapping process; the higher the inconsistency, the higher the conflict factor C uv The larger the value is, the more serious the data information matching or conversion problem is on this side; u With V v Represents the neighborhood fluctuation factor of node u and node v respectively; the neighborhood fluctuation factor describes the fluctuation or instability of the data in the neighborhood of the node; for example, during data migration or mapping, if the data quality, format or semantics of the area where a node is located are very different, its fluctuation factor will be higher; take the average of the two It can comprehensively reflect the overall fluctuation of the path segment connecting the two nodes; λ is a balance parameter with a value range of [0, 1]; by adjusting λ, the metadata conflict factor C can be controlled uv and neighborhood volatility factor The relative importance of λ in the final tuning factor; when the λ value is large (approaching 1), the role of the metadata conflict factor is amplified, and the algorithm pays more attention to data inconsistency issues; when the λ value is small (approaching 0), it pays more attention to the impact of neighborhood data fluctuations.

[0131] Research has shown that traditional methods for quantifying neighborhood data fluctuations directly use the ratio of the data standard deviation to the mean global standard deviation to determine the neighborhood fluctuation factor. However, the standard deviation is sensitive to outliers. Directly using the ratio of the standard deviation to the mean global standard deviation to describe the neighborhood fluctuation factor can distort the fluctuation factor in the presence of noise or outliers. Based on this, the present embodiment proposes a robustness adjustment parameter in step S3222 to optimize the calculation of the neighborhood fluctuation factor. Specifically, by replacing the direct use of the standard deviation with a more robust statistical method, the impact of outliers (noise or outliers) on the results is reduced. Global statistical information is also used to correct local fluctuations, resulting in a more accurate neighborhood fluctuation factor. The specific solution is detailed in subsequent steps S32221 to S32224.

[0132] Specifically, in step S3222, the neighborhood fluctuation factor is obtained based on the Euclidean distance from each node to the first k neighboring nodes with the closest Euclidean distance in the neighborhood graph, including the following steps:

[0133] Step S32221: Obtain the neighboring nodes corresponding to each node in the neighborhood graph; construct a distance set based on the Euclidean distance between the node and the neighboring nodes;

[0134] Step S32222: Sort the elements in the distance set from low to high according to the Euclidean distance to obtain a distance sequence set; select the first k elements of the distance sequence set as the benchmark distance data; calculate the median distance corresponding to the current node based on the benchmark distance data. u and interquartile range IQRD u ;

[0135] It should be noted that in the above embodiment of the present application, the first k elements in the selected distance sequence set are used as the reference distances. These reference distances are used as the benchmark data for calculating the neighborhood fluctuation factor, thereby filtering out the influence of noise points with excessively large distances, so that subsequent analysis and calculations are more focused on more robust data.

[0136] For example, after screening in the above embodiment of the present application, the first 5 neighborhood distances (i.e., benchmark distance data) of node u are: D u =[2.1,3.5,4.0,4.2,15.0]; After calculation and analysis, the median distance of node u is obtained u =4.0, interquartile range IQRD u =Q3-Q1=4.2-3.5=0.7; In the above embodiment of the present application, the median distance is used to replace the data mean in the traditional neighborhood data fluctuation analysis process to reduce the impact of outliers (such as 15.0) on the central trend; and the interquartile range is used to measure the degree of data dispersion, which is more robust to outliers than the standard deviation.

[0137] Step S32223: Based on the median of the distances of all nodes in the current neighborhood graph, MedianD u Statistically obtain the global distance median GMedianD (i.e. the median of the global distance medians);

[0138] For example, assuming that the local medians of the three nodes are 3.0, 4.0, and 5.0 respectively, then GMedianD is 4.0; the overall data distribution characteristics are reflected through global statistics to avoid the absolute value of the local fluctuation factor being affected by the overall scale of the data set.

[0139] Step S32224: Based on the median distance of the current node MedianD u and interquartile range IQRD u And the global distance median GMedianD is used to calculate the neighborhood fluctuation factor;

[0140] The neighborhood fluctuation factor is calculated as follows:

[0141]

[0142] Where δ is the weight for balancing local and global influences;

[0143] The technical solutions adopted in the above embodiments of the present application are described by way of example:

[0144] Assume that the neighborhood graph contains 3 nodes, and its neighborhood distance data is shown in the following table:

[0145]

[0146] Perform global statistics calculation, local median set: 4.0, 1.5, 5.3 → Global_Median_Med = 4.0; local IQR set: 0.7, 0.6, 0.4 → Global_IQR_Med = 0.6;

[0147] Furthermore, the local fluctuation is preliminarily calculated (δ = 0.5)

[0148] Node u: 0.7 / 4.0 + 0.5*(4.0 / 4.0) = 0.175 + 0.5 = 0.675

[0149] Node v: 0.6 / 1.5+0.5*(1.5 / 4.0)≈0.4+0.1875=0.5875

[0150] Node w: 0.4 / 5.3+0.5*(5.3 / 4.0)≈0.075+0.6625=0.7375

[0151] In summary, the technical solution adopted in the embodiment of the present application avoids the interference of outliers (such as 15.0, 20.0) on the fluctuation calculation by using the median and IQR instead of the mean and standard deviation. For example, the neighborhood distance of node u contains 15.0, but MedianD u =4.0 can still reflect the typical distance; further, the global statistic GMedianD is used to eliminate the influence of the overall distribution difference of the data set. For example, the local median of node v is 1.5, which is much smaller than GMedianD=4.0, but its fluctuation factor is corrected by the global proportional term in the formula.

[0152] Example 2

[0153] On the other hand, this embodiment 2 is based on a multi-source heterogeneous data integration method based on dynamic metadata modeling provided in embodiment 1 of the invention, and also provides a storage medium (referred to as storage medium), in which a computer program is stored. When the computer program is executed by the processor, the steps of the above-mentioned multi-source heterogeneous data integration method based on dynamic metadata modeling are implemented.

[0154] In summary, the present invention proposes a multi-source heterogeneous data integration method and storage medium based on dynamic metadata modeling. This method collects structured data (database tables), semi-structured data (JSON / XML logs), and unstructured data (images, videos) through multi-source interfaces (such as APIs, ETL tools, and file system scans). Metadata is then extracted from the original data to capture data format (such as the date format YYYY-MM-DD), structure (table associations), and semantics (field business meaning) information to form a metadata set. This method comprehensively captures data features and provides input for subsequent dynamic metadata modeling. Data heterogeneity is quantified through metadata sets, providing a basis for conflict detection (naming, type, structure, and semantics).

[0155] Furthermore, metadata collections are described uniformly by standardizing field names (e.g., mapping custID to customer ID), defining common data types (e.g., dates are unified to ISO 8601), constraint rules (e.g., non-null constraints), and semantic tags (e.g., "customer ID"). A metadata dictionary is established to record attributes such as field names, types, and semantics, thereby ensuring that the resulting database supports cross-system compatibility. Dynamic models are used to define standard naming rules (e.g., semantic similarity matching), design common mapping schemes (e.g., string-to-integer cleansing rules), merge multiple tables into a unified schema (e.g., linking the order table's foreign key to the customer table), and verify the consistency of field meanings based on a business glossary. This eliminates data silos, provides standardized metadata views, and adapts to differences in multi-source data through dynamic rules to ensure the accuracy of subsequent mapping and integration.

[0156] Furthermore, since traditional dimensionality reduction methods (such as MDS) rely on Euclidean distance and cannot capture the nonlinear manifold structure of high-dimensional data, the embodiment of the present application calculates the Euclidean distance between unified description fields, selects the first k nearest neighbors of each node, constructs a neighborhood graph, and then uses the Dijkstra algorithm to calculate the geodesic distance (real path distance along the manifold) between any two nodes in the neighborhood graph; then, by optimizing the tuning factor, the conflict quantification (S3221) such as naming conflict (semantic similarity score), type conflict (conversion complexity score), structural conflict (reorganization difficulty score) and semantic conflict (business impact score) is calculated and analyzed to obtain a neighborhood fluctuation factor (S3222): based on the median and interquartile range (IQR) of the node neighborhood distance, the impact of outliers is reduced; further, through multidimensional scaling analysis (MDS) and optimization (S33-S34), the high-dimensional geodesic distance is mapped to low-dimensional initial coordinates, solving the problem of insufficient modeling of nonlinear manifold structures by traditional methods; the path cost is dynamically adjusted through the tuning factor to improve the robustness of the shortest path matrix;

[0157] Ultimately, by mapping data and integrating them, a fused database is generated that supports cross-source queries, eliminates data silos, and provides standardized views, thereby supporting efficient data analysis.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-source heterogeneous data integration method based on dynamic metadata modeling, characterized by: The steps are as follows: Collecting and acquiring multi-source heterogeneous data; collecting metadata for the multi-source heterogeneous data to obtain a metadata set; Performing a unified description of the metadata set based on a dynamic metadata model to obtain unified description field information; Performing mapping processing on the multi-source heterogeneous data based on the unified description field information to obtain mapping data; Integration is performed based on the mapping data to obtain a fusion database.

2. A multi-source heterogeneous data integration method based on dynamic metadata modeling according to claim 1, characterized in that: The metadata set includes data format, structure and semantic information of multi-source heterogeneous data; The multi-source heterogeneous data includes structured data, semi-structured data and unstructured data.

3. The multi-source heterogeneous data integration method based on dynamic metadata modeling according to claim 2 is characterized in that: The mapping process of the multi-source heterogeneous data based on the unified description field information to obtain mapping data specifically includes the following steps: A neighborhood graph is constructed based on the Euclidean distance between each two unified description field information; each node in the neighborhood graph represents a multi-source heterogeneous data; and the line between each two nodes represents the Euclidean distance between the two unified description field information; Calculate the shortest distance matrix between every two nodes in the graph based on the neighborhood graph combined with the artificial bee colony algorithm; Applying multidimensional scaling analysis to perform dual centering processing based on the shortest distance matrix to obtain low-dimensional mapping coordinates of each node; The low-dimensional mapping coordinates are optimized based on the shortest distance matrix to obtain mapping data for each node.

4. The multi-source heterogeneous data integration method based on dynamic metadata modeling according to claim 3 is characterized in that: The method of calculating the shortest distance matrix between every two nodes in the graph based on the neighborhood graph combined with the artificial bee colony algorithm includes the following steps: Obtaining data conflict information during the unified description process of the metadata set; Analyze the data conflict information and the Euclidean distances from each node to the nearest k neighboring nodes in the neighborhood graph to obtain a tuning factor for the path segment between each two adjacent nodes; Traversing each of the nodes, taking the node as the starting point and any node other than the current node as the end point, and combining them to obtain multiple initial paths from the starting point to the end point; there are no repeated nodes in the initial paths; Obtaining the edge distance of each initial path; calculating the path fitness based on the edge distance and the tuning factor; traversing each of the initial paths; Replacing the path segments based on the tuning factors of each path segment in the current initial path and the tuning factors of the path segments between the end point and each neighboring node of the end point of the current initial path to obtain a second path; and determining and replacing the second path based on the new fitness of the second path and the fitness of the original initial path to obtain a third path; The fitness of the third path is replaced and optimized according to the distribution of the tuning factors to establish a shortest distance matrix.

5. The multi-source heterogeneous data integration method based on dynamic metadata modeling according to claim 4 is characterized in that: The step of determining and replacing the new fitness of the second path with the fitness of the original initial path to obtain the third path includes the following steps: Calculating based on the second path to obtain a new fitness; When the new fitness is greater than the path fitness, the current second path replaces the current initial path to obtain a third path.

6. The multi-source heterogeneous data integration method based on dynamic metadata modeling according to claim 5 is characterized in that: The fitness based on the third path is replaced and optimized according to the distribution of the tuning factor to establish a shortest distance matrix, including the following steps: Counting the fitness of all third paths, and selecting a high-potential path using a roulette wheel probability distribution method; obtaining an optimized third path by replacing the path segments with high tuning factors with the path segments with the lowest tuning factors among the path segments from each node to the neighboring nodes of each node based on the distribution of tuning factors in the high-potential paths; Determining whether a tuning factor of each path segment in the optimized third path is less than a preset tuning factor threshold; If so, determining the optimized third path as the target path; A shortest distance matrix is ​​established based on all the target paths.

7. The multi-source heterogeneous data integration method based on dynamic metadata modeling according to claim 6 is characterized in that: The step of analyzing the data conflict information in combination with the Euclidean distances from each node to the nearest k neighboring nodes in the neighborhood graph to obtain the tuning factor of each path segment between two adjacent nodes includes the following steps: Performing conflict quantification detection processing on the data conflict information to obtain conflict quantification scores for each piece of data conflict information; and calculating metadata conflict factors based on the conflict quantification scores; A neighborhood fluctuation factor is obtained by analyzing the Euclidean distances of each node in the neighborhood graph to the first k neighboring nodes with the closest Euclidean distances; A tuning factor α is obtained by comprehensive calculation based on the neighborhood fluctuation factor and the metadata conflict factor. uv .

8. The multi-source heterogeneous data integration method based on dynamic metadata modeling according to claim 7 is characterized in that: Also includes: The neighborhood fluctuation factor V of two nodes based on the path segment composed of two adjacent nodes u 、V v Calculate the average value of neighborhood fluctuation factor Based on the metadata conflict factor C uv The average values ​​of the neighborhood fluctuation factors are weighed by the balance parameter λ and then summed to obtain the tuning factor α uv ; The tuning factor α uv The calculation method is: Where, α uv is the tuning factor of the path segment between node u and node v; C uv V is the metadata conflict factor between node u and node v; u is the neighborhood fluctuation factor of node u; V v is the neighborhood fluctuation factor of node v; λ is the balance parameter.

9. The multi-source heterogeneous data integration method based on dynamic metadata modeling according to claim 8 is characterized in that: The step of analyzing the Euclidean distances between each node in the neighborhood graph and the first k neighboring nodes with the closest Euclidean distances to obtain the neighborhood fluctuation factor comprises the following steps: Obtaining neighboring nodes corresponding to each node in the neighborhood graph; constructing a distance set based on the Euclidean distance between the node and the neighboring node; Sort the elements in the distance set from low to high according to the Euclidean distance to obtain a distance sequence set; select the first k elements of the distance sequence set as the benchmark distance data; calculate the median distance corresponding to the current node based on the benchmark distance data u and interquartile range IQRD u ; Based on the median distance of all nodes in the current neighborhood graph, MedianD u Perform statistics to obtain the global median distance GMedianD; Based on the median distance of the current node MedianD u and interquartile range IQRD u And the global distance median GMedianD are used to calculate the neighborhood fluctuation factor.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of a multi-source heterogeneous data integration method based on dynamic metadata modeling as described in any one of claims 1 to 9.

Images