An intelligent data governance method and system based on active metadata

Abstract

The application relates to the technical field of active metadata management, in particular to an intelligent data management method and system based on active metadata. The method comprises the following steps: calculating the structural complexity of two groups of active metadata according to the number of attributes, the correlation measurement value and the combination entropy value of the two groups of active metadata; calculating the stationary measurement value according to the average value and the maximum value point of a group of active metadata, and obtaining a data characteristic measurement value; further obtaining a change characteristic measurement difference value of the two groups of active metadata; performing weighted summation on the structural complexity and the change characteristic measurement difference value to obtain a complexity difference value; setting the calculation mode of a reference distance of a DTW algorithm, and processing the two groups of active metadata by using the DTW algorithm according to the calculation mode of the reference distance selected according to the complexity difference value of the two groups of active metadata. The application can improve the matching and mapping effect of the active metadata, and realizes the consistency management of multi-source active metadata.

Description

Technical Field

[0001] The present invention relates to the technical field of active metadata management, and in particular to an intelligent data governance method and system based on active metadata. Background Art

[0002] Active metadata is a type of metadata that automatically updates, manages, interprets, and interacts with data and operations in a system. It describes data and provides information about it. Currently, intelligent methods are often used to effectively govern various types of active metadata to improve data quality, security, and compliance. This governance is particularly important in environments that span systems, businesses, and data sources. In such environments, active metadata spans multiple systems or platforms, often with varying data structures or attributes. Therefore, matching and mapping active metadata is necessary to achieve consistent governance of multi-source data.

[0003] In existing technologies, the DTW algorithm (Dynamic Time Warping) is commonly used to match and map heterogeneous active metadata across data sources to address the alignment and comparison issues involved in data matching and mapping. This approach efficiently integrates data, eliminates information silos, and improves data quality, thereby promoting the development of intelligent data governance and enhancing the accuracy and efficiency of data-driven decision-making.

[0004] However, when processing multi-source, heterogeneous active metadata, the DTW algorithm introduces errors when constructing the distance accumulation matrix for active metadata with different attributes and dimensions. Furthermore, the DTW algorithm only considers the distance information of individual point pairs in the sequence, ignoring local variation in the sequence. This underestimates the contribution of local features to the distance calculation, which can easily lead to inappropriate path matching. Secondly, the purpose of using the DTW algorithm to match and map active metadata across systems, businesses, and data sources is to achieve cross-data source integration and analysis, which is the primary purpose of active metadata governance methods and systems. Therefore, it is necessary to perform DTW calculations between active metadata while preserving the original data units and scope to integrate multi-source data and discover potential correlation patterns. The DTW algorithm typically uses the Euclidean distance metric. However, in this scenario, due to the complexity of multi-source active metadata and the DTW algorithm's inherent inconsistency in considering local variation in the sequence, this ultimately produces an ineffective optimal curved path. This results in poor data matching quality and the system may not provide accurate data mapping, impacting subsequent data governance, analysis, and decision-making. Summary of the Invention

[0005] In order to solve the above technical problems, the purpose of the present invention is to provide an intelligent data governance method and system based on active metadata. The technical solutions adopted are as follows:

[0006] In a first aspect, an embodiment of the present invention provides an intelligent data governance method based on active metadata, the method comprising:

[0007] Obtaining active metadata from different data sources and preprocessing them to obtain different groups of active metadata;

[0008] The structural complexity of the two sets of active metadata is calculated based on the number of attributes, correlation measures and combined entropy values ​​of the two sets of active metadata;

[0009] Calculating a stationary measure of a set of active metadata based on an average value and a maximum point of the set of active metadata; obtaining a data feature measure based on the stationary measure of the set of active metadata and a normalized sampling frequency;

[0010] Based on the difference of the data characteristic measurement values ​​of the two sets of active metadata, the change characteristic measurement difference value of the two sets of active metadata is obtained; the structural complexity and change characteristic measurement difference value of the two sets of active metadata are weightedly summed to obtain the complexity difference degree value;

[0011] The calculation method of the reference distance of the DTW algorithm is set, and the calculation method of the reference distance is selected according to the complexity difference value of the two sets of active metadata. The two sets of active metadata are processed using the DTW algorithm.

[0012] Preferably, obtaining active metadata from different data sources and preprocessing to obtain different groups of active metadata includes:

[0013] The preprocessing includes data cleaning, data format conversion and data grouping; the active metadata that has undergone data cleaning and format conversion is grouped according to different data sources to obtain different groups of active metadata.

[0014] Preferably, calculating the structural complexity of the two sets of active metadata according to the number of attributes, correlation metrics, and combined entropy values ​​of the two sets of active metadata includes:

[0015] The number of attributes in the two sets of active metadata is normalized to obtain the attribute characteristic value; two straight lines are fitted using the two sets of active metadata respectively, and the correlation measurement value of the two sets of active metadata is obtained according to the slope of the two straight lines; the correlation measurement value is negatively correlated with the exponential function with the natural constant as the base to obtain the correlation characteristic value; the joint entropy value of the two sets of active metadata is normalized to obtain the entropy characteristic value; the average value of the attribute characteristic value, correlation characteristic value and entropy characteristic value of the two sets of active metadata is the structural complexity of the two sets of active metadata.

[0016] Preferably, the calculation formula of the correlation metric value is:

[0017] ,

[0018] in, represents the correlation measure between the i-th group of active metadata and the j-th group of active metadata; and They respectively represent the slope of the straight line obtained by straight-line fitting using the i-th set of active metadata and the slope of the straight line obtained by straight-line fitting using the j-th set of active metadata.

[0019] Preferably, calculating the stationarity metric value of a set of active metadata according to the average value and the maximum value point of the set of active metadata includes:

[0020] The absolute value of the difference between each data value and the average value in a set of active metadata is obtained and summed to obtain the degree of dispersion; a maximum point in the set of active metadata is subtracted from the data adjacent to the left and the data adjacent to the right, and the sum is taken to obtain the local data change value of the maximum point; the local data change values ​​of all maximum points in the set of active metadata are summed to obtain the fluctuation change characteristic value; the degree of dispersion of the set of active metadata and the sum of the fluctuation change characteristic values ​​are normalized to obtain the stationarity measurement value of the set of active metadata.

[0021] Preferably, obtaining the change characteristic metric difference value of the two sets of active metadata based on the difference of the data characteristic metric values ​​of the two sets of active metadata includes:

[0022] The absolute value of the difference between the data feature measurement values ​​of the two sets of active metadata is normalized to obtain the change feature measurement difference value of the two sets of active metadata.

[0023] Preferably, the calculation method of setting the reference distance of the DTW algorithm includes:

[0024] The reference distance calculation method of the DTW algorithm includes two calculation methods: the traditional method and the relative change distance calculation method. The traditional method uses the Euclidean distance as the reference distance; the relative change distance calculation method is:

[0025] ,

[0026] in, Indicates the relative change distance between the i-th data point and the j-th data point; and Represent the data values ​​of the i-th data point and the i-1-th data point respectively; and Represent the data values ​​of the j-th data point and the j-1-th data point respectively.

[0027] Preferably, the calculation method of the reference distance is selected according to the complexity difference value of the two sets of active metadata, and the two sets of active metadata are processed using the DTW algorithm, including:

[0028] A threshold is set. If the complexity difference between the two sets of active metadata is less than or equal to the threshold, the traditional method is used to calculate the reference distance. If the complexity difference between the two sets of active metadata is greater than the threshold, the relative change distance calculation method is used.

[0029] Preferably, obtaining the data feature metric value includes:

[0030] The data feature metric is obtained by adding the stationarity metric and the normalized sampling frequency of a set of active metadata.

[0031] In a second aspect, the present invention also provides an intelligent data governance system based on active metadata, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the system implements the steps of an intelligent data governance method based on active metadata.

[0032] The embodiments of the present invention have at least the following beneficial effects: the present invention pre-processes active metadata from different sources to obtain active metadata of different groups, thereby improving the quality of the active metadata and making subsequent analysis results more accurate; further, the complexity of the two sets of active metadata to be subjected to the DTW algorithm is analyzed by the number of attributes, correlation measurement values, and combined entropy values ​​of the two sets of active metadata to obtain the structural complexity. The analysis from three dimensions can help analyze whether the calculation of the reference distance using the traditional method during the DTW algorithm will be affected; then, a stationarity measurement value of a set of active metadata is obtained, and then the data characteristic measurement value is obtained based on the change distribution characteristics of the data in combination with the sampling frequency, thereby measuring the data characteristics of the two sets of active metadata. The difference in characteristics is calculated to obtain the change feature measurement difference value, which is used to evaluate the comprehensive difference in the stationarity and sampling frequency of the two sets of active metadata. The structural complexity and change feature measurement difference value of the two sets of active metadata are then combined to obtain the complexity difference degree value, which indicates the complexity and difference between the two sets of active metadata. Finally, the calculation method of the reference distance of the DTW algorithm is set, and then the appropriate reference distance calculation method in the DTW algorithm is adaptively selected based on the complexity difference degree value to obtain a more appropriate reference distance, construct an accurate distance matrix to obtain more reasonable path matching, improve the matching and mapping effect of active metadata, realize the consistency management of multi-source active metadata, and ensure the efficient integration, accurate analysis and effective utilization of active metadata. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 A method flow chart of an intelligent data governance method based on active metadata provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] To further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features and effects of an intelligent data governance method and system based on active metadata proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable form.

[0036] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0037] The following describes in detail a specific solution of an intelligent data governance method and system based on active metadata provided by the present invention in conjunction with the accompanying drawings.

[0038] Example 1:

[0039] The main application scenario of this invention is: Given the cross-system, cross-business, and cross-data source characteristics of active metadata, it is necessary to use the DTW algorithm to match and map active metadata to achieve consistency management of active metadata. This is because the DTW algorithm can achieve multi-source heterogeneous data matching and mapping through sequence similarity, without the need for unified data processing, even if the timestamps in different data sources (such as multiple sensor data, log data, etc.) have different sampling frequencies, time offsets, or uneven intervals. However, when using the DTW algorithm to process active metadata, the traditional method only uses Euclidean distance as the reference distance and does not consider the data characteristics of the active metadata. Therefore, it is necessary to improve the calculation method of the reference distance when using the DTW algorithm to process active metadata.

[0040] See also Figure 1 , which shows a method flow chart of an intelligent data governance method based on active metadata provided by one embodiment of the present invention, the method comprising the following steps:

[0041] Step S1: Obtain active metadata from different data sources and perform preprocessing to obtain different groups of active metadata.

[0042] The present invention obtains the complex difference value between two sets of active metadata to be calculated by the DTW algorithm, adaptively selects the calculation method of the reference distance in the DTW algorithm, obtains a more accurate distance matrix and obtains a more reasonable path matching.

[0043] First, active metadata needs to be obtained, and then preprocessing operations need to be performed on the active metadata. The preprocessing operations include data cleaning, data format conversion, and data grouping.

[0044] Data cleaning includes deduplication, handling missing values, and removing outliers. Deduplication involves checking for duplicate records using threshold filtering and removing any duplicates. Missing value handling involves linear interpolation. Outlier removal involves detecting and removing abnormal values ​​or outliers using anomaly detection algorithms (such as KNN). The goal of data cleaning is to remove noise, duplication, or inconsistencies from the data, ensuring the quality of active metadata.

[0045] The purpose of data conversion is to facilitate processing by the DTW algorithm. Metadata from different sources may use different structures (such as JSON, CSV, XML, etc.) and needs to be converted to a unified format, such as CSV, to facilitate processing by the DTW algorithm.

[0046] The purpose of grouping is to group active metadata that has undergone data cleaning and format conversion according to different data sources. The data source of each group of active metadata is the same.

[0047] Step S2: Calculate the structural complexity of the two sets of active metadata based on the number of attributes, correlation metrics, and combined entropy values ​​of the two sets of active metadata.

[0048] In the actual operation process of the DTW algorithm, for active metadata with different attributes and different dimensions, when constructing the distance accumulation matrix, the differences in different dimensions will introduce errors in the calculation of the reference distance. Therefore, we must first consider whether the two sets of active metadata to be subjected to the DTW algorithm contain more attributes and whether their relationship is complex. Then, we can obtain the complex difference degree value of the two sets of active metadata to be subjected to the DTW algorithm. According to the complex difference degree value, we can adaptively obtain the calculation method of the reference distance in the DTW algorithm to obtain a more accurate distance matrix, find the optimal matching path, and obtain a more accurate correspondence between data.

[0049] In a cross-system, cross-business, and cross-data source environment, active metadata is inherently diverse and complex. Therefore, dimensional information is a crucial component of active metadata. However, the increase in dimensionality introduces noise due to excessive redundant features, resulting in reduced accuracy in the calculation of the reference distance (Euclidean distance) in the traditional DTW algorithm. Therefore, it is necessary to analyze the two sets of active metadata that will undergo DTW calculations by introducing multidimensional features. The three most important dimensions of multidimensional features include the number of attributes, overall change trend, and combined entropy. They comprehensively evaluate the dimensional characteristics of the data from the perspectives of structure, dynamic characteristics, and uncertainty, respectively. These dimensions have a profound impact on the accuracy and stability of Euclidean distance calculations. Therefore, by comprehensively obtaining the structural complexity of the two sets of active metadata that will undergo DTW calculations from these three perspectives, we can help analyze whether the Euclidean distance calculation may be affected.

[0050] The structural complexity of the two sets of active metadata is calculated based on the number of attributes, correlation measures, and combined entropy values ​​of the two sets of active metadata. Specifically, the number of attributes in the two sets of active metadata is normalized to obtain attribute eigenvalues; two straight lines are fitted using the two sets of active metadata, and the correlation measures of the two sets of active metadata are obtained based on the slopes of the two straight lines; the correlation eigenvalue is negatively mapped using an exponential function with a natural constant as the base to obtain the correlation eigenvalue; the joint entropy value of the two sets of active metadata is normalized to obtain the entropy eigenvalue; and the average of the attribute eigenvalues, correlation eigenvalues, and entropy eigenvalues ​​of the two sets of active metadata is the structural complexity of the two sets of active metadata.

[0051] The specific calculation formulas for structural complexity and correlation metrics are:

[0052] ,

[0053] ,

[0054] in, represents the structural complexity of the i-th group of active metadata and the j-th group of active metadata; Norm() represents the normalization function, which limits the output result to [0,1]; m(∂) represents the number of attributes of the data in the two groups of active metadata (the i-th group of active metadata and the j-th group of active metadata) to be subjected to the DTW algorithm (for example, if one group of active metadata represents time-sales and the other group represents time-views, then m(∂) is 3 for these two groups of active metadata); represents the correlation measure between the i-th group of active metadata and the j-th group of active metadata, exp(-) represents the inverse proportional normalization function, and exp() represents the exponential function with a natural constant as the base. and represent the slopes of the straight line obtained by linear fitting using the i-th set of active metadata and the slopes of the straight line obtained by linear fitting using the j-th set of active metadata, respectively; represents the joint entropy value of the i-th group of active metadata and the j-th group of active metadata. 、 and They represent the attribute eigenvalue, correlation eigenvalue, and entropy eigenvalue of the two sets of active metadata respectively. The larger the three are, the higher the complexity of the two sets of active metadata.

[0055] Regarding the number of attributes in the two sets of active metadata, the DTW algorithm measures their similarity by comparing the "distance" of the two sequences point by point. If the active metadata contains more attributes, the amplitude or scale of the data changes of different attributes may be greater. In this case, the reference distance calculation of the DTW algorithm will be dominated by the high amplitude or large scale, which will lead to unreasonable path matching and a higher structural complexity of the two sets of active metadata.

[0056] The correlation measure of the trends of two sets of active metadata is intended to capture the degree of similarity of the overall trends of the two sets of active metadata. The least squares method is used for straight line fitting to minimize the sum of the distances between the fitted straight line and the active metadata, reflecting the overall trend of a set of active metadata. The absolute value of the difference between the slopes of the two straight lines is The absolute difference between the slopes of the two straight lines is greater, indicating that the trends of the two sets of active metadata are more different. As a restriction, it ensures that the effect of the difference value will not be amplified when the slope difference is large. , The closer it is to 0, the greater the trend difference between the two sets of active metadata. The DTW algorithm will over-stretch and compress local data points to forcibly align the two sets of active metadata, which may easily lead to unreasonable reference distances.

[0057] The combined entropy of the two sets of active metadata that will be subjected to the DTW algorithm (the entropy calculation formula is the existing formula) reflects the uncertainty between the two sets of active metadata. The larger the entropy value, the more complex the relationship between the two sets of active metadata and the lower the correlation between the data. In this case, the DTW algorithm will minimize the reference distance as much as possible, making the alignment path more complex. The calculated reference distance result cannot accurately reflect the actual relationship between the data, resulting in a large matching error.

[0058] In summary, The closer it is to 1, the lower the correlation between the two sets of active metadata. The more attributes they contain and the less similar the overall change trend, the less accurate the calculation results obtained when the DTW algorithm calculates the reference distance using the traditional method, and the more likely it is to produce incorrect matching paths.

[0059] Step S3: Calculate the stationarity measurement value of a set of active metadata based on the average value and the maximum point of the set of active metadata; and add the stationarity measurement value of the set of active metadata and the normalized sampling frequency to obtain the data feature measurement value.

[0060] In step S2, the structural complexity of the two sets of active metadata that will be used for the DTW calculation is obtained. The purpose is to analyze the complexity of the two sets of active metadata in terms of attributes, trends, and information volume. These characteristics are characteristics of the special scenario problems of active metadata themselves. However, because the DTW algorithm only considers the distance information of a single point pair in the sequence during operation and does not consider the local variation characteristics of the active metadata, this step requires further analysis of the data characteristics of the active metadata to obtain data feature metrics for each set of metadata.

[0061] The stationarity measure of a set of active metadata is calculated based on the average value and maximum points of the set. Specifically, the absolute value of the difference between each data value and the average value in the set of active metadata is calculated and summed to obtain the degree of dispersion. The local data variation value of a maximum point in the set of active metadata is obtained by subtracting the data adjacent to the left and the data adjacent to the right, respectively, and summing the results. The local data variation values ​​of all maximum points in the set of active metadata are summed to obtain the fluctuation characteristic value. The stationarity measure of the set of active metadata is obtained by normalizing the sum of the degree of dispersion and the fluctuation characteristic value of the set of active metadata.

[0062] The specific calculation formula is:

[0063] ,

[0064] in, represents the stability measure of the t-th group of active metadata, which is used to evaluate the data change characteristics of active metadata; tanh() represents the hyperbolic tangent function, which limits the output result to (0,1) and is used for normalization; represents the kth active metadata in the tth group of active metadata; represents the average value of the tth group of active metadata; n represents the number of maximum value points in the tth group of active metadata; represents the αth maximum point in the tth group of active metadata; and They represent the data points adjacent to the left and right of the αth maximum point in the tth group of active metadata.

[0065] In the above formula, Indicates the degree of dispersion of the t-th group of active metadata. A smaller result indicates that the value is close to the mean and the distribution is more concentrated. Conversely, a larger result indicates that the distribution is more dispersed, which may contain more outliers (mutation points). Because these outliers are far away from the mean of the dataset, they amplify the summation result. The local data variation value of the αth maximum point in the tth set of active metadata is accumulated and summed to obtain the fluctuation characteristic value of the tth set of active metadata. This measure aims to reveal the volatility and local variation characteristics of the data set by comparing the value difference between the maximum point and its nearest neighbor. A larger value indicates a greater difference in the value of the maximum point relative to its nearest neighbor, indicating high data volatility and more pronounced local peaks. Conversely, a smaller value indicates relatively stable data with less fluctuation. The data characteristic metric reflects the stationarity of the active metadata set. When the stationarity of data varies significantly, the traditional Euclidean distance calculation is more likely to be affected by the less stationary data, ultimately resulting in unreasonable matching paths.

[0066] In addition, the sampling frequency of active metadata determines the time interval between active metadata. If the sampling frequencies of two sets of active metadata differ significantly, the DTW calculation will be sensitive to the temporal misalignment, which will lead to distortion in the calculation of the Euclidean distance (reference distance), resulting in unreasonable matching results that fail to accurately reflect the similarity between the data. Therefore, it is necessary to incorporate the sampling frequency of active metadata into the analysis and combine the stationarity metric with the sampling frequency of active metadata. The stationarity metric of a set of active metadata and the normalized sampling frequency are added to obtain the data feature metric value. This provides some assistance in subsequently determining the data metric difference between the two sets of data. The calculation formula for the data feature metric value is:

[0067] ,

[0068] in, represents the data feature measurement value of the tth group of active metadata, Indicates the sampling frequency of the active metadata set. From this, we can obtain the data feature measurement values ​​of the two sets of active metadata to be subjected to the DTW algorithm.

[0069] Step S4: obtaining a change feature metric difference value of the two sets of active metadata based on the difference in the data feature metric values ​​of the two sets of active metadata; and performing a weighted summation on the structural complexity and change feature metric difference value of the two sets of active metadata to obtain a complexity difference value.

[0070] In step S3, the data feature metrics of the two sets of active metadata are calculated. To evaluate the combined difference in stationarity and sampling frequency between the two sets of active metadata, the absolute value of the difference between the data feature metrics of the two sets of active metadata is calculated and normalized to obtain the difference in the change feature metrics of the two sets of active metadata. The specific calculation formula is:

[0071] ,

[0072] Where, Indicates the difference value of the change feature measurement between the two sets of active metadata (i-th set of active metadata and j-th set of active metadata) to be calculated by DTW; obtain The purpose is to evaluate the comprehensive difference in stationarity and sampling frequency between the two sets of data. First, the data feature metric ρ includes a stationarity metric for the active metadata. The DTW algorithm constructs a distance matrix by calculating the distance between each pair of time points. However, a large difference in stationarity between the two sets of active metadata will cause DTW to overemphasize changes in global trends when calculating the reference distance, overestimating some data in the distance matrix and affecting the subsequent alignment path selection. Secondly, a large difference in sampling frequency will also cause the DTW algorithm to attempt to nonlinearly adjust these time points during the alignment process due to different time steps, which can mislead the path matching results.

[0073] Therefore, by obtaining the difference value of the change feature measurement of the two sets of active metadata to be calculated by DTW ,think When is small, the two sets of active metadata have similar stability, similar change trends and fluctuations, and similar sampling frequencies. Such two sets of active metadata will not produce large errors when using the Euclidean distance as the reference distance for the DTW algorithm in the traditional way, and will have little effect on the DTW calculation results. On the contrary, When is large, the stationarity differences and sampling frequency differences between the data are large. The traditional method of using Euclidean distance as the reference distance for DTW algorithm may produce large errors and unreasonable path matching.

[0074] Finally, it is necessary to conduct a comprehensive analysis of the structural complexity and change characteristic measurement difference values ​​of the two sets of active metadata. Therefore, the structural complexity and change characteristic measurement difference values ​​of the two sets of active metadata are weighted and summed to obtain the complexity difference value. The specific calculation formula is:

[0075] ,

[0076] in, Indicates the complexity difference value between the two sets of active metadata (the i-th set of active metadata and the j-th set of active metadata) to be calculated by DTW; and Respectively represent weight values; This comprehensively reflects the complexity and data differences between the two sets of active metadata. A larger complexity difference value indicates that the two sets of active metadata are more complex, contain more attributes, and do not have similar overall change trends. At the same time, there are large differences in stationarity and sampling frequency between the data. When traditional DTW uses Euclidean distance as the reference distance for calculation, the calculation results are less accurate and are more likely to produce incorrect matching paths. and The value of needs to be set according to different scenarios and requirements. This solution provides a reference: , .

[0077] Step S5 , setting a calculation method for the reference distance of the DTW algorithm, selecting a calculation method for the reference distance according to the complexity difference values ​​of the two sets of active metadata, and processing the two sets of active metadata using the DTW algorithm.

[0078] In step S4, the complexity difference values ​​of the two sets of active metadata that will be subjected to the DTW algorithm are obtained. For the two sets of active metadata with smaller complexity difference values, the reference distance calculation method in the traditional DTW algorithm can be used; however, for the two sets of active metadata with larger and smaller complexity difference values, the calculation method of their reference distance needs to be determined.

[0079] Furthermore, the calculation method of the reference distance of the DTW algorithm is set. The calculation method of the reference distance of the DTW algorithm includes two calculation methods, namely the traditional method and the relative change distance calculation method. The traditional method uses the Euclidean distance as the reference distance; the relative change distance calculation method is:

[0080] ,

[0081] in, Indicates the relative change distance between the i-th data point and the j-th data point; and Represent the data values ​​of the i-th data point and the i-1-th data point respectively; and Represent the data values ​​of the jth data point and the j-1th data point, respectively. It should be noted that, in the embodiment of the present invention, the i-th data point and the j-th data point are data points in two sequences, that is, they represent the i-th data point of one set of data and the j-th data point of the other set of data in two sets of active metadata.

[0082] When two sets of data When is large, since the Euclidean distance is sensitive to absolute values ​​but ignores relative changes, the relative change distance is introduced to calculate the distance between the current two sets of data to obtain a more accurate distance matrix and a more reasonable path matching.

[0083] The use of relative change distance as the reference distance has the following advantages over the traditional Euclidean distance as the reference distance: the first Euclidean distance is too dependent on the absolute value of the data, and the relative change distance is too dependent on the absolute value of the two sets of active metadata. When the value is large, the dimensional characteristics of active metadata are strong, and the data change characteristics are quite different. This leads to the fact that large-scale or large-range active metadata will dominate the calculation results when obtaining the reference distance through the Euclidean distance calculation method, and the contribution of other dimensions is seriously underestimated. The use of relative change distance eliminates such scale effects because it calculates relative differences, avoids the dimensional disaster caused by multi-dimensional data, and reduces the dependence on absolute numerical differences, so that the influence of each dimension on the distance calculation result is balanced to a certain extent. When larger, it can provide a more reliable reference distance calculation result than the Euclidean distance.

[0084] Specifically, the reference distance calculation method is selected based on the complexity difference between the two sets of active metadata, and a threshold is set. If the complexity difference between the two sets of active metadata is less than or equal to the threshold, the traditional reference distance calculation method is used; if the complexity difference between the two sets of active metadata is greater than the threshold, the relative change distance calculation method is used. The threshold setting in the embodiments of the present invention needs to be adjusted by the implementer based on actual circumstances. The specific implementation environment will require different threshold settings. Depending on the requirements for matching accuracy or computational efficiency, if accuracy is more important, a higher threshold can be considered to ensure a more detailed match. If efficiency is a priority, a smaller threshold can be appropriately selected. This solution strikes a balance between accuracy and efficiency, and a reference threshold of 0.7 is given.

[0085] After selecting the calculation method of the reference distance, the DTW algorithm is performed. The present invention adaptively obtains the DTW distance calculation method, constructs a more accurate distance matrix, finds the optimal matching path, obtains a more accurate correspondence between data, and matches accordingly, thereby completing data mapping, improving the accuracy of data matching and mapping, and improving data quality. The intelligent data governance method and system can better automatically perform tasks such as data fusion and cleaning, thereby improving the efficiency and quality of data governance.

[0086] Example 2:

[0087] This embodiment provides an intelligent data governance system based on active metadata, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of an intelligent data governance method based on active metadata. Since Example 1 has already described an intelligent data governance method based on active metadata in detail, it will not be further elaborated here.

[0088] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. Furthermore, the foregoing descriptions of specific embodiments of this specification are provided. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An intelligent data governance method based on active metadata, characterized in that: The method includes: Obtaining active metadata from different data sources and preprocessing them to obtain different groups of active metadata; The structural complexity of the two sets of active metadata is calculated based on the number of attributes, correlation measures and combined entropy values ​​of the two sets of active metadata; Calculating a stationary measure of a set of active metadata based on an average value and a maximum point of the set of active metadata; obtaining a data feature measure based on the stationary measure of the set of active metadata and a normalized sampling frequency; Based on the difference of the data characteristic measurement values ​​of the two sets of active metadata, the change characteristic measurement difference value of the two sets of active metadata is obtained; the structural complexity and change characteristic measurement difference value of the two sets of active metadata are weightedly summed to obtain the complexity difference degree value; Set the calculation method of the reference distance of the DTW algorithm, select the calculation method of the reference distance according to the complexity difference value of the two sets of active metadata, and use the DTW algorithm to process the two sets of active metadata; The calculating of the structural complexity of the two sets of active metadata according to the number of attributes, correlation measurement values, and combined entropy values ​​of the two sets of active metadata includes: The number of attributes in the two sets of active metadata is normalized to obtain the attribute characteristic value; two straight lines are fitted using the two sets of active metadata respectively, and the correlation measurement value of the two sets of active metadata is obtained according to the slope of the two straight lines; the correlation measurement value is negatively correlated with the exponential function with the natural constant as the base to obtain the correlation characteristic value; the joint entropy value of the two sets of active metadata is normalized to obtain the entropy characteristic value; the average value of the attribute characteristic value, correlation characteristic value and entropy characteristic value of the two sets of active metadata is the structural complexity of the two sets of active metadata.

2. The intelligent data governance method based on active metadata according to claim 1 is characterized in that: The obtaining of active metadata from different data sources and preprocessing to obtain different groups of active metadata includes: The preprocessing includes data cleaning, data format conversion and data grouping; the active metadata that has undergone data cleaning and format conversion is grouped according to different data sources to obtain different groups of active metadata.

3. The intelligent data governance method based on active metadata according to claim 1 is characterized in that: The calculation formula of the correlation metric value is: ， in, represents the correlation measure between the i-th group of active metadata and the j-th group of active metadata; and They respectively represent the slope of the straight line obtained by straight-line fitting using the i-th set of active metadata and the slope of the straight line obtained by straight-line fitting using the j-th set of active metadata.

4. The intelligent data governance method based on active metadata according to claim 1 is characterized in that: Calculating the stationarity measurement value of a set of active metadata according to the average value and the maximum value point of the set of active metadata includes: The absolute value of the difference between each data value and the average value in a set of active metadata is obtained and summed to obtain the degree of dispersion; a maximum point in the set of active metadata is subtracted from the data adjacent to the left and the data adjacent to the right, and the sum is taken to obtain the local data change value of the maximum point; the local data change values ​​of all maximum points in the set of active metadata are summed to obtain the fluctuation change characteristic value; the degree of dispersion of the set of active metadata and the sum of the fluctuation change characteristic values ​​are normalized to obtain the stationarity measurement value of the set of active metadata.

5. The intelligent data governance method based on active metadata according to claim 1 is characterized in that: The method of obtaining the change characteristic measurement difference value of the two sets of active metadata based on the difference of the data characteristic measurement values ​​of the two sets of active metadata includes: The absolute value of the difference between the data feature measurement values ​​of the two sets of active metadata is normalized to obtain the change feature measurement difference value of the two sets of active metadata.

6. The intelligent data governance method based on active metadata according to claim 1 is characterized in that: The calculation method of setting the reference distance of the DTW algorithm includes: The reference distance calculation method of the DTW algorithm includes two calculation methods: the traditional method and the relative change distance calculation method. The traditional method uses the Euclidean distance as the reference distance; the relative change distance calculation method is: ， in, Indicates the relative change distance between the i-th data point and the j-th data point; and Represent the data values ​​of the i-th data point and the i-1-th data point respectively; and Represent the data values ​​of the j-th data point and the j-1-th data point respectively.

7. The intelligent data governance method based on active metadata according to claim 1 is characterized in that: The method of calculating the reference distance selected according to the complexity difference between the two sets of active metadata and processing the two sets of active metadata using the DTW algorithm includes: A threshold is set. If the complexity difference between the two sets of active metadata is less than or equal to the threshold, the traditional method is used to calculate the reference distance. If the complexity difference between the two sets of active metadata is greater than the threshold, the relative change distance calculation method is used.

8. The intelligent data governance method based on active metadata according to claim 1 is characterized in that: The obtaining of the data feature measurement value includes: The data feature metric is obtained by adding the stationarity metric and the normalized sampling frequency of a set of active metadata.

9. An intelligent data governance system based on active metadata, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of an intelligent data governance method based on active metadata as described in any one of claims 1 to 8 are implemented.

Images