Intelligent data governance method for full data AI control system

Abstract

The present invention discloses an intelligent data governance method for a full-data AI management and control system, and relates to the technical field of data processing. The method comprises: step 1: at each time, source data is obtained from all data sources, and the total entropy of the system at each time is calculated according to the running status data of each data source; step 2: based on the total entropy of the current time and the total entropy of the previous time, a three-dimensional quality tensor is constructed according to the quality characterization data of the source data of each data source; step 3: the three-dimensional quality tensor is mapped to a high-dimensional feature space; step 4: according to the classification weight, the governance priority index of each data source is calculated; according to the order of the governance priority index from high to low, data governance is performed on each data source. The present invention realizes the reasonable allocation and efficient governance of resources by intelligently calculating the priority and classification weight.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an intelligent data governance method for a full-data AI management and control system. Background Art

[0002] In the modern information society, data has become an important asset for enterprises, institutions and individuals. With the rapid development of information technology and the widespread application of technologies such as the Internet of Things, big data, and artificial intelligence, the sources and types of data have become more and more diverse, and the complexity and quantity of data have continued to increase. In order to effectively manage and utilize this data, a full data management and control system has emerged. Its core task is to ensure the high quality, accuracy and consistency of data, and to be able to intelligently manage and optimize data in the face of a dynamic and changing data environment. However, existing technologies still have many deficiencies in intelligent data governance and cannot fully adapt to the complexity and dynamic changes of multi-source data environments.

[0003] In traditional data governance systems, data quality management usually relies on static rules and preset algorithms. These methods mainly focus on basic operations such as data cleaning, data deduplication and data verification, and lack the ability to perceive and process complex associations and dynamic changes between data sources in real time. For example, in the process of data cleaning, many existing technologies adopt simple rule-based cleaning strategies, which are difficult to handle complex quality issues of large-scale heterogeneous data, especially in the process of collecting and processing multi-source data, the potential associations between different data sources and the spatiotemporal characteristics between data are often ignored. Based on this, traditional data cleaning methods are prone to the accumulation of data errors and cannot fully guarantee the quality of data. In addition, in terms of data source quality assessment, many existing methods usually only focus on a single quality indicator, such as data accuracy, completeness or consistency, and ignore the comprehensive characterization of multi-dimensional quality characteristics. Especially in the face of a dynamically changing complex data environment, existing methods lack the ability to effectively monitor and characterize the global state of the data source, making it difficult for data governance strategies to adapt to real-time changes. Traditional methods often use static rules or simple weight models to evaluate data quality. This method cannot capture the complex associations and temporal dynamic characteristics between data sources, which greatly reduces the effectiveness of data governance. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent data governance method for a full-data AI management and control system, which realizes comprehensive perception and fine governance of multi-source data by introducing entropy calculation, three-dimensional quality tensor construction, high-dimensional feature space mapping, graph embedding technology and correlation strength analysis. The method can dynamically monitor the status of data sources, comprehensively evaluate data quality, identify complex associations between data sources, and realize reasonable allocation and efficient governance of resources through intelligent calculation of priorities and classification weights. Compared with traditional methods, the present invention improves the accuracy, flexibility and intelligence level of data governance, and significantly enhances the system's adaptability and governance effect in complex data environments.

[0005] In order to solve the above technical problems, the present invention provides an intelligent data governance method for a full data AI management and control system, the method comprising:

[0006] Step 1: At each time, source data is obtained from all data sources, and the total entropy of the system at each time is calculated based on the operating status data of each data source;

[0007] Step 2: Based on the total entropy at the current time and the total entropy at the previous time, a three-dimensional quality tensor is constructed according to the quality characterization data of the source data of each data source;

[0008] Step 3: Map the three-dimensional mass tensor to a high-dimensional feature space; regard each data source as a node, and calculate the association strength using the result of the high-dimensional feature space mapping based on the graph embedding method; construct the classification weight based on the association strength;

[0009] Step 4: Calculate the governance priority index of each data source based on the classification weight; perform data governance on each data source in order of the governance priority index from high to low.

[0010] Furthermore, the method for performing data governance on each data source in step 4 includes, in sequence: data cleaning, data deduplication, data verification and data standardization.

[0011] Furthermore, the operation status data in step 1 includes: the amount of data of the data source at each time, the number of dimensions of the data source, the data update frequency of the data source, the number of fields of the data source, and the number of data types of the data source.

[0012] Furthermore, the total entropy is calculated by the following formula: :

[0013] ;

[0014] in, For the system at time The total entropy of To represent the number of data sources in the system; To indicate the Data sources at time The amount of data at the time To indicate the The number of dimensions of a data source; To indicate the The frequency of data updates for each data source; To indicate the The number of fields in a data source; To indicate the The number of data types of each data source; , and They represent the maximum data update frequency, maximum number of fields, and maximum number of data types for all data sources in the entire system.

[0015] Furthermore, the quality characterization data in step 2 include: completeness, accuracy and consistency; the completeness characterizes the missing state of the source data, and its value is equal to the ratio of the actual number of data items to the expected number of data items; the accuracy characterizes the accuracy of the source data, and its value is equal to the ratio of the accurate number of data items to the actual number of data items; the consistency characterizes the degree of deviation of the data, and its value is equal to the standard deviation of all data items; the three-dimensional quality tensor is represented by the following formula:

[0016] ;

[0017] in, Indicates The three-dimensional quality tensor of each data source; Indicates The completeness rate of each data source; To indicate the The consistency rate of the data sources; Indicates The accuracy of the data source; Indicates that the system is at time The total entropy of .

[0018] Furthermore, in step 3, the three-dimensional mass tensor is mapped to a high-dimensional feature space by the following formula to obtain the result of high-dimensional feature space mapping: :

[0019] ;

[0020] in, is the size of the time window; is the time variable in the integral, representing the past time; is the time attenuation coefficient, which controls the influence of past time on current time. The larger the value, the greater the influence. is the gradient of total entropy; Indicates The divergence of the three-dimensional mass tensor of each data source; For the The result of high-dimensional feature space mapping of the three-dimensional quality tensor of the data sources.

[0021] Furthermore, in step 3, each data source is regarded as a node, and based on the graph embedding method, the process of calculating the association strength using the result of high-dimensional feature space mapping specifically includes: obtaining the cluster center of each data source by calculating the clustering algorithm; calculating the distance between the cluster center of each data source and the cluster center of other data sources, and considering two data sources with a distance lower than a set threshold as connected data sources to construct a graph network; and then calculating the association strength according to the following formula:

[0022] ;

[0023] in, For the The strength of association between the data sources; For the The result of high-dimensional feature space mapping of the three-dimensional quality tensor of each data source; For the Data sources at time Entropy of time; For the Data sources at time Entropy of time; is the high-dimensional feature matrix of the graph network extracted by the convolutional network; is the symmetric normalized graph Laplacian matrix; where

[0024] ;

[0025] ;

[0026] is the identity matrix, is the adjacency matrix, is the degree matrix; is a nonlinear activation function;

[0027] ;

[0028] is the weight matrix, Row, No. Column element equals The cluster center of the data source is The inverse of the distance between the cluster centers of the data sources; is the transpose operation of the matrix; Represents the L1 norm operation.

[0029] Furthermore, the classification weight is constructed through the following formula :

[0030] ;

[0031] in, For the Data sources at time The classification weight when It is Hadamard.

[0032] Furthermore, the governance priority index of each data source is calculated using the following formula: :

[0033] ;

[0034] in, For the Data sources at time Governance priority index at the time of Represents the F-norm operation.

[0035] The intelligent data governance method of the full data AI management and control system of the present invention has the following beneficial effects:

[0036] First, the present invention realizes the global quantitative measurement of the system state by introducing the concept of entropy. In traditional data governance systems, the measurement of data quality usually relies on a single indicator or static rules, which is difficult to fully reflect the dynamic changes of the system. The present invention uses entropy in information theory as a measurement indicator, which can comprehensively reflect the status and change trend of each data source in the system.

[0037] Secondly, the present invention proposes a method for constructing a three-dimensional quality tensor. This innovative structure can characterize the quality status of data sources in multiple dimensions. Traditional data governance methods usually regard data quality as a single-dimensional problem, such as data accuracy or completeness, while ignoring the complex associations and multidimensional features between data. Through the construction of a three-dimensional quality tensor, the present invention not only takes into account multi-dimensional quality characterization data such as data completeness, accuracy and consistency, but also captures the dynamic changes of data sources through the time dimension. By mapping the three-dimensional quality tensor to a high-dimensional feature space, the system can extract high-dimensional features of each data source. These features not only reflect the individual quality information of the data source, but also comprehensively consider the time dimension and association relationship. Based on the high-dimensional feature space, the system calculates the association strength between data sources by graph embedding. The introduction of graph embedding technology enables the system to construct a graph network based on data source nodes, and further extract cluster centers through clustering algorithms to identify similarities and dependencies between data sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 A method flow chart of an intelligent data governance method for a full-data AI management and control system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Example 1, reference Figure 1 : An intelligent data governance method for a full-data AI management and control system, the method comprising:

[0042] Step 1: At each time, source data is obtained from all data sources, and the total entropy of the system at each time is calculated based on the operating status data of each data source;

[0043] The concept of entropy originally came from thermodynamics, and was later used in information theory to measure the degree of disorder and uncertainty of a system. Here, the concept of entropy is used to characterize the state and complexity of different data sources in the full-data AI control system. When each data source is in a stable and normal working state, the total entropy value of the system will be lower, indicating that the degree of order in the system is higher and the data stability is better; on the contrary, if the state of some data sources is abnormal or fluctuates, the total entropy value of the system will increase, which reflects that the overall disorder of the system is increasing. Based on this principle, the operating status of the data source can be grasped in real time by regularly calculating and monitoring the changes in total entropy.

[0044] The calculation of total entropy is not just a mathematical process, it includes a quantitative characterization of the state of each data source. The data state of each data source at a certain point in time can be regarded as a certain "amount of information" contributed by it, and the combination and association of these "amounts of information" will affect the entropy value of the system. In order to more accurately characterize the state of each data source, the system also needs to comprehensively consider multiple factors, such as the activity of the data source, the frequency of data collection, the quality of the collected data, etc. The full-data AI management and control system calculates the state probability distribution of each data source through an intelligent mechanism, and these probability distributions reflect the stability of the operating state of the data source. According to these probability distributions, the system can dynamically adjust the entropy calculation formula to adapt to different data source characteristics and changes in the external environment. The main difference between this method of using entropy to characterize the system state and the prior art is that the full-data AI management and control system of the present invention adopts a dynamic, multi-source data collection and intelligent analysis method, which can adapt to the state changes of different data sources and quickly respond and adjust through changes in entropy values. In traditional methods, data governance systems usually evaluate their status based on predefined rules and static indicators. However, the method of the present invention introduces entropy, a dynamic and global measurement method, so that the system can monitor the status of all data sources in real time and comprehensively consider the mutual influence between the data sources, forming a more intelligent and robust overall evaluation system. In addition, the calculation of entropy not only reflects the single status of the data source, but also helps to build a more refined three-dimensional quality tensor in the subsequent steps through comprehensive measurements at the system level, thereby guiding more accurate data governance strategies.

[0045] Step 2: Based on the total entropy at the current time and the total entropy at the previous time, a three-dimensional quality tensor is constructed according to the quality characterization data of the source data of each data source;

[0046] The construction of the three-dimensional quality tensor is based on the quality characterization data of the source data, and the quality characterization data is actually extracted from the data status of each data source. The extraction process of quality characterization data comprehensively considers indicators of multiple dimensions such as data accuracy, completeness, consistency, and timeliness. In this step, the data quality of each data source is not simply regarded as independent, but as an interrelated whole. In the full-data AI control system, the quality of the data source is characterized by multi-dimensional indicators, and these multi-dimensional quality indicators are integrated into a three-dimensional tensor. The three-dimensional quality tensor not only records the quality status of each data source, but also reflects the quality changes between different time points and the interactive relationship between data sources. The construction of the three-dimensional quality tensor is significantly different from the traditional data governance method. Traditional data governance methods usually view data quality issues in isolation and lack consideration of potential correlations between data sources. However, in the full-data AI control system, the data quality characterization data of each data source are not independent of each other, but are organically linked together through the structure of the three-dimensional quality tensor. This three-dimensional structure tensor construction method can not only describe the data quality changes at different time points, but also characterize the interaction and influence relationship between different data sources. For example, data fluctuations from a certain data source may affect the status of other data sources through a certain dependency within the system, and this impact can be reflected through the quality dimension in the three-dimensional tensor. Furthermore, each dimension of the three-dimensional quality tensor not only reflects the status of the data source, but also carries the implicit relationship between time and quality. The quality dimension here includes not only numerical accuracy in the traditional sense, but also the logical consistency of the data, the reliability of data transmission, and the timeliness of data collection. Through the three-dimensional tensor, the quality changes of each data source at different time points can be fully captured, and the potential correlation patterns between data sources can be revealed through the interaction between the time dimension and the data source. This process actually constructs a comprehensive data quality map, laying the foundation for subsequent feature space mapping and correlation strength calculation.

[0047] Compared with the prior art, the three-dimensional quality tensor in the present invention introduces entropy as a representation of the overall state of the system, while traditional methods often lack such a means of measuring the overall state. Through the change of entropy, the dynamic changes of the state of each data source in the system can be accurately captured, and the three-dimensional quality tensor further refines and expands this characterization capability. It can not only record the state information of each data source at each time point, but also reveal the potential correlation and mutual influence between different data sources through the quality characterization data in the tensor. Existing data governance methods are often limited to static analysis of a single data source, while the full-data AI management and control system achieves a comprehensive grasp of the overall state of the system through this dynamic, multi-dimensional three-dimensional tensor structure, thereby making the data governance process more refined and intelligent. The construction of the three-dimensional quality tensor provides a data basis for subsequent high-dimensional feature space mapping. On the basis of the three-dimensional tensor, the multi-dimensional quality characterization data can be converted into a high-dimensional feature representation through an appropriate mapping method, so that the complex relationship between each data source can be intuitively reflected in the high-dimensional space. At the same time, the construction process of the three-dimensional mass tensor also has a certain degree of adaptability. This adaptability is reflected in its ability to dynamically adjust the mass characterization data in the tensor according to the change of total entropy over time, so that the system can respond to changes in the external environment and data source status in real time.

[0048] Step 3: Map the three-dimensional mass tensor to a high-dimensional feature space; regard each data source as a node, and calculate the association strength using the result of the high-dimensional feature space mapping based on the graph embedding method; construct the classification weight based on the association strength;

[0049] First of all, as a high-order description of the quality of the data source, the three-dimensional quality tensor contains the state information of the data source, the dynamic changes in the time dimension, and the implicit associations between the data sources. However, due to the complexity of the three-dimensional tensor and the high-order nature of the data dimension, directly processing and analyzing the three-dimensional tensor is not an efficient way, so it is necessary to further extract and express these multidimensional information through high-dimensional feature space mapping. In the process of high-dimensional feature space mapping, the system uses feature extraction algorithms to project the complex multidimensional data in the three-dimensional tensor into a high-dimensional feature space. This high-dimensional feature space not only retains the relationship between the various data sources in the original three-dimensional tensor, but also further enhances the interpretability of these relationships, allowing the system to more accurately measure and analyze the quality and association of data sources.

[0050] Once the mapping of the high-dimensional feature space is completed, the system will treat each data source as a node and use graph embedding technology to calculate the association strength. Graph embedding is a technology that maps graph structure data (here refers to data source nodes and the relationship between them) to a low-dimensional vector space. This technology can effectively retain the similarity and topological structure between nodes. In the full-data AI management and control system, the association strength between data source nodes reflects their mutual influence and dependence in terms of data quality, data transmission status, time change, etc. Through graph embedding technology, the system can automatically learn the hidden features between different data source nodes and calculate the association strength based on these hidden features. The calculation of association strength not only considers the direct connection between data source nodes, but also reflects the indirect relationship between data sources through graph structure information, so that the system can comprehensively and finely characterize the interaction between different data sources. After calculating the association strength using graph embedding and high-dimensional feature space mapping, the system further uses these association strengths to construct classification weights. Classification weight refers to the importance of each data source being classified in the entire data governance process. This importance directly affects the priority calculation in subsequent steps. In order to construct reasonable classification weights, the system needs to compare and weigh multiple data sources, and this comparison is not only based on the independent characteristics of the data source, but also needs to consider its degree of correlation with other data sources. By combining the results of high-dimensional feature space mapping and graph embedding, the system can accurately measure the relative importance of data sources, thereby assigning appropriate classification weights to each data source.

[0051] Step 4: Calculate the governance priority index of each data source based on the classification weight; perform data governance on each data source in order of the governance priority index from high to low.

[0052] First, through the construction of classification weights, the relative importance of each data source in the feature space has been obtained. This weight reflects the criticality and influence of each data source in the whole system. However, relying solely on classification weights is not enough to form a complete governance strategy, because data governance is not only about the quality of a single data source, but also needs to comprehensively consider the mutual influence, dynamic changes and resource allocation between various data sources. Therefore, the calculation of the governance priority index needs to further consider these factors to make the entire governance process more intelligent and refined. In the calculation process of the governance priority index, the full data AI management and control system adopts a model based on multi-factor weighting to determine the priority of each data source. In this model, in addition to the classification weights, indicators of multiple dimensions need to be considered, such as the current quality level of each data source, historical quality fluctuations, correlation strength with other data sources, data importance assessment, etc. These indicators are integrated into the calculation of the governance priority index in a weighted and comprehensive way. Specifically, the calculation of the governance priority index not only involves the current status of a single data source, but also requires trend analysis based on the dynamic changes in the time dimension, and identifies the mutual influence between various data sources through the analysis of correlation strength. This process enables the calculation of the priority index to have a more global perspective, thus providing a solid foundation for the intelligentization of data governance.

[0053] The level of governance priority index represents the degree to which each data source needs to be prioritized at the current point in time. The higher the index value, the greater the quality risk of the data source in the current system, or the greater its potential impact on other data sources, so data governance needs to be prioritized. By sorting the governance priority index of all data sources, the system can quickly identify those data sources that need immediate intervention and processing, and then formulate corresponding data governance strategies according to the priority level. This process avoids the waste of resources and blind processing that may exist in traditional data governance methods, because it is based on real-time, global multi-dimensional data analysis results, and can dynamically adjust the governance order, so that the system can achieve the maximum benefit of data governance under limited resources. After determining the governance order, the system will govern the data sources in order from high to low according to the ranking of the governance priority index. This governance order is not only based on snapshot information at a single moment, but also combines multiple factors such as the change trend of each data source at different time points, the dependency relationship with other data sources, and the current system total entropy state. Therefore, this method can adapt to changes in the system and realize dynamic intelligent data governance. For example, at a certain point in time, if the quality problem of a data source suddenly deteriorates, causing its governance priority index to rise sharply, the system can immediately identify this change and prioritize its processing to prevent the quality problem of the data source from spreading to the entire system. This ability to dynamically adjust and respond in real time enables the full-data AI management and control system to always maintain efficient and accurate operation when handling complex data governance tasks.

[0054] Embodiment 2: The method for performing data governance on each data source in step 4 includes, in sequence: data cleaning, data deduplication, data verification and data standardization.

[0055] Specifically, in the first step of data governance, data cleaning is the most basic and critical part of the whole process. The core purpose of data cleaning is to eliminate invalid, erroneous, missing and noisy data, laying the foundation for subsequent data governance. In the full data AI management and control system, data cleaning not only relies on simple rule filtering, but also combines a variety of intelligent data cleaning algorithms. For example, by analyzing the distribution characteristics of the source data, the system can identify abnormal data points, and based on the trend of entropy changes, determine whether these abnormal data are reasonable fluctuations or manifestations of data quality problems. Intelligent processing of data cleaning can effectively reduce the drawbacks of traditional methods that rely heavily on manual labeling and static rules, greatly improving the accuracy and efficiency of data cleaning.

[0056] After data cleaning is completed, the next step is data deduplication. The main goal of data deduplication is to solve the problem of duplicate data between the same data source or different data sources. Duplicate data not only takes up system storage resources, but also misleads data analysis and mining results. In the full data AI management and control system, data deduplication is not a simple character matching and comparison, but an intelligent identification and elimination of redundant data based on the correlation between data sources in the system and the change in entropy. For example, based on the mapping of high-dimensional feature space, the system can automatically determine which data is duplicated and which data has similar features but represents different meanings through similarity analysis between nodes. Such an intelligent data deduplication method can greatly reduce redundant data and improve the overall data quality of the system.

[0057] After the data is deduplicated, it enters the data verification stage. In this stage, the system performs a comprehensive accuracy and consistency check on the cleaned and deduplicated data. Data verification not only checks the correctness of the data format, but also includes multiple dimensions such as the rationality of the data logical relationship and the consistency of the timestamp. In traditional methods, data verification is often implemented through static rules, which is difficult to cope with complex data associations and dynamic changes. In the full data AI management and control system, data verification combines the results of dynamic entropy change analysis and high-dimensional feature space mapping, and can adaptively adjust the verification rules to ensure the integrity and accuracy of the data. For example, when the entropy value of a data source changes significantly, the system can intelligently trigger a stricter data verification strategy, automatically identify possible data quality issues, and avoid bad data from affecting subsequent analysis and decision-making.

[0058] The last step is data standardization, which is the final link of data governance. It aims to unify the format, measurement unit, precision, etc. of data to facilitate unified management and analysis of data. In the full-data AI management and control system, data standardization is not only a simple format conversion process, but also combines the overall global perspective of the system and multi-dimensional data characteristics to intelligently select the optimal standardization solution. For example, the system will dynamically adjust the standardized rules and parameters based on the time series changes in the three-dimensional mass tensor and the correlation between the data sources, so that the data can achieve optimal consistency throughout the system. Through data standardization, the system can seamlessly integrate data from different sources, providing a solid data foundation for subsequent analysis and mining.

[0059] Embodiment 3: The operation status data in step 1 includes: the amount of data of the data source at each time, the number of dimensions of the data source, the data update frequency of the data source, the number of fields of the data source and the number of data types of the data source.

[0060] Specifically, the amount of data of a data source at each time point reflects the activity and information density of the data source at the current time. In a full-data AI control system, the amount of data from a data source is a key indicator because it is directly related to the overall load of the system and the priority of data processing. By continuously monitoring the changes in the amount of data, the system can grasp the state fluctuations of each data source in real time and associate these changes with the changes in the total entropy of the system. When the amount of data from a data source fluctuates dramatically, this may indicate problems with data quality or data integrity. The system can quickly adjust the entropy calculation strategy to capture these potential abnormal changes. The number of dimensions of a data source is another important indicator that reflects the characteristics of a data source. The number of dimensions directly determines the complexity and correlation of the data source. This indicator is particularly important in multi-dimensional data analysis. By continuously monitoring the number of dimensions of a data source, the full-data AI control system can dynamically determine the potential correlation between data sources and provide support for the subsequent construction of a three-dimensional quality tensor. When the number of dimensions of a data source changes, this may indicate that the logical structure or collection method of the data has been adjusted. The system can recalculate the entropy value based on the change in the number of dimensions to maintain an accurate representation of the state of the data source.

[0061] The frequency of data updates is a core indicator that reflects the activity and timeliness of data sources. Data sources with high frequency updates often mean higher activity and stronger real-time requirements, which is particularly critical in the full data system. By real-time monitoring of the update frequency of data sources, the system can identify those data sources that require high priority processing, and thus give different weights in the calculation of entropy. For data sources with low frequency updates, the system will adopt a more robust strategy to ensure that the overall entropy calculation can accurately reflect the update characteristics of different data sources. The number of fields is another important dimension of the complexity of the data source structure. In a multi-source data system, different data sources may contain different numbers of fields, which not only reflect the structural complexity of the data source, but also affect the quality representation of the data. By monitoring the changes in the number of fields, the system can dynamically adjust the entropy calculation model to adapt to the structural characteristics of different data sources. An increase or decrease in the number of fields usually means that the data structure has changed, which may bring potential data consistency or quality issues. Therefore, the system will perform a sensitivity analysis on the change in the number of fields when calculating entropy to ensure that the change in the total entropy can accurately reflect the structural changes of the data source. Finally, the number of data types is a key indicator of the diversity and complexity of data sources. In the full data AI management and control system, the number of data types directly affects the difficulty of data governance and analysis. Different types of data require different processing strategies. Therefore, by monitoring the changes in the number of data types, the system can dynamically adjust the data governance strategy. When the number of data types of a data source changes significantly, it may indicate that the data source has introduced a new data model or data format, which poses new challenges to the consistency and integrity of the data. The system dynamically introduces the parameter of the number of data types in the calculation of entropy to comprehensively consider the impact of data diversity on the system state.

[0062] Example 4: Calculate the total entropy using the following formula: :

[0063] ;

[0064] in, For the system at time The total entropy of To represent the number of data sources in the system; To indicate the Data sources at time The amount of data at the time To indicate the The number of dimensions of a data source; To indicate the The frequency of data updates for each data source; To indicate the The number of fields in a data source; To indicate the The number of data types of each data source; , and They represent the maximum data update frequency, maximum number of fields, and maximum number of data types for all data sources in the entire system.

[0065] Specifically, the first part of the formula shows how to deal with the relative amount of data from each data source. and dimensions .pass This expression system can capture the information complexity of data sources in different dimensions. Represents the time period of each data source The amount of data generated at any given moment, and the number of dimensions It reflects the diversity and complexity of the data. Data sources with larger dimensions usually contain more dimensional data information, so the complexity and potential relevance of the information are also higher. Normalizing the data volume by dimension can ensure that the calculation of entropy can accurately reflect the contribution of different dimensional data sources to the overall system. In particular, when the dimension of the data source is When it is larger, its contribution to the total entropy of the system is amplified, which means that high-dimensional data sources need to receive more attention in data governance. Next, the logarithmic term in the formula further introduces the update frequency , Number of fields and data type number Dynamic characteristics such as these are achieved by maximizing the overall , and Normalize the data so that the features of different data sources can be compared under the same standard. Update frequency It is an important indicator to measure the activity of data sources. Frequently updated data sources usually mean more real-time requirements and higher processing priority. It reflects the structural complexity of the data source. Data sources with more fields usually contain richer information, which may require more sophisticated processing and analysis. It reflects the diversity of data sources. Different types of data have different contributions to the entropy of the system. Therefore, the number of data types is introduced to measure the impact of diversified data on the overall information entropy of the system.

[0066] These features are processed by This is achieved by this item. Logarithmic operation is widely used in information theory to measure the amount of information. It can convert the product of features into an additive form, thereby avoiding the influence of excessive differences between features on entropy calculation. The normalized result allows the characteristics of each data source in terms of its update frequency, number of fields and data type to be compared with the maximum value in the system, providing a unified evaluation standard. The introduction of this normalization and logarithmic transformation enables the system to fairly measure the impact of each data source on the total entropy of the system even when the characteristic values ​​of the data sources differ greatly. This formula is not only effectively expanded under the framework of information theory, but also combined with the actual operation of the system, and has strong application value. Through the calculation of total entropy, the system can monitor the status of data sources in real time, identify those data sources with significant changes in information volume or high feature complexity, and give priority to data governance. This process can greatly improve the accuracy of data governance, ensure that the system can reasonably allocate resources when facing a large number of data sources, and give priority to those data sources that are most likely to affect the stability of the system. More importantly, the entropy calculation in this formula can dynamically respond to changes in data sources in the system. Since the characteristics of the data source (such as update frequency, data volume, etc.) are constantly changing over time, the total entropy of the system is also dynamic, which enables the full-data AI management and control system to respond to changes in the state of the data source in real time. By calculating the total entropy at each point in time, the system can track the changing trends of different data sources and adjust the data governance strategy according to the fluctuations in the entropy value. In other words, the calculation of total entropy is not only a measure of the current system state, but also a predictive tool for future data governance strategies. When the entropy value of certain data sources in the system increases significantly, the system can promptly identify potential risk data sources and prioritize their processing to avoid further spread of data problems.

[0067] Embodiment 5: The quality characterization data in step 2 include: completeness, accuracy and consistency; the completeness characterizes the missing state of the source data, and its value is equal to the ratio of the actual number of data items to the expected number of data items; the accuracy characterizes the accuracy of the source data, and its value is equal to the ratio of the accurate number of data items to the actual number of data items; the consistency characterizes the degree of deviation of the data, and its value is equal to the standard deviation of all data items; the three-dimensional quality tensor is represented by the following formula:

[0068] ;

[0069] in, Indicates The three-dimensional quality tensor of each data source; Indicates The completeness rate of each data source; To indicate the The consistency rate of the data sources; Indicates The accuracy of the data source; Indicates that the system is at time The total entropy of .

[0070] Specifically, the construction of the three-dimensional mass tensor is based on the total entropy of the system The total entropy of the system has been calculated in the previous step through the multi-dimensional characteristics of the data source, such as the dimension, data volume, and update frequency. It represents the overall state of the system at a certain point in time. The introduction of total entropy provides a global and comprehensive measurement tool for the three-dimensional quality tensor, which can balance the weights of different data sources in quality characterization. Specifically, each part of the three-dimensional quality tensor is and the system at the previous point in time The total entropy This normalization operation ensures that the quality characterization of each data source can be reasonably compared and analyzed within the framework of the whole system. The core of quality characterization data lies in three indicators: completeness, accuracy and consistency.

[0071] Completeness It is used to measure the missing data of a data source at the current point in time. Its value is equal to the ratio of the actual number of data items to the expected number of data items. This ratio provides a basic quality measurement standard for the system. The higher the integrity rate of the data source, the less missing data there is, and the system can rely more on the output of the data source when performing data analysis. Therefore, the importance of the integrity rate in the three-dimensional quality tensor is self-evident. It directly affects the system's trust in the data source and the subsequent data governance strategy. Accuracy It is the second important quality characterization indicator, which indicates the accuracy of data of a certain data source at the current point in time, that is, the ratio of the number of accurate data items to the actual number of data items. The higher the accuracy of a data source, the more reliable the data provided by the data source, with less errors and noise. This indicator, together with other features in the three-dimensional quality tensor, helps the system determine which data sources can be prioritized for decision-making and analysis. When the accuracy of a data source is low, the system may reduce its weight or include it in a more stringent data governance process to correct potential errors and deviations. Consistency rate It is an indicator used to characterize the degree of data deviation between data sources. It measures the stability and consistency of data sources in the form of standard deviation. In the full-data AI management and control system, different data sources may provide data of different dimensions and types, so the introduction of consistency rate can help the system identify the degree of deviation between data sources. When the consistency rate of a data source is low, it means that the data fluctuation of the data source is small, and the data is more consistent and reliable; when the consistency rate is high, the system needs to pay attention to the uncertainty that may be caused by the fluctuation of the data source, especially when multi-source data is merged, data sources with higher consistency rates may bring greater uncertainty and data quality issues.

[0072] In the formula of the three-dimensional quality tensor, these three quality characterization indicators are organically combined through the structure of the tensor matrix. The system not only considers the absolute value of each quality indicator, but also associates these indicators with the total entropy through normalization, allowing the system to measure the quality of the data source from a global perspective. For example, in the tensor, each part of the quality characterization is represented by the total entropy. The multiplication operation is performed, while the other elements in the tensor are based on time The total entropy of is normalized. This normalized processing method helps to eliminate the magnitude differences between time intervals and different data sources, so that the system can uniformly measure the quality performance of each data source under different behavior modes and data states. In addition, the three-dimensional quality tensor can also capture the relationship between each quality characterization data through its matrix structure. Quality characterization data is not isolated, and there is often a potential correlation between the completeness, accuracy and consistency of the data source. For example, when the completeness of a data source is high, its accuracy may also be correspondingly high, because the overall quality of the data source is usually more stable when there are fewer missing data. Through the high-dimensional structure of the tensor, the system can analyze the correlation between these complex quality characteristics and optimize the data governance process. In the data governance process, the system can dynamically adjust the priority and strategy of data governance by analyzing the changes in the three-dimensional quality tensor to cope with the real-time changes in the quality of the data source. Compared with traditional data governance methods, the three-dimensional quality tensor formula of the present invention has significant advantages. Traditional methods usually only consider a single quality indicator, such as data volume or update frequency, ignoring the multidimensionality and complexity of data quality. The present invention introduces three key quality characterization data, completeness, accuracy and consistency, and integrates them into a tensor model, which greatly improves the flexibility and accuracy of the system when processing complex data sources. The structured expression of the three-dimensional quality tensor can not only help the system to comprehensively and dynamically perceive the quality status of the data source, but also reveal the complex relationships between data sources, providing strong support for intelligent data governance.

[0073] Example 6: In step 3, the three-dimensional mass tensor is mapped to a high-dimensional feature space by the following formula to obtain the result of the high-dimensional feature space mapping: :

[0074] ;

[0075] in, is the size of the time window; is the time variable in the integral, representing the past time; is the time attenuation coefficient, which controls the influence of past time on current time. The larger the value, the greater the influence. is the gradient of total entropy; Indicates The divergence of the three-dimensional mass tensor of each data source; For the The result of high-dimensional feature space mapping of the three-dimensional quality tensor of the data sources.

[0076] Specifically, the first part of the formula is an integral term with respect to time , which describes how to perform a three-dimensional mass tensor Perform a time-weighted average. The time window size here is determined by the parameter decide, and Indicates a point in time in the past. The core idea of ​​this integral term is to accumulate the quality status of the data source over a period of time in the past, so as to capture the time evolution trend of the quality of the data source. Time decay coefficient In this process, it plays a role in adjusting the impact of historical data on current features. When is larger, the influence of the past data source state on the current moment is weakened, which means that the system pays more attention to the real-time state of the data source; on the contrary, when When the value is smaller, the system will pay more attention to the cumulative effect of historical data. Through this weighted average method, the system can adaptively adjust the degree of attention to historical data, thereby providing more comprehensive time dimension information for feature extraction of data sources. This method avoids relying solely on quality characterization data at a certain moment, but considers the dynamic changes in data source quality through time weighting, reflecting the intelligence and flexibility of the full data system.

[0077] Next item It is the gradient of the total entropy of the system at the current time point. The introduction of the gradient means that the system not only considers the absolute value of the total entropy, but also pays attention to its trend over time. The existence of the gradient enables the system to more keenly perceive the fluctuations in the system state reflected by the change in entropy value. When the total entropy of the system changes rapidly, this may mean that the state between data sources has changed significantly, and the system needs to adjust the governance strategy in time according to these changes. Therefore, the gradient of the total entropy can provide important dynamic information for high-dimensional feature space mapping, helping the system to better capture potential correlations and abnormal states between data sources. The last term of the formula This involves the divergence operation of the three-dimensional mass tensor, which further refines the internal structural characteristics of the mass tensor. By calculating the divergence of the tensor, the system can identify the change patterns and differences in the quality characterization data. This process is similar to the analysis of fields in physics. By calculating the divergence, the system can understand the rate and direction of change of the internal quality characteristics of each data source. Squaring this divergence term and adding it to the high-dimensional feature space mapping can enhance the sensitivity to data sources with drastic quality changes. This is especially important for the system to identify and process abnormal data sources.

[0078] In general, this high-dimensional feature space mapping formula cleverly combines multiple mathematical operations such as time integral, entropy gradient and tensor divergence to achieve multi-dimensional and high-precision analysis of three-dimensional quality tensors. Each part of the formula serves different goals: the integral term captures dynamic changes in time, the entropy gradient term perceives the overall state changes of the system, and the divergence term refines the quality characteristics inside the data source. Through these combinations, the system can extract more detailed and comprehensive feature information in a complex data environment, laying a solid foundation for subsequent data analysis and governance strategies. Compared with traditional data processing methods, the high-dimensional feature space mapping formula of the present invention embodies stronger dynamics and flexibility. Traditional methods usually only analyze data based on the current time point, lacking consideration of the time dimension and historical data. In the full-data AI management and control system, the introduction of time enables the system to capture dynamic changes in data sources over a longer time range, thereby improving the accuracy and stability of data feature extraction. At the same time, by introducing the total entropy gradient, the system can more keenly perceive changes in the global state, especially when the correlation between data sources changes significantly, the gradient information can help the system respond quickly. In addition, the introduction of tensor divergence provides the system with additional spatial dimension analysis tools. Divergence can identify subtle changes and local inconsistencies in the quality representation of the data source. Especially when the quality of the data source fluctuates greatly, the divergence term can effectively enhance the perception of these changes. This local feature extraction method is difficult to achieve in traditional data analysis. Through tensor operations, the present invention can perform more detailed feature capture and analysis in high-dimensional space.

[0079] Embodiment 7: In step 3, each data source is regarded as a node, and the process of calculating the association strength using the result of high-dimensional feature space mapping based on graph embedding specifically includes: obtaining the cluster center of each data source by calculating the clustering algorithm; calculating the distance between the cluster center of each data source and the cluster center of other data sources, and considering two data sources with a distance lower than a set threshold as connected data sources to construct a graph network; and then calculating the association strength according to the following formula:

[0080] ;

[0081] in, For the The strength of association between the data sources; For the The result of high-dimensional feature space mapping of the three-dimensional quality tensor of each data source; For the Data sources at time Entropy of time; For the Data sources at time Entropy of time; is the high-dimensional feature matrix of the graph network extracted by the convolutional network; is the symmetric normalized graph Laplacian matrix; where

[0082] ;

[0083] ;

[0084] is the identity matrix, is the adjacency matrix, is the degree matrix; is a nonlinear activation function;

[0085] ;

[0086] is the weight matrix, Row, No. Column element equals The cluster center of the data source is The inverse of the distance between the cluster centers of the data sources; is the transpose operation of the matrix; Represents the L1 norm operation.

[0087] Specifically, in this method, each data source is regarded as a node in the graph network. In order to accurately characterize the characteristics of each data source, the system first obtains the high-dimensional feature representation of each data source through the high-dimensional feature space mapping formula in the previous step. The characteristics of each data source not only reflect the static quality information of the data source, but also integrate the dynamic changes in the time dimension, which provides a high-precision feature basis for subsequent graph embedding operations. Next, through the clustering algorithm, the system calculates the cluster center of each data source. The introduction of the cluster center is to simplify and summarize the characteristics of the data source in the high-dimensional feature space and reduce the complexity. Specifically, the clustering algorithm can cluster similar data sources together and represent the characteristics of these similar data sources with the center of each cluster. In this way, the system can not only extract the local features of the data source, but also enhance the expression of the global features. Since the characteristics of the data source are represented in a high-dimensional space, choosing a suitable clustering algorithm can effectively improve the expression ability and computational efficiency of the graph network.

[0088] After obtaining the cluster centers, the system determines the similarity between two data sources by calculating the distance between the cluster centers of each data source and the cluster centers of other data sources. If the distance between the cluster centers of two data sources is lower than a preset threshold, it is considered that the two data sources are strongly associated, and a connection is established in the graph network. This process constructs a graph network based on nodes and edges, where nodes represent data sources and edges represent the strong association between data sources. The purpose of constructing a graph network is to better characterize the relationship between data sources in high-dimensional space and to enhance the ability of feature extraction and analysis by utilizing the characteristics of the graph structure. After constructing the graph network, the system further extracts high-dimensional features from the graph network through a convolutional network. The introduction of convolutional networks is to process the structural information in graph networks. The system aggregates the features of neighboring nodes through graph convolution operations to capture the position and association pattern of each data source in the graph network. Output results of the convolutional network It is a high-dimensional feature matrix that contains the feature representation of each data source in the graph network. Here, the role of the convolutional network is to further process the data source features through the graph structure and enhance the modeling ability of complex associations between data sources. The next step is to calculate the association strength. In the formula, the calculation of the association strength mainly depends on the combination of multiple factors such as the high-dimensional features of each data source, entropy value, graph Laplacian matrix and weight matrix. The system first uses the symmetric normalized graph Laplacian matrix , this is to consider the global structural characteristics of the graph network in the calculation, to ensure that the extraction and calculation of features have a certain stability. The normalization operation of the graph Laplacian matrix is ​​done by the degree matrix Each element of the degree matrix represents the number of connections of each node (data source). This normalization step helps to reduce the impact of node degree differences in the network on the calculation of association strength, making the calculation results more fair and reasonable. Next, through the weight matrix The introduction of , the formula further considers the inverse of the distance between cluster centers. The construction of this weight matrix is ​​based on the distance between the cluster centers of two data sources. The closer the distance between the two data sources, the greater the weight in the calculation of the association strength. The introduction of the weight matrix enhances the sensitivity of the formula to local features, so that the calculated association strength can better reflect the closeness between data sources. The activation function in the formula It is a nonlinear activation function used to enhance the nonlinear expression ability of the calculation. The existence of the activation function enables the system to better extract complex feature relationships in high-dimensional feature space and enhance the formula's ability to capture complex data patterns. Finally, the L1 norm operation in the formula By normalizing and standardizing the matrix, the stability and robustness of the calculation results are ensured. The use of the L1 norm helps to suppress the influence of noise and abnormal data, making the calculation results of the association strength more accurate and reliable.

[0089] Example 8: Construct classification weights using the following formula :

[0090] ;

[0091] in, For the Data sources at time The classification weight when It is Hadamard.

[0092] Specifically, the Hadamard product is introduced into the formula ( ), which is a matrix operation symbol for element-level multiplication. The introduction of Hadamard product shows that when constructing classification weights, the system takes into account the interaction between multiple features of different data sources. The calculation of classification weights does not only rely on a single feature or a single indicator, but also incorporates the interaction of multiple features and correlation strength into the calculation to more accurately measure the importance of data sources. First, the calculation of classification weights is based on high-dimensional features. The high-dimensional features represent the feature vectors of each data source in the high-dimensional space. These feature vectors integrate the information of the data source in multiple dimensions, including its quality, relevance, and time dynamic changes. By comparing the high-dimensional feature vectors with the feature vectors of other data sources, the relative position of each data source in the whole system can be obtained. This relative position helps to identify the relationship between data sources, thus providing a comprehensive basis for the construction of classification weights. At the same time, the calculation of classification weights also takes into account the strength of association of each data source. The strength of association reflects the closeness and dependency between a data source and other data sources. When the strength of association between two data sources is high, they may have a strong mutual influence in data analysis and governance decisions. Therefore, in the process of constructing classification weights, the strength of association needs to be incorporated into the calculation of the weights to ensure that the results can reflect the complex associations between data sources.

[0093] Hadamard product in formula It plays the role of combining high-dimensional features with association strength element by element. This operation means that the system multiplies each feature of the data source with the corresponding association strength to generate a comprehensive weight matrix. The introduction of the Hadamard product makes the calculation of weights more detailed and flexible, because it can capture the influence of the specific features of each data source on the entire system and combine it with the association relationship with other data sources. Through this element-by-element operation method, the system can generate a more accurate classification weight that fully reflects the importance of the data source. The construction of classification weights is a process of multi-dimensional feature fusion. The system captures the state of the data source in the high-dimensional feature space, and gradually constructs the classification weight of each data source through the Hadamard product combined with the association strength. In this process, the dynamic changes of high-dimensional features and the real-time adjustment of association strength ensure the accuracy and adaptability of the classification weights. Especially in the scenario of multi-source data governance, the association relationship between data sources may change over time, and the dynamic adjustment mechanism of classification weights can help the system better cope with these changes and make reasonable governance decisions. Compared with traditional methods, the classification weight construction method in the present invention is more flexible and intelligent. Traditional data governance methods are usually based on predefined rules and simple feature comparisons, and it is difficult to fully reflect the complex associations between data sources. In the full data AI management and control system of the present invention, by introducing multiple factors such as high-dimensional feature space, Hadamard product and correlation strength, the system can more accurately evaluate the importance of data sources. The construction of classification weights not only considers the individual characteristics of each data source, but also integrates the mutual influence between data sources, thereby generating a more representative and accurate weight index.

[0094] Example 9: Calculate the governance priority index of each data source using the following formula: :

[0095] ;

[0096] in, For the Data sources at time Governance priority index at the time of Represents the F-norm operation.

[0097] Specifically, the first part of the formula Represents the total entropy of the system Data source Amount of data The purpose of this part is to measure the impact of changes in the amount of data from a certain data source on the total entropy of the entire system. As a key metric in the entire data governance process, the total entropy of the system can reflect the correlation between various data sources and changes in data status. By calculating partial derivatives, we can identify which data sources contribute more to the change in total entropy. When the amount of data from a certain data source changes, if the absolute value of the partial derivative is large, it means that the data source has a strong impact on the overall information of the system, so it needs to be given more attention in the governance process. The next part Representative The F-norm is a metric commonly used in matrix analysis that represents the overall size or energy of a matrix. Here, the use of the F-norm helps to comprehensively measure the overall strength of the classification weight of each data source at a specific point in time. It reflects the relative importance of each data source. This weight is not only based on a single feature, but also combines multiple information such as high-dimensional features and association strength. By performing F-norm calculations on classification weights, the system can more comprehensively evaluate the importance of each data source, thereby more accurately reflecting the relative status of data sources when calculating priorities.

[0098] The next part of the formula The number of dimensions of the data source involved , Number of fields and update frequency . The number of dimensions and the number of fields together determine the complexity and amount of information of the data source, while the update frequency indicates the activity and real-time requirements of the data source. In the process of data governance, data sources with higher complexity and higher activity usually require more resources and higher priority. Therefore, the formula reflects the comprehensive complexity and activity of the data source by combining the product of the number of dimensions and the number of fields with the inverse of the update frequency. A higher number of dimensions and fields means that the data source has a larger amount of information, and a lower update frequency means that the system can accumulate more information over a longer period of time, thereby increasing the priority of the data source. The last part of the formula is an exponential term ,in, Representative The index item is used to measure the correlation strength of each data source. This index item is introduced to comprehensively consider the overall correlation of all data sources in the calculation of governance priority. By taking the average value of the correlation strength and calculating the index, the system can evaluate the overall correlation degree of all data sources. If the average correlation strength of all data sources is high, it means that there is a close correlation between the data sources in the system. At this time, the priority of each data source will be suppressed to a certain extent to avoid over-centralized governance of highly correlated data sources. On the contrary, if the average correlation strength is low, it means that the correlation between the data sources is weak, and the system can allocate governance resources more freely. Therefore, the design of this index item can help the system balance the allocation of governance resources at the global level and avoid conflicts and waste of governance resources between highly correlated data sources.

[0099] The present invention is described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. An intelligent data governance method for a full data AI control system, characterized in that: The method comprises: Step 1: At each time, source data is obtained from all data sources, and the total entropy of the system at each time is calculated based on the operating status data of each data source; Step 2: Based on the total entropy at the current time and the total entropy at the previous time, a three-dimensional quality tensor is constructed according to the quality characterization data of the source data of each data source; Step 3: Map the three-dimensional mass tensor to a high-dimensional feature space; regard each data source as a node, and calculate the association strength using the result of the high-dimensional feature space mapping based on the graph embedding method; construct the classification weight based on the association strength; Step 4: Calculate the governance priority index of each data source according to the classification weight; perform data governance on each data source in descending order of the governance priority index; the operation status data in step 1 includes: the amount of data of the data source at each time, the number of dimensions of the data source, the data update frequency of the data source, the number of fields of the data source, and the number of data types of the data source; The total entropy is calculated by the following formula: : ； in, For the system at time The total entropy of To represent the number of data sources in the system; To indicate the Data sources at time The amount of data at the time To indicate the The number of dimensions of a data source; To indicate the The frequency of data updates for each data source; To indicate the The number of fields in a data source; To indicate the The number of data types of each data source; , and They represent the maximum data update frequency, maximum number of fields, and maximum number of data types for all data sources in the entire system.

2. The intelligent data governance method of the full data AI management and control system according to claim 1, characterized in that: The method of performing data governance on each data source in step 4 includes, in sequence: data cleaning, data deduplication, data verification and data standardization.

3. The intelligent data governance method of the full data AI management and control system according to claim 2, characterized in that: The quality characterization data in step 2 include: completeness, accuracy and consistency; the completeness characterizes the missing of source data, and its value is equal to the ratio of the actual number of data items to the expected number of data items; the accuracy characterizes the accuracy of source data, and its value is equal to the ratio of the accurate number of data items to the actual number of data items; the consistency characterizes the degree of deviation of data, and its value is equal to the standard deviation of all data items; the three-dimensional quality tensor is represented by the following formula: ； in, Indicates The three-dimensional quality tensor of each data source; Indicates The completeness rate of each data source; To indicate the The consistency rate of the data sources; Indicates The accuracy of the data source; Indicates that the system is at time The total entropy of .

4. The intelligent data governance method of the full data AI management and control system according to claim 3, characterized in that: In step 3, the three-dimensional mass tensor is mapped to a high-dimensional feature space using the following formula to obtain the result of high-dimensional feature space mapping: : ； in, is the size of the time window; is the time variable in the integral, representing the past time; is the time attenuation coefficient, which controls the influence of past time on current time. The larger the value, the greater the influence. is the gradient of total entropy; Indicates The divergence of the three-dimensional mass tensor of each data source; For the The result of high-dimensional feature space mapping of the three-dimensional quality tensor of the data sources.

5. The intelligent data governance method of the full data AI management and control system according to claim 4, characterized in that: In step 3, each data source is regarded as a node. Based on the graph embedding method, the process of calculating the association strength using the result of high-dimensional feature space mapping specifically includes: obtaining the cluster center of each data source through a clustering algorithm; calculating the distance between the cluster center of each data source and the cluster center of other data sources, and considering two data sources with a distance lower than a set threshold as connected data sources to construct a graph network; and then calculating the association strength according to the following formula: ； in, For the The strength of association between the data sources; For the The result of high-dimensional feature space mapping of the three-dimensional quality tensor of each data source; For the Data sources at time Entropy of time; For the Data sources at time Entropy of time; is the high-dimensional feature matrix of the graph network extracted by the convolutional network; is the symmetric normalized graph Laplacian matrix; where ； ； is the identity matrix, is the adjacency matrix, is the degree matrix; is a nonlinear activation function; ； is the weight matrix, Row, No. Column element equals The cluster center of the data source is The inverse of the distance between the cluster centers of the data sources; is the transpose operation of the matrix; Represents the L1 norm operation.

6. The intelligent data governance method of the full data AI management and control system according to claim 5, characterized in that: The classification weight is constructed by the following formula : ； in, For the Data sources at time The classification weight when It is Hadamard.

7. The intelligent data governance method of the full data AI management and control system according to claim 6, characterized in that: The governance priority index of each data source is calculated using the following formula: : ； in, For the Data sources at time Governance priority index at the time of Represents the F-norm operation.

Images