Metadata-driven data center data management method and system

Abstract

This invention proposes a metadata-driven data platform management method and system. The method includes: connecting to multiple heterogeneous data sources through various access methods; automatically adjusting and optimizing the metadata model; using machine learning methods to identify patterns in the changes of various fields in historical data and generating evolution rules for metadata evolution; automatically providing feedback on the differences between real-time monitoring data sources and the adaptive metadata model; using the adaptive metadata model to monitor the quality of data sources in real time and designing early warning rules based on the data source quality, generating an early warning rule set; when data quality issues trigger an early warning; designing a machine learning-based repair model for data quality issues and selecting appropriate repair methods for optimization. This invention overcomes the shortcomings of existing technologies by introducing an adaptive metadata evolution mechanism and a multi-level data quality early warning and repair mechanism, providing an efficient, flexible, and intelligent data platform management solution.

Description

Technical Field

[0001] This invention belongs to the field of metadata-driven data management, and in particular relates to a data management method and system for a data middle platform based on metadata-driven data management. Background Technology

[0002] With the rapid development of informatization and digitalization, data has become a core asset for modern enterprise decision-making and operations. To manage and utilize data more efficiently, an increasing number of enterprises are adopting data platform architectures to integrate and manage massive amounts of data scattered across different business systems. Data platforms, through a unified data management platform, aggregate various data resources within an enterprise, providing one-stop data processing, analysis, and service capabilities to support business decision-making and innovation. However, despite the enormous potential of data platforms in data integration and application, the construction and application of most data platforms currently face many challenges, particularly in metadata management, data quality control, and processing workflow optimization, resulting in serious deficiencies in the efficiency and quality of data processing using existing technologies.

[0003] First, most existing data platforms rely on static metadata models. Metadata defines the structure, type, and relationships of data and is the core of data management. However, traditional metadata models are usually manually defined during the initial construction and often struggle to keep up with changes in business needs and data source updates, resulting in metadata lag and inconsistency. For example, when new business requirements emerge or new data sources are added, the existing metadata model may not be updated effectively in a timely manner. This not only leads to a disconnect between the data source and the metadata model but may also affect subsequent data integration, cleaning, and analysis, thereby reducing the overall processing efficiency of the data platform.

[0004] Secondly, data quality has always been a major pain point in data platform management. Data quality involves multiple aspects such as accuracy, completeness, consistency, and timeliness. In traditional data processing workflows, data quality management largely relies on manually set static rules or predefined quality inspection standards. While some technical means can achieve basic monitoring of data quality, such as data cleaning and verification, these methods often fall short when dealing with large-scale, multi-source, heterogeneous data. In particular, when data quality anomalies occur, traditional methods often lack real-time automatic repair mechanisms, leading to problem backlog or delayed repairs. This not only affects the reliability of data quality but also increases the cost and risk of manual intervention.

[0005] Finally, existing data platform systems also have shortcomings in terms of automation and intelligence in data processing. Currently, most systems rely on fixed ETL (Extract, Transform, Load) processes for data integration, and these processes are generally pre-configured and lack flexible adaptive capabilities. As business changes, traditional data processing processes may fail to adjust in a timely manner, leading to bottlenecks, repetitive operations, and inefficient processing, thus affecting the real-time performance and efficiency of data processing. Furthermore, real-time monitoring and intelligent decision-making mechanisms for data quality are not yet fully implemented, and the lack of automatic adjustment and optimization capabilities based on dynamic feedback further increases the complexity and difficulty of data platform management.

[0006] In summary, existing technologies have varying degrees of shortcomings in metadata management, data quality control, and intelligent optimization of data processing workflows. Especially when facing constantly changing business needs and diverse data sources, traditional systems often prove inflexible, inaccurate, and inefficient. Therefore, a new technological solution is urgently needed to address these issues, enabling data platforms to operate more intelligently and efficiently to meet rapidly changing business demands and high-quality data management requirements. Summary of the Invention

[0007] The purpose of this invention is to propose a data management method and system for a data middle platform based on metadata. By introducing an adaptive metadata evolution mechanism and a multi-level data quality early warning and repair mechanism, it overcomes the shortcomings of existing technologies and provides an efficient, flexible and intelligent data middle platform management solution.

[0008] To achieve the above objectives, a metadata-driven data management method for a data middle platform is provided in a first aspect of the present invention, the method comprising the following steps:

[0009] By connecting to various heterogeneous data sources through multiple access methods, raw data is obtained, and a pre-set preliminary metadata model is used to perform preliminary analysis on the raw data.

[0010] Based on the historical flow patterns, change rules, and business needs of the original data, the metadata model is automatically adjusted and optimized to obtain an adaptive metadata model after adaptive evolution.

[0011] Machine learning methods are used to identify patterns in the changes of various fields in historical data, generating evolution rules for metadata evolution; wherein, the optimization objective of the evolution rules is to maximize the predictive accuracy and adaptability of the adaptive metadata model;

[0012] It will monitor the differences between the data source and the adaptive metadata model in real time, and automatically provide feedback and adjust the adaptive metadata model when a deviation is detected.

[0013] The adaptive metadata model is used to monitor the quality of the data source in real time, and early warning rules are designed based on the quality of the data source to generate an early warning rule set; the early warning rules include:

[0014] When the data quality of a certain field exceeds the preset tolerance range, an alarm will be automatically triggered. When generating an alarm rule, the deweighting of the field will be assigned different weight values ​​according to the importance of the field based on the business weight in the metadata model. At the same time, based on the quality assessment results of each field, the data problems with the highest severity will be monitored with the highest priority. If the missing values ​​of a certain field exceed 30% and it is a core business field, an alarm will be triggered immediately.

[0015] When a data quality issue triggers an alert, a feedback mechanism will be used to repair the data and output a repair feedback dataset. The specific repair method depends on the field type and the severity of the data anomaly.

[0016] If data in a certain field is missing, its historical data trend will be checked first, and the missing data will be filled based on the trend or average value.

[0017] If the value of a field exceeds the predetermined reasonable range, an attempt will be made to correct it;

[0018] For time series data, time windows and historical data will be used for forecasting to ensure data consistency.

[0019] Based on the repair feedback dataset, a machine learning-based repair model for data quality issues is designed. Appropriate repair methods are selected and optimized for different field types and specific data repair methods to generate the final repaired dataset.

[0020] The design addresses data quality issues using a machine learning-based remediation model, including:

[0021] If data in a certain field is missing, a multi-layered adaptive repair mechanism is designed to fill the missing data, including:

[0022] If data is missing in a certain field, an appropriate regression model or neural network will be selected to predict and fill the missing data based on the correlation between fields and the distribution of the data.

[0023] If outliers occur, a deep learning-based variational autoencoder model is introduced to perform anomaly repair based on the deep structure of the data.

[0024] For time series data, time series modeling is used to correct the time field to ensure the consistency of the data in time series.

[0025] Furthermore, the various access methods include API, database connection, and file import; the data source type is one of relational database, non-relational database, file system, and data lake.

[0026] Furthermore, the preliminary metadata model includes the data source's structural information, field names, field types, and field relationships.

[0027] Furthermore, during the evolution of the initial metadata model, the patterns of change in each field are captured in real time, and the metadata model is dynamically adjusted using the following formula:

[0028] ;

[0029] in, This represents the evolved metadata model, recording the changes in fields and their relationships; It is the initial metadata model; Indicates a field at a specific point in time. Changes on; is the change weighting coefficient, representing the influence of changes in different fields; n is the total number of fields.

[0030] Furthermore, the evolution rule set for metadata evolution includes rules for adjusting and optimizing the metadata model, denoted as follows: ,in, It is The evolution rules describe how to handle changes in field types or adjustments to relationships; This indicates the time when the rule was generated; the evolving rule will be updated over time based on changes in actual data.

[0031] When a deviation is detected in the adaptive metadata model, automatic feedback is provided and the adaptive metadata model is adjusted, as follows:

[0032] Assuming that in the feedback mechanism, based on changes in business needs or data quality issues, the metadata model is ultimately adjusted as follows:

[0033] ;

[0034] in, This is the final adjusted metadata model; It is a metadata model obtained through adaptive evolution; These are adjustments based on real-time data feedback.

[0035] Furthermore, the adaptive quality assessment algorithm introduced in the real-time monitoring of data source quality using the adaptive metadata model dynamically adjusts the quality assessment criteria according to different data sources and business needs, as expressed as:

[0036] ;

[0037] in, This indicates the quality of the data source, containing quality information for each field, including metrics such as completeness, inconsistency, and accuracy. The format is as follows: ,in Representative Quality assessment results for each field; It is an evaluation function that combines field data. Metadata model and quality standards To calculate the quality of each field; It is an adaptive metadata model; It is a quality standard for each field.

[0038] Furthermore, the process for generating the early warning rule is as follows:

[0039] ;

[0040] in, It is the first Warning rules; It is an alert generation function that combines field data. Field weights and threshold To generate early warning rules; It is a field Importance weights; This is the quality threshold for that field;

[0041] The data repair via feedback mechanism is intelligent repair based on historical trends and machine learning, expressed as:

[0042] ;

[0043] in, It involves repairing the feedback dataset; This is data after a quality assessment; It is an incremental fix for the data.

[0044] Furthermore, further repairs are performed on the repair feedback dataset, as follows:

[0045] ;

[0046] in, This is the final repaired dataset, containing all the data that has been repaired; It is a repair feedback dataset, which includes the results of quality assessment and preliminary repairs; This indicates the incremental improvements made after optimization, i.e., the further improvements made.

[0047] Furthermore, a regularization term is introduced into the adaptive weighted regression model. To prevent overfitting, it is expressed as:

[0048] ;

[0049] in, It is a regularization loss function used to constrain model parameters; It is the regularization coefficient, which controls the strength of regularization; It is the first in the model The weights of each parameter.

[0050] Combined with regularization terms The goal of the adaptive weighted regression model is to minimize the sum of the prediction error and the regularization loss. The loss function is... Represented as:

[0051] ;

[0052] in, These are the predicted missing values; These are the actual missing values;

[0053] The loss function of the variational autoencoder model is expressed as:

[0054] ;

[0055] in, It is the loss function of the variational autoencoder; It is the distribution of the latent variables generated by the encoder; It is the distribution of sample data generated by the decoder; It is the KL divergence, which measures the difference between the encoder distribution and the prior distribution.

[0056] A second aspect of the present invention provides a metadata-driven data platform data management system, the system comprising:

[0057] The data source access unit is used to connect to various heterogeneous data sources through multiple access methods, obtain raw data, and perform preliminary analysis on the raw data using a preset preliminary metadata model.

[0058] The initial metamodel building unit is used to automatically adjust and optimize the metadata model based on the historical flow patterns, change rules, and business needs of the original data, so as to obtain an adaptive metadata model after adaptive evolution.

[0059] The metadata evolution unit is used to identify patterns in the changes of various fields in historical data using machine learning methods, and to generate evolution rules for metadata evolution; wherein, the optimization objective of the evolution rules is to maximize the prediction accuracy and adaptability of the adaptive metadata model;

[0060] The metamodel adjustment unit is used to automatically provide feedback and adjust the adaptive metadata model when the difference between the real-time monitoring data source and the adaptive metadata model is detected.

[0061] A metadata early warning unit is used to monitor the quality of the data source in real time using the adaptive metadata model, and to design early warning rules based on the quality of the data source, generating an early warning rule set; the early warning rules include:

[0062] When the data quality of a certain field exceeds the preset tolerance range, an alarm will be automatically triggered. When generating an alarm rule, the deweighting of the field will be assigned different weight values ​​according to the importance of the field based on the business weight in the metadata model. At the same time, based on the quality assessment results of each field, the data problems with the highest severity will be monitored with the highest priority. If the missing values ​​of a certain field exceed 30% and it is a core business field, an alarm will be triggered immediately.

[0063] The metadata repair unit is used to repair data through a feedback mechanism when a data quality issue triggers an alert, outputting a repair feedback dataset. The specific repair method is determined based on the field type and the degree of data anomaly.

[0064] If data in a certain field is missing, its historical data trend will be checked first, and the missing data will be filled based on the trend or average value.

[0065] If the value of a field exceeds the predetermined reasonable range, an attempt will be made to correct it;

[0066] For time series data, time windows and historical data will be used for forecasting to ensure data consistency.

[0067] The metadata re-repair unit is used to design a machine learning-based repair model for data quality issues based on the repair feedback dataset. It selects and optimizes appropriate repair methods for different field types and specific data repair methods, and generates the final repaired dataset.

[0068] The design addresses data quality issues using a machine learning-based remediation model, including:

[0069] If data in a certain field is missing, a multi-layered adaptive repair mechanism is designed to fill the missing data, including:

[0070] If data is missing in a certain field, an appropriate regression model or neural network will be selected to predict and fill the missing data based on the correlation between fields and the distribution of the data.

[0071] If outliers occur, a deep learning-based variational autoencoder model is introduced to perform anomaly repair based on the deep structure of the data.

[0072] The data correction unit is used to correct the time field of time series data using time series modeling to ensure the consistency of the data in time series.

[0073] The beneficial technical effects of the present invention are at least as follows:

[0074] This invention addresses the problem of outdated metadata by introducing an adaptive metadata evolution mechanism. This mechanism automatically derives and updates the metadata model based on data flow and changes in business needs, ensuring it remains synchronized with data sources and business requirements. This mechanism not only reduces manual intervention but also allows for timely adjustments to the metadata structure and rules when new data sources are added or business needs change, preventing data from becoming disconnected from the metadata model and improving the efficiency and accuracy of data processing.

[0075] This invention overcomes the limitations of traditional static data quality management through a multi-layered data quality management system. Under this mechanism, the system can monitor data quality in real time at the field, record, and dataset levels, detecting issues such as missing values, outliers, and duplicate data. Simultaneously, when data quality problems are detected, the system can automatically trigger a repair mechanism, automatically filling in missing values, correcting outliers, and removing duplicate data according to preset rules, ensuring data accuracy and consistency. This mechanism significantly reduces manual intervention and improves the efficiency and automation level of data quality repair.

[0076] Through its intelligent decision-making module, this invention can automatically optimize data flow paths and data cleaning rules based on real-time feedback during data processing, thereby improving the automation and efficiency of data processing. The intelligent decision-making module can flexibly adjust data integration and processing flows based on real-time feedback, optimizing the ETL process, reducing unnecessary repetitive operations and bottlenecks in data processing, and thus improving overall data processing speed and accuracy.

[0077] In summary, this invention effectively solves the problems of metadata lag, insufficient data quality management, and inefficient data processing automation in existing technologies through an adaptive metadata evolution mechanism and a real-time multi-level data quality early warning and repair mechanism. It improves the flexibility, intelligence level, and processing efficiency of the data platform, thereby providing enterprises with more efficient and reliable data management capabilities. Attached Figure Description

[0078] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0079] Figure 1 This is a flowchart of the data management method for an intelligent data platform based on metadata-driven technology according to the present invention. Detailed Implementation

[0080] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0081] like Figure 1 As shown in the embodiment of the present invention, a data management method for an intelligent data platform based on metadata-driven architecture is provided. The method includes:

[0082] 1011. Connect to various heterogeneous data sources through multiple access methods to obtain raw data, and perform preliminary analysis on the raw data using a preset preliminary metadata model.

[0083] Specifically, this step first connects to various heterogeneous data sources through multiple access methods (such as API, database connection, file import, etc.) to obtain the raw data. The data source type can be a relational database, a non-relational database, a file system, a data lake, etc.

[0084] A preliminary analysis of the accessed data source is conducted, focusing primarily on the data's basic attributes, such as field names, data types, and data structures. This information will be transformed into a preliminary metadata model. Through the data metadata structure extraction process, field types, relationships between fields, and the presence of redundant or missing fields can be identified.

[0085] The output is a preliminary metadata model. It records the structure information, field names, field types, and field relationships of the original data source.

[0086] For example, suppose when accessing a data source, the system recognizes that it contains the following fields:

[0087] Field 1: user_id (Type: Integer)

[0088] Field 2: timestamp (type: timestamp)

[0089] Field 3: order_amount (Type: Floating value)

[0090] This field information will build a preliminary metadata model, including field names, data types, and relationships between fields.

[0091] 1012. Based on the historical flow patterns, change rules, and business needs of the original data, automatically adjust and optimize the metadata model to obtain an adaptively evolved metadata model.

[0092] Specifically, because data changes at different points in time with changes in the external environment and business needs, the initial metadata model usually has certain biases or incompleteness.

[0093] Dynamic data flow analysis: Based on historical data flow, the system can identify patterns in field changes. For example, if a field has different names or data types across multiple data sources, the system can determine the dynamic mapping relationship between fields based on pattern recognition.

[0094] Model evolution rule generation: By continuously tracking historical data and trends, the system will automatically generate rules for metadata evolution. For example, if a field named `order_amount` frequently changes its data type from integer to floating-point values, the system will generate rules for dynamically adjusting the data type of that field.

[0095] The evolution mechanism not only focuses on changes in data structure, but also predicts and adapts to future data structures based on changes in business needs and data update cycles. For example, if a business scenario is to add new fields or undergo changes in data patterns, the system will predict the evolution trend of the data structure based on past data flow patterns and business needs.

[0096] Metadata model after adaptive evolution of output variables It has been adjusted according to the changing patterns of the data source and business needs.

[0097] Furthermore, during the evolution of the metadata model, the system will capture the patterns of change in each field in real time and use the following formula to dynamically adjust the metadata model:

[0098] ;

[0099] in, This represents the evolved metadata model, recording the changes in fields and their relationships; It is the initial metadata model; Indicates a field at a specific point in time. The changes in the data may include changes in field type, addition of fields, deletion of fields, etc. This is the change weighting coefficient, representing the impact of changes in different fields. It is dynamically adjusted based on factors such as the frequency of historical data and the importance of the changes. (The formula contains...) Part of it innovatively introduces a weighted variation coefficient. This coefficient reflects the impact of changes in different fields on the final model adjustment. This weighted coefficient is based not only on the frequency of field changes but also on the priority of business requirements. For example, a field might be a core business field, and its changes will be given higher weight, thus affecting the focus of model evolution.

[0100] 1013. Use machine learning methods to identify the patterns of change in each field in historical data and generate evolution rules for metadata evolution; wherein, the optimization objective of the evolution rules is to maximize the prediction accuracy and adaptability of the adaptive metadata model.

[0101] Specifically, the evolution rules are not static but are optimized based on new data changes as the system continues to run. Each data change triggers feedback adjustments to the existing rules. For example, if a field does not achieve the expected results under past rule adjustments, the system will automatically learn this mismatch and update the evolution rules.

[0102] The optimization objective of the evolutionary rules is to maximize the predictive accuracy and adaptability of the data model. By constructing a global optimization algorithm, the system can adjust the rules based on feedback from a large amount of historical data, thereby improving the model's adaptability.

[0103] Output variable: Evolutionary rule set It contains rules for adjusting and optimizing the metadata model. Evolutionary rule set. It can be represented as a set of dynamically updated rules:

[0104]

[0105] in, It is The evolution rules describe how to handle changes in field types or adjustments to relationships; This indicates the time when the rule is generated; the evolutionary rule will be updated over time according to changes in actual data. In the formula, the evolutionary rule... Continuous updates over time demonstrate the dynamic adaptability of the rules. This is an innovative mechanism that adjusts rules in real time based on changes in data flow and business needs. Through continuous learning, the system can effectively avoid the problems of traditional rules becoming outdated or failing to meet actual needs.

[0106] Furthermore, dynamic weights and time-series optimizations (such as weighted time-series loss functions) make rule generation and optimization more aligned with business objectives and data change patterns.

[0107] 1014. Real-time monitoring of the differences between the data source and the adaptive metadata model; when a deviation is detected in the adaptive metadata model, automatic feedback and adjustment of the adaptive metadata model will be provided.

[0108] Specifically, this step introduces a feedback and repair algorithm. Based on changes in the new data, feedback update items are automatically generated to adjust the metadata model in real time to reduce the difference between the model and the actual data.

[0109] The output variables are the final metadata model adjusted based on real-time feedback. This ensures a high degree of matching with the actual data environment.

[0110] Assuming that in the feedback mechanism, based on changes in business needs or data quality issues, the metadata model is ultimately adjusted as follows:

[0111]

[0112] in, This is the final adjusted metadata model; It is a metadata model obtained through adaptive evolution; These are adjustments based on real-time data feedback.

[0113] The innovation of the feedback and repair algorithm lies in the introduction of adaptive adjustment and business-demand-driven feedback terms. Feedback terms It goes beyond simply supplementing data based on changes; it also incorporates dynamic changes in business needs, ensuring that the data model is not only technically accurate but also meets the requirements of business objectives.

[0114] 1015. Use the adaptive metadata model to monitor the quality of the data source in real time, and design early warning rules based on the quality of the data source to generate an early warning rule set.

[0115] Specifically, data quality assessment and monitoring are fundamental to ensuring the stable operation of the data processing system. In this step, the system will use the adaptive metadata model generated in the previous step ( To monitor data quality, focus on checking the following dimensions:

[0116] Data integrity: based on The system first assesses the integrity of each data field, checking for missing values, duplicate values, etc. In the metadata model, each field has a predefined "integrity standard," for example, the proportion of missing values ​​in a certain field cannot exceed 30%.

[0117] Data consistency: Based on business logic, the system determines whether changes in field values ​​are as expected. For example, the data in the timestamp field should strictly increase in chronological order. If reversed or skipped time values ​​are found, the field is considered to have a consistency problem.

[0118] Data accuracy: Accuracy is the standard for evaluating whether data matches its actual object or historical record. By comparing data with historical data, if a field value deviates from the normal range, it will be marked as an anomaly.

[0119] These quality assessments are based on a comparison of the data with the metadata model, using field information, data relationships, and business logic within the model to determine whether the quality meets the standards.

[0120] The output variable is It contains quality information for each field, including metrics such as completeness, inconsistency, and accuracy, in the following format: ,in Representing the Quality assessment results for each field.

[0121] Furthermore, this step introduces an adaptive quality assessment algorithm. This algorithm can dynamically adjust the quality assessment standards according to different data sources and business needs, greatly enhancing the system's adaptability in complex and dynamic environments. Traditional methods typically use fixed quality standards, making them difficult to adapt to rapidly changing business environments, while the system of this invention can intelligently adjust the quality assessment rules. The quality assessment process based on the metadata model can be represented by the following formula:

[0122] ;

[0123] in, It is an evaluation function that combines field data. Metadata model and quality standards To calculate the quality of each field. It is an adaptive metadata model that includes information such as field definitions, data types, and standard values. These are quality standards for each field, which may include missing value thresholds, outlier ranges, time series standards, etc.

[0124] Furthermore, based on the assessed quality indicators, the system will intelligently generate a set of early warning rules. When the data quality of a certain field exceeds the preset tolerance range, the system will automatically trigger an alarm. When generating early warning rules, the system will consider the following factors:

[0125] Field Importance: Based on business weights in the metadata model, the importance of a field is assigned different weight values. For example, order_amount is a critical field, and its exceptions have a significant impact, so it has a high weight.

[0126] Severity of quality issues: Based on the quality assessment results of each field, data issues with higher severity will be monitored first. If the missing values ​​of a field exceed 30% and it is a core business field, the system should immediately trigger an alert.

[0127] Furthermore, the generation of early warning rules not only relies on the current data quality assessment but also dynamically adjusts based on historical data trends. Through historical data analysis, the system can determine which fields have exhibited quality issues in the past and generate corresponding early warning mechanisms.

[0128] Furthermore, the output variable represents the generated set of early warning rules. It contains multiple rules, each of which can define a threshold, impact weight, and triggering condition for a quality issue.

[0129] This solution innovatively introduces a dynamic early warning generation mechanism based on field weights. Compared with traditional monitoring systems based on fixed rules, this system is more flexible and can automatically adjust early warning strategies according to business importance, reducing unnecessary early warnings and improving the accuracy and effectiveness of monitoring. The early warning rule generation process is as follows:

[0130] ;

[0131] in, It is Warning rules; It is an alert generation function that combines field data. Field weights and threshold To generate early warning rules; It is a field The importance weights are derived from the metadata model; This is the quality threshold for the field; an alert is triggered when the proportion of missing values ​​exceeds 30%.

[0132] 1016. When a data quality issue triggers an alert, the data will be repaired through a feedback mechanism, and a repair feedback dataset will be output.

[0133] Specifically, once a data quality issue triggers an alert, the system will repair the data through a feedback mechanism. The specific repair method depends on the field type and the severity of the data anomaly.

[0134] Missing value repair: If data in a field is missing, the system will first check its historical data trend and fill in the missing data based on the trend or average. For example, if the average value of order_amount over the past month is... The missing data may be filled with... .

[0135] Outlier Correction: If the value of a field exceeds the predetermined reasonable range, the system will attempt to correct it. For example, if an `order_amount` exceeds the maximum preset value (e.g., ...), the system will attempt to correct it. The system will predict the correction value using the trained model.

[0136] Consistency Restoration: For time series data, the system will use time windows and historical data for prediction to ensure data consistency. For example, if the timestamp field has a reverse order issue, the system will re-sort it and use the trend of previous and subsequent time points to fill in the data.

[0137] Furthermore, the repaired data will re-enter the quality assessment process to ensure that the data quality is restored to the normal range, and will eventually be provided to the data processing module for continued use.

[0138] The output variable is This indicates the repaired dataset, which includes all data quality issues that have been fixed through the alert and feedback mechanisms.

[0139] The innovation of this step lies in the fact that the repair strategy does not rely solely on simple interpolation or mean filling, but introduces an intelligent repair mechanism based on historical trends and machine learning, making it particularly suitable for high-quality data repair in dynamic business environments. The data repair process can be represented by the following formula:

[0140] ;

[0141] in, This is the repaired data; This is data after a quality assessment; It is an incremental data repair tool that includes filling in missing values ​​and correcting outliers.

[0142] 1017. Based on the repair feedback dataset, design a machine learning-based repair model for data quality issues. Select appropriate repair methods for different field types and specific data repair methods, optimize them, and generate the final repaired dataset.

[0143] Specifically, the core task of data repair is to select appropriate repair methods for different types of data quality problems through intelligent mechanisms. These quality problems typically include, but are not limited to, missing values, outliers, duplicate data, and inconsistent data.

[0144] Missing value repair: A multi-level adaptive repair mechanism is designed for missing data. Specifically, if data in a certain field is missing, the system will select an appropriate regression model or neural network to predict and fill in the missing data based on the correlation between fields and the distribution of the data.

[0145] Outlier Repair: In addition to traditional anomaly detection methods (such as Isolation Forest and K-Nearest Neighbors), this invention introduces a deep learning-based Variational Autoencoder (VAE) model for outlier repair. This model can not only effectively detect outliers but also repair them based on the deep structure of the data.

[0146] Consistency Repair: If there are inconsistencies in the time series of data (e.g., abnormal order of timestamp fields), the system will use time series modeling (such as LSTM, ARIMA, etc.) to correct the time fields and ensure the consistency of the data in time series.

[0147] Furthermore, based on these repair strategies, the goal of this invention is to ultimately output a high-quality dataset that has undergone comprehensive repair and optimization. This dataset will provide a reliable foundation for subsequent data analysis, prediction, and decision-making.

[0148] Furthermore, Formula 1: A formalized representation of the data repair process, the repaired dataset can be represented as:

[0149] ;

[0150] in, This represents the final repaired dataset, which contains all the data that has been repaired. This indicates the preliminary repair dataset, which includes the results of quality assessment and initial repair. This indicates the incremental repair after optimization, that is, the further repair part, such as filling missing values, correcting outliers, etc.

[0151] Furthermore, to achieve efficient repair, this invention employs a series of machine learning algorithms targeting data quality issues. These algorithms can automatically select the most suitable repair model based on the data type and quality problem:

[0152] Missing Value Repair Model: This invention employs an adaptive weighted regression model. This model calculates weighted coefficients based on the relationship between the missing field and other relevant fields, and then uses these coefficients to make weighted predictions for the missing values. Furthermore, this invention introduces a regularization term to prevent overfitting and ensure the model's generalization ability.

[0153] Outlier Detection and Repair: In terms of outlier repair, traditional isolated forest methods may ignore the deep features of the data. Therefore, this invention introduces a variational autoencoder (VAE) as an outlier repair model. The VAE can learn the distribution from the latent space of the data and repair outliers based on the learned latent structure.

[0154] Consistency Repair: To address the inconsistency problem in time series, this invention employs a method based on Long Short-Term Memory (LSTM) networks. This method can learn the long-term dependencies of time fields, thereby ensuring the sequential consistency of time fields.

[0155] Furthermore, an adaptive weighted regression and regularized missing value repair model: To improve the accuracy of missing value repair, this invention designs a weighted regression model that dynamically calculates weights based on the correlation between each data point and other points, and uses a regularization term to prevent overfitting. The regularization term designed in this invention is as follows:

[0156] ;

[0157] in, This represents the regularization loss function, used to constrain model parameters. This represents the regularization coefficient, which controls the strength of regularization. In the model, the first The weights of each parameter.

[0158] Objective function: Including the regularization term, the ultimate goal is to minimize the sum of prediction error and regularization loss.

[0159] ;

[0160] in, This indicates the predicted missing values. This represents the actual missing values ​​(used to calculate error during training). This repair method can dynamically adjust weights based on data diversity, thus effectively handling different data quality issues.

[0161] For outlier repair, a variational autoencoder (VAE) is used as the repair model. By learning the latent space of the data, the VAE can effectively detect and repair outliers. This invention uses the following model:

[0162] ;

[0163] in, This represents the loss function of the variational autoencoder. This represents the distribution of latent variables generated by the encoder. This indicates the distribution of sample data generated by the decoder. This represents the KL divergence, which measures the difference between the encoder distribution and the prior distribution.

[0164] 1018. For time series data, use time series modeling to correct the time field to ensure the consistency of the data in time series.

[0165] Specifically, once the data is repaired and an optimized dataset is generated... The system will then enter a feedback and self-optimization phase. To ensure the long-term effectiveness of the repair process, the system will automatically provide quality feedback and update the model, forming a continuous optimization mechanism. The repaired dataset will then enter an automatic quality assessment system, which will automatically evaluate whether the data quality meets the expected standards. If the data quality still does not meet the standards, the system will automatically select other repair models or update the existing model.

[0166] Furthermore, repair model It will continuously train and adjust based on new data feedback. Whenever new data is corrected and fed back, the corrected model will be updated to adapt to the new data features.

[0167] The iterative update of the repair model can be represented as:

[0168] ;

[0169] in, This indicates the updated and repaired model. This indicates the current repair model. This indicates the incremental update of the model, adjusting the model parameters based on the newly corrected feedback data.

[0170] This invention also provides a metadata-driven data platform data management system, characterized in that the system includes:

[0171] The data source access unit is used to connect to various heterogeneous data sources through multiple access methods, obtain raw data, and perform preliminary analysis on the raw data using a preset preliminary metadata model.

[0172] The initial metamodel building unit is used to automatically adjust and optimize the metadata model based on the historical flow patterns, change rules, and business needs of the original data, so as to obtain an adaptive metadata model after adaptive evolution.

[0173] The metadata evolution unit is used to identify patterns in the changes of various fields in historical data using machine learning methods, and to generate evolution rules for metadata evolution; wherein, the optimization objective of the evolution rules is to maximize the prediction accuracy and adaptability of the adaptive metadata model;

[0174] The metamodel adjustment unit is used to automatically provide feedback and adjust the adaptive metadata model when the difference between the real-time monitoring data source and the adaptive metadata model is detected.

[0175] A metadata early warning unit is used to monitor the quality of the data source in real time using the adaptive metadata model, and to design early warning rules based on the quality of the data source, generating an early warning rule set; the early warning rules include:

[0176] When the data quality of a certain field exceeds the preset tolerance range, an alarm will be automatically triggered. When generating an alarm rule, the deweighting of the field will be assigned different weight values ​​according to the importance of the field based on the business weight in the metadata model. At the same time, based on the quality assessment results of each field, the data problems with the highest severity will be monitored with the highest priority. If the missing values ​​of a certain field exceed 30% and it is a core business field, an alarm will be triggered immediately.

[0177] The metadata repair unit is used to repair data through a feedback mechanism when a data quality issue triggers an alert, outputting a repair feedback dataset. The specific repair method is determined based on the field type and the degree of data anomaly.

[0178] If data in a certain field is missing, its historical data trend will be checked first, and the missing data will be filled based on the trend or average value.

[0179] If the value of a field exceeds the predetermined reasonable range, an attempt will be made to correct it;

[0180] For time series data, time windows and historical data will be used for forecasting to ensure data consistency.

[0181] The metadata re-repair unit is used to design a machine learning-based repair model for data quality issues based on the repair feedback dataset. It selects and optimizes appropriate repair methods for different field types and specific data repair methods, and generates the final repaired dataset.

[0182] The design addresses data quality issues using a machine learning-based remediation model, including:

[0183] If data in a certain field is missing, a multi-layered adaptive repair mechanism is designed to fill the missing data, including:

[0184] If data is missing in a certain field, an appropriate regression model or neural network will be selected to predict and fill the missing data based on the correlation between fields and the distribution of the data.

[0185] If outliers occur, a deep learning-based variational autoencoder model is introduced to perform anomaly repair based on the deep structure of the data.

[0186] The data correction unit is used to correct the time field of time series data using time series modeling to ensure the consistency of the data in time series.

[0187] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0188] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0189] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0190] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A metadata-driven data platform data management method, characterized in that, The method comprises the following steps: By connecting to various heterogeneous data sources through multiple access methods, raw data is obtained, and a pre-set preliminary metadata model is used to perform preliminary analysis on the raw data. Based on the historical flow patterns, change rules, and business needs of the original data, the metadata model is automatically adjusted and optimized to obtain an adaptive metadata model after adaptive evolution. Machine learning methods are used to identify patterns in the changes of various fields in historical data, generating evolution rules for metadata evolution; wherein, the optimization objective of the evolution rules is to maximize the predictive accuracy and adaptability of the adaptive metadata model; It will monitor the differences between the data source and the adaptive metadata model in real time, and automatically provide feedback and adjust the adaptive metadata model when a deviation is detected. The adaptive metadata model is used to monitor the quality of the data source in real time, and early warning rules are designed based on the quality of the data source to generate an early warning rule set; the early warning rule set includes: When the data quality of a certain field exceeds the preset tolerance range, an alarm will be automatically triggered based on factors, including the importance of the field and the severity of the data quality problem. The importance of the fields is determined by assigning different weight values ​​to fields based on their importance through business weights in the metadata model. The severity of the data quality issues is determined by triggering an alert immediately if a field has more than 30% missing values ​​and is a core business field, based on the quality assessment results of each field. When a data quality issue triggers an alert, a feedback mechanism will be used to repair the data and output a repair feedback dataset. The specific repair method depends on the field type and the severity of the data anomaly. If data in a certain field is missing, its historical data trend will be checked first, and the missing data will be filled based on the trend or average value. If the value of a field exceeds the predetermined reasonable range, an attempt will be made to correct it; For time series data, time windows and historical data will be used for forecasting to ensure data consistency. Based on the repair feedback dataset, a machine learning-based repair model for data quality issues is designed. Appropriate repair methods are selected and optimized for different field types and specific data repair methods to generate the final repaired dataset. The design addresses data quality issues using a machine learning-based remediation model, including: If data in a certain field is missing, a multi-layered adaptive repair mechanism is designed to fill the missing data, including: If data is missing in a certain field, an appropriate regression model or neural network will be selected to predict and fill the missing data based on the correlation between fields and the distribution of the data. If outliers occur, a deep learning-based variational autoencoder model is introduced to perform anomaly repair based on the deep structure of the data. For time series data, time series modeling is used to correct the time field to ensure the consistency of the data in time series.

2. The metadata-driven data management method for a data middle platform according to claim 1, characterized in that, The various access methods include API, database connection, and file import; the data source type is one of relational database, non-relational database, file system, and data lake.

3. The metadata-driven data management method for a data middle platform according to claim 1, characterized in that, The preliminary metadata model includes the data source's structural information, field names, field types, and field relationships.

4. The metadata-driven data management method for a data middle platform according to claim 1, characterized in that, During the evolution of the initial metadata model, the patterns of change in each field are captured in real time, and the metadata model is dynamically adjusted using the following formula: ； in, This represents the evolved metadata model, recording the changes in fields and their relationships; It is the initial metadata model; Indicates a field at a specific point in time. Changes on; is the change weighting coefficient, representing the influence of changes in different fields; n is the total number of fields.

5. The metadata-driven data management method for a data middle platform according to claim 4, characterized in that, The set of evolutionary rules for metadata evolution includes rules for adjusting and optimizing the metadata model, denoted as follows: ,in, It is The evolution rules describe how to handle changes in field types or adjustments to relationships; This indicates the time when the rule was generated; the evolving rule will be updated over time based on changes in actual data. When a deviation is detected in the adaptive metadata model, automatic feedback is provided and the adaptive metadata model is adjusted, as follows: Assuming that in the feedback mechanism, based on changes in business needs or data quality issues, the metadata model is ultimately adjusted as follows: ； in, This is the final adjusted metadata model; It is a metadata model obtained through adaptive evolution; These are adjustments based on real-time data feedback.

6. The metadata-driven data management method for a data middle platform according to claim 1, characterized in that, In the process of using the adaptive metadata model to monitor the quality of data sources in real time, an adaptive quality assessment algorithm is introduced to dynamically adjust the quality assessment criteria according to different data sources and business needs, expressed as: ； in, This indicates the quality of the data source, containing quality information for each field, including completeness, inconsistency, and accuracy metrics, in the following format: ,in Representing the Quality assessment results for each field; It is an evaluation function that combines Metadata model and quality standards To calculate the quality of each field; It is an adaptive metadata model; It is a quality standard for each field.

7. The metadata-driven data management method for a data middle platform according to claim 6, characterized in that, The process of generating the early warning rules is as follows: ； in, It is Warning rules; It is an early warning generation function, combining... Importance weight and threshold To generate early warning rules; It is a field Importance weights; This is the quality threshold for that field; The data repair via feedback mechanism is intelligent repair based on historical trends and machine learning, expressed as: ； in, It involves repairing the feedback dataset; This is data after a quality assessment; It is an incremental fix for the data.

8. The metadata-driven data management method for a data middle platform according to claim 7, characterized in that, Further repairs are performed on the repair feedback dataset, as follows: ； in, This is the final repaired dataset, containing all the data that has been repaired; It is a repair feedback dataset, which includes the results of quality assessment and preliminary repairs; This indicates the incremental improvements made after optimization, i.e., the further improvements made.

9. The metadata-driven data management method for a data middle platform according to claim 1, characterized in that, A regularization term is introduced into the machine learning-based repair model. To prevent overfitting, it is expressed as: ； in, It is a regularization loss function used to constrain model parameters; It is the regularization coefficient, which controls the strength of regularization; It is the first in the model The weights of each parameter; Combined with regularization terms The goal of machine learning's repair model is to minimize the sum of prediction error and regularization loss. The loss function is... Represented as: ； in, These are the predicted missing values; These are the actual missing values; The loss function of the variational autoencoder model is expressed as: ； in, It is the loss function of the variational autoencoder; It is the distribution of the latent variables generated by the encoder; It is the distribution of sample data generated by the decoder; It is the KL divergence, which measures the difference between the encoder distribution and the prior distribution.

10. A metadata-driven data platform data management system, characterized in that, The system includes: The data source access unit is used to connect to various heterogeneous data sources through multiple access methods, obtain raw data, and perform preliminary analysis on the raw data using a preset preliminary metadata model. The initial metamodel building unit is used to automatically adjust and optimize the metadata model based on the historical flow patterns, change rules, and business needs of the original data, so as to obtain an adaptive metadata model after adaptive evolution. The metadata evolution unit is used to identify patterns in the changes of various fields in historical data using machine learning methods, and to generate evolution rules for metadata evolution; wherein, the optimization objective of the evolution rules is to maximize the prediction accuracy and adaptability of the adaptive metadata model; The metamodel adjustment unit is used to automatically provide feedback and adjust the adaptive metadata model when the difference between the real-time monitoring data source and the adaptive metadata model is detected. A metadata early warning unit is used to monitor the quality of the data source in real time using the adaptive metadata model, and to design early warning rules based on the quality of the data source, generating an early warning rule set; the early warning rule set includes: When the data quality of a certain field exceeds the preset tolerance range, an alarm will be automatically triggered based on factors, including the importance of the field and the severity of the data quality problem. The importance of the fields is determined by assigning different weight values ​​to fields based on their importance through business weights in the metadata model. The severity of the data quality issues is determined by triggering an alert immediately if a field has more than 30% missing values ​​and is a core business field, based on the quality assessment results of each field. The metadata repair unit is used to repair data when a data quality issue triggers an alert, outputting a repair feedback dataset through a feedback mechanism. The specific repair method depends on the field type and the severity of the data anomaly. If data in a certain field is missing, its historical data trend will be checked first, and the missing data will be filled based on the trend or average value. If the value of a field exceeds the predetermined reasonable range, an attempt will be made to correct it; For time series data, time windows and historical data will be used for forecasting to ensure data consistency. The metadata re-repair unit is used to design a machine learning-based repair model for data quality issues based on the repair feedback dataset. It selects and optimizes appropriate repair methods for different field types and specific data repair methods, and generates the final repaired dataset. The design addresses a machine learning-based remediation model for data quality issues, including: If data in a certain field is missing, a multi-layered adaptive repair mechanism is designed to fill the missing data, including: If data is missing in a certain field, an appropriate regression model or neural network will be selected to predict and fill the missing data based on the correlation between fields and the distribution of the data. If outliers occur, a deep learning-based variational autoencoder model is introduced to perform anomaly repair based on the deep structure of the data. The data correction unit is used to correct the time field of time series data using time series modeling to ensure the consistency of the data in time series.

Images