Adaptive metadata collection and change tracking system

Abstract

The present invention discloses an adaptive metadata collection and change tracking system, which relates to the field of metadata processing technology, and includes a data management center, wherein the data management center is communicatively connected with a data source identification module, a data analysis module, an incremental data collection module, a real-time data processing module, a metadata warehouse module, a change tracking and history recording module, a data quality monitoring module, and a user interface module, wherein electrical signals are connected between the modules. The present invention ensures the accuracy and consistency of data by real-time monitoring and evaluating data quality, and reduces the need for manual intervention and the risk of human error by automatically identifying and collecting new metadata sources, including unknown or dynamically changing data. The system can continuously track and record the change history of metadata, and timely discover and repair data quality problems, thereby effectively improving data quality.

Description

Technical Field

[0001] The present invention relates to the field of metadata processing technology, and in particular to an adaptive metadata collection and change tracking system. Background Art

[0002] With the acceleration of the digitalization process, the amount of data faced by enterprises is growing exponentially, and enterprises usually have a large number of data sources, including databases, APIs, files, IoT devices, etc., but data sources are diverse and dynamic. There are both known and fixed data sources and unknown and dynamically changing data sources. With the increase in data volume and changes in data sources, the complexity and challenges of data management are also increasing. Therefore, it is necessary to manage dynamically changing data sources and track the change history of data to ensure data availability and security.

[0003] In the prior art, when faced with unknown or dynamically changing data sources, it takes a certain amount of time to adapt and process, which can easily lead to delayed data collection or incomplete data. As a result, it is difficult to record all change details when tracking and managing metadata change history, especially when the change frequency is high or the changes involve multiple data sources, which may lead to incomplete change history records. Therefore, how to process unknown or dynamically changing data, effectively track and manage metadata change history, and reduce change tracking errors are the problems we need to solve. To this end, an adaptive metadata collection and change tracking system is proposed. Summary of the invention

[0004] The present invention aims to provide an adaptive metadata collection and change tracking system to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] The adaptive metadata collection and change tracking system comprises a data management center, wherein the data management center is communicatively connected with a data source identification module, a data analysis module, an incremental data collection module, a real-time data processing module, a metadata warehouse module, a change tracking and history recording module, a data quality monitoring module and a user interface module, wherein the modules are connected by electrical signals;

[0007] The data source identification module is used to identify data sources, including databases, file systems, and APIs, and to connect and adapt to various unknown and dynamically changing data sources, and convert data from the data sources into a format that can be processed by the system;

[0008] The data parsing module is used to parse and standardize data from various data sources, including structured, semi-structured and unstructured data;

[0009] The incremental data acquisition module is used to incrementally acquire data and only processes the changed data to reduce the delay of data acquisition;

[0010] The real-time data processing module is used to process data streams in real time, update metadata and change history records, and reduce change tracking errors;

[0011] The metadata warehouse module is used to manage and store metadata, including data table structure, field information, data type, data source, update frequency, and provide a unified metadata view to facilitate user query and management, helping users to better understand and use data;

[0012] The change tracking and history recording module is used to track and record the change history of metadata, including metadata addition, deletion and modification operations, to reduce change tracking errors;

[0013] The data quality monitoring module is used to monitor the quality and integrity of data in real time and provide a feedback mechanism to promptly discover and fix data quality issues to ensure data accuracy and consistency;

[0014] The user interface module is used to provide a visual interactive interface to assist users in querying data, viewing metadata, and tracking change history.

[0015] A further improvement of the technical solution of the present invention is that: in the data source identification module, the data source identification and data conversion process of the data source includes:

[0016] Automatically detect potential data sources by scanning the network environment and system configuration, and identify the types of data sources, including databases, file systems, and APIs;

[0017] Conduct a preliminary analysis of unknown data sources to determine whether they are suitable as the input data source of the system, and establish a connection with the data source based on the type and configuration information of the data source. For database data sources, you need to provide the database address, port, user name, and password information. For file system data sources, you need to specify the file path and access permissions. For API data sources, you need to understand the API interface specifications, authentication methods, and call frequency.

[0018] Analyze the data structure and format of the data source, determine the rules and mapping relationships for data conversion, and use data conversion tools to convert the data of the data source into a format that the system can process. For structured data, directly perform field mapping and type conversion. For semi-structured and unstructured data, perform parsing, cleaning, and reconstruction operations. It also supports dynamic addition, modification, and deletion of data source configurations. When a new data source is added, it can automatically identify and connect to the data source. When the configuration information of the data source changes, it can be updated in real time and the connection can be re-established.

[0019] After the data conversion is completed, the data format is verified and the data validation rule tool is used to verify the integrity, accuracy and consistency of the data to ensure that the converted data meets the system requirements.

[0020] A further improvement of the technical solution of the present invention is that: in the data parsing module, the parsing process of various data sources includes:

[0021] Clarify the data types transmitted by the data source, including structured, semi-structured and unstructured data, and collect structured, semi-structured and unstructured data for preprocessing and parsing; structured data includes tables, JSON, XML, etc., which are data organized in a clearly defined format; semi-structured data includes log files, HTML pages, etc.; unstructured data includes text, images, audio, etc. Among them, the preprocessing process handles missing values, outliers, and duplicate data to ensure the accuracy and reliability of the data;

[0022] Different parsing strategies are used to parse data according to the data type. For structured data, the corresponding parser is used to parse the data and convert it into a data structure that the system can understand; for semi-structured data, natural language processing technology is used to extract the required information; for unstructured data, corresponding analysis technologies such as text analysis, image recognition, speech recognition, etc. are used to process and parse it to extract useful information.

[0023] A further improvement of the technical solution of the present invention is that: in the data parsing module, the standardization process of various data sources includes:

[0024] Standardize the parsed data and convert it into a unified format and structure so that the system can understand and process it, including data type conversion, field naming specifications, and data unit unification;

[0025] The parsed and standardized data is stored in the database for subsequent metadata collection and change tracking. Quality checks and verifications are performed by comparing the original data with the parsed data to ensure the accuracy and completeness of the data.

[0026] A further improvement of the technical solution of the present invention is that: in the incremental data acquisition module, the process of incremental data acquisition includes:

[0027] Perform a complete initial synchronization of the data source, capture the basic state of the data, and record the state of historically successfully collected data, including the timestamp of the collection and the amount of data collected. The time point of historically successfully collected data is the time point of the last successful data collection.

[0028] Compare the timestamp in the data source with the value recorded in the last successful collection status, interact with the data source, and identify the data that has changed;

[0029] According to the identified changed data, only the changed part of the data, i.e. the incremental data, is collected, and the collected incremental data is processed and stored;

[0030] After successfully completing the incremental data collection and processing, update the collection status and record the current collection timestamp and data volume information as a reference for the next collection.

[0031] A further improvement of the technical solution of the present invention is that in the real-time data processing module, the process of updating metadata and change history records includes:

[0032] Monitor data sources of various types in real time, capture real-time data streams, and preprocess and parse the captured data. Preprocessing includes data cleaning, format conversion, and denoising operations. Parse the data and convert it into a format that the system can recognize.

[0033] The pre-processed and parsed data is fed into the real-time processing flow to identify the changes in the data by comparing the current data with the last stored data;

[0034] Update metadata based on data changes. Metadata includes data table structure, field type, index information, and records data change history. Change history includes change time, change type, and change content. This helps the system track data change history, identify anomalies and problems, and handle them.

[0035] A further improvement of the technical solution of the present invention is that in the change tracking and history recording module, the process of tracking and recording the change history of metadata includes:

[0036] Set up structures and fields in the metadata warehouse to store metadata and its change history, and monitor change events in the metadata warehouse, including addition, deletion, and modification operations, and capture detailed information about the changes, including change time, change type, and change content; record the captured metadata change information in a dedicated change history table to ensure that each change record contains sufficient detailed information, and build a change history view to display the metadata change history.

[0037] A further improvement of the technical solution of the present invention is that in the data quality monitoring module, the process of data quality assessment and analysis includes:

[0038] Define metrics for assessing data quality, including accuracy, completeness, consistency, and timeliness, and randomly select data samples from data sources for quality checks;

[0039] Conduct quality assessment and analysis on the collected data samples to check whether they meet the defined data quality indicators and obtain the scores of each data quality indicator;

[0040] Extract the number of abnormal samples, the total number of samples, the weight of each indicator, and the score of each indicator, and comprehensively obtain the quality assessment coefficient to obtain the overall data quality score;

[0041] Based on the assessment results, identify existing data quality issues, conduct in-depth analysis of the identified data quality issues, and formulate a data quality repair plan based on the analysis results.

[0042] A further improvement of the technical solution of the present invention is that the calculation formula of the quality assessment coefficient is:

[0043]

[0044] Among them, Q i is the quality assessment coefficient, indicating the overall data quality score, w i is the weight of each indicator, n is the number of indicators, A i The score of each indicator reflects the performance of the indicator in data quality. is the impact factor of the abnormal situation, indicating the impact of the abnormal situation on the overall score, w y is the abnormal situation weight, w z is the sum of all indicator weights, is the proportion of abnormal situations, indicating the proportion of abnormal situations in the data, N y is the number of abnormal samples, N z is the total sample size. It should be noted that if Q i If the value trend of Q continues to be stable or gradually increases, it means that the data quality is improving or maintaining a good state. i If the value trend of continues to decline or fluctuates greatly, it means that there may be problems with the data quality and further analysis and processing are needed.

[0045] Due to the adoption of the above technical solution, the present invention has the following technical advances compared with the prior art:

[0046] 1. The present invention provides an adaptive metadata collection and change tracking system, which ensures data accuracy and consistency by real-time monitoring and evaluation of data quality, and reduces the need for manual intervention and the risk of human error by automatically identifying and collecting new metadata sources, including unknown or dynamically changing data. The system can continuously track and record the change history of metadata, promptly discover and fix data quality problems, thereby effectively improving data quality.

[0047] 2. The present invention provides an adaptive metadata collection and change tracking system, which establishes a complete metadata database, records and manages metadata information of various data sources, and tracks data change history to ensure data traceability and compliance. At the same time, it automatically performs data quality checks and compliance verifications to ensure the legality, integrity and security of data, reducing compliance risks in the data management process. By recording the change history of metadata, metadata version control and history tracking become more convenient and reliable, greatly reducing the complexity and workload of metadata management and improving management efficiency.

[0048] 3. The present invention provides an adaptive metadata collection and change tracking system, which automatically identifies and collects new data sources, monitors changes in data quality, and performs automatic processing and adjustment according to pre-set rules and policies, thereby reducing the cost and risk of manual intervention, saving human resources, and improving the efficiency and reliability of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0050] Figure 1 It is a module structure diagram of the present invention;

[0051] Figure 2 A flow chart of the identification of data sources and data conversion of data sources of the present invention;

[0052] Figure 3 This is a flow chart of incremental data collection of the present invention;

[0053] Figure 4 A flow chart for updating metadata and change history for the present invention;

[0054] Figure 5 The figure is a flow chart of data quality assessment and analysis of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Examples, such as Figure 1-5 As shown, the present invention provides an adaptive metadata collection and change tracking system, including a data management center, the data management center is communicatively connected with a data source identification module, a data analysis module, an incremental data collection module, a real-time data processing module, a metadata warehouse module, a change tracking and history recording module, a data quality monitoring module and a user interface module, wherein the modules are connected by electrical signals;

[0057] The data source identification module is used to identify data sources, including databases, file systems, and APIs, and to connect and adapt to various unknown and dynamically changing data sources, converting the data of the data source into a format that the system can process, ensuring that the system can discover new data sources in a timely manner and provide a basis for subsequent data collection and processing. The data source identification and data conversion process includes: automatically detecting potential data sources by scanning the network environment and system configuration, and identifying the type of data source. The data source types include databases, file systems, and APIs. Preliminary analysis is performed on unknown data sources to determine whether they are suitable as input data sources for the system, and establishing a connection with the data source based on the type and configuration information of the data source. For database data sources, the database address, port, user name, and password information need to be provided. For file system data sources, the file path and Access rights: For API data sources, you need to understand the API interface specifications, authentication methods, and calling frequencies, analyze the data structure and format of the data source, determine the rules and mapping relationships for data conversion, and use data conversion tools to convert the data from the data source into a format that the system can process. For structured data, directly perform field mapping and type conversion. For semi-structured and unstructured data, you need to perform parsing, cleaning, and reconstruction operations. It also supports dynamic addition, modification, and deletion of data source configurations. When a new data source is added, it can automatically identify and connect to the data source. When the configuration information of the data source changes, it can be updated in real time and re-established. After the data conversion is completed, perform data format verification and use data verification rule tools to verify the integrity, accuracy, and consistency of the data to ensure that the converted data meets the system requirements.

[0058] The data parsing module is used to parse and standardize data from various data sources, including structured, semi-structured, and unstructured data, to ensure that the system can understand and process data from various data sources, thereby providing a reliable foundation for subsequent metadata collection and change tracking. The parsing process of data from various data sources includes:

[0059] Clarify the data types transmitted by the data source, including structured, semi-structured and unstructured data, and collect structured, semi-structured and unstructured data for preprocessing and parsing; structured data includes tables, JSON, XML, etc., which are data organized in a clearly defined format; semi-structured data includes log files, HTML pages, etc.; unstructured data includes text, images, audio, etc., where the preprocessing process handles missing values, outliers, and duplicate data to ensure the accuracy and reliability of the data; different parsing strategies are used for data parsing according to the data type; for structured data, use the corresponding parser to parse the data and convert it into a data structure that the system can understand; for semi-structured data, use natural language processing technology to extract the required information; for unstructured data, use corresponding analysis technologies, such as text analysis, image recognition, speech recognition, etc. to process and parse it and extract useful information from it;

[0060] Furthermore, the standardization process of data from various data sources includes:

[0061] Standardize the parsed data and convert it into a unified format and structure so that the system can understand and process it, including data type conversion, field naming specifications, and data unit unification. Store the parsed and standardized data in the database for subsequent metadata collection and change tracking. Perform quality checks and verification by comparing the original data with the parsed data to ensure data accuracy and completeness.

[0062] The incremental data collection module is used to collect data incrementally, only processing the changed data, reducing the delay of data collection, and improving the efficiency and accuracy of data collection. The process of incremental data collection includes:

[0063] Perform a complete initial synchronization of the data source, capture the basic state of the data, and record the state of historically successfully collected data, including the timestamp of the collection and the amount of data collected. The time point of historically successfully collected data is the time point of the last successful data collection. Compare the timestamp in the data source with the value recorded in the last successful collection state, interact with the data source, identify the changed data, and based on the identified changed data, only collect the changed part of the data, i.e., the incremental data, and process and store the collected incremental data. After successfully completing the incremental data collection and processing, update the collection state, record the current collection timestamp and data amount information, as a reference for the next collection;

[0064] The real-time data processing module is used to process data streams in real time, update metadata and change history records, ensure that the system can capture data changes in a timely manner, and reduce change tracking errors. The process of updating metadata and change history records includes:

[0065] Real-time monitoring of data sources of various types, capturing real-time data streams, and preprocessing and parsing the captured data. Preprocessing includes data cleaning, format conversion, and denoising operations. Parsing data converts it into a format that the system can recognize, and inputs the preprocessed and parsed data into the real-time processing flow. By comparing the current data with the last stored data, the changed parts in the data are identified, and metadata is updated according to data changes. Metadata includes data table structure, field type, index information, and records the data change history. The change history includes change time, change type, and change content, which helps the system track the change history of data and find anomalies and problems for processing;

[0066] The metadata warehouse module is used to manage and store metadata, including data table structure, field information, data type, data source, and update frequency. It also provides a unified metadata view to facilitate user query and management, help users better understand and use data, ensure the consistency and accuracy of metadata, and provide a reliable basis for data analysis and decision-making.

[0067] The change tracking and history module is used to track and record the change history of metadata, including metadata addition, deletion, and modification operations, to help users track data changes, understand the data evolution process, and reduce change tracking errors. The process of tracking and recording metadata change history includes:

[0068] Set up structures and fields in the metadata warehouse to store metadata and its change history, and monitor change events in the metadata warehouse, including addition, deletion, and modification operations, capture detailed information about the changes, including change time, change type, and change content, and record the captured metadata change information in a dedicated change history table to ensure that each change record contains enough detailed information for subsequent query and analysis, and build a change history view to display the metadata change history;

[0069] The data quality monitoring module is used to monitor the quality and integrity of data in real time and provide a feedback mechanism to promptly discover and fix data quality issues, ensure data accuracy and consistency, and reduce errors in data collection and change tracking. The data quality assessment and analysis process includes:

[0070] Define indicators for evaluating data quality, including accuracy, completeness, consistency, and timeliness, and randomly select data samples from the data source for quality inspection. Perform quality assessment and analysis on the collected data samples to check whether they meet the defined data quality indicators, obtain the scores of each data quality indicator, extract the number of abnormal samples, the total number of samples, the weight of each indicator, and the score of each indicator, and comprehensively obtain the quality assessment coefficient to obtain the overall data quality score. Based on the assessment results, identify the existing data quality problems, conduct in-depth analysis of the identified data quality problems, and formulate a data quality repair plan based on the analysis results;

[0071] Furthermore, the calculation formula of the quality assessment coefficient is:

[0072]

[0073] Among them, Q i is the quality assessment coefficient, indicating the overall data quality score, w i is the weight of each indicator, n is the number of indicators, A i The score of each indicator reflects the performance of the indicator in data quality. is the impact factor of the abnormal situation, indicating the impact of the abnormal situation on the overall score, w y is the abnormal situation weight, w z is the sum of all indicator weights, is the proportion of abnormal situations, indicating the proportion of abnormal situations in the data, N y is the number of abnormal samples, N z is the total sample size. It should be noted that if Q i If the value trend of Q continues to be stable or gradually increases, it means that the data quality is improving or maintaining a good state. i If the value trend of continues to decline or fluctuates greatly, it means that there may be problems with the data quality and further analysis and processing are needed; the user interface module is used to provide a visual interactive interface to assist users in querying data, viewing metadata, and tracking change history.

[0074] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An adaptive metadata collection and change tracking system, including a data management center, characterized by: The data management center is communicatively connected with a data source identification module, a data analysis module, an incremental data acquisition module, a real-time data processing module, a metadata warehouse module, a change tracking and history recording module, a data quality monitoring module and a user interface module, wherein the modules are connected by electrical signals; The data source identification module is used to identify the data source, connect and adapt various unknown and dynamically changing data sources, and convert the data of the data source into a format that can be processed by the system. In the data source identification module, the data source identification and data conversion process of the data source includes: Automatically detect potential data sources by scanning the network environment and system configuration, and identify the types of data sources, including databases, file systems, and APIs; Perform a preliminary analysis on unknown data sources to determine whether they are suitable as the input data source of the system, and establish a connection with the data source based on the type and configuration information of the data source; Analyze the data structure and format of the data source, determine the rules and mapping relationships for data conversion, and use data conversion tools to convert the data from the data source into a format that the system can process; After the data conversion is completed, the data format is verified and the data integrity, accuracy and consistency are verified using data verification rule tools; The data parsing module is used to parse and standardize data from various data sources, including structured, semi-structured and unstructured data; The incremental data acquisition module is used to incrementally acquire data and only processes the changed data. In the incremental data acquisition module, the process of incremental data acquisition includes: Perform a complete initial synchronization of the data source, capture the basic state of the data, and record the state of historically successfully collected data, including the timestamp of the collection and the amount of data collected. The time point of historically successfully collected data is the time point of the last successful data collection. Compare the timestamp in the data source with the value recorded in the last successful acquisition status, interact with the data source, and identify the data that has changed; According to the identified changed data, only the changed part of the data, i.e. the incremental data, is collected, and the collected incremental data is processed and stored; After successfully completing incremental data collection and processing, update the collection status and record the current collection timestamp and data volume information as a reference for the next collection; The real-time data processing module is used to process data streams in real time, update metadata and change history records. In the real-time data processing module, the process of updating metadata and change history records includes: Monitor data sources of various types in real time, capture real-time data streams, and pre-process and analyze the captured data; The pre-processed and parsed data is fed into the real-time processing flow to identify the changes in the data by comparing the current data with the last stored data; Update metadata based on data changes. Metadata includes data table structure, field type, index information, and record data change history. Change history includes change time, change type, and change content. Deal with anomalies and problems when they are found. The metadata warehouse module is used to manage and store metadata and provide a unified metadata view; The change tracking and history recording module is used to track and record the change history of metadata, including the addition, deletion and modification operations of metadata. In the change tracking and history recording module, the process of tracking and recording the change history of metadata includes: Set up structures and fields in the metadata warehouse, store metadata and its change history, and monitor change events in the metadata warehouse, including addition, deletion, and modification operations, and capture detailed information about the changes; Record the captured metadata change information in a dedicated change history table, ensure that each change record contains sufficient detailed information, and build a change history view to display the metadata change history; The data quality monitoring module is used to monitor the quality and integrity of data in real time and provide a feedback mechanism to promptly discover and repair data quality problems. In the data quality monitoring module, the process of data quality assessment and analysis includes: Define metrics for assessing data quality, including accuracy, completeness, consistency, and timeliness, and randomly select data samples from data sources for quality checks; Conduct quality assessment and analysis on the collected data samples to check whether they meet the defined data quality indicators and obtain the scores of each data quality indicator; Extract the number of abnormal samples, the total number of samples, the weight of each indicator, and the score of each indicator, and comprehensively obtain the quality assessment coefficient to obtain the overall data quality score; Based on the assessment results, identify existing data quality issues, conduct in-depth analysis of the identified data quality issues, and formulate a data quality repair plan based on the analysis results; The user interface module is used to provide a visual interactive interface to assist users in querying data, viewing metadata, and tracking change history.

2. The adaptive metadata collection and change tracking system according to claim 1, characterized in that: In the data analysis module, the analysis process of various data sources includes: Identify the types of data transmitted by data sources, including structured, semi-structured, and unstructured data, and collect structured, semi-structured, and unstructured data for pre-processing and analysis; Different parsing strategies are used to parse data according to the data type. For structured data, the corresponding parser is used to parse the data and convert it into a data structure that the system can understand. For semi-structured data, natural language processing technology is used to extract the required information. For unstructured data, the corresponding analysis technology is used to process and parse it to extract useful information.

3. The adaptive metadata collection and change tracking system according to claim 2, characterized in that: In the data analysis module, the standardization process of various data sources includes: Standardize the parsed data and convert it into a unified format and structure, including data type conversion, field naming specifications, and data unit unification; The parsed and standardized data are stored in a database, and quality checks and verifications are performed by comparing the original data with the parsed data.

4. The adaptive metadata collection and change tracking system according to claim 3, characterized in that: The calculation formula of the quality assessment coefficient is: Among them, Q i is the quality assessment coefficient, indicating the overall data quality score, w i is the weight of each indicator, n is the number of indicators, A i To score each indicator, is the impact factor of the abnormal situation, indicating the impact of the abnormal situation on the overall score, w y is the abnormal situation weight, w z is the sum of all indicator weights, is the proportion of abnormal situations, indicating the proportion of abnormal situations in the data, N y is the number of abnormal samples, N z is the total sample size.

Images