Metadata version control and tracing method and system

Abstract

The invention relates to the technical field of data processing, in particular to a metadata version control and tracing method and system. The method comprises the following steps: establishing a first metadata warehouse, and obtaining the type of metadata in the first metadata warehouse; quantitatively analyzing the distribution condition of the metadata in the first metadata warehouse; a second metadata warehouse is constructed, the metadata are backed up, and the backed-up metadata are stored in the second metadata warehouse; timestamp fields are added to backup metadata of the second metadata warehouse, and the change history of the metadata is tracked. According to the method, quality detection and correction can be carried out on the data through an automatic tool or algorithm, the requirement for manual intervention is reduced, the efficiency and accuracy of data processing are improved, and meanwhile, the metadata version control can automatically record and manage the change history of the data and provide the traceability and transparency of the data; dependence on computing resources is reduced, and particularly when a large-scale data set is processed, the computing resources can be utilized more efficiently.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method and system for metadata version control and traceability. Background Technology

[0002] In today's rapidly changing business information processing environment, real-time data warehouses have become an important tool for enterprises to respond quickly to market changes and make data-driven decisions. With the acceleration of data processing speeds and the surge in data volume, data version control has become a critical element in ensuring data consistency, integrity, and traceability. Metadata, as data describing data, plays a crucial role in the era of digital transformation and big data. It provides information about data structure, content, source, quality, and other aspects, which is essential for ensuring data availability and value. However, as enterprise data environments become increasingly complex, effectively managing changes in metadata has become a pressing issue. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes a method and system for metadata version control and tracing, used to trace the versions of metadata.

[0004] On one hand, this invention discloses a method for metadata version control and traceability, including: Establish a primary metadata repository and obtain the types of metadata within the primary metadata repository; Based on the types of metadata in the first metadata repository, the distribution of each type of metadata is quantitatively analyzed; Build a second metadata repository, back up metadata of the same type, and store the backup metadata in the second metadata repository; Add timestamp fields to the backup metadata of the second metadata repository and use the timestamp fields to track the change history of the metadata.

[0005] On the other hand, the present invention also provides a metadata version control and traceability system, the system comprising the following modules: The first metadata repository generation module is used to establish the first metadata repository and obtain the types of metadata in the first metadata repository. The metadata analysis module is used to quantitatively analyze the distribution of each type of metadata based on the type of metadata in the first metadata repository; The second metadata repository generation module is used to build a second metadata repository, back up metadata of the same type, and store the backup metadata in the second metadata repository; The metadata traceability module is used to add timestamp fields to the backup metadata of the second metadata repository and to use the timestamp fields to track the change history of the metadata.

[0006] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: The proposed solution uses automated tools or algorithms to perform quality checks and corrections on metadata, reducing the need for manual intervention and improving the efficiency and accuracy of data processing. At the same time, metadata version control can automatically record and manage the data change history, providing data traceability and transparency. It also reduces the dependence on computing resources, especially when processing large-scale datasets, enabling more efficient use of computing resources and facilitating the widespread application of data chains in multiple fields. Attached Figure Description Figure 1 This is a flowchart illustrating the metadata version control and traceability method provided by the present invention; Figure 2 This is a schematic diagram of the framework of the metadata version control and traceability system provided by the present invention. Detailed Implementation

[0007] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Based on the embodiments in this application, other similar embodiments obtained by those skilled in the art without creative effort should all fall within the scope of protection of this application.

[0008] Example 1 like Figure 1 As shown, this invention provides a metadata version control and traceability method, which includes: Step S01: Establish the first metadata repository and obtain the types of metadata in the first metadata repository; The first metadata repository is used to comprehensively collect and integrate various metadata resources. After the metadata resources in the first metadata repository are integrated, the metadata can be classified and managed in the subsequent process.

[0009] Furthermore, the first metadata repository has data structures that can be used to store and record user data editing behavior on the target metadata.

[0010] By recording user editing behavior in the first metadata repository, it is possible to track the source and process of each metadata change in detail. At the same time, the data structure that records user editing behavior provides a foundation for quality monitoring of a large amount of metadata.

[0011] Step S02: Based on the types of metadata in the first metadata repository, quantitatively analyze the distribution of each type of metadata; In a preferred embodiment, Apache Atlas can be used to classify, search, and version manage metadata, thereby quantitatively analyzing the distribution of various types of metadata.

[0012] Apache Atlas is an open-source metadata management and governance tool. As a preferred embodiment, the use of Apache Atlas for metadata classification, searching, and version management, and for quantitative analysis of the distribution of various types of metadata, includes: Integrate the Apache Atlas server with the first metadata repository; Create classification rules in Apache Atlas; When metadata is imported into the first metadata repository, Apache Atlas automatically scans the attributes of the metadata and categorizes it according to predefined classification rules. Furthermore, based on the classification results of Apache Atlas, the amount of metadata in each type can be counted to generate a distribution report; Based on the steps described above, this embodiment of Apache Atlas not only enables the classification, search, and version management of metadata, but also provides an intuitive display of metadata distribution through quantitative analysis, thus providing strong support for data governance and management.

[0013] Step S03: Build a second metadata repository, back up metadata of the same type, and store the backup metadata in the second metadata repository; Step S04: Add a timestamp field to the backup metadata of the second metadata repository, and use the timestamp field to record changes in the metadata.

[0014] Specifically, during the backup of the same type of metadata, a timestamp field is added to the backup metadata in the second metadata repository. This timestamp field is used to record the creation, modification, and deletion times of the backup metadata in the second metadata repository, as well as to record the data backup time.

[0015] In the second metadata repository, backup metadata of the same type is integrated with the corresponding timestamps to track the change history of metadata.

[0016] As a preferred embodiment, during the backup process, a timestamp field is added to the second metadata repository to record the creation, modification, and deletion times of the backup metadata, and to record the backup time, including: Back up metadata of the same type in the first metadata repository, generate a backup metadata dataset, and store the backup metadata dataset in the second metadata repository; In the table structure of the second metadata warehouse, a timestamp field is added to each backup metadata item. The initial value of the timestamp field adopts the timestamp information of the corresponding metadata in the first metadata warehouse. Whenever the target metadata is modified or deleted in the first metadata repository, the timestamp information of the modification or deletion operation is captured, and the timestamp information is updated in the corresponding backup metadata in the second metadata repository.

[0017] As a preferred embodiment, when the target metadata changes, Apache Atlas automatically records the changes and creates a new version. Furthermore, it can add timestamps to each version of the metadata in the first metadata repository to achieve version control of the metadata.

[0018] In a preferred embodiment, while backing up the metadata in the first metadata warehouse, a timestamp is added, and corresponding backup metadata is generated and placed in the second metadata warehouse. The backup metadata is also given the same timestamp, so that the backup metadata and the target metadata always remain consistent.

[0019] Metadata version control not only records the changes to metadata, but also provides contextual information about those changes. This detailed record makes the metadata change process completely transparent, allowing users to clearly understand the source and evolution of the data, thus enhancing data traceability and credibility.

[0020] According to the above steps, this embodiment can completely record the lifecycle of metadata, including the time information of creation, modification, deletion and backup; these timestamp fields provide accurate time basis for data traceability and version management, ensuring the traceability and transparency of metadata.

[0021] Furthermore, when a user edits the target metadata, the system first retrieves the latest target metadata from the first metadata repository and checks whether the backup metadata in the second metadata repository conflicts with the latest target metadata in the first metadata repository. If a conflict exists, the conflicting content is communicated to the user, allowing the user to modify the target metadata again.

[0022] This embodiment effectively avoids data conflicts caused by editing operations by checking the consistency of the latest target metadata in the first metadata repository and the backup metadata in the second metadata repository in real time. Furthermore, when the user edits the target metadata, it can prompt the conflicting modification content and remind the user to readjust the editing operation, thereby improving the accuracy of metadata modification.

[0023] Example 2 This embodiment is a metadata version control and traceability system. The system is composed of module units corresponding to the steps of a metadata version control and traceability method in any of the foregoing embodiments, for controlling and tracing metadata versions. The system includes the following modules: The first metadata repository generation module is used to establish the first metadata repository and obtain the types of metadata in the first metadata repository. The metadata analysis module is used to quantitatively analyze the distribution of each type of metadata based on the type of metadata in the first metadata repository; The second metadata repository generation module is used to build a second metadata repository, back up metadata of the same type, and store the backup metadata in the second metadata repository; The metadata traceability module is used to add timestamp fields to the backup metadata of the second metadata repository and use the timestamp fields to record changes in the metadata.

[0024] In a preferred embodiment, such as Figure 2 As shown, the first metadata repository and the second metadata repository are stored in the same storage terminal, which can be a network disk, network hard drive or other storage device.

[0025] Example 3 This embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause a computer to implement the metadata version control and traceability method described in any of the foregoing embodiments.

[0026] Example 4 This embodiment provides a computer program product, which includes a computer program that, when executed by a processor, is used for the metadata version control and tracing method described in any of the foregoing embodiments.

[0027] In summary, the metadata version control and traceability scheme provided in this invention can perform quality inspection and correction of metadata through automated tools or algorithms, reducing the need for manual intervention and improving the efficiency and accuracy of data processing. At the same time, metadata version control can automatically record and manage the data change history, providing data traceability and transparency; it also reduces the dependence on computing resources, especially when processing large-scale datasets, enabling more efficient use of computing resources.

[0028] All features disclosed in this specification, or steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps. This invention is not limited to the specific embodiments described above. This invention extends to any new feature or any new combination disclosed in this specification, as well as any new step or any new combination of any disclosed method or process.

Claims

1. A method for metadata version control and traceability, characterized in that, Includes the following steps: Establish a primary metadata repository and obtain the types of metadata within the primary metadata repository; Based on the types of metadata in the first metadata repository, the distribution of each type of metadata is quantitatively analyzed; Build a second metadata repository, back up metadata of the same type, and store the backup metadata in the second metadata repository; Add timestamp fields to the backup metadata of the second metadata repository and use the timestamp fields to track the change history of the target metadata.

2. The metadata version control and traceability method as described in claim 1, characterized in that, The first metadata repository is used to comprehensively collect and integrate various metadata resources.

3. The metadata version control and traceability method as described in claim 1, characterized in that, The first metadata repository has a data structure for storing and recording user actions of editing target metadata.

4. The metadata version control and traceability method as described in claim 1, characterized in that, Use a data management system to classify, search, and version control metadata.

5. The metadata version control and traceability method as described in claim 4, characterized in that, Use a data management system to classify, search, and version control metadata, including: Integrate the data management system server with the first metadata repository; Create classification rules in the data management system; When metadata is imported into the first metadata repository, the data management system automatically scans the attributes of the metadata and categorizes the metadata according to predefined classification rules. Based on the classification results of the data management system, the amount of metadata in each type is counted, and a distribution report is generated; When the target metadata changes, the data management system automatically records the changes and creates a new version, while adding a timestamp to each version.

6. The metadata version control and traceability method as described in claim 1, characterized in that, When backing up metadata of the same type, a timestamp field is added to the backup metadata of the second metadata warehouse. This timestamp field is used to record the creation, modification and deletion time of the backup metadata in the second data warehouse, as well as to record the data backup time.

7. The metadata version control and traceability method as described in claim 6, characterized in that, Back up metadata of the same type in the first metadata repository, generate a backup metadata dataset, and store the backup metadata dataset in the second metadata repository; In the table structure of the second metadata warehouse, add a timestamp field to each backup metadata item; Whenever the target metadata is modified or deleted in the first metadata repository, the timestamp information of the target metadata modification or deletion operation is captured, and the timestamp information is updated to the corresponding backup metadata in the second metadata repository.

8. The metadata version control and traceability method as described in claim 1, characterized in that, When a user edits the target metadata, the system first retrieves the latest target metadata from the first metadata repository and then checks whether the corresponding backup metadata in the second metadata repository conflicts with the latest target metadata in the first metadata repository.

9. A metadata version control and traceability system, characterized in that, The system is used to implement the metadata version control and traceability method according to any one of claims 1-8, and includes the following modules: The first metadata repository generation module is used to establish the first metadata repository and obtain the types of metadata within the first metadata repository. The metadata analysis module is used to quantitatively analyze the distribution of each type of metadata based on the type of metadata in the first metadata repository; The second metadata repository generation module is used to build a second metadata repository, back up metadata of the same type, and store the backed-up metadata in the second metadata repository; The metadata traceability module is used to add timestamp fields to the backup metadata of the second metadata repository and to use the timestamp fields to track the change history of the metadata.

10. A metadata version control and traceability system as described in claim 9, characterized in that, The first metadata repository and the second metadata repository have the same storage location.

Images