Metadata management method, device and system, equipment and medium

Abstract

The invention discloses a metadata management method, device, system, equipment and medium, and the method comprises the steps: automatically obtaining a to-be-upgraded version which is a data version obtained by traversing a metadata set of a business database of a business system under a catalog, or a researcher compiles the business metadata according to the format of the lake and warehouse database to obtain a metadata upgrading version; and updating the lake and warehouse database according to the version to be upgraded, so that the data of the lake and warehouse database is kept consistent with the data of the business database. Through automatic updating processing, the data of the lake and warehouse database and the data of the business database can be kept consistent, the updating and synchronization processing efficiency is improved, the workload of technicians and the maintenance cost of metadata are reduced, and the processing efficiency and flexibility are also improved.

Description

Background Art

[0002] With the in-depth digital transformation, the data environment faced by enterprises has become increasingly complex. Each enterprise has gradually converted the content of its offline business into online processing. The rapid growth of the data volume, data scale, and data types involved in the business not only increases the difficulty of data processing but also poses new requirements for data storage, management, and analysis.

[0003] For this reason, Lakehouse, a technical architecture that combines the advantages of Data Lake and Data Warehouse, has emerged. This architecture not only improves the speed and flexibility of data processing but also further strengthens data security protection and convenient access by means of metadata management technology.

[0004] However, the currently commonly used methods have the following technical problems: There are differences in the metadata format and storage method between the Lakehouse architecture and the enterprise business system. Each time metadata is used, technical personnel need to rewrite it according to the metadata of the business processing system so that the metadata can be adapted to the Lakehouse architecture. This not only increases the workload of technical personnel and the maintenance cost of metadata but also reduces the processing efficiency and flexibility. Summary of the Invention

[0005] The present invention provides a method, device, system, equipment, and medium for managing metadata. The method can solve the technical problems in the prior art that the use, update, and maintenance of data are difficult and inefficient due to the inconsistent metadata formats of different system databases.

[0006] In the first aspect of the embodiments of the present invention, a method for managing metadata is provided. The method includes:

[0007] Automatically obtain the version to be upgraded, where the version to be upgraded is the data version obtained by traversing the metadata set in the business database of the business system in the directory, or the metadata upgrade version obtained by R & D personnel writing business metadata according to the format of the Lakehouse database;

[0008] Perform update processing on the Lakehouse database according to the version to be upgraded to make the data in the Lakehouse database consistent with the data in the business database.

[0009] In combination with the first aspect, in one implementation, the performing update processing on the Lakehouse database according to the version to be upgraded includes:

[0010] Parse the version to be upgraded to obtain change information, where the change information includes: change type, change object, and change content;

[0011] Convert the metadata of the business database according to the change information to obtain converted data;

[0012] Use the converted data to upgrade and update the lakehouse database, and the upgrade and update includes: updating one or more of the table structure definition, field mapping relationship, and index of the lakehouse database.

[0013] Combined with the first aspect, in one implementation, after the step of using the converted data to upgrade and update the lakehouse database, the method further includes:

[0014] Push the change information to the Kafka message queue, and then pull the accumulated change information from the Kafka message queue;

[0015] Sort and deduplicate according to the timestamp or version number of the change information to obtain sorted and deduplicated information, so as to perform data migration and data synchronization on the data of the business database according to the sorted and deduplicated information.

[0016] Combined with the first aspect, in one implementation, after the step of performing data migration and data synchronization on the data of the business database according to the sorted and deduplicated information, the method further includes:

[0017] Obtain migration parameters, where the migration parameters are obtained by real-time monitoring of logs and metrics during the processing of data migration and data synchronization;

[0018] Determine the failed tasks from the migration parameters, and trigger a retry mechanism or an alarm notification according to the failed tasks.

[0019] Combined with the first aspect, in one implementation, after the step of using the converted data to upgrade and update the lakehouse database, the method further includes:

[0020] Export the business metadata snapshot from the business system database and export the lakehouse metadata snapshot from the lakehouse database;

[0021] Perform data verification on the business metadata snapshot and the lakehouse metadata snapshot;

[0022] If the result of the data verification is inconsistent, repair the metadata of the lakehouse database.

[0023] Combined with the first aspect, in one implementation, the operation of obtaining the version to be upgraded includes:

[0024] After the R & D personnel upload the metadata iterative upgrade file they write to the code development library and the project manager completes the review, configure the upgrade directory path of the metadata file to be upgraded in the code product library;

[0025] Extract the version content of the metadata file to be upgraded from the code product library according to the upgrade directory path to obtain the version to be upgraded.

[0026] The second aspect of the embodiments of the present invention provides a metadata management device, and the device includes:

[0027] An acquisition module, configured to automatically acquire the version to be upgraded, where the version to be upgraded is a data version obtained by traversing the metadata set under the directory of the business database of the business system, or a metadata upgrade version obtained by a developer writing business metadata according to the format of the lakehouse database;

[0028] An update management module, configured to perform an update process on the lakehouse database according to the version to be upgraded, so that the data in the lakehouse database is consistent with the data in the business database.

[0029] In combination with the second aspect, in an implementation manner, the update management module is further configured to:

[0030] Parse the version to be upgraded to obtain change information, where the change information includes: change type, change object, and change content;

[0031] Convert the metadata of the business database according to the change information to obtain conversion data;

[0032] Use the conversion data to upgrade and update the lakehouse database, and the upgrade and update include: updating one or more of the table structure definition, field mapping relationship, and index of the lakehouse database.

[0033] In combination with the second aspect, in an implementation manner, the device further includes:

[0034] A pull module, configured to, after the step of using the conversion data to upgrade and update the lakehouse database, push the change information to the Kafka message queue, and then pull the accumulated change information from the Kafka message queue;

[0035] A synchronization module, configured to sort and deduplicate according to the timestamp or version number of the change information to obtain sorted and deduplicated information, so as to perform data migration and data synchronization on the data of the business database according to the sorted and deduplicated information.

[0036] In combination with the second aspect, in an implementation manner, the device further includes:

[0037] A parameter module, configured to, after the step of performing data migration and data synchronization on the data of the business database according to the sorted and deduplicated information, obtain migration parameters, where the migration parameters are obtained by real-time monitoring of logs and metrics during the process of data migration and data synchronization;

[0038] An alarm module, configured to determine a task with processing failure from the migration parameters, and trigger a retry mechanism or an alarm notification according to the task with processing failure.

[0039] In combination with the second aspect, in an implementation, the apparatus further includes:

[0040] An export module, configured to export a business metadata snapshot from the business system database and export a lakehouse metadata snapshot from the lakehouse database after the step of upgrading and updating the lakehouse database by using the conversion data;

[0041] A verification module, configured to perform data verification on the business metadata snapshot and the lakehouse metadata snapshot;

[0042] A repair module, configured to repair the metadata of the lakehouse database if the result of the data verification is inconsistent.

[0043] In combination with the second aspect, in an implementation, the obtaining module is further configured to:

[0044] After a developer uploads a written metadata iterative upgrade file to the code development library and the project manager completes the review, configure the upgrade directory path of the metadata file to be upgraded in the code product library;

[0045] Extract the version content of the metadata file to be upgraded from the code product library according to the upgrade directory path to obtain the version to be upgraded.

[0046] A third aspect of the embodiments of the present invention provides a metadata management system, where the system includes: a metadata management apparatus, a business database, and a lakehouse database;

[0047] The metadata management apparatus communicates with the business database and the lakehouse database respectively;

[0048] The metadata management apparatus executes the metadata management method as described above.

[0049] Compared with the prior art, a metadata management method, apparatus, system, device, and medium provided by the embodiments of the present invention have the beneficial effects that: the present invention can automatically obtain the data version obtained from the metadata set in the directory of the business database of the business system or the metadata upgrade version written by the developer according to the format of the lakehouse database; update the lakehouse database according to the version to be upgraded. Through the automatic update process, the data of the lakehouse database can be kept consistent with that of the business database, which not only improves the processing efficiency of update and synchronization, reduces the workload of technical personnel and the maintenance cost of metadata, but also improves the processing efficiency and flexibility. Description of the Drawings

[0050] Figure 1 It is a schematic flowchart of a method for managing metadata provided by an embodiment of the present invention;

[0051] Figure 2 It is an operation flowchart of automatic monitoring of Flink CDC provided by an embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of an operation interface of a metadata management device provided by an embodiment of the present invention;

[0053] Figure 4 It is an operation flowchart of automatic upgrade of a metadata management device provided by an embodiment of the present invention;

[0054] Figure 5 It is an operation flowchart of upgrade and update processing provided by an embodiment of the present invention;

[0055] Figure 6 It is a schematic structural diagram of a metadata management device provided by an embodiment of the present invention;

[0056] Figure 7 It is a schematic structural diagram of a metadata management system provided by an embodiment of the present invention. Detailed implementation manners

[0057] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0058] With the deepening of digital transformation, the data environment faced by enterprises is becoming increasingly complex. Each enterprise gradually converts the content of its offline business into online processing. The rapid growth of the data volume, data scale, and data types involved in the business not only increases the difficulty of data processing but also poses new requirements for data storage, management, and analysis.

[0059] For this reason, Lakehouse, as a technical architecture that combines the advantages of Data Lake and Data Warehouse, has emerged as the times require. This architecture not only improves the speed and flexibility of data processing but also further strengthens data security protection and convenient access by means of metadata management technology.

[0060] However, the commonly used methods at present have the following technical problems: There are differences in the format and storage method of metadata between the lakehouse integration architecture and the enterprise business system. Every time metadata is used, technicians need to rewrite it according to the metadata of the business processing system so that the metadata can be adapted to the lakehouse integration architecture. This not only increases the workload of technicians and the maintenance cost of metadata, but also reduces the processing efficiency and flexibility.

[0061] To solve the above problems, the following will introduce and explain in detail a metadata management method, device, system, equipment and medium provided by the embodiments of the present application through specific embodiments below.

[0062] Refer to Figure 1 , which shows a schematic flowchart of a metadata management method provided by an embodiment of the present invention.

[0063] In one embodiment, the method is applicable to a metadata management device, and the metadata management device can be a device or system set in a computer. Users or technicians can perform operations such as metadata management, update, modification, and editing through the metadata management device.

[0064] In one embodiment, the metadata management device can communicate with the business database of the enterprise's business system and can also communicate with the lakehouse database of the lakehouse integration architecture.

[0065] Among them, as an example, the metadata management method may include:

[0066] S11. Automatically obtain the version to be upgraded, where the version to be upgraded is the data version obtained by traversing the metadata set in the directory of the business database of the business system, or the metadata upgrade version obtained by the R & D personnel writing business metadata according to the format of the lakehouse database.

[0067] In one embodiment, the metadata management device can communicate with the business database of the business system to obtain the version of the metadata in the business database of the business system and obtain the version to be upgraded. The metadata management device can also directly obtain the metadata upgrade version edited by the R & D personnel to obtain the version to be upgraded. Among them, the version to be upgraded can be the metadata management device traversing the metadata set in the directory of the business database of the business system to obtain the data version of the metadata. The data version of the metadata can be the current version or the version of the metadata that needs to be changed during the business processing. The version to be upgraded can also be the metadata upgrade version obtained by the R & D personnel writing business metadata according to the format of the lakehouse database and uploading it to the metadata management device.

[0068] After determining the data version of the business database or the upgraded version of the metadata to be updated in the lakehouse database edited by the R & D personnel, the version can be automatically updated according to the version to be upgraded. Through automatic update, the data versions of the two databases, namely the lakehouse database and the business database, can be made the same. This can not only unify the format of the metadata in the database, but also simplify the update processing flow through automatic operation, reduce the workload of technical personnel and lower the maintenance cost of metadata, thus improving the processing efficiency and flexibility. Specifically, the data version of the lakehouse database can be adjusted according to the new upgraded version to make the versions of the two databases consistent.

[0069] In an optional embodiment, the version to be upgraded may be the changed data version after the metadata stored in the business database of the business system changes. In order to timely detect whether the metadata format of the business database has changed, the metadata management device can pre-set a metadata change monitoring mechanism for monitoring the metadata changes in the business database of the business system. Once there is a change, the data version of the business database can be obtained to get the version to be upgraded.

[0070] Referring to Figure 2 , it shows the operation flow chart of Flink CDC automatic monitoring provided by an embodiment of the present invention.

[0071] Specifically, the metadata change can be monitored through the stream data integration tool (Flink CDC), and Flink CDC can be configured to monitor the metadata changes in the business system database. The core of this mechanism lies in using the database's WAL (Write-Ahead Log) technology. The core concept of the WAL technology is that before any database persistence change is made, these changes are first recorded in a log file. Flink CDC can capture and process the metadata change events recorded in the WAL in real time.

[0072] The metadata in the business database, including information such as the structure, attributes, and relationships of the data, is the basis for data management and data application. When these metadata change, such as the adjustment of the table structure, the addition or deletion of fields, etc., they need to be timely monitored and identified.

[0073] Flink CDC has the ability to capture and parse database changes in real time. When it detects that the metadata in the business system database has changed, the metadata event recognition mechanism of Flink CDC will be automatically triggered. This mechanism captures the metadata change events by parsing the database change log or using the change capture tool provided by the database.

[0074] Once a metadata event is identified, Flink CDC parses the event and extracts the changed metadata information, which includes the type of change (such as addition, deletion, modification), the object of change (such as table, field), the content of the change, etc.

[0075] Example of change rules: Suppose there is a user information table (user) in the business system, and its fields include user primary key (userId), user name (userName), and email. When the field structure of this table changes, for example, a new field "phone number" (phoneNumber) is added, this change will be recorded in the WAL. Flink CDC will listen for and capture this change event, and then process it according to preset rules (such as storing the information of the new field in a specific metadata table).

[0076] In this embodiment, during the development process of the business system functions, the management and maintenance of metadata are crucial. To ensure that the metadata in the business database and the lakehouse database can be upgraded consistently and efficiently, a metadata management device is developed to meet the metadata maintenance requirements during the development process of the business system functions. It is necessary to update the format or content of the metadata in the two databases to unify the metadata of the two databases. Technicians can pre-edit the upgraded version of the metadata to be updated. To obtain the version of the updated metadata written by the R & D personnel and get the version to be upgraded, where, as an example, the operation of obtaining the version to be upgraded may include the following sub-steps:

[0077] S111. After the R & D personnel upload the iteratively upgraded metadata file to the code development library and the project manager completes the review, configure the upgrade directory path of the metadata file to be upgraded in the code product library.

[0078] S112. Extract the version content of the metadata file to be upgraded from the code product library according to the upgrade directory path to obtain the version to be upgraded.

[0079] In an operation mode, the R & D personnel upload the iteratively upgraded metadata file to the code development library and the project manager conducts a review. When the project manager completes the review, the metadata management device can configure the upgrade directory path of the metadata file to be upgraded in the code product library.

[0080] Then, the metadata management device can extract the version content of the metadata file to be upgraded from the code product library according to the upgrade directory path to obtain the version to be upgraded.

[0081] Refer to Figure 3 , which shows a schematic diagram of the operation interface of the metadata management device provided by an embodiment of the present invention. In an embodiment, the metadata management device may have the following functions:

[0082] First, the database setting function: It supports multiple database types (such as MySQL, PostgreSQL, Oracle, KingbaseES, and DM).

[0083] Second, the function of configuring the metadata set path: According to the requirements of project managers for adjusting project business requirements or improving business functions, R & D personnel will be required to adjust business metadata and iterate and upgrade metadata versions.

[0084] Third, the one - key upgrade function: The device reads the metadata upgrade file and automatically upgrades the metadata.

[0085] Specifically, the specific process for R & D personnel to write and configure is as follows:

[0086] The first step is that R & D personnel are responsible for adjusting business metadata and writing metadata version iteration and upgrade files.

[0087] The second step is that after the metadata version iteration is completed, R & D personnel upload the iterated and upgraded metadata file to the code development library.

[0088] The third step is that the project manager reviews the metadata upgrade file in the code development library to ensure that it meets business requirements.

[0089] The fourth step is that after the metadata upgrade content is reviewed and approved, the project manager publishes the iterated and upgraded metadata file from the code development library to the code product library.

[0090] The fifth step is that after receiving the instruction from the project manager to upgrade the application system, the implementer downloads the metadata upgrade file from the code product library to the application upgrade directory.

[0091] The sixth step is that then, the implementer configures the metadata set path in the metadata management device to set the metadata file upgrade directory path to ensure that the device can correctly read and use these metadata. After obtaining these metadata file upgrade directory paths, the upgraded metadata and its data version can be obtained according to these paths to get the version to be upgraded. Similarly, the business database can also obtain the upgraded metadata and its data version according to these paths for business processing. At this time, it can also be determined that there is a metadata version update through the above - mentioned monitoring mechanism, so that the version to be upgraded can also be obtained.

[0092] It should be noted that the metadata written by R & D personnel can define metadata specifications and version number rules; it can determine metadata requirements, including: analyzing business requirements and defining metadata models. Among them, analyzing business requirements can be to clarify which metadata are needed to describe business objects, attributes, and their relationships. Defining metadata models can be to design metadata models according to business requirements, including entities, attributes, relationships, etc.

[0093] The R & D personnel can also determine the metadata format. Specifically, they can design the file format. For example, they can choose the JSON format as the format of the metadata file. The R & D personnel can also design the file structure: design the file structure according to the metadata model to ensure that the metadata can clearly express the business objects and their attributes.

[0094] When the R & D personnel write the metadata file, they can create the file in the metadata management device. Specifically, they can create the metadata file and name it with the version number. Then, they can fill in the metadata in the metadata management device. Specifically, according to the designed file structure, they can fill in the corresponding metadata in the file. This may include entity names, attribute names, data types, constraints, etc.

[0095] After the R & D personnel write the metadata file, they can verify the metadata. Specifically, after writing is completed, they can use appropriate tools or methods to verify the correctness and integrity of the metadata.

[0096] At the same time, after the R & D personnel complete writing the metadata, they submit the metadata file to the code development library, which is convenient for viewing the submission history of the metadata file and understanding the changes of the file.

[0097] Subsequently, the technical personnel or the implementation personnel can regularly download the latest metadata set file from the code product library to obtain the version content of the metadata file to be upgraded to get the version to be upgraded, and then carry out the implementation and upgrade work;

[0098] Refer to Figure 3 , the implementation personnel start the metadata management device, set the connection of the business database and configure the metadata set file path to ensure that the metadata set can be read smoothly. Finally, in the intuitive interface of the device, they can easily click the [One - click Upgrade] button to start the subsequent metadata upgrade process.

[0099] S12. Update the lakehouse database according to the version to be upgraded so that the data in the lakehouse database is consistent with the data in the business database.

[0100] After the technical personnel or the implementation personnel click the [One - click Upgrade] button of the metadata management device, the metadata management device can respond to the click operation of the technical personnel and update the lakehouse database according to the version to be upgraded so that the data in the lakehouse database is consistent with the data in the business database.

[0101] Refer to Figure 4 , which shows the operation flow chart of the automatic upgrade of the metadata management device provided by an embodiment of the present invention.

[0102] The metadata management device automatically compares the initial metadata version in the business database with the version to be upgraded in the lakehouse database by executing the [One - click Upgrade] process, upgrades each metadata item in sequence and cyclically, and synchronously updates the business data metadata and version information to ensure that all metadata is successfully migrated to the latest version of the business database. The operations are as follows:

[0103] In the first step, the metadata management device is started.

[0104] In the second step, the metadata management device connects to the business database of the business system.

[0105] In the third step, the metadata management device traverses the metadata set in the directory of the business database of the business system and automatically identifies the metadata version.

[0106] In the fourth step, the metadata management device extracts the metadata file set to be upgraded.

[0107] In the fifth step, the metadata management device traverses and executes the metadata files to be upgraded.

[0108] In the sixth step, the metadata management device verifies the upgrade result.

[0109] In order to accurately perform the update process according to the version to be upgraded, as an example, the update process of the lakehouse database according to the version to be upgraded may include the following sub - steps:

[0110] S121. Parse the version to be upgraded to obtain change information, where the change information includes: change type, change object, and change content.

[0111] S122. Convert the metadata of the business database according to the change information to obtain conversion data.

[0112] S123. Use the conversion data to upgrade and update the lakehouse database, and the upgrade and update includes: updating one or more of the table structure definition, field mapping relationship, and index in the lakehouse database.

[0113] Specifically, the metadata management device can parse the version to be upgraded to obtain the change information required to change the metadata version. Among them, the change information may include: change type, change object, and change content.

[0114] Then, the metadata of the business database can be converted according to the change information to obtain conversion data. Finally, the business database is upgraded and updated using the conversion data. Specifically, the processing of this upgrade and update may include: updating one or more of the table structure definition, field mapping relationship, and index in the lakehouse database. For example, update the table structure definition and field mapping relationship of the lakehouse database according to the conversion data.

[0115] It should be noted that the above parsing and conversion processing can be to let the Flink CDC metadata event task parse and convert the collected metadata, and convert the metadata of the business database into a format suitable for lakehouse storage and query according to the storage format and requirements of the lakehouse database.

[0116] In addition, when upgrading and updating the lakehouse database, the metadata after parsing and conversion will be used to upgrade and update the lakehouse, which may include updating the table structure definition, field mapping relationship, index, etc. in the lakehouse. Through these updates, it is ensured that the data in the lakehouse is consistent and accurate with the data in the business system database.

[0117] In one embodiment, after the update, in order to unify the data of the two databases and facilitate subsequent calls, as an example, after the step of upgrading and updating the lakehouse database using the converted data, the method further includes the following steps:

[0118] S124. After pushing the change information to the Kafka message queue, pull the accumulated change information from the Kafka message queue.

[0119] S125. Sort and deduplicate according to the timestamp or version number of the change information to obtain sorted and deduplicated information, so as to perform data migration and data synchronization on the data of the business database according to the sorted and deduplicated information.

[0120] When processing events for upgrading and changing real-time metadata, it is also possible to process batch historical data migration and synchronization tasks for the two databases. In this way, the data in the lakehouse can be managed more flexibly and efficiently. The step process of stream-batch integration is as follows:

[0121] The first step, real-time stream processing, can capture metadata change events in the database through the Flink CDC tool, parse and convert them into a format suitable for lakehouse storage in real time, and push the parsed metadata events to a message queue such as Kafka for subsequent processing.

[0122] Then, it is possible to pull the accumulated metadata events from a message queue such as Kafka regularly or triggeringly, sort and deduplicate them according to the timestamp or version number of the events to ensure the orderliness and accuracy of batch processing. Batch update the metadata in the lakehouse, such as table structure definition, field mapping relationship, index, etc. For historical data migration and synchronization tasks, the batch processing method is also adopted to ensure data consistency and integrity.

[0123] During the process of migrating data, it is necessary to monitor the status of the two databases to improve the efficiency of data update and synchronization. Among them, as an example, after the step of upgrading and updating the lakehouse database with the converted data, the method further includes the following steps:

[0124] S126. Obtain migration parameters, where the migration parameters are obtained by monitoring logs and metrics in real time during the processing of data migration and data synchronization.

[0125] S127. Determine the tasks that failed to be processed from the migration parameters, and trigger a retry mechanism or an alarm notification based on the tasks that failed to be processed.

[0126] Specifically, the logs and metrics in the stream batch processing process, such as processing speed, success rate, error rate, etc., can be monitored in real time. For tasks that failed to be processed, a retry mechanism or an alarm notification is provided to ensure that all metadata events can be correctly processed.

[0127] After the update, it is necessary to verify whether the updated data is consistent to ensure that the data in the two databases is the same. Among them, as an example, after the step of upgrading and updating the lakehouse database with the converted data, the method further includes the following steps:

[0128] S13. Export the business metadata snapshot from the business system database and export the lakehouse metadata snapshot from the lakehouse database.

[0129] S14. Perform data verification on the business metadata snapshot and the lakehouse metadata snapshot.

[0130] S15. If the result of the data verification is inconsistent, repair the metadata of the lakehouse database.

[0131] In one embodiment, the consistency verification can be a data source comparison. Specifically, the metadata snapshot can be exported from the business database of the business system regularly or in a triggered manner, and the exported metadata snapshot is compared with the metadata in the lakehouse database to check for differences.

[0132] For the metadata with differences, detailed verification and analysis are performed, such as whether the field types, lengths, default values, etc. are consistent. For complex data structures (such as nested tables, arrays, etc.), recursive or traversal methods are used for verification.

[0133] If the verification result is inconsistent, consistency repair can be performed. Specifically, according to the verification result, the metadata in the lakehouse database is repaired to ensure that the metadata in the lakehouse database is consistent with the metadata in the business database, and the logs and change records during the repair process are recorded for subsequent auditing and traceability.

[0134] In the actual operation mode, the metadata management device can perform automated verification. Specifically, an automated verification tool or script can be set to regularly execute the above verification process to ensure continuous monitoring and maintenance of data consistency.

[0135] Throughout the process, precise change capture and parsing mechanisms, as well as strict metadata management and update processes, are adopted to avoid data loss, duplication, or inconsistency.

[0136] Refer to Figure 5 , which shows the operation flowchart of the upgrade and update process provided by an embodiment of the present invention.

[0137] In one embodiment, the operations of the metadata management device for the upgrade and update process may include the following steps:

[0138] First step, the metadata management device can parse the version of the metadata to be upgraded this time and extract the change information (type, object, and content).

[0139] Second step, the metadata parsing and conversion module of the metadata management device can convert the metadata into the storage format of the lakehouse database.

[0140] Third step, the lakehouse module of the metadata management device can update the table structure definition, field mapping relationship, index, etc. of the lakehouse database.

[0141] Fourth step, the metadata management device can, during the update process, process metadata changes and batch historical data migration in real time.

[0142] Fifth step, the metadata management device can verify the data consistency and integrity during the real-time processing of metadata changes and batch historical data migration.

[0143] Adopting the technical solution of the present invention has the following advantages:

[0144] First, unified and efficient metadata management: Through the metadata management device, centralized management and one-key upgrade of metadata are achieved, simplifying the complexity of metadata maintenance and improving the work efficiency of implementers. At the same time, it supports multiple database types, improving the compatibility and flexibility of the system.

[0145] Second, reducing the cost of data verification and consistency maintenance: Using Flink CDC to capture and parse metadata change events in real time ensures the consistency of metadata between the business system database and the lakehouse integration. Through precise change capture and parsing mechanisms, data loss, duplication, or inconsistency are avoided, reducing the cost of data verification and consistency maintenance.

[0146] Third, improve the flexibility of technology integration and operation and maintenance efficiency: The metadata management device provides an intuitive operation interface and one-key upgrade function, reducing the difficulty and complexity of technology integration, and realizing a processing mode that combines streaming and batch processing. It can not only process real-time metadata change events but also handle batch historical data migration and synchronization tasks, improving the work efficiency of operation and maintenance personnel.

[0147] Fourth, meet the data management requirements in the digital and intelligent environment: The technical solution of the present invention can adapt to the high requirements for data management in the digital and intelligent environment, including aspects such as data timeliness, accuracy, consistency, and integrity. Through intelligent management means, it improves the efficiency and accuracy of data management, providing strong support for the digital and intelligent transformation of enterprises.

[0148] In addition, during the process of metadata synchronization between the business database and the lakehouse database, Flink CDC not only listens for metadata change events in the database but also defines and executes corresponding synchronization tasks (tasks) according to different event types (such as addition, modification, deletion, etc.). These tasks are responsible for synchronizing the changed metadata from the business system database to the corresponding storage area in the lakehouse architecture.

[0149] For the storage and optimization of metadata, classified storage can be carried out. For example, different storage areas can be designed according to the type or importance of metadata. Frequently updated metadata (such as business table structure change information) is classified as type-a data and stored in a high-performance storage area (such as SSD) to improve access efficiency; while less-changed metadata (such as static configuration information) is classified as type-b data and stored in a more cost-effective storage area (such as HDD) to optimize storage costs.

[0150] Type-a data can be metadata related to core business logic and frequently changed, such as business table structure, field type, etc., and is stored in the high-performance storage SSD area.

[0151] Type-b data can be relatively stable metadata, such as business rule descriptions, data dictionaries, etc., and is stored in the cost-effective storage HDD area.

[0152] The partitioning strategy can partition and store according to the attributes of metadata (such as business line, data type, etc.) to improve query efficiency.

[0153] Optionally, for each metadata update, record its version number and timestamp for easy traceability and auditing.

[0154] In this embodiment, the embodiment of the present invention provides a method for managing metadata, and its beneficial effects are as follows: The present invention can automatically obtain the data version of the metadata set in the business database of the business system under the directory, or the metadata upgrade version written by the R & D personnel according to the format of the lakehouse database; update the lakehouse database according to the version to be upgraded. Through automatic update processing, the data in the lakehouse database can be kept consistent with the data in the business database, which not only improves the processing efficiency of update and synchronization, reduces the workload of technical personnel and the maintenance cost of metadata, but also improves the processing efficiency and flexibility.

[0155] The embodiment of the present invention also provides a device for managing metadata. Refer to Figure 6 , which shows a schematic structural diagram of a device for managing metadata provided by an embodiment of the present invention.

[0156] Among them, by way of example, the device for managing metadata may include:

[0157] An acquisition module 201, configured to automatically acquire the version to be upgraded, where the version to be upgraded is the data version obtained by traversing the metadata set in the business database of the business system under the directory, or the metadata upgrade version written by the R & D personnel according to the format of the lakehouse database;

[0158] An update management module 202, configured to update the lakehouse database according to the version to be upgraded, so as to keep the data in the lakehouse database consistent with the data in the business database.

[0159] Optionally, the update management module is further configured to:

[0160] Parse the version to be upgraded to obtain change information, where the change information includes: change type, change object, and change content;

[0161] Convert the metadata of the business database according to the change information to obtain converted data;

[0162] Use the converted data to upgrade and update the lakehouse database, where the upgrade and update includes: updating one or more of the table structure definition, field mapping relationship, and index of the lakehouse database.

[0163] Optionally, the device further includes:

[0164] A pulling module, configured to, after the step of using the converted data to upgrade and update the lakehouse database, push the change information to the Kafka message queue, and then pull the accumulated change information from the Kafka message queue;

[0165] A synchronization module, configured to sort and deduplicate according to the timestamp or version number of the change information to obtain sorted and deduplicated information, so as to perform data migration and data synchronization on the data in the business database according to the sorted and deduplicated information.

[0166] Optionally, the device further includes:

[0167] A parameter module, configured to obtain migration parameters after the step of performing data migration and data synchronization on the data in the business database according to the sorted and deduplicated information, where the migration parameters are obtained by real-time monitoring of logs and metrics during the processing of data migration and data synchronization;

[0168] An alarm module, configured to determine a failed task from the migration parameters and trigger a retry mechanism or an alarm notification according to the failed task.

[0169] Optionally, the device further includes:

[0170] An export module, configured to export a business metadata snapshot from the business system database and export a lakehouse metadata snapshot from the lakehouse database after the step of upgrading and updating the lakehouse database by using the converted data;

[0171] A verification module, configured to perform data verification on the business metadata snapshot and the lakehouse metadata snapshot;

[0172] A repair module, configured to repair the metadata of the lakehouse database if the result of the data verification is inconsistent.

[0173] Optionally, the obtaining module is further configured to:

[0174] After a developer uploads a metadata iterative upgrade file written to the code development library and the project manager completes the review, configure the upgrade directory path of the metadata file to be upgraded in the code product library;

[0175] Extract the version content of the metadata file to be upgraded from the code product library according to the upgrade directory path to obtain the version to be upgraded.

[0176] An embodiment of the present invention further provides a metadata management system. Refer to Figure 7 , which shows a schematic structural diagram of a metadata management system provided by an embodiment of the present invention.

[0177] Among them, by way of example, the metadata management system may include: a metadata management device, a business database, and a lakehouse database;

[0178] The metadata management device communicates with the business database and the lakehouse database respectively;

[0179] The metadata management device executes the metadata management method as described in the above embodiments.

[0180] Among them, the lakehouse database can be a database with a lakehouse architecture.

[0181] Those skilled in the art can clearly understand that for the convenience of description and simplicity, the specific working process of the device described above can refer to the corresponding process in the foregoing method embodiments, and will not be described herein again.

[0182] Furthermore, an embodiment of the present application also provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the metadata management method as described in the above embodiments.

[0183] Furthermore, an embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer-executable program, and the computer-executable program is used to cause a computer to execute the metadata management method as described in the above embodiments.

[0184] In the description of the embodiments of the present invention, it should be noted that the orientation or positional relationship indicated by terms such as "upper" and "lower" is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the embodiments of the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. When an element such as a layer, region, or substrate is referred to as being "on" or "above" another element, it can be directly on the other element, or there can also be an intermediate element. On the contrary, when an element is referred to as being "directly on" or "above" another element, there is no intermediate element. It should also be understood that when an element is referred to as being "under" or "below" another element, it can be directly under or below the other element, or there can also be an intermediate element. On the contrary, when an element is referred to as being "directly under" or "below" another element, there is no intermediate element. Unless otherwise clearly defined and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0185] Those skilled in the art should understand that the embodiments of the present application may also provide a computer program product. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0186] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), devices, and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0187] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0189] The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A metadata management method, characterized in that: The method comprises: Automatically obtain the version to be upgraded, wherein the version to be upgraded is a data version obtained by traversing the metadata set of the business database of the business system under the directory, or a metadata upgrade version obtained by R&D personnel writing business metadata according to the format of the Hucang database; The lake warehouse database is updated according to the version to be upgraded to ensure that the data in the lake warehouse database is consistent with the data in the business database.

2. The metadata management method according to claim 1, characterized in that: The updating process of the lake warehouse database according to the version to be upgraded includes: Parsing the version to be upgraded to obtain change information, wherein the change information includes: change type, change object and change content; Convert the metadata of the business database into a format according to the change information to obtain converted data; The conversion data is used to upgrade and update the lake warehouse database, and the upgrade and update includes: updating one or more of the table structure definition, field mapping relationship and index of the lake warehouse database.

3. The metadata management method according to claim 2, characterized in that: After the step of upgrading the lake warehouse database using the conversion data, the method further includes: After pushing the change information to the Kafka message queue, pulling the accumulated change information from the Kafka message queue; The change information is sorted and deduplicated according to the timestamp or version number to obtain sorted and deduplicated information, so as to perform data migration and data synchronization on the business database according to the sorted and deduplicated information.

4. The metadata management method according to claim 3, characterized in that: After the step of performing data migration and data synchronization on the business database according to the sorting and deduplication information, the method further includes: Obtain migration parameters, which are obtained by real-time monitoring of logs and indicators during the data migration and data synchronization process; The failed processing task is determined from the migration parameter, and a retry mechanism or an alarm notification is triggered according to the failed processing task.

5. The metadata management method according to claim 3, characterized in that: After the step of upgrading the lake warehouse database using the conversion data, the method further includes: Export business metadata snapshots from the business system database and export lake warehouse metadata snapshots from the lake warehouse database; Performing data verification on the business metadata snapshot and the lake warehouse metadata snapshot; If the result of the data verification is inconsistent, the metadata of the lake warehouse database is repaired.

6. The metadata management method according to claim 1, characterized in that: The operation of obtaining the version to be upgraded includes: After the R&D personnel upload the metadata iterative upgrade file to the code development library and the project manager completes the review, configure the upgrade directory path of the metadata file to be upgraded in the code product library; The version content of the metadata file to be upgraded is extracted from the code product library according to the upgrade directory path to obtain the version to be upgraded.

7. A metadata management device, characterized in that: The device comprises: An acquisition module is used to automatically acquire the version to be upgraded, wherein the version to be upgraded is a data version obtained by traversing the metadata set of the business database of the business system under the directory, or a metadata upgrade version obtained by R&D personnel writing business metadata according to the format of the lake warehouse database; The update management module is used to update the lake warehouse database according to the version to be upgraded so that the data of the lake warehouse database and the business database are consistent.

8. A metadata management system, characterized in that: The system comprises: a metadata management device, a business database and a lake warehouse database; The metadata management device communicates with the business database and the lake warehouse database respectively; The metadata management device executes the metadata management method according to any one of claims 1 to 6.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the metadata management method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer-executable program, and the computer-executable program is used to enable a computer to execute the metadata management method according to any one of claims 1 to 6.

Images