Metadata management method, metadata management device and computer storage medium

Abstract

The invention provides a metadata management method, a metadata management device and a computer storage medium. The metadata management method comprises the steps that in response to a service instruction, service processing is conducted on a database through a calculation engine connected with a user; calling a metadata management service to generate metadata change information; filtering calculation engine information in the metadata change information, and extracting metadata change data; and packaging the metadata change data according to a preset data format of the metadata management service. By means of the metadata management method, the metadata of the computing engine is managed in a centralized mode through the metadata management service, and the consistency and reliability of the data are ensured; and filtering the information of the computing engines by utilizing a metadata management service, and unifying metadata updating contents of different computing engines.

Description

Technical Field

[0001] The present application relates to the field of metadata processing technology, and in particular to a metadata management method, a metadata management device, and a computer storage medium. Background Art

[0002] With the continuous development of big data technology, the requirements for data management and processing are becoming increasingly complex. In a multi-engine, multi-user big data environment, metadata management and push have become key issues.

[0003] The rapid development of big data platforms has made data processing and management increasingly complex. In practical applications, different tasks require the use of different processing engines, such as Hive, Spark, and Flink. These engines each excel at different processing tasks, but current metadata management technologies cannot achieve unified metadata management across multiple engines, hindering data consistency and security. Summary of the Invention

[0004] To solve the above technical problems, the present application proposes a metadata management method, a metadata management device, and a computer storage medium.

[0005] To solve the above technical problems, this application proposes a metadata management method, which includes:

[0006] In response to business instructions, the computing engine connected by the user performs business processing on the database;

[0007] Call metadata management service to generate metadata change information;

[0008] Filtering the computing engine information in the metadata change information to extract the metadata change data;

[0009] The metadata change data is encapsulated according to a preset data format of the metadata management service.

[0010] After encapsulating the metadata change data according to a preset data format of the metadata management service, the metadata management method further includes:

[0011] Send the encapsulated metadata change data to the hook topic data of the distributed message subscription queue;

[0012] Obtain the bound user of the database in the hook theme data;

[0013] The encapsulated metadata change data is sent to the bound user for synchronization.

[0014] Wherein, after obtaining the bound user of the database in the hook theme data, the metadata management method further includes:

[0015] Obtaining permission information of the bound user;

[0016] Obtain the target bound user whose permission information is administrator permission;

[0017] The encapsulated metadata change data is sent to the target binding user for synchronization.

[0018] The metadata change data includes: adding operation users, operation types, and / or task scheduling node information.

[0019] The step of calling the metadata management service to generate metadata change information includes:

[0020] Calling the event listener of the metadata management service to listen to business processing information;

[0021] Determining whether there is new metadata information or changed metadata information in the business processing information;

[0022] If so, classifying the business processing information into task types based on changes in the metadata;

[0023] The metadata change information is generated according to the task type of the business processing information.

[0024] The metadata management method further includes:

[0025] Create users and databases;

[0026] Binding the user to the database and setting the permission information for the user to operate the database;

[0027] The step of performing business processing on the database by the computing engine connected to the user in response to the business instruction includes:

[0028] Responding to the service instruction, obtaining the permission information of the user;

[0029] When the permission information of the user meets the access permission conditions, setting the permission for the user to use the computing engine;

[0030] The computing engine performs business processing on the database specified by the business instruction.

[0031] The performing of business processing on the database specified by the business instruction by the computing engine includes:

[0032] Obtaining the service type of the service instruction;

[0033] When the service type is a data modification type, obtaining permission information of the user to operate the database specified by the service instruction;

[0034] Determining whether the permission information meets the permission modification conditions;

[0035] If so, the computing engine is executed to perform business processing on the database specified by the business instruction.

[0036] The metadata of all computing engines are stored in the same database through the metadata management service.

[0037] To solve the above technical problems, the present application also proposes a metadata management device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the metadata management method as described above.

[0038] In order to solve the above technical problems, the present application also proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above metadata management method.

[0039] Compared with existing technologies, the present application has the following advantages: the metadata management device, in response to business instructions, performs business processing on the database through the computing engine connected by the user; calls the metadata management service to generate metadata change information; filters the computing engine information in the metadata change information to extract the metadata change data; and encapsulates the metadata change data according to the preset data format of the metadata management service. Through the above-mentioned metadata management method, the metadata of the computing engines is centrally managed through the metadata management service, ensuring data consistency and reliability; and the metadata management service is used to filter the computing engine information and unify the metadata update content of different computing engines. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] in:

[0042] Figure 1 This is a flowchart of the first embodiment of the metadata management method provided by this application;

[0043] Figure 2 This is a flowchart of the multi-engine unified metadata provided by this application;

[0044] Figure 3 yes Figure 1 The specific flow diagram of step S12 of the metadata management method is shown;

[0045] Figure 4 This is a flowchart of the second embodiment of the metadata management method provided by this application;

[0046] Figure 5 It is a schematic diagram of the multi-user security process provided by this application;

[0047] Figure 6 This is a flowchart of the third embodiment of the metadata management method provided by this application;

[0048] Figure 7 This is the platform architecture diagram of the metadata management method provided by this application;

[0049] Figure 8 This is a schematic structural diagram of an embodiment of a metadata management device provided by the present application;

[0050] Figure 9 It is a structural diagram of an embodiment of a computer storage medium provided by this application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] The terms "first," "second," "third," "fourth," etc. (if any) in the specification and claims of the present application and in the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential sequence. It should be understood that the numbers used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or apparatus.

[0053] In order to solve the problem that current metadata management technology cannot achieve unified management of metadata in a multi-engine environment and cannot ensure data consistency and security, this application proposes a processing method for unified metadata management based on permissions.

[0054] The technical terms involved in the metadata management method of this application include but are not limited to the following:

[0055] Hive: It is a data warehouse tool based on Hadoop. When no other computing engines are called, it is based on MapReduce (programming model) services and is suitable for offline data processing and batch processing of large-scale historical data.

[0056] Spark: A memory-based, highly fault-tolerant computing engine that supports more complex operation types, processing offline tasks through large batches and real-time tasks through small batches.

[0057] Flink: A distributed real-time and offline computing engine used for stateful computation on both unbounded and bounded data streams. It processes offline tasks using bounded streams and real-time tasks using unbounded streams.

[0058] Ranger: A centralized security management framework that is primarily used to address authorization and auditing issues in the Hadoop (Hadoop Distributed File System) ecosystem.

[0059] Metastore: Used to store and manage metadata in the Hive data warehouse. Currently, Spark and Flink can also configure the Metastore metadata management service. Combined with the Ranger service, it can manage the relationship between users and databases to achieve consistent user access and metadata consistency across Hive, Spark, and Flink engines.

[0060] ConfigCenter: Self-developed software used for heartbeat maintenance between various big data components, service restart, configuration modification, and external query interfaces, user systems, resource configuration, etc.

[0061] In a big data platform, different tasks require different processing engines, including but not limited to Hive, Spark, and Flink, as described above. These processing or computing engines in a big data platform operate independently but share the same metadata to ensure data consistency and accessibility. The metadata management method provided in this application effectively achieves these technical benefits.

[0062] Please refer to the following for details: Figure 1 and Figure 2 , Figure 1 This is a flowchart of the first embodiment of the metadata management method provided by this application. Figure 2 This is a flowchart of the multi-engine unified metadata provided by this application.

[0063] The metadata management method of the present application is applied to a metadata management device. The metadata management device of the present application can be a server, a terminal device, or a system comprising a server and a terminal device. Accordingly, the various components of the metadata management device, such as the various units, subunits, modules, and submodules, can be entirely located in the server, entirely located in the terminal device, or separately located in the server and the terminal device.

[0064] Furthermore, the server described above may be either hardware or software. When the server is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it may be implemented as multiple software programs or software modules, such as software or software modules for providing a distributed server, or as a single software program or software module, without further limitation.

[0065] like Figure 1 As shown, the specific steps are as follows:

[0066] Step S11: In response to the business instruction, the computing engine connected by the user performs business processing on the database.

[0067] In the embodiments of this application, Figure 2 As shown, the user inputs business instructions to the computing engine, and the computing engine Figure 2 Hive in the database performs business processing tasks on the database. For example, when a business product requires a computing engine, the user connects to a computing engine and then processes data in the database corresponding to the business product. The business product can perform data processing on the database using, for example, DDL statements. DDL (Data Definition Language) is the language used to create, delete, and modify objects within the database.

[0068] During a DDL task, the data content within the database will change, and the metadata describing the data content will also change accordingly.

[0069] Step S12: calling the metadata management service to generate metadata change information.

[0070] In an embodiment of the present application, the metadata management device calls the Metastore metadata management service to generate corresponding metadata change information for the metadata change event occurring in step S11, and records it in a unified database, such as Mysql, to update the metadata in the database.

[0071] It should be noted that the metadata of all computing engines in this application are stored in the same database through the Metastore metadata management service.

[0072] Before generating metadata change information, the metadata management device also needs to monitor whether metadata changes occur during the business processing in step S11. If the business product in step S11 involves data processing, there will be metadata changes; if the business product in step S11 only involves data access, there will be no metadata changes.

[0073] Please refer to the following for details: Figure 3 , Figure 3 yes Figure 1 Detailed flowchart of step S12 of the metadata management method is shown.

[0074] like Figure 3 As shown, the specific steps are as follows:

[0075] Step S121: calling the event listener of the metadata management service to listen to business processing information.

[0076] In the embodiment of the present application, the metadata management device calls the eventListener (event listener) of the metadata management service to listen to the business processing information, that is, the update status of the metadata in the database.

[0077] Step S122: Determine whether there is any new metadata information or changed metadata information in the business processing information.

[0078] In this embodiment of the present application, the metadata management device determines whether there is any new metadata information or metadata change information in the metadata update status in the database, where the change information includes metadata deletion, metadata replacement, etc. If so, the process proceeds to step S123; if not, it indicates that the metadata in the database has not changed.

[0079] Step S123: classify the business processing information into task types based on changes in the metadata.

[0080] In the embodiment of the present application, the metadata management device classifies the task types of business processing, and the task types include but are not limited to: creatTable (addition of data table), alterTable (replacement of data table), deleteTable (deletion of data table), etc.

[0081] Step S124: Generate metadata change information according to the task type of the business processing information.

[0082] In an embodiment of the present application, the metadata management device generates metadata change information corresponding to the task type.

[0083] Step S13: filtering the computing engine information in the metadata change information and extracting the metadata change data.

[0084] Step S14: Encapsulate the metadata change data according to the preset data format of the metadata management service.

[0085] In an embodiment of the present application, the metadata management device filters out statistical information changes based on DDLFilter, performs compatibility processing due to different update logic of different engines, and assembles the data that meets the expected push, adds operation users, operation types, task scheduling node information, etc. through a certain data format.

[0086] Among them, DDLFilter is a mechanism for filtering DDL (Data Definition Language) operations, which is mainly used in database synchronization and data replication to ensure that only DDL operations that meet specific conditions are synchronized to the target database.

[0087] The metadata management device can filter the computing engine information in the metadata change information through DDLFilter, and only extract the data content and data type generated by all computing engines, thereby unifying the metadata change data generated by different computing engines.

[0088] Furthermore, the metadata management device encapsulates the filtered metadata change data according to a data format preset in the metadata management service, thereby unifying the metadata formats of all computing engines.

[0089] Finally, if Figure 2 As shown, after the assembled metadata change data passes the Ranger authority verification, the metadata management device sends the assembled data to the HOOK topic of Kafka, thus completing the multi-engine unified metadata process.

[0090] Figure 2 The multi-engine unified metadata process shown in the figure adds event listeners to each engine (Hive, Spark, and Flink) to simplify filtering DDL syntax. However, each time a computing engine is added to the unified metadata, the engine must simultaneously implement a DDL syntax filter to push metadata.

[0091] use Figure 2 The multi-engine unified metadata architecture shown has the following two advantages:

[0092] 1) Reduce the cost of adding new computing engines such as Trino and Tez to the unified metadata architecture. New computing engines can use the unified Metastore-side DDL syntax filter.

[0093] 2) Because the event monitoring is based on the same MySQL database, multiple engines can still ensure the timing of metadata change push.

[0094] In this application, a metadata management device responds to business instructions by performing business processing on a database through a user-connected computing engine; calls a metadata management service to generate metadata change information; filters the computing engine information in the metadata change information to extract the metadata change data; and encapsulates the metadata change data in accordance with a preset data format of the metadata management service. Through this metadata management method, the metadata of the computing engines is centrally managed through the metadata management service, ensuring data consistency and reliability; and the metadata management service is used to filter the computing engine information and unify the metadata updates of different computing engines.

[0095] Furthermore, the metadata management method of this application also involves a multi-user security process, please refer to Figure 4 and Figure 5 , Figure 4 This is a flowchart of the second embodiment of the metadata management method provided by this application. Figure 5 It is a schematic diagram of the multi-user security process provided by this application.

[0096] like Figure 5 As shown, ConfigCenter has super administrator privileges for Ranger, Metastore, and Kafka. Metastore needs to restructure its user system to use ConfigCenter's user system for unified user and security management.

[0097] like Figure 4 As shown, the specific steps are as follows:

[0098] Step S21: Send the encapsulated metadata change data to the hook topic data of the distributed message subscription queue.

[0099] In the embodiment of the present application, Figure 2 At the end of the multi-engine unified metadata process shown, the metadata management device sends the assembled data to the HOOK topic of Kafka.

[0100] Step S22: Obtain the bound user of the database in the hook theme data.

[0101] In an embodiment of the present application, the metadata management device determines all bound users who subscribe to and monitor the HOOK topic. Specifically, the metadata management device analyzes the database involved in the metadata change data in the HOOK topic and then determines at least one bound user that has a binding relationship with the database.

[0102] Step S23: Send the encapsulated metadata change data to the bound user for synchronization.

[0103] In an embodiment of the present application, the metadata management device sends metadata change data to the bound users who have a binding relationship with the database, thereby achieving metadata synchronization effect among multiple users.

[0104] Furthermore, since there are at least two types of permission binding relationships between the database and the user: administrator permission and guest permission. Administrator permission grants the user the permission to access and manage the data in the database, while guest permission only grants the user the permission to access the database.

[0105] Therefore, when synchronizing metadata change data, the metadata management device only needs to synchronize it to the target bound user who has administrator authority over the database, and does not need to synchronize it to the bound user with visitor authority.

[0106] The following combination Figure 5 The multi-user security flow shown continues with the process of binding permissions between the database and the user:

[0107] Specifically, during initial deployment, ConfigCenter creates a super administrator user and other built-in user information and stores it in MySQL. Ranger, Kafka, and Metastore are all registered with ConfigCenter and adapted to a unified user system.

[0108] When a user logs in to ConfigCenter and creates a new user A and A password, and user B and B password, the user is not bound to any database or topic and will not call the Ranger service.

[0109] The user creates database C on the ConfigCenter page, calls the Metastore interface to create database B, and Metastore calls the MySQL interface to store the new database in MySQL.

[0110] Then, on the ConfigCenter page, the user binds user A as the administrator and user B as the guest user to database C. ConfigCenter then calls the MySQL interface to store this association information in the MySQL database. Simultaneously, ConfigCenter calls the Ranger interface to set permissions for user A and user B to operate database C. User A has administrator permissions for database C, and user B has guest permissions for database C.

[0111] In a multi-user environment, data security and isolation are particularly important. Multiple users may access the same data simultaneously, so it's crucial to ensure that each user can only access data within their permissions. Furthermore, when table structures are changed, relevant users must be notified promptly to avoid data inconsistencies and improper operations.

[0112] The permission information bound in the above process also applies to the business processing process, that is, the permission verification must be completed before the business processing. Figure 6 , Figure 6 This is a flowchart of the third embodiment of the metadata management method provided by this application.

[0113] like Figure 6 As shown, the specific steps are as follows:

[0114] Step S31: Responding to the service instruction, obtaining the user's authority information.

[0115] Step S32: When the user's permission information meets the access permission conditions, set the user's permission to use the computing engine.

[0116] Step S33: Perform business processing on the database specified by the business instruction through the computing engine.

[0117] In an embodiment of the present application, when the computing engine (any one of Hive, Spark, or Flink) uses user A to create data table D in database C, or executes any DDL statement, the metadata management device will trigger the Ranger permission verification plug-in. At this time, since user A has administrator privileges on database C, the permission verification passes.

[0118] When the computing engine (any one of Hive, Spark, or Flink) uses user B to create table E in database C or execute any DDL statement, the metadata management device triggers the Ranger permission verification plug-in. At this time, since user B only has guest permissions for database C, the permission verification fails.

[0119] When the computing engine (any one of Hive, Spark, or Flink) uses user B to query table information or read data in database C, or execute query statements, the metadata management device will trigger the Ranger permission verification plug-in. At this time, since user B only has guest permissions to database C, the permission verification passes and the data can be read.

[0120] Combined with the above Figure 5 In the multi-user security process, ConfigCenter consumes Hook topic data. In this message, user A operates on database C, which meets the user system's push conditions. The Kafka interface is called to send the data to A_DATABASE_C_TOPIC. When user A creates a Kafka consumer client connection and consumes data from A_DATABASE_C_TOPIC, the Ranger plug-in is triggered, granting permission to read the data.

[0121] When user B creates a Kafka consumer client connection and consumes data from A_DATABASE_C_TOPIC, the Ranger plug-in is triggered. However, the user's permissions are not granted, resulting in an error and failure to read the data. This is because the guest user only has permission to access the database, not to monitor database changes.

[0122] Metadata management for this application is a core function of the big data platform and mainly includes the following requirements:

[0123] Consistency: Ensures that metadata accessed by different engines and users is consistent.

[0124] Real-time: timely perception and push of metadata change information.

[0125] Security: Ensure data security and isolation between users.

[0126] Scalability: Supports the expansion needs of multiple engines and various users.

[0127] The metadata management method provided by this application can well meet the above requirements. Figure 7 , Figure 7 This is the platform architecture diagram of the metadata management method provided by this application.

[0128] like Figure 7 As shown, the platform system that implements the metadata management method primarily includes three processes: a multi-engine unified metadata process, a multi-user security and isolation process, and a metadata management process. It should be noted that for ease of understanding, the terms computing engine, user management, and metadata management are essentially services or capabilities.

[0129] Combine Figure 7The overall process of the metadata management method for this application is as follows:

[0130] The user calls ConfigCenter through the visualization page to create user A and database B.

[0131] The metadata management device creates a database for users to save user information to ConfigCenter.

[0132] The metadata management device creates a database and updates the metadata information through the Metastore.

[0133] The metadata management device uses ConfigCenter to bind users to databases and designate user A as the administrator of database B. Users not bound to a database cannot query or use the database through the compute engine. Users with guest permissions to the database can only query the database but cannot operate on it or its tables. This functionality is implemented through Ranger's permission management.

[0134] When a business product needs to use a computing engine, the metadata management device connects to a computing engine through user A and performs DDL execution business tasks on database B.

[0135] The metadata management device sends the request to the Metastore metadata management service and records the metadata change information to Mysql.

[0136] The metadata management device filters out DDL-related operations through the Metastore event listener, assembles the change information and sends it to a pre-created single-partition topic named C_TOPIC in Kafka (a distributed messaging publish-subscribe system). The single partition can ensure the time sequence of the data.

[0137] ConfigCenter then consumes the C_TOPIC data and parses the database information in the data. When parsing to database B, it finds that the database is bound to user A in the user system as an administrator. Therefore, it creates a single-partition A_DATABASE_B_TOPIC and sends the DDL data to this topic.

[0138] ConfigCenter calls the Ranger service interface and sets A_DATABASE_B_TOPIC. User A has consumption permissions.

[0139] User A consumes data from Kafka's A_DATABASE_B_TOPIC and obtains metadata change information that complies with time series. This information can be used for interface display or other security-related applications. Other users do not have permission to consume this topic.

[0140] Specifically, to solve the metadata consistency problem between different computing engines, this application will use a unified Metastore system for metadata management. The specific implementation steps are as follows:

[0141] Centralized metadata management: Metastore is used as the metadata management center for all computing engines. All metadata for Hive, Spark, and Flink are managed through Metastore.

[0142] Underlying storage: Metastore metadata is stored in a MySQL database to ensure that all engines use the same metadata, avoiding version inconsistencies caused by data dispersion.

[0143] By adopting a single metadata storage solution, this application will ensure that all engines have access to the same dataset, eliminating the risk of inconsistency caused by data duplication or dispersion.

[0144] To achieve real-time awareness of metadata changes, this application will introduce a transaction hook mechanism and implement data push through Kafka:

[0145] Transaction hooks: Implement transaction hooks in the Metastore to capture metadata operations. Specifically, this application inserts hooks into the Metastore transaction processing process to capture all metadata modification operations.

[0146] Filtering and identification: From the operation information captured by transaction hooks, we filter out statistically relevant change information and retain only DDL (Data Definition Language)-related operations. We further identify specific operation types, such as CREATE, ALTER, and DROP.

[0147] Message Queue: This sends the identified operation information to the Kafka HOOK topic in a specific data structure. As a high-performance message queue, Kafka ensures real-time message delivery.

[0148] Data timing: The HOOK topic uses a partition to ensure that all messages are sent in order, thereby ensuring the timing of the data.

[0149] Through the above steps, the system can efficiently and promptly perceive metadata changes made by all engines, meeting real-time requirements.

[0150] To ensure data security and isolation between users, this application introduces a user system and implements it through the following measures:

[0151] User system: Each user can be set as an administrator or guest user for one or more databases. The administrator user has higher permissions and can perform DDL operations on the database, while the guest user only has access permissions.

[0152] User binding: The administrator user bound to the database will automatically be bound to the consumption permissions of the topic associated with this database and can obtain push messages about database DDL changes.

[0153] Metadata change forwarding: For a database bound to an administrator user, metadata change information for all tables will be forwarded to a new topic named A_DATABASE_C_TOPIC, where A is the user name and C is the database name.

[0154] Permission management: The ConfigCenter service is connected to the computing engine user system to ensure that only valid users have permission to use engines such as Hive, Spark, and Flink. By associating ConfigCenter users, the Metastore database, and Kafka topics with Rager permissions, different levels of users have different operation permissions on the Metastore database, and designated users have consumption or production permissions on Kafka topics.

[0155] To support the expansion requirements of multiple engines and multiple users, this application designs the following mechanisms:

[0156] Metastore-compatible: Any other computing engines that are compatible with Metastore can be managed uniformly using the same metadata management solution. This includes, but is not limited to, new computing engines (such as Trino) or custom computing frameworks.

[0157] Flexible Expansion: The Metastore design allows for easy addition of new computing engines by simply implementing the corresponding metadata operation interface. This ensures system scalability to meet future development needs.

[0158] User Management: The user system supports dynamic addition and deletion of users, as well as flexible configuration of user permissions. Super administrators can manage user binding relationships and permissions through a simple interface.

[0159] Through the above technical solutions, this application achieves metadata consistency, real-time performance, security, and scalability. Unified Metastore management ensures data consistency across different engines. The combination of transaction hooks and Kafka ensures real-time awareness of metadata changes. The user system and Ranger service ensure data security and isolation between users. Flexible expansion mechanisms support the expansion needs of multiple engines and users. This solution not only improves the overall performance of the system but also provides a solid foundation for future expansion and maintenance.

[0160] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0161] In order to implement the above metadata management method, this application also proposes a metadata management device, please refer to Figure 8 , Figure 8 It is a structural diagram of an embodiment of the metadata management device provided by this application.

[0162] The metadata management apparatus 400 of this embodiment includes a processor 41 , a memory 42 , an input / output device 43 , and a bus 44 .

[0163] The processor 41 , memory 42 , and input / output device 43 are respectively connected to a bus 44 . The memory 42 stores program data, and the processor 41 is used to execute the program data to implement the metadata management method described in the above embodiment.

[0164] In the embodiment of the present application, the processor 41 may also be referred to as a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip having signal processing capabilities. The processor 41 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or the processor 41 may be any conventional processor.

[0165] This application also provides a computer storage medium, please continue to refer to Figure 9 , Figure 96 is a schematic structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by a processor, it is used to implement the metadata management method of the above embodiment.

[0166] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0167] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A metadata management method, characterized in that: The metadata management method includes: In response to business instructions, the computing engine connected by the user performs business processing on the database; Call metadata management service to generate metadata change information; Filtering the computing engine information in the metadata change information to extract the metadata change data; The metadata change data is encapsulated according to a preset data format of the metadata management service.

2. The metadata management method according to claim 1, characterized in that: After encapsulating the metadata change data according to the preset data format of the metadata management service, the metadata management method further includes: Send the encapsulated metadata change data to the hook topic data of the distributed message subscription queue; Obtain the bound user of the database in the hook theme data; The encapsulated metadata change data is sent to the bound user for synchronization.

3. The metadata management method according to claim 2, characterized in that: After obtaining the bound user of the database in the hook theme data, the metadata management method further includes: Obtaining permission information of the bound user; Obtain the target bound user whose permission information is administrator permission; The encapsulated metadata change data is sent to the target binding user for synchronization.

4. The metadata management method according to claim 1, wherein: The metadata change data includes: adding the operation user, the operation type, and / or the task scheduling node information.

5. The metadata management method according to claim 4, characterized in that: The calling of the metadata management service to generate metadata change information includes: Calling the event listener of the metadata management service to listen to business processing information; Determining whether there is new metadata information or changed metadata information in the business processing information; If so, classifying the business processing information into task types based on changes in the metadata; The metadata change information is generated according to the task type of the business processing information.

6. The metadata management method according to claim 1, wherein: The metadata management method further includes: Create users and databases; Binding the user to the database and setting the permission information for the user to operate the database; The step of performing business processing on the database by the computing engine connected to the user in response to the business instruction includes: Responding to the service instruction, obtaining the permission information of the user; When the permission information of the user meets the access permission conditions, setting the permission for the user to use the computing engine; The computing engine performs business processing on the database specified by the business instruction.

7. The metadata management method according to claim 6, characterized in that: The performing business processing on the database specified by the business instruction by the computing engine includes: Obtaining the service type of the service instruction; When the service type is a data modification type, obtaining permission information of the user to operate the database specified by the service instruction; Determining whether the permission information meets the permission modification conditions; If so, the computing engine is executed to perform business processing on the database specified by the business instruction.

8. The metadata management method according to claim 1, wherein: The metadata of all computing engines are stored in the same database through the metadata management service.

9. A metadata management device, characterized in that: The metadata management device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the metadata management method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the metadata management method according to any one of claims 1 to 8.

Images