Metadata cloning method, device, system, computer equipment and storage medium

Abstract

The present application relates to a metadata cloning method, device, system, computer equipment, storage medium and computer program product. The method comprises: using a metadata object owner to connect to a source data source, and calling a pre-deployed to-be-cloned object list extraction function in the source data source, extracting the to-be-cloned object from the source data source based on the metadata object owner and the metadata object type; calling a pre-deployed metadata definition information extraction function in the source data source, extracting metadata definition information of each to-be-cloned object from the source data source, and storing the metadata definition information in a first metadata dump table in the source data source; transmitting the target metadata information stored in the first metadata dump table to a second metadata dump table in the destination data source, and reconstructing the metadata information stored in the second metadata dump table to obtain the target metadata. The use of this method can improve data security.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a metadata cloning method, apparatus, system, computer equipment, storage medium and computer program product. Background Art

[0002] In a database, metadata is the basic framework of data content. In important scenarios such as read-write separation, load balancing, multi-active disaster recovery, and verification preparation, there are a lot of demands to support Gaussian database users to initiate and control the cloning and reconstruction of metadata of multiple types of objects within the scope of their permissions.

[0003] Taking GaussDB as an example, the relevant technology uses a "gs_dump (export) + gs_restore (import)" tool provided by the database manufacturer itself to import and export data. This solution only supports the backup and migration of the entire database covering the data content, resulting in serious resource consumption and the risk of data leakage, and poor data security. Summary of the invention

[0004] Based on this, it is necessary to provide a metadata cloning method, apparatus, system, computer device, computer-readable storage medium and computer program product that can improve data security in response to the above technical problems.

[0005] In a first aspect, the present application provides a metadata cloning method, the method comprising:

[0006] When the start condition of the metadata cloning workflow is met, for the source data source and the destination data source specified by the metadata cloning workflow, the metadata object owner specified by the metadata cloning workflow is used to connect to the source data source, and a to-be-cloned object list extraction function pre-deployed in the source data source is called, and the to-be-cloned object list extraction function is used to extract the to-be-cloned object list from the source data source based on the metadata object owner and the metadata object type specified by the metadata cloning workflow;

[0007] Calling a metadata definition information extraction function pre-deployed in the source data source, extracting metadata definition information of each object to be cloned from the source data source through the metadata definition information extraction function, and storing the metadata definition information in a first metadata dump table in the source data source;

[0008] The target metadata information stored in the first metadata dump table is transferred to the second metadata dump table in the destination data source, and the metadata information stored in the second metadata dump table is reconstructed to obtain the target metadata.

[0009] In one embodiment, the method further comprises:

[0010] In response to the target user's input operation on each clone parameter item in the workflow creation interface, obtaining parameter information of each clone parameter item;

[0011] In response to a target user's creation operation on a metadata clone workflow in a workflow creation interface, a workflow template is called, and the metadata clone workflow is created based on the workflow template and parameter information of each of the clone parameter items.

[0012] In one of the embodiments, the cloning parameter items include: scheduling owner user, belonging scheduling project, source data source, destination data source, metadata object type, metadata object owner.

[0013] In one embodiment, the method further comprises:

[0014] Determine whether a metadata dump table has been created in the source data source and the destination data source. If the metadata dump table has not been created in the source data source and / or the destination data source, create the metadata dump table in the source data source and / or the destination data source.

[0015] In one embodiment, the transferring the target metadata information stored in the first metadata dump table to the second metadata dump table in the destination data source includes:

[0016] For any target metadata information stored in the first metadata dump table, determining whether there is metadata information to be updated that overlaps with the target metadata information in the second metadata dump table in the target data source;

[0017] When the metadata information to be updated exists in the second metadata dump table, updating the metadata information to be updated according to the target metadata information;

[0018] When the metadata information to be updated does not exist in the second metadata dump table, the target metadata information is stored in the second metadata dump table.

[0019] In one embodiment, the clone parameter item further includes a dump condition, and the method further includes:

[0020] According to the dump condition specified by the metadata cloning workflow, target metadata information is determined from the first metadata dump table.

[0021] In a second aspect, the present application further provides a metadata cloning system, the system comprising: a cloning device, a source data source, and a target data source, wherein the source data source is deployed with a to-be-cloned object extraction module and a metadata definition information extraction module, the to-be-cloned object extraction module is deployed with a to-be-cloned object list extraction function, and the metadata definition information extraction module is deployed with a metadata definition information extraction function, wherein:

[0022] The cloning device is used to, when the start condition of the metadata cloning workflow is met, for the source data source and the destination data source specified by the metadata cloning workflow, use the metadata object owner specified by the metadata cloning workflow to connect to the source data source, and call the object extraction module to be cloned and the metadata definition information extraction module;

[0023] The to-be-cloned object extraction module is used to extract a list of to-be-cloned objects from the source data source based on the metadata object owner and the metadata object type specified by the metadata cloning workflow, wherein the list of to-be-cloned objects includes at least one to-be-cloned object;

[0024] The metadata definition information extraction module is used to extract the metadata definition information of each object to be cloned from the source data source, and store the metadata definition information in a first metadata dump table in the source data source;

[0025] The cloning device is further used to transfer the target metadata information stored in the first metadata dump table to the second metadata dump table in the destination data source, and to reconstruct the metadata information stored in the second metadata dump table to obtain the target metadata.

[0026] In a third aspect, the present application further provides a metadata cloning device, the device comprising:

[0027] A first processing module is used for, when a start condition of a metadata cloning workflow is met, for a source data source and a destination data source specified by the metadata cloning workflow, using a metadata object owner specified by the metadata cloning workflow to connect to the source data source, and calling a to-be-cloned object list extraction function pre-deployed in the source data source, and extracting a to-be-cloned object list from the source data source through the to-be-cloned object list extraction function based on the metadata object owner and the metadata object type specified by the metadata cloning workflow, wherein the to-be-cloned object list includes at least one to-be-cloned object;

[0028] a second processing module, configured to call a metadata definition information extraction function pre-deployed in the source data source, extract metadata definition information of each object to be cloned from the source data source through the metadata definition information extraction function, and store the metadata definition information in a first metadata dump table in the source data source;

[0029] The dump reconstruction module is used to transfer the target metadata information stored in the first metadata dump table to the second metadata dump table in the destination data source, and to reconstruct the metadata information stored in the second metadata dump table to obtain the target metadata.

[0030] In one embodiment, the device further comprises:

[0031] An acquisition module, configured to acquire parameter information of each clone parameter item in response to an input operation of a target user on each clone parameter item in a workflow creation interface;

[0032] The creation module is used to call the workflow template in response to the target user's creation operation on the metadata clone workflow in the workflow creation interface, and create the metadata clone workflow based on the workflow template and the parameter information of each clone parameter item.

[0033] In one of the embodiments, the cloning parameter items include: scheduling owner user, belonging scheduling project, source data source, destination data source, metadata object type, metadata object owner.

[0034] In one embodiment, the device further comprises:

[0035] The third processing module is used to determine whether a metadata dump table has been created in the source data source and the destination data source, and if the metadata dump table has not been created in the source data source and / or the destination data source, create the metadata dump table in the source data source and / or the destination data source.

[0036] In one of the embodiments, the dump reconstruction module is further used for:

[0037] For any target metadata information stored in the first metadata dump table, determining whether there is metadata information to be updated that overlaps with the target metadata information in the second metadata dump table in the target data source;

[0038] When the metadata information to be updated exists in the second metadata dump table, updating the metadata information to be updated according to the target metadata information;

[0039] When the metadata information to be updated does not exist in the second metadata dump table, the target metadata information is stored in the second metadata dump table.

[0040] In one embodiment, the cloning parameter item further includes a dump condition, and the device further includes:

[0041] The determination module is used to determine target metadata information from the first metadata dump table according to the dump condition specified by the metadata cloning workflow.

[0042] In a fourth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above metadata cloning methods when executing the computer program.

[0043] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above metadata cloning methods.

[0044] In a sixth aspect, the present application also provides a computer program product, including a computer program, which implements any of the above metadata cloning methods when executed by a processor.

[0045] The above-mentioned metadata cloning method, apparatus, system, computer equipment, storage medium and computer program product can, when the start condition of the metadata cloning workflow is met, for the source data source and the destination data source specified by the metadata cloning workflow, use the metadata object owner specified by the metadata cloning workflow to connect to the source data source, and call the to-be-cloned object list extraction function pre-deployed in the source data source, and extract the to-be-cloned object list from the source data source based on the metadata object owner and the metadata object type specified by the metadata cloning workflow through the to-be-cloned object list extraction function; call the metadata definition information extraction function pre-deployed in the source data source, and extract the metadata definition information of each to-be-cloned object from the source data source through the metadata definition information extraction function, and store the metadata definition information in a first metadata dump table in the source data source; transfer the target metadata information stored in the first metadata dump table to the second metadata dump table in the destination data source, and reconstruct the metadata information stored in the second metadata dump table to obtain the target metadata. By adopting the metadata cloning method, apparatus, system, computer equipment, storage medium and computer program product provided in the embodiments of the present disclosure, a workflow can be created to implement metadata cloning. Through the workflow, in the metadata cloning process, only the metadata definition information of the specified data object is transferred out to reconstruct the metadata. The small amount of exported data can reduce resource consumption, and only the metadata definition information is exported without involving the export of specific data content, which can reduce the risk of data leakage and greatly improve data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of a metadata processing method in one embodiment;

[0048] Figure 2 A schematic diagram of a metadata processing method in one embodiment;

[0049] Figure 3 is a schematic diagram of a metadata processing method in one embodiment;

[0050] Figure 4 is a flow chart of step 106 in one embodiment;

[0051] Figure 5 is a schematic diagram of the structure of a metadata processing system in one embodiment;

[0052] Figure 6 is a schematic diagram of a metadata processing method in another embodiment;

[0053] Figure 7 is a structural block diagram of a metadata processing device in one embodiment;

[0054] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] When using the "gs_dump (export) + gs_restore (import)" tool provided by the database vendor to import and export data, the following problems often occur:

[0057] The data content must be exported and imported together. Cloning and rebuilding only the metadata part is not supported. There is a huge resource consumption redundancy. The data information file needs to be dumped in other locations or media storage space outside the database itself. The software, hardware and operation and maintenance service costs are high, and the migration efficiency is low. The dump file method has the risk of data leakage and retention, and requires additional special supervision and timely cleanup. It does not meet the security and compliance requirements of most application scenarios. The gs_dump and gs_restore tool commands can only be executed by database operating system users. It is not flexible enough to meet the needs of ordinary users to clone metadata on demand.

[0058] When the "gs_dump (export) + gs_restore (import)" tool cannot meet user needs, the usual practice is to manually query and collect metadata information and build them one by one. This solution is complex to operate, inefficient, prone to errors, and has high labor costs.

[0059] The embodiment of the present disclosure provides a metadata cloning method to solve the above problems.

[0060] In one embodiment, Figure 1 As shown, a metadata cloning method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0061] Step 102: When the start conditions of the metadata cloning workflow are met, for the source data source and the destination data source specified by the metadata cloning workflow, the metadata object owner specified by the metadata cloning workflow is used to connect to the source data source, and a to-be-cloned object list extraction function pre-deployed in the source data source is called. The to-be-cloned object list extraction function extracts the to-be-cloned object list from the source data source based on the metadata object owner and the metadata object type specified by the metadata cloning workflow, and the to-be-cloned object list includes at least one to-be-cloned object.

[0062] In the disclosed embodiment, the metadata cloning workflow can be a workflow that is created in advance and has a specified start time, or it can be a workflow that is created and started in real time. That is, after the metadata cloning workflow is created, if the metadata cloning workflow is directly triggered to start, it is determined that the start condition of the metadata cloning workflow is currently met; or, when the start time is specified when the metadata cloning workflow is created, when the current time meets the start time, it is determined that the start condition of the metadata cloning workflow is currently met.

[0063] When the starting condition of the metadata clone workflow is met, the metadata clone workflow can be started.

[0064] In an exemplary embodiment, referring to Figure 2 As shown, the above method may also include:

[0065] Step 202, in response to the target user's input operation on each clone parameter item in the workflow creation interface, obtaining parameter information of each clone parameter item;

[0066] Step 204 , in response to the target user's creation operation for the metadata clone workflow in the workflow creation interface, the workflow template is called, and the metadata clone workflow is created based on the workflow template and parameter information of each clone parameter item.

[0067] In the disclosed embodiment, a workflow template can be created in advance, and the user can input parameter information of each clone parameter item through input operation in the workflow creation interface. The clone parameter items may include: scheduling owner user, belonging scheduling project, source data source, destination data source, metadata object type, metadata object owner. Among them, the scheduling owner user can be the user who created the metadata clone workflow, the belonging scheduling project can be the project to which the metadata clone workflow belongs, the source data source provides the data source of the metadata to be cloned, the destination data source can be the data source of the metadata to be cloned and reconstructed, and the metadata object type can include: schema, function / stored procedure (function), sequence (sequence), table (table), view (view) and other database object types, and the metadata object owner is the owner of the metadata to be cloned.

[0068] Reference Figure 3 In the workflow creation interface shown, the user can enter or select the corresponding parameter information in each clone parameter item in the workflow creation interface. After completing the setting of the parameter information of each clone parameter item, the creation control in the interface can be triggered to call the workflow template, add the parameter information of each clone parameter item to the workflow template, and complete the creation of the metadata clone workflow. Among them, the creation control can include a create and start control and a create workflow control. After triggering the create and start control, the metadata clone workflow can be directly created and started at the same time; after triggering the create workflow control, the metadata clone workflow can be created but not started. Optionally, you can jump to the start condition setting interface to set the start condition of the metadata clone workflow.

[0069] After the metadata cloning workflow is started, the metadata object owner specified by the metadata cloning workflow can be used to connect to the source data source specified by the metadata cloning workflow, and the to-be-cloned object list extraction function pre-deployed in the source data source can be called. The to-be-cloned object list extraction function can be used to extract the to-be-cloned objects from the source data source based on the metadata object owner and the metadata object type specified by the metadata cloning workflow, and obtain a to-be-cloned object list. The to-be-cloned object list is used to record each to-be-cloned object, that is, the to-be-cloned object list extraction function can extract data objects of the metadata object type that the metadata object owner can operate as to-be-cloned objects, and construct a to-be-cloned object list.

[0070] The function for extracting the list of objects to be cloned and the function for extracting metadata definition information may be functions pre-deployed by a database administrator or user at the source data source.

[0071] Step 104: call the metadata definition information extraction function pre-deployed in the source data source, extract the metadata definition information of each object to be cloned from the source data source through the metadata definition information extraction function, and store the metadata definition information in the first metadata dump table in the source data source.

[0072] In the embodiment of the present disclosure, after obtaining the list of objects to be cloned, the metadata definition information extraction function pre-deployed in the source data source can be called to extract metadata definition information of each object to be cloned from the source data source through the metadata definition information extraction function. The extracted metadata definition information can be the metadata object type specified by the metadata cloning workflow. The extracted metadata definition information can be stored in the first metadata dump table in the source data source.

[0073] In an exemplary embodiment, before extracting metadata definition information, the following operations may be performed:

[0074] Determine whether a metadata dump table has been created in the source data source and the destination data source. If the metadata dump table has not been created in the source data source and / or the destination data source, create the metadata dump table in the source data source and / or the destination data source.

[0075] In the disclosed embodiment, before extracting metadata definition information, a metadata dump table can be searched in the source data source and the destination data source. If the metadata dump table is not created in the source data source, a metadata dump table is created in the source data source, or, if the metadata dump table is not created in the destination data source, a metadata dump table is created in the destination data source. The metadata dump table is used to store metadata information, and may include fields such as data object, metadata object type, metadata definition information, metadata object owner, and timestamp. The metadata dump table created by the source data source is the first metadata dump table, and the metadata dump table created by the destination data source is the second metadata dump table.

[0076] Step 106: transfer the target metadata information stored in the first metadata dump table to the second metadata dump table in the destination data source, and reconstruct the metadata information stored in the second metadata dump table to obtain the target metadata.

[0077] In the disclosed embodiment, the target metadata information can be obtained from the first resource data storage table, and the target metadata information in the first metadata dump table in the source data source can be incrementally transferred and merged into the second metadata dump table of the same structure under the destination data source by using data synchronization components such as DataX. Among them, batch operations strictly control the upper limit of the number of concurrent operations to avoid exceeding the limit of the number of database connections and affecting the stability and availability of the cluster.

[0078] In the disclosed embodiment, by reconstructing the metadata information in the second metadata dump table, the cloned target metadata can be obtained.

[0079] The metadata cloning method provided by the embodiment of the present disclosure can, when the start condition of the metadata cloning workflow is met, for the source data source and the destination data source specified by the metadata cloning workflow, use the metadata object owner specified by the metadata cloning workflow to connect to the source data source, and call the to-be-cloned object list extraction function pre-deployed in the source data source, and extract the to-be-cloned objects from the source data source based on the metadata object owner and the metadata object type specified by the metadata cloning workflow through the to-be-cloned object list extraction function to obtain the to-be-cloned object list; call the metadata definition information extraction function pre-deployed in the source data source, and extract the metadata definition information of each to-be-cloned object from the source data source through the metadata definition information extraction function, and store the metadata definition information in a first metadata dump table in the source data source; transfer the target metadata information stored in the first metadata dump table to a second metadata dump table in the destination data source, and reconstruct the metadata information stored in the second metadata dump table to obtain the target metadata. By adopting the metadata cloning method provided in the embodiment of the present disclosure, a workflow can be created to implement metadata cloning. Through the workflow, in the metadata cloning process, only the metadata definition information of the specified data object is transferred out to reconstruct the metadata. The small amount of exported data can reduce resource consumption, and only the metadata definition information is exported without involving the export of specific data content, which can reduce the risk of data leakage and greatly improve data security.

[0080] In an exemplary embodiment, the cloning parameter item may further include a dump condition, and the above method may further include:

[0081] According to the dump condition specified by the metadata cloning workflow, target metadata information is determined from the first metadata dump table.

[0082] In the disclosed embodiment, the dump condition specified by the metadata cloning workflow can be pre-set, and the dump condition is a condition for determining the metadata information to be cloned. For example, the extraction time condition of the metadata information to be cloned can be set, and the metadata information that meets the extraction time condition is used as the target metadata information. For example, if the dump condition is "metadata information extracted on the same day", the metadata information with the timestamp information belonging to the same day in the first metadata dump table can be used as the target metadata information, which can avoid the repeated transmission of the metadata information stored in the first metadata dump table to the second metadata dump table, thereby reducing resource consumption.

[0083] In an exemplary embodiment, referring to Figure 4As shown, in step 106, the target metadata information stored in the first metadata dump table is transferred to the second metadata dump table in the destination data source, including:

[0084] Step 402: for any target metadata information stored in the first metadata dump table, determine whether there is metadata information to be updated that overlaps with the target metadata information in the second metadata dump table in the target data source;

[0085] Step 404: when there is metadata information to be updated in the second metadata dump table, update the metadata information to be updated according to the target metadata information;

[0086] Step 406: When there is no metadata information to be updated in the second metadata dump table, the target metadata information is stored in the second metadata dump table.

[0087] In the embodiment of the present disclosure, when writing target metadata information into the second metadata dump table, it can be determined whether there is metadata information to be updated that overlaps with the target metadata information in the second metadata dump table. For example, if the target metadata information is "data object: object 1, metadata object type: view, metadata object owner: owner1, metadata definition information: ddl1, timestamp: t1", if there is metadata information in the second metadata dump table whose data object is "object 1", metadata object type is "view", and metadata object owner is "owner1", it can be determined that the metadata information overlaps with the target metadata information and can be used as the metadata information to be updated. The metadata definition information and timestamp in the metadata information to be updated can be updated according to the metadata definition information and timestamp of the target metadata information. If there is no metadata information to be updated in the second metadata dump table, the target metadata information can be directly stored in the second metadata dump table.

[0088] Read the metadata information of the object to be cloned in the second metadata dump table in the destination data source, and execute the reconstruction script in batches to complete the metadata reconstruction of the object to be cloned. Among them, the batch operation strictly controls the upper limit of the number of concurrent operations to avoid exceeding the limit of the number of database connections and affecting the stability and availability of the cluster.

[0089] The metadata cloning method provided by the embodiment of the present disclosure can ensure that only the latest metadata information of the object to be cloned is retained through duplication checking, and can avoid redundancy or conflict in subsequent metadata reconstruction.

[0090] In one embodiment, Figure 5As shown, a metadata cloning system is provided, the system includes: a cloning device 502, a source data source 504, and a target data source 506. The source data source is deployed with a to-be-cloned object extraction module 5042 and a metadata definition information extraction module 5044. The to-be-cloned object extraction module 5042 is deployed with a to-be-cloned object list extraction function, and the metadata definition information extraction module 5044 is deployed with a metadata definition information extraction function, wherein:

[0091] The cloning device 502 is used to connect the source data source 504 and the destination data source 506 specified by the metadata cloning workflow using the metadata object owner specified by the metadata cloning workflow, and call the object extraction module 5042 and the metadata definition information extraction module 5044 when the start condition of the metadata cloning workflow is met;

[0092] The to-be-cloned object extraction module 5042 is used to extract a list of to-be-cloned objects from the source data source 504 based on the metadata object owner and the metadata object type specified by the metadata cloning workflow, wherein the list of to-be-cloned objects includes at least one to-be-cloned object;

[0093] The metadata definition information extraction module 5044 is used to extract the metadata definition information of each object to be cloned from the source data source 504, and store the metadata definition information in the first metadata dump table in the source data source 504;

[0094] The cloning device 502 is further used to transfer the target metadata information stored in the first metadata dump table to the second metadata dump table in the destination data source 506, and to reconstruct the metadata information stored in the second metadata dump table to obtain the target metadata.

[0095] In the embodiment of the present disclosure, the operations performed by the cloning device 502, the source data source 504, the target data source 506, the to-be-cloned object extraction module 5042 and the metadata definition information extraction module 5044 deployed in the source data source may refer to the relevant description of the aforementioned embodiment, and are not specifically limited in the embodiment of the present disclosure.

[0096] By adopting the metadata cloning system provided by the embodiment of the present disclosure, a workflow can be created to implement metadata cloning. Through the workflow, in the metadata cloning process, only the metadata definition information of the specified data object is exported to reconstruct the metadata. The small amount of exported data can reduce resource consumption, and only the metadata definition information is exported without involving the export of specific data content, which can reduce the risk of data leakage and greatly improve data security.

[0097] In order to enable those skilled in the art to better understand the embodiments of the present disclosure, the embodiments of the present disclosure are described below through specific examples.

[0098] Reference Figure 6 As shown, the system supporter prepares a dedicated metadata cloning workflow template on the scheduling management platform and launches it on the platform front-end interface. The database administrator or user manually creates the "Extract function for the list of objects to be cloned" and the "Extract function for metadata definition information" on the source data source, and optionally creates a metadata dump table on the source data source and the destination data source.

[0099] The user configures six related parameters on the platform front-end interface: "Scheduling Owner User", "Scheduling Project", "Source Data Source", "Destination Data Source", "Metadata Object Type", and "Metadata Object Owner". After clicking Submit, the system backend generates a custom workflow based on a set of dedicated workflow templates on the scheduling management platform to the specified project location, and starts the workflow temporarily or scheduled as needed.

[0100] The system checks whether the source data source and the destination data source have metadata dump tables. If not, they are automatically created. Use the metadata object owner user to connect to the source data source. First, call the function to extract the list of objects to be cloned. According to the specified metadata object type, one of the five types of database objects, "schema", "function / stored procedure", "sequence", "table", and "view", extract the range of objects to be cloned to form a list of objects to be cloned. Then use the metadata definition information extraction function to batch extract the metadata definition information of the objects to be cloned to the metadata dump table. Batch operations strictly control the upper limit of the number of concurrent operations to avoid exceeding the limit of database connections and affecting the stability and availability of the cluster.

[0101] The system uses data synchronization components such as DataX to incrementally transfer and merge the metadata information of the object to be cloned in the metadata dump table of the source data source to the metadata dump table of the same structure under the destination data source. During the transmission, the system will first verify the part of the metadata dump table of the destination data source that overlaps with the object to be cloned and update it, retaining only the latest metadata information of the object to be cloned to avoid redundancy or conflict in subsequent reconstruction.

[0102] The system reads the metadata information of the object to be cloned in the metadata dump table of the destination data source, and executes the script in batches to complete the metadata reconstruction of this part of the object. Batch operations strictly control the upper limit of the number of concurrent operations to avoid exceeding the limit of database connections and affecting the stability and availability of the cluster.

[0103] The metadata cloning method provided in the disclosed embodiment can avoid the waste of resources and leakage risks caused by database content migration. Relying on the existing scheduling management platform capabilities of data research and development, and using the self-developed task component for batch execution of SQL (Structured Query Language, database language) tasks, users can generate workflows with one click after simple configuration, complete the entire process of metadata cloning and reconstruction, and fully meet the needs of users to clone metadata independently and flexibly. It has the following advantages: it avoids migrating database content and supports cloning and rebuilding only object metadata, greatly saving resources and costs, and improving migration efficiency; it avoids the dump file link to ensure that data content is not involved and meet security and compliance requirements; it meets the needs of users to initiate and control the cloning and reconstruction of metadata of multiple types of objects within the scope of their authority; it supports users to simply configure and generate workflows with one click to complete the entire process of metadata cloning and reconstruction, saving labor costs and greatly improving the interactive experience.

[0104] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0105] Based on the same inventive concept, the embodiment of the present application also provides a metadata cloning device for implementing the metadata cloning method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more metadata cloning devices provided below can refer to the limitations of the metadata cloning method above, and will not be repeated here.

[0106] In an exemplary embodiment, Figure 7 As shown, a metadata cloning device is provided, including: a first processing module 702, a second processing module 704 and a dump reconstruction module 706, wherein:

[0107] The first processing module 702 is used for, when the start condition of the metadata cloning workflow is met, for the source data source and the destination data source specified by the metadata cloning workflow, using the metadata object owner specified by the metadata cloning workflow to connect to the source data source, and calling the to-be-cloned object list extraction function pre-deployed in the source data source, and extracting the to-be-cloned object list from the source data source through the to-be-cloned object list extraction function based on the metadata object owner and the metadata object type specified by the metadata cloning workflow, wherein the to-be-cloned object list includes at least one to-be-cloned object;

[0108] The second processing module 704 is used to call the metadata definition information extraction function pre-deployed in the source data source, extract the metadata definition information of each object to be cloned from the source data source through the metadata definition information extraction function, and store the metadata definition information in the first metadata dump table in the source data source;

[0109] The dump reconstruction module 706 is used to transfer the target metadata information stored in the first metadata dump table to the second metadata dump table in the destination data source, and reconstruct the metadata information stored in the second metadata dump table to obtain the target metadata.

[0110] By adopting the metadata cloning device provided in the embodiment of the present disclosure, a workflow can be created to realize metadata cloning. Through the workflow, in the metadata cloning process, only the metadata definition information of the specified data object is transferred out to reconstruct the metadata. The small amount of exported data can reduce resource consumption, and only the metadata definition information is exported without involving the export of specific data content, which can reduce the risk of data leakage and greatly improve data security.

[0111] In one embodiment, the device further comprises:

[0112] An acquisition module, configured to acquire parameter information of each clone parameter item in response to an input operation of a target user on each clone parameter item in a workflow creation interface;

[0113] The creation module is used to call the workflow template in response to the target user's creation operation on the metadata clone workflow in the workflow creation interface, and create the metadata clone workflow based on the workflow template and the parameter information of each clone parameter item.

[0114] In one of the embodiments, the cloning parameter items include: scheduling owner user, belonging scheduling project, source data source, destination data source, metadata object type, metadata object owner.

[0115] In one embodiment, the device further comprises:

[0116] The third processing module is used to determine whether a metadata dump table has been created in the source data source and the destination data source, and if the metadata dump table has not been created in the source data source and / or the destination data source, create the metadata dump table in the source data source and / or the destination data source.

[0117] In one embodiment, the dump reconstruction module 706 is further configured to:

[0118] For any target metadata information stored in the first metadata dump table, determining whether there is metadata information to be updated that overlaps with the target metadata information in the second metadata dump table in the target data source;

[0119] When the metadata information to be updated exists in the second metadata dump table, updating the metadata information to be updated according to the target metadata information;

[0120] When the metadata information to be updated does not exist in the second metadata dump table, the target metadata information is stored in the second metadata dump table.

[0121] In one embodiment, the cloning parameter item further includes a dump condition, and the device further includes:

[0122] The determination module is used to determine target metadata information from the first metadata dump table according to the dump condition specified by the metadata cloning workflow.

[0123] Each module in the metadata cloning device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0124] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a metadata cloning method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0125] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0127] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0128] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A metadata cloning method, characterized in that: The method comprises: In response to input operations for each clone parameter item in the workflow creation interface, parameter information of each clone parameter item is obtained; the clone parameter item includes a scheduling owner user, a scheduling project, a source data source, a destination data source, a metadata object type, and a metadata object owner; the scheduling owner user is a user who creates a metadata clone workflow, and the metadata object owner is an owner of the metadata to be cloned; In response to a creation operation for a metadata clone workflow in a workflow creation interface, a workflow template is called, and a metadata clone workflow is created based on the workflow template and parameter information of each of the clone parameter items; When the start condition of the metadata cloning workflow is met, for the source data source and the destination data source specified by the metadata cloning workflow, the metadata object owner specified by the metadata cloning workflow is used to connect to the source data source, and a to-be-cloned object list extraction function pre-deployed in the source data source is called, and the to-be-cloned object list extraction function is used to extract the to-be-cloned object list from the source data source based on the metadata object owner and the metadata object type specified by the metadata cloning workflow, wherein the to-be-cloned object list includes at least one to-be-cloned object; Determine whether a metadata dump table has been created in a source data source and a destination data source, and if a metadata dump table has not been created in the source data source and / or the destination data source, create a first metadata dump table in the source data source, and / or create a second metadata dump table in the destination data source; Calling a metadata definition information extraction function pre-deployed in the source data source, extracting metadata definition information of each object to be cloned from the source data source through the metadata definition information extraction function, and storing the metadata definition information in a first metadata dump table in the source data source; the extracted metadata definition information is a metadata object type specified by the metadata cloning workflow; After checking the metadata information of the object to be cloned for duplication, the target metadata information stored in the first metadata dump table is transferred to the second metadata dump table in the destination data source, and the metadata information stored in the second metadata dump table is reconstructed to obtain the target metadata; wherein the target metadata information includes the metadata definition information.

2. The method according to claim 1, characterized in that The transferring the target metadata information stored in the first metadata dump table to the second metadata dump table in the destination data source includes: For any target metadata information stored in the first metadata dump table, determining whether there is metadata information to be updated that overlaps with the target metadata information in the second metadata dump table in the destination data source; When the metadata information to be updated exists in the second metadata dump table, updating the metadata information to be updated according to the target metadata information; When the metadata information to be updated does not exist in the second metadata dump table, the target metadata information is stored in the second metadata dump table.

3. The method according to claim 1, characterized in that The clone parameter item also includes a dump condition, and the method further includes: According to the dump condition specified by the metadata cloning workflow, target metadata information is determined from the first metadata dump table; the dump condition is a condition for determining metadata information to be cloned.

4. A metadata cloning system, characterized in that: The system comprises: a cloning device, a source data source, and a destination data source. The source data source is provided with an object extraction module to be cloned and a metadata definition information extraction module. The object extraction module is provided with an object list extraction function to be cloned. The metadata definition information extraction module is provided with a metadata definition information extraction function. The cloning device, in response to an input operation for each clone parameter item in a workflow creation interface, obtains parameter information of each clone parameter item; the clone parameter item includes a scheduling owner user, a belonging scheduling project, a source data source, a destination data source, a metadata object type, and a metadata object owner; in response to a creation operation for a metadata clone workflow in the workflow creation interface, calls a workflow template, and creates a metadata clone workflow based on the workflow template and the parameter information of each clone parameter item; the scheduling owner user is a user who creates the metadata clone workflow, and the metadata object owner is an owner of the metadata to be cloned; The cloning device is used to, when the start condition of the metadata cloning workflow is met, for the source data source and the destination data source specified by the metadata cloning workflow, use the metadata object owner specified by the metadata cloning workflow to connect to the source data source, and call the object extraction module to be cloned and the metadata definition information extraction module; The to-be-cloned object extraction module is used to extract a list of to-be-cloned objects from the source data source based on the metadata object owner and the metadata object type specified by the metadata cloning workflow, wherein the list of to-be-cloned objects includes at least one to-be-cloned object; The metadata definition information extraction module is used to extract the metadata definition information of each object to be cloned from the source data source, and store the metadata definition information in a first metadata dump table in the source data source; the extracted metadata definition information is a metadata object type specified by the metadata cloning workflow; The cloning device is further used to determine whether a metadata dump table has been created in the source data source and the destination data source, and if the metadata dump table has not been created in the source data source and / or the destination data source, create a first metadata dump table in the source data source, and / or create a second metadata dump table in the destination data source; The cloning device is also used to transfer the target metadata information stored in the first metadata dump table to the second metadata dump table in the destination data source after checking the metadata information of the to-be-cloned object for duplication, and to reconstruct the metadata information stored in the second metadata dump table to obtain the target metadata; wherein the target metadata information includes the metadata definition information.

5. A metadata cloning device, characterized in that: The device comprises: An acquisition module is used to obtain parameter information of each clone parameter item in response to an input operation for each clone parameter item in the workflow creation interface; the clone parameter item includes a scheduling owner user, a scheduling project, a source data source, a destination data source, a metadata object type, and a metadata object owner; the scheduling owner user is a user who creates a metadata clone workflow, and the metadata object owner is an owner of the metadata to be cloned; A creation module, configured to call a workflow template in response to a creation operation for a metadata clone workflow in a workflow creation interface, and create a metadata clone workflow based on the workflow template and parameter information of each of the clone parameter items; A first processing module is used for, when a start condition of a metadata cloning workflow is met, for a source data source and a destination data source specified by the metadata cloning workflow, using a metadata object owner specified by the metadata cloning workflow to connect to the source data source, and calling a to-be-cloned object list extraction function pre-deployed in the source data source, and extracting a to-be-cloned object list from the source data source through the to-be-cloned object list extraction function based on the metadata object owner and the metadata object type specified by the metadata cloning workflow, wherein the to-be-cloned object list includes at least one to-be-cloned object; A third processing module is used to determine whether a metadata dump table has been created in the source data source and the destination data source, and if the metadata dump table has not been created in the source data source and / or the destination data source, create a first metadata dump table in the source data source and / or create a second metadata dump table in the destination data source; The second processing module is used to call the metadata definition information extraction function pre-deployed in the source data source, extract the metadata definition information of each object to be cloned from the source data source through the metadata definition information extraction function, and store the metadata definition information in the first metadata dump table in the source data source; the extracted metadata definition information is the metadata object type specified by the metadata cloning workflow; A dump reconstruction module is used to transfer the target metadata information stored in the first metadata dump table to the second metadata dump table in the destination data source after checking the metadata information of the to-be-cloned object for duplication, and to reconstruct the metadata information stored in the second metadata dump table to obtain the target metadata; wherein the target metadata information includes the metadata definition information.

6. The device according to claim 5, characterized in that The dump reconstruction module is also used for: For any target metadata information stored in the first metadata dump table, determining whether there is metadata information to be updated that overlaps with the target metadata information in the second metadata dump table in the destination data source; When the metadata information to be updated exists in the second metadata dump table, updating the metadata information to be updated according to the target metadata information; When the metadata information to be updated does not exist in the second metadata dump table, the target metadata information is stored in the second metadata dump table.

7. The device according to claim 5, characterized in that The cloning parameter item also includes a dump condition, and the device also includes: The determination module is used to determine target metadata information from the first metadata dump table according to the dump condition specified by the metadata cloning workflow; the dump condition is a condition for determining the metadata information to be cloned.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images