Cross-cluster metadata synchronization method and device, equipment and medium

Abstract

The invention discloses a cross-cluster metadata synchronization method and device, equipment and a medium, and relates to the technical field of artificial intelligence. The method comprises the steps of determining standard source cluster metadata, standard target cluster metadata and a preset scheduling synchronization rule corresponding to a metadata synchronization task based on target data synchronization information corresponding to the metadata synchronization task; performing data comparison on the standard source cluster metadata and the standard target cluster metadata based on a preset difference detection algorithm and a preset hierarchy verification rule, and determining a target synchronization data list corresponding to the metadata synchronization task; and performing data synchronization on the target cluster based on the target synchronization data list and a preset scheduling synchronization rule to realize cross-cluster metadata synchronization. Through the technical scheme of the invention, cross-cluster metadata synchronization can be carried out among clusters with different architectures and data formats, and the efficiency of cross-cluster metadata synchronization is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for synchronizing metadata across clusters. Background Art

[0002] With the rapid development of distributed computing and big data technologies, distributed clusters are gradually being applied to various business scenarios. However, due to the different storage data between clusters, data synchronization between multiple clusters becomes very complicated.

[0003] In the prior art, metadata synchronization is usually performed using mapping relationships between clusters. However, due to the different architectures and data formats between different clusters, if metadata synchronization is performed using mapping relationships, it cannot meet the synchronization requirements of different cluster architectures and data formats, thus reducing the efficiency of metadata synchronization. Therefore, how to perform cross-cluster metadata synchronization between clusters with different architectures and data formats and improve the efficiency of cross-cluster metadata synchronization is a problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a method, device, equipment and medium for synchronizing metadata across clusters, which can solve the problem of low efficiency of synchronizing metadata across clusters.

[0005] According to one aspect of the present invention, a method for synchronizing metadata across clusters is provided, comprising:

[0006] Get the target data synchronization information corresponding to the metadata synchronization task;

[0007] Determine, based on the target data synchronization information, the standard source cluster metadata, the standard target cluster metadata, and the preset scheduling synchronization rule corresponding to the metadata synchronization task;

[0008] Compare the standard source cluster metadata and the standard target cluster metadata based on a preset difference detection algorithm and a preset hierarchical verification rule, and determine a target synchronization data list corresponding to the metadata synchronization task;

[0009] The target cluster is synchronized with data based on the target synchronization data list and the preset scheduling synchronization rules to achieve cross-cluster metadata synchronization.

[0010] According to another aspect of the present invention, a device for synchronizing metadata across clusters is provided, comprising:

[0011] An information acquisition module is used to acquire target data synchronization information corresponding to the metadata synchronization task;

[0012] A data acquisition module, used to determine the standard source cluster metadata, the standard target cluster metadata and the preset scheduling synchronization rules corresponding to the metadata synchronization task based on the target data synchronization information;

[0013] A data comparison module, used to compare the standard source cluster metadata and the standard target cluster metadata based on a preset difference detection algorithm and a preset hierarchical verification rule, and determine a target synchronization data list corresponding to the metadata synchronization task;

[0014] The data synchronization module is used to synchronize the data of the target cluster based on the target synchronization data list and the preset scheduling synchronization rules to achieve cross-cluster metadata synchronization.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for synchronizing metadata across clusters described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for synchronizing cross-cluster metadata according to any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method for synchronizing metadata across clusters according to any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention determines the standard source cluster metadata, standard target cluster metadata and preset scheduling synchronization rules corresponding to the metadata synchronization task based on the target data synchronization information corresponding to the metadata synchronization task. Furthermore, based on the preset difference detection algorithm and the preset hierarchical verification rules, data comparison is performed on the standard source cluster metadata and the standard target cluster metadata to determine the target synchronization data list corresponding to the metadata synchronization task. Finally, data synchronization is performed on the target cluster based on the target synchronization data list and the preset scheduling synchronization rules to achieve cross-cluster metadata synchronization. Since the standard format metadata of clusters of different architecture types is obtained according to the data synchronization information, and cross-cluster metadata synchronization is performed according to the differences between the standard format metadata, the problem of low efficiency of cross-cluster metadata synchronization is solved, and cross-cluster metadata synchronization can be performed between clusters of different architectures and data formats, thereby improving the efficiency of cross-cluster metadata synchronization.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a flowchart of a method for synchronizing metadata across clusters provided according to Embodiment 1 of the present invention;

[0025] Figure 2 This is a flowchart of a method for synchronizing metadata across clusters provided according to Embodiment 2 of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of a device for synchronizing metadata across clusters provided according to Embodiment 3 of the present invention;

[0027] Figure 4 It is a structural schematic diagram of an electronic device for implementing the method for synchronizing metadata across clusters according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", "target", "base", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] It is worth noting that in the technical solution of the present application, the information collected is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse; if the user chooses to refuse, the expert decision-making process will be entered.

[0031] Embodiment 1

[0032] Figure 1 A flowchart of a method for cross-cluster metadata synchronization provided in Embodiment 1 of the present invention. This embodiment is applicable to cross-cluster metadata synchronization of clusters with different cluster architectures and data formats. The method can be executed by a cross-cluster metadata synchronization device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Exemplarily, it can be configured in a synchronization engine. Figure 1 As shown, the method includes:

[0033] S110: Obtain target data synchronization information corresponding to the metadata synchronization task.

[0034] Metadata may refer to information data used to describe the content, quality, representation, spatial reference, management method, and other attribute characteristics of a dataset. A metadata synchronization task may refer to a task of synchronizing metadata between two different clusters.

[0035] Among them, the target data synchronization information may refer to the configuration parameter information required to implement the metadata synchronization task. In an optional implementation, the target data synchronization information includes: metadata synchronization task name, source cluster name, source cluster type, target cluster name, target cluster type and metadata synchronization frequency. The source cluster may refer to the source-end cluster containing the metadata to be synchronized. The source cluster name may refer to the cluster name of the source cluster. The source cluster type may refer to the architecture type of the source cluster. The target cluster may refer to the target-end cluster that stores the metadata to be synchronized. The target cluster name may refer to the cluster name of the target cluster. The target cluster type may refer to the architecture type of the target cluster. Usually, the cluster name and architecture type can be obtained from the original parameters of each cluster. The metadata synchronization frequency may refer to the frequency of metadata synchronization operations between the source cluster and the target cluster. Usually, the metadata synchronization frequency can be formulated according to the actual application scenario. Therefore, by refining the target data synchronization information, the specific content information contained in the target data synchronization information can be determined, providing an effective basis for subsequent operations.

[0036] S120. Determine, based on the target data synchronization information, standard source cluster metadata, standard target cluster metadata, and preset scheduling synchronization rules corresponding to the metadata synchronization task.

[0037] The standard source cluster metadata may refer to the metadata in a standard format contained in the source cluster. The standard target cluster metadata may refer to the metadata in a standard format contained in the target cluster. Generally, clusters of different architecture types have different metadata formats. Therefore, in the embodiment of the present invention, standard metadata in a unified data format is used to synchronize metadata, thereby improving the accuracy of metadata synchronization.

[0038] The preset scheduling synchronization rule may refer to a preset metadata synchronization rule. Exemplarily, the preset scheduling synchronization rule may be a metadata synchronization frequency, or may be a metadata synchronization rule adjusted according to user needs. Specifically, the preset scheduling synchronization rule may be constructed according to the metadata synchronization frequency in the target data synchronization information, so that subsequent metadata synchronization tasks are performed according to the preset scheduling synchronization rule, thereby improving the efficiency of metadata synchronization.

[0039] S130 . Compare the standard source cluster metadata and the standard target cluster metadata based on a preset difference detection algorithm and a preset hierarchical verification rule to determine a target synchronization data list corresponding to the metadata synchronization task.

[0040] The preset difference detection algorithm may refer to a preset algorithm for detecting difference metadata between a source cluster and a target cluster. The preset level verification rule may refer to a preset rule for limiting the detection scope of the preset difference detection algorithm. Exemplarily, the preset level verification rule may include each detection level corresponding to the detection scope, and may also include a priority order between each detection level.

[0041] The synchronization data list may refer to a collection of metadata information that needs to be synchronized. Exemplarily, the synchronization data list may include a list of data to be synchronized, or a list of data to be deleted. The target synchronization data list may refer to the synchronization data list corresponding to the metadata synchronization task.

[0042] S140: Synchronize data on the target cluster based on the target synchronization data list and the preset scheduling synchronization rule to achieve cross-cluster metadata synchronization.

[0043] Specifically, after comparing the standard source cluster metadata and the standard target cluster metadata based on the preset difference detection algorithm and the preset hierarchical verification rules and determining the target synchronization data list corresponding to the metadata synchronization task, the metadata to be synchronized in the source cluster can be obtained based on the target synchronization data list, and then the target cluster can be synchronized based on the preset scheduling synchronization rules and the metadata to be synchronized.

[0044] It is worth noting that in the embodiment of the present invention, data synchronization may include operations such as adding, deleting and modifying metadata, and the embodiment of the present invention does not specifically limit this.

[0045] The technical solution of the embodiment of the present invention determines the standard source cluster metadata, standard target cluster metadata and preset scheduling synchronization rules corresponding to the metadata synchronization task based on the target data synchronization information corresponding to the metadata synchronization task. Furthermore, based on the preset difference detection algorithm and the preset hierarchical verification rules, data comparison is performed on the standard source cluster metadata and the standard target cluster metadata to determine the target synchronization data list corresponding to the metadata synchronization task. Finally, data synchronization is performed on the target cluster based on the target synchronization data list and the preset scheduling synchronization rules to achieve cross-cluster metadata synchronization. Since the standard format metadata of clusters of different architecture types is obtained according to the data synchronization information, and cross-cluster metadata synchronization is performed according to the differences between the standard format metadata, the problem of low efficiency of cross-cluster metadata synchronization is solved, and cross-cluster metadata synchronization can be performed between clusters of different architectures and data formats, thereby improving the efficiency of cross-cluster metadata synchronization.

[0046] Embodiment 2

[0047] Figure 2 A flowchart of a method for cross-cluster metadata synchronization is provided for the second embodiment of the present invention. This embodiment is refined based on the above-mentioned embodiment. In this embodiment, the operation of determining the standard source cluster metadata and the standard target cluster metadata corresponding to the metadata synchronization task based on the target data synchronization information is specifically refined, which may include: determining the first adapter and the first converter corresponding to the source cluster based on the source cluster type and the preset plug-in system, and determining the second adapter and the second converter corresponding to the target cluster based on the target cluster type and the preset plug-in system; determining the basic source cluster metadata corresponding to the source cluster based on the source cluster name and the first adapter, and determining the basic target cluster metadata corresponding to the target cluster based on the target cluster name and the second adapter; formatting the basic source cluster metadata based on the first converter to obtain the standard source cluster metadata corresponding to the source cluster, and formatting the basic target cluster metadata based on the second converter to obtain the standard target cluster metadata corresponding to the target cluster. Figure 2 As shown, the method includes:

[0048] S210. Obtain target data synchronization information corresponding to the metadata synchronization task; wherein the target data synchronization information includes: metadata synchronization task name, source cluster name, source cluster type, target cluster name, target cluster type and metadata synchronization frequency.

[0049] S220: Determine a first adapter and a first converter corresponding to the source cluster based on the source cluster type and the preset plug-in system, and determine a second adapter and a second converter corresponding to the target cluster based on the target cluster type and the preset plug-in system.

[0050] Among them, the preset plug-in system may refer to a preset system for storing extensions. Exemplarily, the preset plug-in system may include preset adapters and converters. An adapter may refer to a device that interacts with a specific type of cluster to obtain metadata in the cluster. Exemplarily, an adapter may include methods for connection, authentication, data acquisition, and data push. A converter may refer to a device that converts the format of metadata in a cluster. The converter may include conversion rules between different types of clusters, different types of data structures, and common data structures. Different types of metadata can be converted into a common standard format through a converter. Exemplarily, a standard format may include common data structures, attributes, and source clusters, etc.

[0051] Among them, the data structure may include data type, data format and data storage location, etc. The data type may be structured, semi-structured or unstructured, and the data storage location may be a local file system, cloud storage or database, etc. If the data type is structured data, the data structure may also include information such as data fields and constraints. Attributes may include the owner or creator of the data, as well as the access rights to the data, such as: public, private or restricted, etc. The source cluster may include the unique identifier of the cluster, the name of the cluster, the description of the cluster and the type of the cluster. Usually, the adapters and converters corresponding to the same cluster architecture match each other, that is, the adapters and converters exist in pairs. The first adapter may refer to the adapter corresponding to the source cluster. The first converter may refer to the converter corresponding to the source cluster. The second adapter may refer to the adapter corresponding to the target cluster. The second converter may refer to the converter corresponding to the target cluster.

[0052] In an optional embodiment, before determining the first adapter and the first converter corresponding to the source cluster based on the source cluster type and the preset plug-in system, and determining the second adapter and the second converter corresponding to the target cluster based on the target cluster type and the preset plug-in system, it also includes: obtaining the preset cluster type, and determining the basic adapter and the basic converter corresponding to the preset cluster type based on the preset cluster type and historical configuration specifications; combining and processing the preset cluster type, the basic adapter and the basic converter to generate a basic plug-in sequence corresponding to the preset cluster type, and statistically processing the basic plug-in sequences corresponding to each preset cluster type to generate a preset plug-in system.

[0053] Among them, the preset cluster type may refer to a preset cluster architecture type. Exemplarily, each preset cluster type may be determined according to the data synchronization scenario within the historical time period. The historical configuration specification may refer to the plug-in configuration relationship corresponding to the preset cluster type in the data synchronization scenario within the historical time period. The basic adapter may refer to the adapter corresponding to the preset cluster type. The basic converter may refer to the converter corresponding to the preset cluster type. The basic plug-in sequence may refer to the information combination consisting of the same preset cluster type and the corresponding basic adapter and basic converter.

[0054] Specifically, before determining the first adapter and the first converter corresponding to the source cluster based on the source cluster type and the preset plug-in system, and determining the second adapter and the second converter corresponding to the target cluster based on the target cluster type and the preset plug-in system, the plug-in system can be constructed by using the preset cluster type and the historical configuration specifications of the known adapters and converters. Thus, when it is necessary to determine the first adapter and the first converter corresponding to the source cluster, and the second adapter and the second converter corresponding to the target cluster, the source cluster type and the target cluster type can be directly used to search for the architecture type in the preset plug-in system, providing an effective basis for subsequent metadata acquisition and improving the efficiency of metadata acquisition.

[0055] It is worth noting that, in the embodiment of the present invention, the conversion rules may be configured in a database table, and the conversion rules may be periodically and dynamically cached in a memory through a converter to implement the corresponding conversion rules.

[0056] S230: Determine basic source cluster metadata corresponding to the source cluster based on the source cluster name and the first adapter, and determine basic target cluster metadata corresponding to the target cluster based on the target cluster name and the second adapter.

[0057] The basic source cluster metadata may refer to the original unprocessed metadata in the source cluster, and the basic target cluster metadata may refer to the original unprocessed metadata in the target cluster.

[0058] Specifically, after determining the first adapter and the first converter corresponding to the source cluster based on the source cluster type and the preset plug-in system, and determining the second adapter and the second converter corresponding to the target cluster based on the target cluster type and the preset plug-in system, the first adapter can be used to connect to the source cluster to obtain the basic source cluster metadata corresponding to the source cluster, and the second adapter can be used to connect to the target cluster to obtain the basic target cluster metadata corresponding to the target cluster.

[0059] It is worth noting that in the embodiment of the present invention, after the source cluster is connected based on the source cluster name and the first adapter, the metadata in the source cluster can be parsed using a metadata parser to obtain the basic source cluster metadata corresponding to the source cluster. Similarly, after the target cluster is connected based on the target cluster name and the second adapter, the metadata in the target cluster can be parsed using a metadata parser to obtain the basic target cluster metadata corresponding to the target cluster.

[0060] S240. Based on the first converter, the basic source cluster metadata is formatted to obtain standard source cluster metadata corresponding to the source cluster; based on the second converter, the basic target cluster metadata is formatted to obtain standard target cluster metadata corresponding to the target cluster.

[0061] Specifically, after determining the basic source cluster metadata corresponding to the source cluster based on the source cluster name and the first adapter, and determining the basic target cluster metadata corresponding to the target cluster based on the target cluster name and the second adapter, the first converter can be used to determine the conversion rules for formatting the basic source cluster metadata, and the basic source cluster metadata is formatted according to the conversion rules to obtain standard source cluster metadata. Similarly, the second converter is used to determine the conversion rules for formatting the basic target cluster metadata, and the basic target cluster metadata is formatted according to the conversion rules to obtain standard target cluster metadata. Thus, the corresponding adapter and converter are determined by the cluster type, the cluster metadata is obtained through the adapter, and the cluster metadata is formatted through the converter. It can adapt to clusters with different architectures and data formats, providing an effective basis for subsequent metadata synchronization.

[0062] S250. Determine a preset scheduling synchronization rule corresponding to the metadata synchronization task based on the target data synchronization information.

[0063] Specifically, the metadata synchronization frequency in the target data synchronization information may be used to determine a preset scheduling synchronization rule corresponding to the metadata synchronization task, so that subsequent metadata synchronization operations are implemented according to the preset scheduling synchronization rule.

[0064] S260: Acquire a source cluster table structure in standard source cluster metadata and a target cluster table structure in standard target cluster metadata.

[0065] The table structure may refer to the structure of a database table consisting of basic attributes such as data types, fields, primary keys, foreign keys, and indexes. Usually, the table structure is based on columns. The source cluster table structure may refer to the table structure corresponding to the source cluster. The target cluster table structure may refer to the table structure corresponding to the target cluster.

[0066] S270: Perform data comparison on the source cluster table structure and the target cluster table structure based on a preset difference detection algorithm to generate a table structure synchronization list.

[0067] The table structure synchronization list may refer to a set of table structures that require metadata synchronization.

[0068] Specifically, after obtaining the source cluster table structure in the standard source cluster metadata and the target cluster table structure in the standard target cluster metadata, the preset difference detection algorithm can be used to compare the source cluster table structure and the target cluster table structure, and compare the table structures with the same library name and table name in the two clusters one by one. If there is table A in the source cluster and there is no table A in the target cluster, all metadata in table A need to be synchronized, and table A can be added to the table structure synchronization list.

[0069] S280: Determine a candidate source cluster field in the standard source cluster metadata and a candidate target cluster field in the standard target cluster metadata based on the table structure synchronization list.

[0070] The source cluster field to be selected may refer to a field in the source cluster that needs to be compared with the field, and the target cluster field to be selected may refer to a field in the target cluster that needs to be compared with the field.

[0071] Specifically, when using the preset difference detection algorithm to compare the source cluster table structure and the target cluster table structure, and comparing the table structures of the same library name and table name in the two clusters one by one, if there is table B in the source cluster and table B in the target cluster, the fields in table B under the source cluster can be used as candidate source cluster fields, and the fields in table B under the target cluster can be used as candidate target cluster fields. This is equivalent to using all fields in the remaining tables except the table structure synchronization list as candidate fields.

[0072] S290: Perform data comparison on the candidate source cluster fields and the candidate target cluster fields based on a preset difference detection algorithm to generate a field synchronization list.

[0073] The field synchronization list may refer to a set of fields that require metadata synchronization.

[0074] Specifically, after determining the candidate source cluster fields in the standard source cluster metadata and the candidate target cluster fields in the standard target cluster metadata based on the table structure synchronization list, the preset difference detection algorithm can be used to perform data comparison on the candidate source cluster fields and the candidate target cluster fields, and compare the candidate source cluster fields and the candidate target cluster fields one by one, and add the different fields between the candidate source cluster fields and the candidate target cluster fields to the field synchronization list.

[0075] S2100. Combine and process the table structure synchronization list and the field synchronization list to generate a target synchronization data list corresponding to the metadata synchronization task.

[0076] Specifically, after generating the table structure synchronization list and the field synchronization list, the table structure synchronization list and the field synchronization list can be combined and processed to obtain the target synchronization data list corresponding to the metadata synchronization task. Thus, the table structure level and the field level between the metadata are compared respectively through the difference detection algorithm to obtain the synchronization data list. This can provide an effective basis for subsequent data synchronization, ensure the accuracy of the synchronization data list, and improve the efficiency of metadata synchronization.

[0077] S2110. Perform data synchronization on the target cluster based on the target synchronization data list and the preset scheduling synchronization rule to achieve cross-cluster metadata synchronization.

[0078] Specifically, after generating the target synchronization data list corresponding to the metadata synchronization task, the table structure synchronization list in the target synchronization data list can be used to determine the candidate table, and all metadata in the candidate table can be added to the target cluster. Also, the field synchronization list in the target synchronization data list can be used to determine the candidate fields, and according to the field comparison result between the source cluster and the target cluster, the candidate fields are synchronized to the target cluster. Exemplarily, if the field comparison result between the source cluster and the target cluster is that the source cluster has field C and the target cluster does not have field C, then the C field is added to the target cluster. If the field comparison result between the source cluster and the target cluster is that the target cluster has field D and the source cluster does not have field D, then the D field is deleted from the target cluster. If the field comparison result between the source cluster and the target cluster is that the source cluster has field E with a content of e1, and the target cluster also has field E with a content of e2, then the content e2 of field E in the target cluster is modified to e1.

[0079] It is worth noting that in an embodiment of the present invention, metadata synchronization can be performed based on a preset security mechanism. The preset security mechanism may include a data transmission encryption mechanism, a data storage encryption mechanism, and a user verification mechanism. Specifically, when a user establishes a metadata synchronization task, the user's identity can be verified based on the user verification mechanism to ensure that only authorized users can access specific operations. When metadata transmission is performed, a data transmission encryption mechanism can be used to protect the transmission of metadata in the network. When metadata is stored, the metadata can be encrypted using a data storage encryption mechanism to ensure the confidentiality of the metadata. In this way, the security and integrity of the metadata during transmission and storage are ensured.

[0080] In an optional embodiment, after synchronizing the target cluster data based on the target synchronization data list and the preset scheduling synchronization rules to achieve cross-cluster metadata synchronization, it also includes: determining the candidate source cluster metadata and the candidate target cluster metadata corresponding to the metadata synchronization task based on the target data synchronization information; performing a consistency comparison between the candidate source cluster metadata and the candidate target cluster metadata to generate a comparison result, and generating warning information corresponding to the metadata synchronization task based on the comparison result.

[0081] Among them, the metadata of the candidate source cluster may refer to the metadata that needs to be verified in the source cluster after metadata synchronization. The metadata of the candidate target cluster may refer to the metadata that needs to be verified in the target cluster after metadata synchronization. The comparison result may refer to the difference result between the metadata of the candidate source cluster and the corresponding metadata of the candidate target cluster. Exemplarily, the comparison result may be relatively consistent or relatively inconsistent. The warning information may refer to the information that warns the comparison result. Exemplarily, the warning information may be the specific difference content corresponding to the comparison result when the comparison result is relatively inconsistent.

[0082] Specifically, after realizing cross-cluster metadata synchronization, the first adapter and the first converter corresponding to the source cluster, and the second adapter and the second converter corresponding to the target cluster can be determined according to the target data synchronization information. The first adapter is used to connect to the source cluster, obtain the metadata corresponding to the source cluster, and the first converter is used to perform data conversion on the metadata to obtain the metadata of the selected source cluster, and the second adapter is used to connect to the target cluster, obtain the metadata corresponding to the target cluster, and the second converter is used to perform data conversion on the metadata to obtain the metadata of the selected target cluster. After that, the table structures of the same library name and table name in the metadata of the selected source cluster and the metadata of the selected target cluster are compared one by one using a preset difference detection algorithm, and the field data structures in the metadata of the selected source cluster and the metadata of the selected target cluster are compared field by field to see if they are consistent. If they are consistent, a relatively consistent comparison result is generated; if they are inconsistent, a relatively inconsistent comparison result is generated, and a corresponding warning information is generated. Therefore, after metadata synchronization is performed, the accuracy of metadata synchronization can be guaranteed by checking the synchronized metadata.

[0083] The technical solution of the embodiment of the present invention is to obtain the target data synchronization information corresponding to the metadata synchronization task, including the metadata synchronization task name, source cluster name, source cluster type, target cluster name, target cluster type and metadata synchronization frequency. Then, based on the source cluster type and the preset plug-in system, determine the first adapter and the first converter corresponding to the source cluster, based on the target cluster type and the preset plug-in system, determine the second adapter and the second converter corresponding to the target cluster, based on the source cluster name and the first adapter, determine the basic source cluster metadata corresponding to the source cluster, based on the target cluster name and the second adapter, determine the basic target cluster metadata corresponding to the target cluster, and based on the first converter, format conversion is performed on the basic source cluster metadata to obtain the standard source cluster metadata corresponding to the source cluster, and based on the second converter, format conversion is performed on the basic target cluster metadata to obtain the standard target cluster metadata corresponding to the target cluster. Further, the preset scheduling synchronization rule corresponding to the metadata synchronization task is determined based on the target data synchronization information. The source cluster table structure in the standard source cluster metadata and the target cluster table structure in the standard target cluster metadata are obtained. Data comparison is performed on the source cluster table structure and the target cluster table structure based on the preset difference detection algorithm to generate a table structure synchronization list. Based on the table structure synchronization list, the candidate source cluster fields in the standard source cluster metadata and the candidate target cluster fields in the standard target cluster metadata are determined. Based on the preset difference detection algorithm, data comparison is performed on the candidate source cluster fields and the candidate target cluster fields to generate a field synchronization list, and the table structure synchronization list and the field synchronization list are combined to generate a target synchronization data list corresponding to the metadata synchronization task. Finally, data synchronization is performed on the target cluster based on the target synchronization data list and the preset scheduling synchronization rules to achieve cross-cluster metadata synchronization. Since the standard format metadata of clusters of different architecture types is obtained according to the data synchronization information, and cross-cluster metadata synchronization is performed according to the differences between the standard format metadata, the problem of low efficiency of cross-cluster metadata synchronization is solved, and cross-cluster metadata synchronization can be performed between clusters with different architectures and data formats, thereby improving the efficiency of cross-cluster metadata synchronization.

[0084] Continuing with the above implementation, in an embodiment of the present invention, an interface for implementing a method for cross-cluster metadata synchronization may also be constructed. The interface may include: a synchronization configuration page, a synchronization monitoring page, and a synchronization trigger button. The synchronization configuration page is used to receive user instructions for adding, modifying, and deleting synchronization rules, and to adjust the preset scheduling synchronization rules according to the instructions. The synchronization monitoring page is used to display the status and log of the metadata synchronization task. The synchronization trigger button is used to receive a metadata synchronization task trigger instruction sent by a user to start the metadata synchronization task based on the instruction.

[0085] Embodiment 3

[0086] Figure 3This is a schematic diagram of the structure of a device for synchronizing metadata across clusters provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an information acquisition module 310, a data acquisition module 320, a data comparison module 330 and a data synchronization module 340;

[0087] The information acquisition module 310 is used to acquire the target data synchronization information corresponding to the metadata synchronization task;

[0088] A data acquisition module 320 is used to determine the standard source cluster metadata, the standard target cluster metadata and the preset scheduling synchronization rule corresponding to the metadata synchronization task based on the target data synchronization information;

[0089] A data comparison module 330 is used to compare the standard source cluster metadata and the standard target cluster metadata based on a preset difference detection algorithm and a preset hierarchical verification rule, and determine a target synchronization data list corresponding to the metadata synchronization task;

[0090] The data synchronization module 340 is used to synchronize data on the target cluster based on the target synchronization data list and the preset scheduling synchronization rule to achieve cross-cluster metadata synchronization.

[0091] The technical solution of the embodiment of the present invention determines the standard source cluster metadata, standard target cluster metadata and preset scheduling synchronization rules corresponding to the metadata synchronization task based on the target data synchronization information corresponding to the metadata synchronization task. Furthermore, based on the preset difference detection algorithm and the preset hierarchical verification rules, data comparison is performed on the standard source cluster metadata and the standard target cluster metadata to determine the target synchronization data list corresponding to the metadata synchronization task. Finally, data synchronization is performed on the target cluster based on the target synchronization data list and the preset scheduling synchronization rules to achieve cross-cluster metadata synchronization. Since the standard format metadata of clusters of different architecture types is obtained according to the data synchronization information, and cross-cluster metadata synchronization is performed according to the differences between the standard format metadata, the problem of low efficiency of cross-cluster metadata synchronization is solved, and cross-cluster metadata synchronization can be performed between clusters of different architectures and data formats, thereby improving the efficiency of cross-cluster metadata synchronization.

[0092] Optionally, target data synchronization information includes: metadata synchronization task name, source cluster name, source cluster type, target cluster name, target cluster type, and metadata synchronization frequency.

[0093] Optionally, the data acquisition module 320 may be specifically used for:

[0094] Determine a first adapter and a first converter corresponding to the source cluster based on the source cluster type and the preset plug-in system, and determine a second adapter and a second converter corresponding to the target cluster based on the target cluster type and the preset plug-in system;

[0095] Determine the basic source cluster metadata corresponding to the source cluster based on the source cluster name and the first adapter, and determine the basic target cluster metadata corresponding to the target cluster based on the target cluster name and the second adapter;

[0096] The basic source cluster metadata is formatted based on the first converter to obtain standard source cluster metadata corresponding to the source cluster, and the basic target cluster metadata is formatted based on the second converter to obtain standard target cluster metadata corresponding to the target cluster.

[0097] Optionally, the device for synchronizing metadata across clusters may further include: a plug-in system generation module, which is used to obtain a preset cluster type before determining the first adapter and the first converter corresponding to the source cluster based on the source cluster type and the preset plug-in system, and determining the second adapter and the second converter corresponding to the target cluster based on the target cluster type and the preset plug-in system, and determine the basic adapter and the basic converter corresponding to the preset cluster type based on the preset cluster type and historical configuration specifications; combine and process the preset cluster type, the basic adapter and the basic converter to generate a basic plug-in sequence corresponding to the preset cluster type, and statistically process the basic plug-in sequences corresponding to each preset cluster type to generate a preset plug-in system.

[0098] Optionally, the data comparison module 330 may be specifically used for:

[0099] Obtain the source cluster table structure in the standard source cluster metadata and the target cluster table structure in the standard target cluster metadata;

[0100] Compare the source cluster table structure and the target cluster table structure based on a preset difference detection algorithm to generate a table structure synchronization list;

[0101] Determine, based on the table structure synchronization list, a candidate source cluster field in the standard source cluster metadata and a candidate target cluster field in the standard target cluster metadata;

[0102] Compare the data of the candidate source cluster field and the candidate target cluster field based on a preset difference detection algorithm to generate a field synchronization list;

[0103] The table structure synchronization list and the field synchronization list are processed in combination to generate a target synchronization data list corresponding to the metadata synchronization task.

[0104] Optionally, the device for cross-cluster metadata synchronization may further include: a data verification module, which is used to determine the candidate source cluster metadata and the candidate target cluster metadata corresponding to the metadata synchronization task based on the target data synchronization information after performing data synchronization on the target cluster based on the target synchronization data list and the preset scheduling synchronization rules to achieve cross-cluster metadata synchronization; perform consistency comparison on the candidate source cluster metadata and the candidate target cluster metadata to generate a comparison result, and generate warning information corresponding to the metadata synchronization task based on the comparison result.

[0105] The device for cross-cluster metadata synchronization provided by the embodiment of the present invention can execute the method for cross-cluster metadata synchronization provided by any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0106] Embodiment 4

[0107] Figure 4 A schematic diagram of an electronic device 410 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0108] like Figure 4 As shown, the electronic device 410 includes at least one processor 420, and a memory connected to the at least one processor 420 in communication, such as a read-only memory (ROM) 430, a random access memory (RAM) 440, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 420 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 430 or the computer program loaded from the storage unit 490 to the random access memory (RAM) 440. In the RAM 440, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 420, the ROM 430, and the RAM 440 are connected to each other via a bus 450. An input / output (I / O) interface 460 is also connected to the bus 450.

[0109] Multiple components in the electronic device 410 are connected to the I / O interface 460, including: an input unit 470, such as a keyboard, a mouse, etc.; an output unit 480, such as various types of displays, speakers, etc.; a storage unit 490, such as a disk, an optical disk, etc.; and a communication unit 4100, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 4100 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0110] The processor 420 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 420 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 420 performs the various methods and processes described above, such as a method for synchronizing metadata across clusters.

[0111] The method includes:

[0112] Get the target data synchronization information corresponding to the metadata synchronization task;

[0113] Determine, based on the target data synchronization information, the standard source cluster metadata, the standard target cluster metadata, and the preset scheduling synchronization rule corresponding to the metadata synchronization task;

[0114] Compare the standard source cluster metadata and the standard target cluster metadata based on a preset difference detection algorithm and a preset hierarchical verification rule, and determine a target synchronization data list corresponding to the metadata synchronization task;

[0115] The target cluster is synchronized with data based on the target synchronization data list and the preset scheduling synchronization rules to achieve cross-cluster metadata synchronization.

[0116] In some embodiments, the method for cross-cluster metadata synchronization may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 490. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 430 and / or the communication unit 4100. When the computer program is loaded into the RAM 440 and executed by the processor 420, one or more steps of the method for cross-cluster metadata synchronization described above may be performed. Alternatively, in other embodiments, the processor 420 may be configured to perform the method for cross-cluster metadata synchronization in any other appropriate manner (e.g., by means of firmware).

[0117] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0119] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0122] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0123] The present application also discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for synchronizing cross-cluster metadata provided by any embodiment of the present application is implemented. The program product and the method for synchronizing cross-cluster metadata disclosed in each embodiment of the present application belong to the same inventive concept, so they are not described here.

[0124] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0125] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for synchronizing metadata across clusters, characterized in that: include: Get the target data synchronization information corresponding to the metadata synchronization task; Determine, based on the target data synchronization information, the standard source cluster metadata, the standard target cluster metadata, and the preset scheduling synchronization rule corresponding to the metadata synchronization task; Compare the standard source cluster metadata and the standard target cluster metadata based on a preset difference detection algorithm and a preset hierarchical verification rule, and determine a target synchronization data list corresponding to the metadata synchronization task; The target cluster is synchronized with data based on the target synchronization data list and the preset scheduling synchronization rules to achieve cross-cluster metadata synchronization.

2. The method according to claim 1, characterized in that The target data synchronization information includes: metadata synchronization task name, source cluster name, source cluster type, target cluster name, target cluster type and metadata synchronization frequency.

3. The method according to claim 2, characterized in that The determining, based on the target data synchronization information, the standard source cluster metadata and the standard target cluster metadata corresponding to the metadata synchronization task includes: Determine a first adapter and a first converter corresponding to the source cluster based on the source cluster type and the preset plug-in system, and determine a second adapter and a second converter corresponding to the target cluster based on the target cluster type and the preset plug-in system; Determine the basic source cluster metadata corresponding to the source cluster based on the source cluster name and the first adapter, and determine the basic target cluster metadata corresponding to the target cluster based on the target cluster name and the second adapter; The basic source cluster metadata is formatted based on the first converter to obtain standard source cluster metadata corresponding to the source cluster, and the basic target cluster metadata is formatted based on the second converter to obtain standard target cluster metadata corresponding to the target cluster.

4. The method according to claim 3, characterized in that Before determining the first adapter and the first converter corresponding to the source cluster based on the source cluster type and the preset plug-in system, and determining the second adapter and the second converter corresponding to the target cluster based on the target cluster type and the preset plug-in system, the method further includes: Obtaining a preset cluster type, and determining a basic adapter and a basic converter corresponding to the preset cluster type based on the preset cluster type and a historical configuration specification; The preset cluster type, the basic adapter and the basic converter are combined and processed to generate a basic plug-in sequence corresponding to the preset cluster type, and the basic plug-in sequences corresponding to each preset cluster type are statistically processed to generate a preset plug-in system.

5. The method according to claim 1, characterized in that The step of comparing the standard source cluster metadata and the standard target cluster metadata based on a preset difference detection algorithm and a preset hierarchical verification rule to determine a target synchronization data list corresponding to the metadata synchronization task includes: Obtain the source cluster table structure in the standard source cluster metadata and the target cluster table structure in the standard target cluster metadata; Compare the source cluster table structure and the target cluster table structure based on a preset difference detection algorithm to generate a table structure synchronization list; Determine the candidate source cluster field in the standard source cluster metadata and the candidate target cluster field in the standard target cluster metadata based on the table structure synchronization list; Compare the data of the candidate source cluster field and the candidate target cluster field based on a preset difference detection algorithm to generate a field synchronization list; The table structure synchronization list and the field synchronization list are processed in combination to generate a target synchronization data list corresponding to the metadata synchronization task.

6. The method according to claim 1, characterized in that After synchronizing the target cluster data based on the target synchronization data list and the preset scheduling synchronization rule to achieve cross-cluster metadata synchronization, the method further includes: Determine, based on the target data synchronization information, the candidate source cluster metadata and the candidate target cluster metadata corresponding to the metadata synchronization task; The metadata of the candidate source cluster and the metadata of the candidate target cluster are compared for consistency to generate a comparison result, and warning information corresponding to the metadata synchronization task is generated based on the comparison result.

7. A device for synchronizing metadata across clusters, characterized in that: include: An information acquisition module is used to acquire target data synchronization information corresponding to the metadata synchronization task; A data acquisition module, used to determine the standard source cluster metadata, the standard target cluster metadata and the preset scheduling synchronization rules corresponding to the metadata synchronization task based on the target data synchronization information; A data comparison module, used to compare the standard source cluster metadata and the standard target cluster metadata based on a preset difference detection algorithm and a preset hierarchical verification rule, and determine a target synchronization data list corresponding to the metadata synchronization task; The data synchronization module is used to synchronize the data of the target cluster based on the target synchronization data list and the preset scheduling synchronization rules to achieve cross-cluster metadata synchronization.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method for cross-cluster metadata synchronization according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for cross-cluster metadata synchronization according to any one of claims 1 to 6 when executed.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for cross-cluster metadata synchronization according to any one of claims 1 to 6.

Images