A data export method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a data export method and device, electronic equipment and storage medium, which receive the snapshot metadata; when the target cluster exports the to-be-sent data through the snapshot export script parameter, and determines that the snapshot metadata is incompatible through the specified configuration file, determine the available snapshot metadata from the snapshot metadata based on the specified configuration file; and export the available snapshot metadata to generate target to-be-sent data corresponding to the available snapshot metadata, thereby realizing snapshot migration for table data in the case of incompatible metadata information, and improving the availability of snapshot migration.

Description

Technical Field

[0001] This invention relates to the field of data export technology, and in particular to a data export method, a data export device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] HBase is a commonly used distributed, column-oriented open-source database in the big data field. When migrating HBase table data, snapshot migration is usually used. Snapshot migration has the advantages of being simple, easy to operate, and having little impact on the production cluster. It is equivalent to completely copying the source cluster table to the target cluster. However, it can also bring some problems to table data migration in certain special scenarios. For example, in cross-version and cross-platform scenarios, due to the differences in HBase version, platform, and self-developed features, the metadata information of the table may be incompatible, resulting in the table data being unusable in the target cluster after being exported from the source cluster.

[0003] Therefore, how to perform snapshot migration of table data is a problem that those skilled in the art need to overcome. Summary of the Invention

[0004] The present invention provides a data export method, apparatus, electronic device, and computer-readable storage medium to solve the problem of how to perform snapshot migration when metadata information is incompatible.

[0005] This invention discloses a data export method applied to a target cluster, wherein the target cluster has a corresponding initial cluster. The initial cluster is used to: determine the data to be sent and snapshot export script parameters for the data to be sent; construct snapshot metadata for the data to be sent based on the snapshot export script parameters; the snapshot metadata includes a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster; and send the snapshot metadata to the target cluster, which may include:

[0006] Receive the snapshot metadata;

[0007] When the target cluster exports the data to be sent through the snapshot export script parameters, and determines that the snapshot metadata is incompatible through the specified configuration file, the available snapshot metadata is determined from the snapshot metadata based on the specified configuration file;

[0008] Export the available snapshot metadata to generate target data to be sent corresponding to the available snapshot metadata.

[0009] Optionally, the snapshot export script parameters may include: table name information and authentication information for the data to be sent, and Internet Protocol address information and export path information for the target cluster.

[0010] Optionally, the snapshot metadata has a corresponding first snapshot table, and the step of determining the available snapshot metadata from the snapshot metadata based on the specified configuration file may include:

[0011] A second snapshot table for the available snapshot metadata is constructed using the specified configuration file and the first snapshot table;

[0012] Available snapshot metadata is determined from the snapshot metadata based on the second snapshot table.

[0013] Optionally, it may also include:

[0014] The directory hierarchy for the snapshot metadata is determined based on the exported path information;

[0015] The available snapshot metadata is stored in the directory hierarchy.

[0016] Optionally, the initial cluster and the target cluster are distributed databases HBase, and the snapshot metadata includes data display files, and may also include:

[0017] When the target cluster exports the data to be sent through the snapshot export script parameters, and the snapshot metadata is determined to be compatible through the specified configuration file, the data display file is read; the data display file includes region information and HFile, which is the underlying data unit file information for the data to be sent.

[0018] The data to be sent is exported based on the region information (region) and the underlying data unit file information (HFile).

[0019] This invention also discloses a data export method applied to an initial cluster, wherein the initial cluster has a corresponding target cluster, and may include:

[0020] Determine the data to be sent, and the parameters for exporting a snapshot of the data to be sent;

[0021] Snapshot metadata for the data to be sent is constructed based on the snapshot export script parameters; the snapshot metadata includes a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster;

[0022] The snapshot metadata is sent to the target cluster, which receives the snapshot metadata. When the target cluster exports the data to be sent through the snapshot export script parameters and determines that the snapshot metadata is incompatible through the specified configuration file, the available snapshot metadata is determined from the snapshot metadata based on the specified configuration file. The available snapshot metadata is exported to generate the target data to be sent corresponding to the available snapshot metadata.

[0023] This invention also discloses a data export device applied to a target cluster, the target cluster having a corresponding initial cluster, the initial cluster being used to: determine data to be sent, and snapshot export script parameters for the data to be sent; construct snapshot metadata for the data to be sent based on the snapshot export script parameters; the snapshot metadata includes a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster; sending the snapshot metadata to the target cluster may include:

[0024] A snapshot metadata receiving module is used to receive the snapshot metadata;

[0025] The available snapshot metadata determination module is used to determine available snapshot metadata from the snapshot metadata based on the specified configuration file when the target cluster exports the data to be sent through the snapshot export script parameters and determines that the snapshot metadata is incompatible through the specified configuration file.

[0026] A snapshot metadata export module is available for exporting the available snapshot metadata to generate target data to be sent corresponding to the available snapshot metadata.

[0027] Optionally, the snapshot export script parameters may include: table name information and authentication information for the data to be sent, and Internet Protocol address information and export path information for the target cluster.

[0028] Optionally, the snapshot metadata has a corresponding first snapshot table, and the available snapshot metadata determination module may include:

[0029] The second snapshot table construction submodule is used to construct a second snapshot table for the available snapshot metadata using the specified configuration file and the first snapshot table.

[0030] The available snapshot metadata determination submodule is used to determine available snapshot metadata from the snapshot metadata based on the second snapshot table.

[0031] Optionally, it may also include:

[0032] The directory hierarchy determination module is used to determine the directory hierarchy for the snapshot metadata based on the exported path information;

[0033] A snapshot metadata storage module is available for storing the available snapshot metadata in the directory hierarchy.

[0034] Optionally, the initial cluster and the target cluster are distributed databases HBase, and the snapshot metadata includes data display files, and may also include:

[0035] The data display file reading module is used to read the data display file when the target cluster exports the data to be sent through the snapshot export script parameters and the snapshot metadata is determined to be compatible through the specified configuration file; the data display file includes region information and HFile, which is the underlying data unit file information for the data to be sent;

[0036] The data export module is used to export the data to be sent based on the region information and the underlying data unit file information HFile.

[0037] This invention also discloses a data export device applied to an initial cluster, the initial cluster having a corresponding target cluster, which may include:

[0038] The data to be sent determination module is used to determine the data to be sent and the snapshot export script parameters for the data to be sent;

[0039] A snapshot metadata construction module is used to construct snapshot metadata for the data to be sent based on the snapshot export script parameters; the snapshot metadata includes a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster.

[0040] A snapshot metadata sending module is used to send the snapshot metadata to the target cluster, and the target cluster is used to receive the snapshot metadata. When the target cluster exports the data to be sent through the snapshot export script parameters, and determines that the snapshot metadata is incompatible through the specified configuration file, it determines the available snapshot metadata from the snapshot metadata based on the specified configuration file; and exports the available snapshot metadata to generate the target data to be sent corresponding to the available snapshot metadata.

[0041] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0042] The memory is used to store computer programs;

[0043] When the processor executes a program stored in the memory, it implements the method described in the embodiments of the present invention.

[0044] This invention also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.

[0045] The embodiments of the present invention have the following advantages:

[0046] In this embodiment of the invention, by receiving the snapshot metadata; when the target cluster exports the data to be sent through the snapshot export script parameters, and determines that the snapshot metadata is incompatible through the specified configuration file, the available snapshot metadata is determined from the snapshot metadata based on the specified configuration file; the available snapshot metadata is exported to generate the target data to be sent corresponding to the available snapshot metadata, thereby realizing snapshot migration of table data in the case of incompatible metadata information, and improving the availability of snapshot migration. Attached Figure Description

[0047] Figure 1 This is a flowchart of the steps of a data export method provided in Embodiment 1 of the present invention;

[0048] Figure 2 This is a schematic diagram of a special scenario HBase cross-cluster migration scheme provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of a snapshot-compatible parsing module provided in an embodiment of the present invention;

[0050] Figure 4 This is a flowchart of the steps of a data export method provided in Embodiment 2 of the present invention;

[0051] Figure 5 This is a structural block diagram of a data export device provided in Embodiment 3 of the present invention;

[0052] Figure 6 This is a structural block diagram of a data export device provided in Embodiment 4 of the present invention;

[0053] Figure 7 This is a hardware structure block diagram of an electronic device provided in various embodiments of the present invention;

[0054] Figure 8 This is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] HBase is a distributed, column-oriented open-source database. Unlike typical relational databases, HBase is suitable for storing unstructured data. Furthermore, HBase uses a column-based rather than row-based schema. HBase contains database tables, and data is stored in tables as the organizational unit. Table data is defined by the data type specified by the table fields. In HBase, snapshot migration is commonly used for table data migration, and it's the most frequently used method. Snapshot migration includes both the table's metadata and the actual data files. When using snapshot migration for cross-cluster data migration in HBase, the new and old cluster tables may fail to use snapshot migration due to storage strategies and / or schema incompatibility, or the metadata information may be incompatible after snapshot migration, making the data unusable in the target cluster after being exported from the source cluster. This invention provides a data export method that combines snapshot export script parameters and a specified configuration file to determine usable snapshot metadata from the snapshot metadata, thereby improving the availability of table data snapshot migration when metadata information is incompatible.

[0057] Example 1

[0058] Reference Figure 1 The diagram illustrates a flowchart of a data export method provided in Embodiment 1 of the present invention, which may specifically include the following steps:

[0059] Step 101: Receive the snapshot metadata;

[0060] Step 102: When the target cluster exports the data to be sent through the snapshot export script parameters, and determines that the snapshot metadata is incompatible through the specified configuration file, the available snapshot metadata is determined from the snapshot metadata based on the specified configuration file.

[0061] Step 103: Export the available snapshot metadata to generate target data to be sent corresponding to the available snapshot metadata.

[0062] In practical applications, the embodiments of the present invention can be applied to a target cluster, which may have a corresponding initial cluster. The initial cluster can be used to: determine the data to be sent and the snapshot export script parameters for the data to be sent; construct snapshot metadata for the data to be sent based on the snapshot export script parameters; the snapshot metadata may include a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster; and send the snapshot metadata to the target cluster. For example, the target cluster may be a target HBase cluster, and the initial cluster corresponding to the target HBase cluster may be a source HBase cluster. The source HBase cluster can be used to determine the data to be sent and the export script for the data to be sent. Parameters, for example, the data to be sent can be the original metadata in the source HBase cluster, and the export script parameters can be "table name, authentication information, target cluster Internet protocol address, target export path". When the export script parameters are "table name, authentication information, target cluster Internet protocol address, target export path", a snapshot can be created based on "table name, authentication information, target cluster Internet protocol address, target export path" and the data to be sent. The snapshot is used as snapshot metadata. The snapshot metadata can include a specified configuration file for expressing the exclusivity between the target HBase cluster and the source HBase cluster, such as a configuration file composed of information such as "vendor, version, self-developed features".

[0063] In a specific implementation, the target cluster of this invention can be used to receive snapshot metadata. When the target cluster exports data to be sent through the snapshot export script parameters and determines that the snapshot metadata is incompatible through the specified configuration file, it can determine the available snapshot metadata from the snapshot metadata based on the specified configuration file. The available snapshot metadata is then exported to generate the target data to be sent corresponding to the available snapshot metadata. For example, when the target cluster is a target HBase cluster, the snapshot metadata is snapshot metadata constructed based on the export script parameters "table name, authentication information, target cluster Internet Protocol address, target export path", and the specified configuration file is "vendor, version, self-developed features", the target HBase cluster can receive the snapshot metadata. When the target HBase cluster exports data to be sent through the export script parameters "table name, authentication information, target cluster Internet Protocol address, target export path", and determines that the snapshot metadata is incompatible through the specified configuration file "vendor, version, self-developed features", it can select metadata compatible with the target HBase cluster from the snapshot metadata based on the configuration file "vendor, version, self-developed features" as available snapshot metadata. Then, the available snapshot metadata can be exported to generate the target data to be sent corresponding to the available snapshot metadata.

[0064] Of course, the above is only an example. Those skilled in the art can use parameters including but not limited to manufacturer, version, and self-developed features as the basis for determining the compatibility of snapshot metadata. In this regard, the embodiments of the present invention do not impose any restrictions.

[0065] In this embodiment of the invention, by receiving the snapshot metadata; when the target cluster exports the data to be sent through the snapshot export script parameters, and determines that the snapshot metadata is incompatible through the specified configuration file, the available snapshot metadata is determined from the snapshot metadata based on the specified configuration file; the available snapshot metadata is exported to generate the target data to be sent corresponding to the available snapshot metadata, thereby realizing snapshot migration of table data in the case of incompatible metadata information, and improving the availability of snapshot migration.

[0066] Based on the above embodiments, modified embodiments of the above embodiments are proposed. It should be noted that, in order to keep the description brief, only the differences from the above embodiments are described in the modified embodiments.

[0067] In an optional embodiment of the present invention, the snapshot export script parameters include: table name information and authentication information for the data to be sent, and Internet Protocol address information and export path information for the target cluster.

[0068] In practical applications, authentication information can ensure the confidentiality, reliability, and accuracy of information, and achieve the purpose of preventing eavesdropping and tampering.

[0069] In a specific implementation, the snapshot export script parameters in this embodiment of the invention may include table name information and authentication information for the data to be sent, as well as Internet Protocol address information and export path information for the target cluster. For example, when the initial cluster is the source HBase cluster, the target cluster is the target HBase cluster, and the data to be sent is the original metadata in the source HBase cluster of the initial cluster, the snapshot export script parameters may be table name information and authentication information for the original metadata, as well as IP address and target export path for the target HBase cluster. The IP address can be used as Internet Protocol address information, and the target export path can be used as export path information.

[0070] This invention improves the security and accuracy of table data migration by including the table name and authentication information for the data to be sent, as well as the Internet Protocol address information and export path information for the target cluster, in the snapshot export script parameters. At the same time, the export path information also creates the conditions for saving the available snapshot metadata at the directory level in the subsequent table data migration process.

[0071] In an optional embodiment of the present invention, the snapshot metadata has a corresponding first snapshot table, and the step of determining available snapshot metadata from the snapshot metadata based on the specified configuration file includes:

[0072] A second snapshot table for the available snapshot metadata is constructed using the specified configuration file and the first snapshot table;

[0073] Available snapshot metadata is determined from the snapshot metadata based on the second snapshot table.

[0074] In a specific implementation, snapshot metadata can have a corresponding first snapshot table. The snapshot table corresponding to the snapshot metadata can be the first snapshot table. In this embodiment of the invention, a second snapshot table for available snapshot metadata can be constructed using a specified configuration file and the first snapshot table. Based on the second snapshot table, available snapshot metadata is determined from the snapshot metadata. For example, when the first snapshot table is a "snapshot table" and the specified configuration file is "vendor, version, self-developed features," the second snapshot table for available snapshot metadata can be constructed using "vendor, version, self-developed features" and the first snapshot table "snapshot table." Specifically, when the initial cluster is the source HBase cluster and the target cluster is the target HBase cluster, the configuration file "vendor, version, self-developed features" is read. After analyzing and comparing the exclusive differences in vendor, version, and self-developed features between the source HBase cluster and the target HBase cluster, configurations with exclusive differences can be removed from the first snapshot table "snapshot table" and converted to configurations with the same effect as the target cluster, generating "snapshot table_new" as the second snapshot table. Then, based on the second snapshot table "snapshot table_new," available snapshot metadata can be determined from the snapshot metadata.

[0075] In this embodiment of the invention, a second snapshot table is constructed using the specified configuration file and the first snapshot table for the available snapshot metadata. Based on the second snapshot table, available snapshot metadata is determined from the snapshot metadata. By eliminating exclusive differential configurations and converting them into configurations with the same effect as the target cluster, the parameter configuration of the source table is preserved to the greatest extent.

[0076] In an optional embodiment of the present invention, it further includes:

[0077] The directory hierarchy for the snapshot metadata is determined based on the exported path information;

[0078] The available snapshot metadata is stored in the directory hierarchy.

[0079] In specific implementations, embodiments of the present invention can determine the directory hierarchy for snapshot metadata based on export path information; and store available snapshot metadata in the directory hierarchy. For example, when the initial cluster is the source HBase cluster, the target cluster is the target HBase cluster, the export path information is the target export path, the first snapshot table is the "snapshot table", and the configuration file is "vendor, version, self-developed features", the configuration file "vendor, version, self-developed features" can be read. After analyzing and comparing the exclusive differences in vendor, version, and self-developed features between the source HBase cluster and the target HBase cluster, configurations with exclusive differences can be removed from the first snapshot table "snapshot table" and converted into configurations with the same effect as the target cluster, generating "snapshot table_new" as the second snapshot table. Then, based on the second snapshot table "snapshot table_new", available snapshot metadata can be determined from the snapshot metadata. The directory hierarchy for snapshot metadata can be determined based on the target export path, and the available snapshot metadata determined from the snapshot metadata based on the second snapshot table "snapshot table_new" can be stored in the directory hierarchy.

[0080] This invention determines the directory hierarchy for the snapshot metadata based on the exported path information; and saves the available snapshot metadata at the directory hierarchy. This avoids serious problems such as snapshots being unrecoverable and / or recovery causing the target cluster to crash.

[0081] In an optional embodiment of the present invention, the initial cluster and the target cluster are distributed databases HBase, and the snapshot metadata includes data display files, and further includes:

[0082] When the target cluster exports the data to be sent through the snapshot export script parameters, and the snapshot metadata is determined to be compatible through the specified configuration file, the data display file is read; the data display file includes region information and HFile, which is the underlying data unit file information for the data to be sent.

[0083] The data to be sent is exported based on the region information (region) and the underlying data unit file information (HFile).

[0084] In practical applications, because HBase is distributed and has many nodes, a large HBase table cannot be stored on a single node. Therefore, HBase tables are split into rows and stored on various nodes. The split parts are called Regions. A Region is the smallest unit of distributed storage and load balancing in HBase. Different Regions can reside on different Region Servers. HFile is the file organization format for storing data in HBase. In HDFS (Hadoop Distributed File System), it is represented as HFile files. HFile is the storage format for KeyValue data in HBase and is a binary file format.

[0085] In a specific implementation, the initial cluster and target cluster in this embodiment of the invention can be a distributed database HBase. This embodiment can export the data to be sent from the target cluster using snapshot export script parameters, and read the data display file when the snapshot metadata is deemed compatible by a specified configuration file. The data display file includes region information and underlying data unit file information (HFile) for the data to be sent. The data to be sent is exported based on the region information and the underlying data unit file information (HFile). For example, when the target cluster is a target HBase cluster, the data to be sent is the original metadata from the initial cluster's source HBase cluster, the snapshot export script parameters are table name information and authentication information for the original metadata, and the specified configuration file is "vendor, version, self-developed features," when the target HBase cluster exports the data via snapshot... The script parameters "table name information and authentication information for the original metadata" export the data to be sent. When the snapshot metadata is deemed compatible by specifying the configuration file "vendor, version, and self-developed features", the data display file, denoted as "data.manifest", can be read based on the existing snapshot metadata file ".snapshotinfo / data.manifest". The data display file "data.manifest" includes region information, denoted as "region information", and HFile information, which is the underlying data unit file information for the data to be sent, denoted as "HFile file information". Based on "region information" and "HFile file information", the exported HFile data files are imported in batches into the regions of the specified table to achieve the export of the data to be sent.

[0086] In this embodiment of the invention, when the target cluster exports the data to be sent through the snapshot export script parameters, and the snapshot metadata is determined to be compatible through the specified configuration file, the data display file is read. The data display file includes region information and HFile, which are the underlying data unit file information for the data to be sent. The data to be sent is exported based on the region information and the underlying data unit file information HFile, thereby realizing the batch import of the exported data files into the table, completing the cross-cluster migration of the table, and improving the efficiency of snapshot migration.

[0087] To enable those skilled in the art to better understand the embodiments of the present invention, a complete example is provided below to illustrate the embodiments of the present invention.

[0088] Reference Figure 2 , Figure 2 This diagram illustrates a cross-cluster migration scheme for HBase in a specific scenario, as provided in Embodiment 1 of the present invention.

[0089] Snapshot migration, a common solution for cross-cluster migration of HBase data, is the preferred method due to its simplicity, ease of operation, and minimal impact on production clusters. This method essentially copies the tables from the source cluster to the target cluster. However, this can present challenges in certain scenarios, such as cross-version or cross-platform migrations. In these cases, incompatibility in table metadata due to HBase versions or platform-specific features can lead to data unavailability on the target cluster after exporting from the source cluster.

[0090] The snapshot export module is used to perform:

[0091] S1. Construct a script applicable to both the source and target HBase clusters. This script executes hbase shell commands and snapshot export commands to create and export snapshots of specified tables in the source HBase cluster. During snapshot export, parameters can be flexibly configured to control migration speed and concurrency.

[0092] Specifically, it may include:

[0093] After configuring the necessary table names, authentication information, target cluster IP address, and target export path in the S11.shell script, execute the script.

[0094] S12. First, the script will call the HBase shell in the background to perform the snapshot creation operation.

[0095] S13. Next, based on the configured target cluster IP and target export path, export the table data to the specified directory of the target cluster.

[0096] Reference Figure 3 , Figure 3 A schematic diagram of a snapshot-compatible parsing module provided in Embodiment 1 of the present invention is shown:

[0097] The snapshot-compatible parsing module is used to perform:

[0098] S2. This module parses the exported snapshot's metadata file, analyzes and compares the exclusive differences between the source and target clusters, such as versions and self-developed features, and ultimately determines compatibility. If incompatible, it automatically exports metadata usable by the target cluster and places it in the same directory. This avoids serious problems such as snapshots being unrecoverable or causing the target cluster to crash during recovery.

[0099] Specifically, it may include:

[0100] S21. In the specified configuration file, configure the vendor / platform name of the source HBase cluster, such as platform=A; version=1.3.1; self-developed features such as secondary index tableIndexEnabled=true, etc.

[0101] S22. Execute the snapshot parsing toolkit, read the configuration file, analyze and compare the exclusive differences between the source cluster and the target cluster, such as version and self-developed features, and output the analysis results: whether the snapshot metadata is available (true / false). If it is not available, automatically export the metadata available in the target cluster and place it in the same directory. This avoids serious problems such as snapshots being unrecoverable or causing the target cluster to crash during recovery. During the parsing process, only exclusive configurations are processed and converted into configurations with the same effect as the target cluster, preserving the parameter configurations of the source table to the greatest extent possible.

[0102] The snapshot recovery module is used to perform:

[0103] S3. Modify the HBase snapshot recovery logic. Before restoring a snapshot, first check the snapshot metadata for compatibility. If compatible, proceed with the normal process; if incompatible, read the metadata file exported by S2 and complete the table recovery operation on the target cluster. Then, batch import the exported data files into the table to complete the cross-cluster migration of the table.

[0104] Specifically, it may include:

[0105] S31. First, read the compatibility analysis results and determine whether the original snapshot metadata is available. If it is available, then execute S33.

[0106] S32. If unavailable, load the processed snapshot table metadata file;

[0107] S33. Based on the existing snapshot metadata file .snapshotinfo / data.manifest, first restore the table in the target cluster. Then, according to the region information / HFile file information recorded in the data.manifest file, batch import the exported HFile data files into the specified table's region to complete the cross-cluster migration of the table.

[0108] The above method prevents snapshot migration failures due to vendor / platform version / self-developed features when migrating HBase across clusters. The snapshot compatibility parsing function parses the metadata files in the exported snapshot, removes exclusive parameter configurations, and converts them to configurations equivalent to those in the target cluster, preserving the source table's parameter configurations to the greatest extent possible. Finally, the data files are imported into the tables in batches.

[0109] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0110] Example 2

[0111] Reference Figure 4 The diagram illustrates a data export method according to Embodiment 2 of the present invention, which may specifically include the following steps:

[0112] Step 401: Determine the data to be sent and the snapshot export script parameters for the data to be sent;

[0113] Step 402: Construct snapshot metadata for the data to be sent based on the snapshot export script parameters;

[0114] Step 403: Send the snapshot metadata to the target cluster.

[0115] In a specific implementation, this invention can be applied to an initial cluster, which can be used to determine the data to be sent and the snapshot export script parameters for the data to be sent; snapshot metadata for the data to be sent is constructed based on the snapshot export script parameters; the snapshot metadata includes a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster; the snapshot metadata is sent to the target cluster, and in this embodiment, the snapshot metadata can include a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster, and the target cluster can be used to receive the snapshot metadata; when the target cluster exports the data to be sent through the snapshot export script parameters and determines that the snapshot metadata is incompatible through the specified configuration file, the available snapshot metadata is determined from the snapshot metadata based on the specified configuration file; the available snapshot metadata is exported to generate the target data to be sent corresponding to the available snapshot metadata.

[0116] As for Embodiment 2, since it is basically similar to Embodiment 1, the description is relatively simple. For relevant details, please refer to the description of the method embodiment.

[0117] Example 3

[0118] Reference Figure 5 The diagram shows a structural block diagram of a data export device provided in Embodiment 3 of the present invention, which may specifically include the following modules:

[0119] Snapshot metadata receiving module 501 is used to receive the snapshot metadata;

[0120] The available snapshot metadata determination module 502 is used to determine available snapshot metadata from the snapshot metadata based on the specified configuration file when the target cluster exports the data to be sent through the snapshot export script parameters and determines that the snapshot metadata is incompatible through the specified configuration file.

[0121] The available snapshot metadata export module 503 is used to export the available snapshot metadata to generate target data to be sent corresponding to the available snapshot metadata.

[0122] Optionally, the snapshot export script parameters may include: table name information and authentication information for the data to be sent, and Internet Protocol address information and export path information for the target cluster.

[0123] Optionally, the snapshot metadata has a corresponding first snapshot table, and the available snapshot metadata determination module may include:

[0124] The second snapshot table construction submodule is used to construct a second snapshot table for the available snapshot metadata using the specified configuration file and the first snapshot table.

[0125] The available snapshot metadata determination submodule is used to determine available snapshot metadata from the snapshot metadata based on the second snapshot table.

[0126] Optionally, it may also include:

[0127] The directory hierarchy determination module is used to determine the directory hierarchy for the snapshot metadata based on the exported path information;

[0128] A snapshot metadata storage module is available for storing the available snapshot metadata in the directory hierarchy.

[0129] Optionally, the initial cluster and the target cluster are distributed databases HBase, and the snapshot metadata includes data display files, and may also include:

[0130] The data display file reading module is used to read the data display file when the target cluster exports the data to be sent through the snapshot export script parameters and the snapshot metadata is determined to be compatible through the specified configuration file; the data display file includes region information and HFile, which is the underlying data unit file information for the data to be sent;

[0131] The data export module is used to export the data to be sent based on the region information and the underlying data unit file information HFile.

[0132] Example 4

[0133] Reference Figure 6 The diagram shows a structural block diagram of a data export device provided in Embodiment 4 of the present invention, which may specifically include the following modules:

[0134] The data to be sent determination module 601 is used to determine the data to be sent and the snapshot export script parameters for the data to be sent;

[0135] The snapshot metadata construction module 602 is used to construct snapshot metadata for the data to be sent based on the snapshot export script parameters; the snapshot metadata includes a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster.

[0136] The snapshot metadata sending module 603 is used to send the snapshot metadata to the target cluster, and the target cluster is used to receive the snapshot metadata. When the target cluster exports the data to be sent through the snapshot export script parameters and determines that the snapshot metadata is incompatible through the specified configuration file, the available snapshot metadata is determined from the snapshot metadata based on the specified configuration file. The available snapshot metadata is exported to generate the target data to be sent corresponding to the available snapshot metadata.

[0137] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0138] In addition, this invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described data export method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0139] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described data export method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0140] Figure 7 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0141] The electronic device 700 includes, but is not limited to, components such as: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, a processor 710, and a power supply 711. Those skilled in the art will understand that... Figure 7 The electronic device structures shown are not intended to limit the electronic device. An electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. In embodiments of the present invention, the electronic device includes, but is not limited to, mobile phones, tablet computers, laptops, PDAs, in-vehicle terminals, wearable devices, and pedometers.

[0142] It should be understood that, in this embodiment of the invention, the radio frequency unit 701 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink data from the base station and processes it with the processor 710; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 701 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, etc. Furthermore, the radio frequency unit 701 can also communicate with networks and other devices through a wireless communication system.

[0143] Electronic devices provide users with wireless broadband internet access through network module 702, such as helping users send and receive emails, browse web pages, and access streaming media.

[0144] The audio output unit 703 can convert audio data received by the radio frequency unit 701 or the network module 702 or stored in the memory 709 into audio signals and output them as sound. Furthermore, the audio output unit 703 can also provide audio output related to specific functions performed by the electronic device 700 (e.g., call signal reception sound, message reception sound, etc.). The audio output unit 703 includes a speaker, a buzzer, and a receiver, etc.

[0145] Input unit 704 is used to receive audio or video signals. Input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on display unit 706. The image frames processed by GPU 7041 can be stored in memory 709 (or other storage medium) or transmitted via radio frequency unit 701 or network module 702. Microphone 7042 can receive sound and process such sound into audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via radio frequency unit 701 in telephone call mode.

[0146] The electronic device 700 also includes at least one sensor 705, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 7061 according to the ambient light level, and the proximity sensor can turn off the display panel 7061 and / or backlight when the electronic device 700 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used to identify the posture of the electronic device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. The sensor 705 may also include a fingerprint sensor, pressure sensor, iris sensor, molecular sensor, gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc., which will not be described in detail here.

[0147] The display unit 706 is used to display information input by the user or information provided to the user. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0148] User input unit 707 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of electronic devices. Specifically, user input unit 707 includes a touch panel 7071 and other input devices 7072. Touch panel 7071, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 7071). Touch panel 7071 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 710, which receives and executes commands from the processor 710. In addition, touch panel 7071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. Besides touch panel 7071, user input unit 707 may also include other input devices 7072. Specifically, other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0149] Furthermore, the touch panel 7071 can cover the display panel 7061. When the touch panel 7071 detects a touch operation on or near it, it transmits the information to the processor 710 to determine the type of touch event. Subsequently, the processor 710 provides corresponding visual output on the display panel 7061 based on the type of touch event. Although in Figure 7 In this embodiment, the touch panel 7071 and the display panel 7061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 7071 and the display panel 7061 can be integrated to realize the input and output functions of the electronic device. The specific implementation is not limited here.

[0150] Interface unit 708 serves as an interface for connecting external devices to electronic device 700. For example, external devices may include a wired or wireless headphone port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 708 can be used to receive input from external devices (e.g., data, power, etc.) and transmit the received input to one or more components within electronic device 700, or it can be used to transmit data between electronic device 700 and external devices.

[0151] The memory 709 can be used to store software programs and various data. The memory 709 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 709 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0152] The processor 710 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 709, and by calling data stored in the memory 709, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. The processor 710 may include one or more processing units; preferably, the processor 710 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 710.

[0153] The electronic device 700 may also include a power supply 711 (such as a battery) for supplying power to various components. Preferably, the power supply 711 is logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0154] In addition, the electronic device 700 includes some functional modules not shown, which will not be described in detail here.

[0155] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0157] like Figure 8 As shown, in another embodiment of the present invention, a computer-readable storage medium 801 is also provided, which stores instructions that, when executed on a computer, cause the computer to perform a data export method as described in the above embodiment.

[0158] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

[0159] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0161] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0164] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0165] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data export method, characterized in that, Applied to a target cluster, the target cluster has a corresponding initial cluster, the initial cluster is used to determine the data to be sent, and to export script parameters for a snapshot of the data to be sent; Snapshot metadata for the data to be sent is constructed based on the snapshot export script parameters. The snapshot metadata includes a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster; Sending the snapshot metadata to the target cluster includes: Receive the snapshot metadata; When the target cluster exports the data to be sent through the snapshot export script parameters, and determines that the snapshot metadata is incompatible through the specified configuration file, the available snapshot metadata is determined from the snapshot metadata based on the specified configuration file; the specified configuration file is a configuration file consisting of at least vendor, version, and self-developed features; the available snapshot metadata is metadata compatible with the target cluster; Export the available snapshot metadata to generate target data to be sent corresponding to the available snapshot metadata; The snapshot export script parameters include: table name information and authentication information for the data to be sent, and Internet Protocol address information and export path information for the target cluster; The snapshot metadata has a corresponding first snapshot table, and the step of determining the available snapshot metadata from the snapshot metadata based on the specified configuration file includes: A second snapshot table for the available snapshot metadata is constructed using the specified configuration file and the first snapshot table; Available snapshot metadata is determined from the snapshot metadata based on the second snapshot table; The initial cluster and the target cluster are both distributed databases, HBase. The snapshot metadata includes data display files and also includes: When the target cluster exports the data to be sent through the snapshot export script parameters, and the snapshot metadata is determined to be compatible through the specified configuration file, the data display file is read; the data display file includes region information and HFile, which is the underlying data unit file information for the data to be sent. The data to be sent is exported based on the region information (region) and the underlying data unit file information (HFile).

2. The method according to claim 1, characterized in that, Also includes: The directory hierarchy for the snapshot metadata is determined based on the exported path information; The available snapshot metadata is stored in the directory hierarchy.

3. A data export method, characterized in that, Applied to an initial cluster, the initial cluster having a corresponding target cluster, including: The data to be sent is determined, as well as the snapshot export script parameters for the data to be sent; the snapshot export script parameters include: table name information and authentication information for the data to be sent, and Internet Protocol address information and export path information for the target cluster; Based on the snapshot export script parameters, snapshot metadata for the data to be sent is constructed; the snapshot metadata includes a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster; the specified configuration file is a configuration file consisting of at least vendor, version, and self-developed features; The snapshot metadata is sent to the target cluster, which receives the snapshot metadata. When the target cluster exports the data to be sent using the snapshot export script parameters and determines that the snapshot metadata is incompatible using the specified configuration file, available snapshot metadata is determined from the snapshot metadata based on the specified configuration file. The available snapshot metadata is exported to generate target data to be sent corresponding to the available snapshot metadata. The available snapshot metadata is metadata compatible with the target cluster. The snapshot metadata has a corresponding first snapshot table. The step of determining available snapshot metadata from the snapshot metadata based on the specified configuration file includes: constructing a second snapshot table for the available snapshot metadata using the specified configuration file and the first snapshot table; and determining available snapshot metadata from the snapshot metadata based on the second snapshot table. The initial cluster and the target cluster are distributed databases HBase. The snapshot metadata includes a data display file and further includes: when the target cluster exports the data to be sent through the snapshot export script parameters and determines that the snapshot metadata is compatible through the specified configuration file, reading the data display file; the data display file includes region information and HFile, which are the underlying data unit file information for the data to be sent; the data to be sent is exported based on the region information and the underlying data unit file information HFile.

4. A data export device, characterized in that, Applied to a target cluster, the target cluster has a corresponding initial cluster, the initial cluster is used to determine the data to be sent, and to export script parameters for a snapshot of the data to be sent; Snapshot metadata for the data to be sent is constructed based on the snapshot export script parameters. The snapshot metadata includes a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster; Sending the snapshot metadata to the target cluster includes: A snapshot metadata receiving module is used to receive the snapshot metadata; The available snapshot metadata determination module is used to determine available snapshot metadata from the snapshot metadata based on the specified configuration file when the target cluster exports the data to be sent through the snapshot export script parameters and determines that the snapshot metadata is incompatible through the specified configuration file; the specified configuration file is a configuration file consisting of at least vendor, version, and self-developed features; the available snapshot metadata is metadata compatible with the target cluster; A snapshot metadata export module is available for exporting the available snapshot metadata to generate target data to be sent corresponding to the available snapshot metadata; The snapshot export script parameters include: table name information and authentication information for the data to be sent, and Internet Protocol address information and export path information for the target cluster; The snapshot metadata has a corresponding first snapshot table, and the available snapshot metadata determination module includes: The second snapshot table construction submodule is used to construct a second snapshot table for the available snapshot metadata using the specified configuration file and the first snapshot table. The available snapshot metadata determination submodule is used to determine available snapshot metadata from the snapshot metadata based on the second snapshot table; The initial cluster and the target cluster are both distributed databases, HBase. The snapshot metadata includes data display files and also includes: The data display file reading module is used to read the data display file when the target cluster exports the data to be sent through the snapshot export script parameters and the snapshot metadata is determined to be compatible through the specified configuration file; the data display file includes region information and HFile, which is the underlying data unit file information for the data to be sent; The data export module is used to export the data to be sent based on the region information and the underlying data unit file information HFile.

5. A data export device, characterized in that, Applied to an initial cluster, the initial cluster having a corresponding target cluster, including: The data to be sent determination module is used to determine the data to be sent and the snapshot export script parameters for the data to be sent; the snapshot export script parameters include: table name information and authentication information for the data to be sent, and Internet Protocol address information and export path information for the target cluster; A snapshot metadata construction module is used to construct snapshot metadata for the data to be sent based on the snapshot export script parameters; the snapshot metadata includes a specified configuration file for expressing the exclusivity between the target cluster and the initial cluster; the specified configuration file is a configuration file consisting of at least vendor, version, and self-developed features; A snapshot metadata sending module is used to send the snapshot metadata to the target cluster, and the target cluster is used to receive the snapshot metadata. When the target cluster exports the data to be sent through the snapshot export script parameters, and determines that the snapshot metadata is incompatible through the specified configuration file, it determines available snapshot metadata from the snapshot metadata based on the specified configuration file; exports the available snapshot metadata to generate target data to be sent corresponding to the available snapshot metadata; the available snapshot metadata is metadata compatible with the target cluster; wherein, the snapshot metadata has a corresponding first snapshot table, and the step of determining the available snapshot metadata from the snapshot metadata based on the specified configuration file includes: constructing a second snapshot table for the available snapshot metadata using the specified configuration file and the first snapshot table; and determining the available snapshot metadata from the snapshot metadata based on the second snapshot table. The initial cluster and the target cluster are distributed databases HBase. The snapshot metadata includes a data display file and further includes: when the target cluster exports the data to be sent through the snapshot export script parameters and determines that the snapshot metadata is compatible through the specified configuration file, reading the data display file; the data display file includes region information and HFile, which are the underlying data unit file information for the data to be sent; the data to be sent is exported based on the region information and the underlying data unit file information HFile.

6. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-2 or 3.

7. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-2 or 3.

Images