Data backup and recovery method, device, and electronic device

Abstract

The application discloses a data backup recovery method, tool, device, electronic equipment and computer readable storage medium, the method comprises the following steps: obtaining the object parameter of the specified object; obtaining the metadata corresponding to the specified object from the first shard cluster according to the object parameter, and extracting the configuration information of the specified object; extracting the data file and the index file of the specified object from the first shard cluster according to the metadata corresponding to the specified object; establishing the mapping relationship between the first data shard in the first shard cluster and the first number of data shards in the second shard cluster according to the configuration information, and writing the configuration information into the second shard cluster; based on the mapping relationship, write the data file and the index file of the specified object into the second shard cluster, and write the metadata into the second shard cluster. The method solves the technical problems of large backup recovery granularity and low efficiency in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a data backup and recovery method, tool, device, electronic device, and computer readable storage medium. BACKGROUND

[0002] The document type distributed database is widely applied in complex data storage business, such as storing online game data, due to its high data storage flexibility, easy expansion, automatic load balancing and other characteristics. Among them, the sharding cluster is a distributed architecture to ensure the high storage performance and scalability of the document type distributed database.

[0003] Backup and recovery of data is a powerful guarantee for business data security. At present, the data backup and recovery under the sharding cluster mainly includes two schemes: physical backup and recovery scheme and logical backup and recovery scheme. The physical backup and recovery scheme is mainly based on file system snapshot implementation, which has the advantage of high backup and recovery efficiency, but it can only perform backup and recovery at the whole cluster level, and cannot realize backup and recovery at the database / table level in the cluster, so that when backup and recovery is needed, not only a large capacity of snapshot space is needed, but also the problem of synchronously recovering the database / table that does not need to be recovered exists. The logical backup and recovery scheme is mainly based on the command line tool provided by the document type distributed database, which has the advantages of small backup and recovery granularity and can backup and recover at the database / table level without the need to backup and recover the whole cluster, but because the backup and recovery process is long and the configuration information and index file of the sharding cluster cannot be backed up and recovered, there are problems of low backup and recovery efficiency and the need for manual participation in creation.

[0004] Therefore, there is an urgent need for a method that can realize physical backup and recovery at the database / table level to solve the technical problems of large backup and recovery granularity and low efficiency existing in the prior art. SUMMARY

[0005] The present application provides a data backup and recovery method, tool, device, electronic device, and computer readable storage medium to at least partially solve the technical problems of large backup and recovery granularity and low efficiency existing in the prior art.

[0006] In a first aspect, the embodiments of the present application provide a data backup and recovery method, the method comprising: in response to a backup and recovery instruction, obtaining object parameters of a specified object, the object parameters being used to indicate a first shard cluster to which the specified object belongs and a namespace of the specified object, the specified object including a database or a data table in the first shard cluster; obtaining metadata corresponding to the specified object from a metadata list of the first shard cluster according to the object parameters, and extracting configuration information of the specified object from a configuration shard of the first shard cluster, the metadata at least recording a correspondence between the specified object and data files and index files of the specified object; extracting the data files and the index files of the specified object from a first data shard of the first shard cluster according to the metadata corresponding to the specified object, the first data shard being at least one data shard of a plurality of data shards included in the first shard cluster and storing the data files and the index files of the specified object; establishing a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in a second shard cluster according to the configuration information, and writing the configuration information of the specified object into a configuration shard of the second shard cluster, the second shard cluster being specified by the backup and recovery instruction, the first number being a number of the first data shard; and based on the mapping relationship, writing the data files and the index files of the specified object into corresponding data shards of the second shard cluster, and writing the metadata corresponding to the specified object into a metadata list of the second shard cluster.

[0007] In a second aspect, the embodiments of the present application provide a data backup and recovery tool, the tool comprising a data backup module, a configuration backup module, a data recovery module, and a configuration recovery module. The data backup module is configured to obtain metadata corresponding to a specified object from a metadata list of a first sharded cluster according to object parameters of the specified object, and extract data files and index files of the specified object from a first data shard of the first sharded cluster according to the metadata corresponding to the specified object. The specified object comprises a database or a data table in the first sharded cluster. The object parameters are obtained in response to a backup and recovery instruction, and are used to indicate the first sharded cluster to which the specified object belongs and a namespace of the specified object. The metadata at least records a correspondence between the specified object and the data files and index files of the specified object. The first data shard is at least one data shard of a plurality of data shards included in the first sharded cluster, and stores the data files and index files of the specified object. The configuration backup module is configured to extract configuration information of the specified object from a configuration shard of the first sharded cluster according to the object parameters, and establish a mapping relationship between the first data shard of the first sharded cluster and a first number of data shards of a second sharded cluster according to the configuration information. The second sharded cluster is specified by the backup and recovery instruction, and the first number is the number of the first data shard. The data recovery module is configured to write the data files and index files of the specified object into corresponding data shards of the second sharded cluster based on the mapping relationship, and write the metadata corresponding to the specified object into a metadata list of the second sharded cluster. The configuration recovery module is configured to write the configuration information of the specified object into a configuration shard of the second sharded cluster.

[0008] In a third aspect, an embodiment of the present application provides a data backup and recovery apparatus, the apparatus comprising: an obtaining unit, a first extracting unit, a second extracting unit, a first writing unit, and a second writing unit; the obtaining unit is configured to obtain object parameters of a specified object in response to a backup and recovery instruction, the object parameters being used to indicate a first shard cluster to which the specified object belongs and a namespace of the specified object, the specified object comprising a database or a data table in the first shard cluster; the first extracting unit is configured to obtain metadata corresponding to the specified object from a metadata list of the first shard cluster according to the object parameters, and extract configuration information of the specified object from a configuration shard of the first shard cluster, the metadata at least recording a correspondence between the specified object and data files and index files of the specified object; the second extracting unit is configured to extract the data files and the index files of the specified object from a first data shard of the first shard cluster according to the metadata corresponding to the specified object, the first data shard being at least one data shard of a plurality of data shards included in the first shard cluster and storing the data files and the index files of the specified object; the first writing unit is configured to establish a mapping relationship between the first data shard in the first shard cluster and a first quantity of data shards in a second shard cluster according to the configuration information, and write the configuration information of the specified object into a configuration shard of the second shard cluster, the second shard cluster being specified by the backup and recovery instruction, and the first quantity being a quantity of the first data shard; and the second writing unit is configured to write the data files and the index files of the specified object into corresponding data shards of the second shard cluster based on the mapping relationship, and write the metadata corresponding to the specified object into a metadata list of the second shard cluster.

[0009] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor; the memory is configured to store one or more computer instructions; and the processor is configured to execute the one or more computer instructions to implement the method described above.

[0010] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium having one or more computer instructions stored thereon, the instructions being executed by a processor to perform the method described above.

[0011] Compared with the prior art, the data backup recovery method provided by the application comprises: in response to a backup recovery instruction, obtaining object parameters of a specified object, the object parameters being used to indicate a first sharding cluster to which the specified object belongs and a namespace of the specified object, the specified object comprising a database or a data table in the first sharding cluster; obtaining metadata corresponding to the specified object from a metadata list of the first sharding cluster according to the object parameters, and extracting configuration information of the specified object from a configuration shard of the first sharding cluster, the metadata at least recording a corresponding relationship between the specified object and data files and index files of the specified object; extracting the data files and the index files of the specified object from a first data shard of the first sharding cluster according to the metadata corresponding to the specified object, the first data shard being at least one data shard of a plurality of data shards included in the first sharding cluster and storing the data files and the index files of the specified object; establishing a mapping relationship between the first data shard of the first sharding cluster and a first number of data shards of a second sharding cluster according to the configuration information, and writing the configuration information of the specified object into a configuration shard of the second sharding cluster; and based on the mapping relationship, writing the data files and the index files of the specified object into corresponding data shards of the second sharding cluster, and writing the metadata corresponding to the specified object into a metadata list of the second sharding cluster. The method is a scheme that combines the efficient advantages of a physical backup scheme and the fine-grained advantages of a logical backup scheme, and can perform physical backup and recovery at the database / table level on a sharding cluster. First, compared with existing physical backup recovery schemes, the method realizes backup and recovery at the database / table level based on the use of metadata, greatly reducing the backup recovery granularity; second, compared with existing logical backup recovery schemes, the method is physical backup at the database / table level, greatly shortening the backup recovery process and having high backup recovery efficiency; third, the method backs up and recovers the configuration information and the index files at the same time as the data files are backed up and recovered, avoiding the step of manual participation in the creation of configuration information and index files. Therefore, the data backup recovery method provided by the application at least partially solves the technical problems of large backup recovery granularity and low efficiency existing in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is an application system diagram of the data backup recovery method provided by the embodiment of the application;

[0013] Figure 2 is a flowchart of the data backup recovery method provided by the first embodiment of the application;

[0014] Figure 3 is a schematic diagram of the data backup recovery method provided by the second embodiment of the application;

[0015] Figure 4 is a flowchart of the data file physical backup provided by the second embodiment of the application;

[0016] Figure 5 is a flowchart of physical recovery of data file provided by the second embodiment of the present application;

[0017] Figure 6 is a flowchart of physical backup of configuration information provided by the second embodiment of the present application;

[0018] Figure 7 is a flowchart of physical recovery of configuration information provided by the second embodiment of the present application;

[0019] Figure 8 is a flowchart of verifying target cluster provided by the second embodiment of the present application;

[0020] Figure 9 is a schematic diagram of data backup recovery tool provided by the third embodiment of the present application;

[0021] Figure 10 is a structural schematic diagram of data backup recovery device provided by the fourth embodiment of the present application;

[0022] Figure 11 is a structural schematic diagram of electronic device provided by the fifth embodiment of the present application. DETAILED DESCRIPTION

[0023] In the following description, a lot of specific details are set forth in order to facilitate a thorough understanding of the present application. However, the present application can be practiced in many different ways from those described herein, and it is therefore intended that the present application not be limited in scope to the specific implementations described herein. Indeed, the specific implementations described herein have been presented for the purposes of illustration and so that the present application can be thoroughly understood and can have a general appreciation of the inventive concepts.

[0024] With the development of information technology and the Internet, data information presents explosive growth, and document type distributed database is widely used in complex data storage business, such as storing online game data, because of its high data storage flexibility, easy expansion, automatic load balancing and other characteristics. Among them, the sharding cluster is a distributed architecture that guarantees the high storage performance and scalability of the document type distributed database.

[0025] Backup and recovery of data in a document-type distributed database is a powerful guarantee for business data security. An efficient backup and recovery mechanism can build a data security line for business. For example, after a game project is tested publicly, after game maintenance, or in the case of business anomalies, the data and index of specified libraries / tables in the database need to be rolled back to a specific time point of a backup. The efficiency and security of data recovery are the core requirements of data rollback, which requires rolling back the data and index of specified libraries / tables to a specified time point in the shortest time without affecting the online database service as much as possible. In this process, the preferred implementation scheme should include the following requirements: first, only the libraries / tables that need to be restored are rolled back, rather than restoring the entire cluster, to avoid rolling back some data that needs to be preserved. Second, the data backup and recovery process is efficient, and the cluster is usually still running externally during the data backup and recovery process. An efficient backup and recovery process can greatly reduce the impact on the normal operation of the cluster, such as query delay. Third, the data backup and recovery process is safe and verifiable to ensure the correctness of data rollback. However, the existing data backup and recovery methods for sharded clusters cannot simultaneously achieve efficient and safe data backup and recovery at the library / table level.

[0026] Currently, data backup and recovery methods mainly include two schemes: physical backup and recovery scheme and logical backup and recovery scheme.

[0027] The physical backup and recovery scheme is mainly based on file system snapshots or instance snapshots. When backing up, the file system snapshots of each data shard and configuration shard of the entire cluster (source cluster) are saved. When restoring, a target cluster with the same number of data shards and configuration shards as the source cluster is created and started, and each shard of the target cluster is one-to-one corresponding to each shard of the source cluster. The physical backup of each shard of the source cluster is downloaded to each shard of the target cluster to realize the backup and recovery of the entire cluster to the target cluster. Since the physical backup scheme uniformly backs up and restores data files, index files, and metadata, it has the advantage of high backup and recovery efficiency. However, the physical backup and recovery scheme can only perform backup and recovery at the entire cluster level and does not support backup and recovery at the library / table level in the cluster. The backup and recovery granularity is too large, which not only requires a large amount of snapshot space during backup and recovery, but also causes problems such as recovery redundancy (e.g., only 1% of the data in the cluster needs to be restored, but the other 99% of the data is also restored synchronously) or data rollback errors (e.g., some data that needs to be preserved is also rolled back).

[0028] The logical backup and recovery scheme is mainly based on the command line tools provided by the document type distributed database, such as mongodump and mongorestore in the MongoDB document type distributed database, wherein the mongodump is responsible for logical backup of the source cluster, specifically, the relevant information (such as the host name, port, user name, password, etc. of the source cluster) and the library / table to be backed up are specified through the command line parameters, the mongodump extracts data from the specified library / table and converts it into BSON format data, and the exported BSON format data is written to the local file system. The formed files are usually stored in a directory, and each table corresponds to a.bson file. In addition, the mongodump will also generate a.metadata.json file in the local file system, which is used to store the index definition and other metadata of the table, but the mongodump itself does not support backup of the index file of the table. The mongorestore is responsible for restoring the data backed up by the mongodump to the target cluster, specifically, the relevant information (such as the host name, port, user name, password, etc. of the target cluster) and the backup directory are specified through the command line parameters, the mongorestore connects to the target cluster, reads the stored.bson file and.metadata.json file in the backup directory specified by the command line, and then inserts the BSON format data in the read.bson file into the corresponding library / table of the target cluster. Finally, according to the index definition contained in the.metadata.json file, the index file is recreated to realize the backup and recovery at the library / table level. As can be seen from the above, the logical backup and recovery scheme has the advantages of small backup and recovery granularity, can backup and restore at the library / table level, and does not need to backup and restore the entire cluster. However, the overall backup and recovery process is relatively long, the data backup and recovery efficiency is low, and it often affects the normal processing of requests by the database during data backup and recovery. In addition, the logical backup and recovery scheme cannot backup and restore configuration information such as shard key and index file, and manual participation is required for creation and recovery.

[0029] Taking the backup and recovery of online game data as an example, the current game data backup and recovery usually includes the following steps: first, obtaining the full backup file of the entire cluster from the source cluster; second, performing full physical recovery in an offline environment; third, using the logical backup and recovery scheme to restore the specified library / table to the online target cluster; fourth, restoring the normal operation of the online game and the business access of the target cluster.

[0030] In summary, there is an urgent need for a high-efficiency and secure shard cluster backup and recovery method at the library / table level to solve the above technical problems existing in the current backup and recovery scheme in the data backup and recovery of the shard cluster.

[0031] Therefore, the application provides a data backup and recovery method, which is a combination of the high efficiency of a physical backup scheme and the fine granularity of a logical backup scheme, and can perform a library / table level physical backup and recovery scheme on a sharded cluster. First, compared with the existing physical backup and recovery scheme, the method realizes library / table level backup and recovery based on the use of metadata, greatly reducing the backup and recovery granularity. Second, compared with the existing logical backup and recovery scheme, the method is a physical backup at the library / table level, greatly shortening the backup and recovery process and having high backup and recovery efficiency. Third, the method backs up and recovers configuration information and index files while backing up and recovering data files, avoiding the step of manually participating in the creation of configuration information and index files.

[0032] Before specifically introducing the application, first introduce some technical terms related to the application, so as to better understand the technical scheme of the application.

[0033] 1. Document-type distributed database

[0034] The document-type distributed database is different from the traditional relational database. The document-type distributed database does not store data in tables, but stores data in JSON format or BSON format documents. These documents can contain various types of data structures, such as strings, integers, objects, arrays, etc., making the document-type distributed database very suitable for handling complex data types. MongoDB is an open source document-type distributed database, which has the characteristics of flexible storage of document structure, high performance, high availability, easy expansion, etc., and is particularly suitable for application scenarios that need to handle a large amount of unstructured or semi-structured data, such as content management systems, Internet of Things applications, big data processing, etc.

[0035] 2. WiredTiger

[0036] WiredTiger is a high-performance, scalable database storage engine, which is the default storage engine of MongoDB document-type distributed database. WiredTiger aims to fully utilize the performance advantages of modern hardware (such as multi-core CPU, large memory, high-speed disk) to provide efficient data storage and retrieval capabilities.

[0037] 3. Sharded cluster

[0038] A sharded cluster is a distributed architecture that improves the performance and scalability of a system by horizontally splitting data (sharding) and distributing it across multiple nodes. This architecture is particularly suitable for handling very large data sets and high-throughput operations. The core idea of a sharded cluster is to alleviate the pressure on individual nodes by distributing data across multiple nodes, thereby achieving higher performance and better resource utilization. The main components of a sharded cluster include: (1) Shards, which are the actual storage units of data, a sharded cluster usually includes a configuration shard and multiple data shards, the configuration shard is used to store configuration information, and the data shard is used to store data files and index files; (2) Config Servers, which store metadata information about the sharded cluster, such as shard list, shard range, shard key, etc.; (3) Routing process (mongos), which is the front-end interface for clients to interact with the sharded cluster, can route client requests to the correct shard and combine the results from multiple shards to return to the client.

[0039] 4、mongos

[0040] Mongos (routing process) is a stateless node in a sharded cluster that provides direct access to the outside world, mainly responsible for request routing, query distribution, data distribution management, aggregation, load balancing and other operations.

[0041] 5、mongod

[0042] Mongod (daemon process) is a service node inside a sharded cluster that actually executes user requests, stores, replicates and synchronizes data, receives user requests distributed by mongos, and replies to mongos after processing is completed.

[0043] 6、namespace

[0044] Namespace (namespace) is the standard name of a collection or index in a sharded cluster, which is a combination of database name and collection or index name, used to uniquely identify a collection or index, ensuring that different collections or indexes in the same database can be distinguished by unique names and there will be no name conflicts.

[0045] 7、Shard Server

[0046] Shard Server is a data sharding service node in a sharded cluster, responsible for data storage, replication and synchronization. Each Shard Server is responsible for storing a part of the data, dispersing the load by horizontally partitioning the data, improving the performance and scalability of the system. Shard Server is composed of multiple mongod instances, and in a sharded cluster, each Shard Server is usually a replica set containing multiple mongod instances. These mongod instances collectively form a Shard Server, responsible for storing and managing a part of the data.

[0047] 8、Shard Key

[0048] Shard Key is a field or combination of fields used to shard and distribute data across multiple shards. The value of the shard key determines how data chunks are distributed among shards, ensuring even distribution of data, optimizing query performance, and supporting data migration.

[0049] 9、Config Server

[0050] Config Server is a service node in a sharded cluster responsible for managing cluster configuration data, used to store metadata about the sharded cluster, including shard list, shard key range, data chunk location, etc.

[0051] 10、Replica Set

[0052] Replica Set is composed of one or more nodes within a single data shard / configuration shard, which synchronize data through replication protocols to ensure high availability and fault tolerance of data. Each node in the replica set replicates the data of the primary node (Primary), ensuring that there are multiple copies of the data, so that if a node fails, the data is still available.

[0053] 11、db

[0054] db (database) is a single database in a sharded cluster, db is a logical container used to organize and store related data, each database has its own independent storage space and namespace.

[0055] 11、collection

[0056] collection (collection) is a table in a sharded cluster, collection is a logical container used to store a group of documents, each collection has a unique name and belongs to a specific db.

[0057] 12、BSON

[0058] BSON is a binary format similar to JSON, used to represent structured data, which is a binary encoding form of JSON, provides more rich data types and more efficient storage performance than JSON, aims to provide more efficient data storage and transmission.

[0059] The data backup recovery method, tool, device, electronic equipment and computer readable storage medium provided in the application will be further described in detail below in combination with specific embodiments and drawings.

[0060] Figure 1 is the application system diagram of the data backup recovery method provided by the embodiment of the application. As shown in Figure 1 , the application system includes a user terminal 101 and a server 102, and the user terminal 101 can be any device such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant (Personal Digital Assistant, PDA) and the like. The server 102 can be a stand-alone server, deploy the data backup recovery method provided by the application, or be a server group composed of multiple servers, wherein each server deploys one module of the data backup recovery method provided by the application. Of course, the server 102 can also be a cloud server, and the data backup recovery method provided by the application is deployed on the cloud server. The server 102 receives the data backup recovery request sent by the user terminal 101, and realizes efficient and secure data physical backup recovery at the library / table level based on the method.

[0061] The first embodiment of the application provides a data backup recovery method, which can be deployed in the server 102 as shown in Figure 1 .

[0062] Figure 2 is the flowchart of the data backup recovery method provided by the embodiment. The data backup recovery method provided by the embodiment will be described in detail below. Figure 2 The embodiments described below are used to explain the technical solutions of the application and are not limited as actual use.

[0063] As shown in Figure 2 , the data backup recovery method provided by the embodiment includes the following steps S210 to S250:

[0064] Step S210, in response to a backup recovery instruction, obtaining object parameters of a specified object, the object parameters being used to indicate a first sharding cluster to which the specified object belongs and a namespace of the specified object, and the specified object including a database or a data table in the first sharding cluster.

[0065] The backup and recovery instruction can be understood as information sent by the user end to the service end, indicating that the specified object needs to be backed up and recovered. The backup and recovery instruction can be information generated by the user end in response to a user operation, or information automatically generated by the user end based on a business event (such as a business exception). Specifically, the backup and recovery instruction carries an object parameter of a specified object (such as a user-specified backup and recovery object, an object causing a business exception, etc.). The object parameter indicates the namespace of the specified object and the shard cluster to which the specified object belongs. In this embodiment, the shard cluster to which the specified object belongs is defined as the first shard cluster, and the first shard cluster can also be understood as the source cluster.

[0066] In this embodiment, the specified object can be one or more databases or data tables in the first shard cluster, so as to implement backup and recovery at the library / table level in this embodiment.

[0067] In step S220, the metadata corresponding to the specified object is obtained from the metadata list of the first shard cluster according to the object parameter, and the configuration information of the specified object is extracted from the configuration shard of the first shard cluster. The metadata at least records the correspondence between the specified object and the data files and index files corresponding to the specified object.

[0068] The metadata can be understood as data describing data, that is, information about data. Source data provides information about the context, structure, content, quality, and purpose of data, and plays an important role in the fields of data management, data exchange, and data analysis. Metadata mainly includes descriptive metadata (used to describe the content of data, such as creation date, etc.), structural metadata (used to describe the organization and structure of data, such as file format, table and field definition, etc.), and management metadata (used to manage and maintain data, such as access permission, data source, etc.). The shard cluster involves multiple types of metadata, which are collectively referred to as a metadata list in this embodiment.

[0069] Since the namespace of the specified object is indicated in the object parameter, the metadata describing the specified object can be located in the metadata list based on the namespace, and then the metadata of the specified object is extracted from the metadata list of the first shard cluster. Specifically, the extracted metadata can include multiple data types, at least recording the correspondence between the specified object and the data files and index files corresponding to the specified object. In addition, in order to enable the target cluster to correctly identify and load the specified object backed up and recovered from the source cluster, the metadata also needs to include other data types.

[0070] In an optional implementation, the metadata corresponding to the specified object is obtained from the metadata list of the first shard cluster according to the object parameter, which can specifically include steps S221-1 to S221-3:

[0071] Step S221-1, parse the metadata list of the first shard cluster.

[0072] Step S221-2, according to the namespace of the specified object, extract the metadata corresponding to the specified object from the metadata list, the metadata at least includes the first metadata for recording the correspondence between the specified object and the data file and the index file corresponding to the specified object, the second metadata for recording the file status of the data file and the index file of the specified object, the third metadata for recording the creation information of the first shard cluster, the fourth metadata for recording the file information of the first shard cluster, the fifth metadata for recording the checkpoint information of the data file and the index file of the specified object in the first shard cluster.

[0073] Step S221-3, encapsulate the metadata corresponding to the specified object into a metadata document, the metadata document at least includes the first sub-data document corresponding to the first metadata, the second sub-data document corresponding to the second metadata, the third sub-data document corresponding to the third metadata, the fourth metadata, and the fifth metadata, the fourth sub-data document corresponding to the index metadata of the specified object, the index metadata being the metadata related to the index in the metadata corresponding to the specified object.

[0074] In summary, the metadata of the specified object includes five parts of content, wherein the first metadata records the correspondence between the specified object and the data file and the index file corresponding to the specified object. The second metadata records the file status of the data file and the index file, such as the file size of the data file and the index file. The third metadata records the creation information of the first shard cluster, such as the creation time of each collection of the database in the first shard cluster, the name at the time of creation, etc. The fourth metadata records the file information of the first shard cluster, such as the physical location and size of each file in the first shard cluster. The fifth metadata records the checkpoint information of the data file and the index file in the first shard cluster, such as the timestamp and status of the checkpoint.

[0075] In the embodiment, after the above metadata of the specified object is extracted from the metadata list, the above metadata is processed and encapsulated, specifically, the first metadata is processed into the first sub-data document, the second metadata is processed into the second sub-data document, the third, fourth, and fifth metadata are processed into the third sub-data document, the metadata related to the index in the first, second, third, fourth, and fifth metadata is processed into the fourth sub-data document, and then the above sub-data document is encapsulated into the metadata document corresponding to the specified object.

[0076] In addition, in this step, the configuration information of the specified object is extracted from the configuration shard of the first shard cluster based on the namespace of the specified object. In an optional implementation, the following steps S222-1 to S222-3 can be included:

[0077] In step S222-1, the configuration information list stored in the configuration shard of the first shard cluster is parsed, and at least the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table are extracted from the configuration information list.

[0078] In step S222-2, the configuration information corresponding to the specified object is extracted from the extracted shard configuration information table, database configuration information table, data table configuration information table, and data block configuration information table according to the namespace of the specified object.

[0079] In step S222-3, the extracted configuration information corresponding to the specified object is processed and encapsulated into a configuration information document.

[0080] The configuration shard stores various configuration information, which is collectively referred to as a configuration information list in this embodiment. In order to ensure that the target cluster can be accessed correctly after the backup of the specified object is restored to the target cluster, the shard configuration information table (used to store the shard configuration information of the shard cluster), the database configuration information table (used to store the database configuration information of the shard cluster, such as the primary shard, whether the shard is enabled, etc.), the data table configuration information table (used to store the table configuration information of the shard cluster, such as the shard key, etc.), and the data block configuration information table (used to store the data block distribution information of the shard cluster, used for reasoning) need to be extracted from the parsed configuration information list. After the above configuration information tables are extracted, the multiple configuration information related to the specified object can be extracted from the above configuration information tables based on the namespace of the specified object, and the configuration information is processed and encapsulated into a configuration information document.

[0081] In step S230, the data file and the index file of the specified object are extracted from the first data shard of the first shard cluster according to the metadata corresponding to the specified object, and the first data shard is at least one data shard of the multiple data shards included in the first shard cluster and storing the data file and the index file of the specified object.

[0082] The metadata records the correspondence between the specified object and the data file and the index file corresponding thereto, and specifically records the correspondence between the specified object and the storage data shards of the data file and the index file corresponding thereto. Therefore, based on the correspondence recorded by the metadata, the data shards storing the data file and the index file of the specified object can be determined, and then the data file and the index file of the specified object can be extracted therefrom. In this embodiment, the data shard storing the specified correspondence is defined as a first data shard. Specifically, the first data shard can be one data shard, or can be multiple data shards, depending on whether the data file and the index file of the specified object are stored in shards.

[0083] Based on this, in an optional implementation, the data file and the index file of the specified object are extracted from the first data shard of the first shard cluster according to the metadata corresponding to the specified object, and specifically can include the following steps S231 to S233:

[0084] In step S231, the storage path of the data file and the index file of the specified object in the first shard cluster is determined according to the correspondence between the specified object and the data file and the index file of the specified object recorded by the metadata corresponding to the specified object.

[0085] In step S232, the first data shard storing the data file and the index file of the specified object is determined according to the storage path.

[0086] In step S233, the data file and the index file of the specified object are extracted from the first data shard.

[0087] In step S240, a mapping relationship between the first data shard in the first shard cluster and the first number of data shards in the second shard cluster is established according to the configuration information, and the configuration information of the specified object is written into the configuration shard of the second shard cluster, the second shard cluster is specified by the backup recovery instruction, and the first number is the number of the first data shard.

[0088] After the configuration information corresponding to the specified object is extracted from the first shard cluster, a mapping relationship between the first data shard storing the data file and the index file of the specified object in the first shard cluster and the data shard in the target shard cluster can be established based on the configuration information, so as to realize the recovery of the data file and the index file of the specified object extracted from the first data shard of the first shard cluster into the corresponding data shard of the target shard cluster. In this embodiment, the target shard cluster for recovering the specified object is defined as a second shard cluster. Since there can be multiple first data shards storing the data file and the index file of the specified object in the first shard cluster, a mapping relationship between the same number of data shards in the second shard cluster and the first data shards needs to be established.

[0089] In an optional implementation, the mapping relationship between the first data shard in the first shard cluster and the first number of data shards in the second shard cluster is established according to the configuration information, which can specifically include the following steps S241-1 to S241-2:

[0090] In step S241-1, the configuration information list stored in the configuration shard of the second shard cluster is parsed, and the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table are extracted from the configuration information list.

[0091] In step S241-2, the mapping relationship between the first data shard in the first shard cluster and the data shard in the second shard cluster is determined according to the configuration information corresponding to the specified object in the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table of the second shard cluster.

[0092] The configuration information list is also stored in the configuration shard of the second shard cluster. In this embodiment, the configuration information consistent with the configuration information extracted from the configuration information list of the first shard cluster, i.e., the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table, can be extracted from the configuration information list. Thus, the mapping relationship between the first data shard in the first shard cluster and the corresponding data shard in the second shard cluster is established based on the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the first shard cluster and the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the second shard cluster.

[0093] In this embodiment, after the mapping relationship between the first shard cluster and the second shard cluster is established based on the configuration information, the configuration information is written into the configuration shard of the second shard cluster. In an optional implementation, the configuration information of the specified object is written into the configuration shard of the second shard cluster, which can specifically include the following steps S242-1 to S242-2:

[0094] In step S242-1, the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the configuration shard of the second shard cluster are updated according to the configuration information document change.

[0095] Step S242-2, deleting the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table in the configuration information list stored in the configuration shard of the second shard cluster, and inserting the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table changed according to the configuration information document into the configuration information list stored in the configuration shard of the second shard cluster.

[0096] Step S250, based on the mapping relationship, writing the data file and the index file of the specified object into the corresponding data shard of the second shard cluster, and writing the metadata corresponding to the specified object into the metadata list of the second shard cluster.

[0097] After the mapping relationship between the first data shard in the first shard cluster and the corresponding data shard in the second shard cluster is established, the corresponding data file and index file can be written into the second shard cluster to realize the recovery of the data file and the index file in the second shard cluster. Optionally, if the data file and the index file exist in the corresponding data shard of the second shard cluster, the data file and the index file in the corresponding data shard can be cleared first, and then the data file and the index file of the specified object can be written into the cleared corresponding data shard.

[0098] In the embodiment, the metadata of the specified object also needs to be written into the metadata list of the second shard cluster. Optionally, after the data file and the index file are recovered in the second shard cluster, the metadata document is written into the metadata list of the second shard cluster.

[0099] Accordingly, in the backup and recovery process described above, not only the backup and recovery of the data file of the specified object are realized, but also the backup and recovery of the index file, the configuration information, and the metadata of the specified object are realized.

[0100] In an optional implementation manner provided in the embodiment, the method provided in the embodiment not only realizes the physical backup and recovery of the specified object, but also realizes the verification of the backup and recovery process to ensure the security of the backup and recovery process, the integrity of the backup and recovery data, and the accessibility of the second shard cluster.

[0101] In a specific implementation manner, after the step of extracting the data file and the index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object, the method provided in the embodiment further includes: performing file state comparison on the data file and the index file of the specified object extracted from the first data shard of the first shard cluster and the data file and the index file of the specified object stored in the first data shard of the first shard cluster to verify the integrity of the extracted data file and index file of the specified object.

[0102] The purpose of the check is to check the integrity of the physical backup, and the data files and index files of the specified object extracted from the first sharded cluster are compared with the data files and index files of the specified object stored in the first data shard of the first sharded cluster. Specifically, file size and file attributes are compared to check whether the data backed up from the first sharded cluster is complete. If it is complete, the check is successful, and the subsequent steps can be executed. If it is not complete, the check fails, and the current backup task fails.

[0103] In a specific implementation, after the step of writing the data files and index files of the specified object into the corresponding data shard of the second sharded cluster based on the mapping relationship, and writing the metadata corresponding to the specified object into the metadata list of the second sharded cluster, the method provided by the embodiment further includes the following execution content:

[0104] Execution content one: comparing the metadata corresponding to the specified object written into the metadata list of the second sharded cluster with the metadata corresponding to the specified object obtained from the metadata list of the first sharded cluster to check the integrity of the metadata corresponding to the specified object written into the metadata list of the second sharded cluster.

[0105] The purpose of the check is to check the integrity of the metadata before and after the physical recovery. The data state of the metadata corresponding to the specified object written into the metadata list of the second sharded cluster is compared with the data state of the metadata corresponding to the specified object obtained from the metadata list of the first sharded cluster. Specifically, data size is compared to check whether the metadata is complete before and after the recovery. If it is complete, the check is successful, and the subsequent steps can be executed. If it is not complete, the check fails, and the current recovery task fails.

[0106] Execution content two: comparing the data files and index files of the specified object written into the corresponding data shard of the second sharded cluster with the data files and index files of the specified object extracted from the first data shard of the first sharded cluster to check the integrity of the data files and index files of the specified object written into the corresponding data shard of the second sharded cluster.

[0107] The purpose of the check is to check the integrity of the data files and index files before and after the physical recovery. The data files and index files of the specified object written into the corresponding data shard of the second sharded cluster are compared with the data files and index files of the specified object extracted from the first data shard of the first sharded cluster. Specifically, file size and file attributes are compared to check whether the data files and index files are complete before and after the recovery. If it is complete, the check is successful, and the subsequent steps can be executed. If it is not complete, the check fails, and the current recovery task fails.

[0108] The third execution content is to generate recovery statistics, which is used to record the number of configuration information, data files, index files and metadata written into the second shard cluster. For example, how many index files, data files, etc. are recovered in the second shard cluster for a specified object.

[0109] In a specific implementation, the method provided by the embodiment further includes the following verification steps:

[0110] The first verification step is to compare the number of configuration information, data files, index files and metadata of the specified object recovered in the second shard cluster with the number of configuration information, data files, index files and metadata written into the second shard cluster recorded in the recovery statistics, so as to verify the integrity of the specified object recovered in the second shard cluster.

[0111] The purpose of the verification is to verify the integrity of the specified object recovered in the second shard cluster, mainly taking the number of configuration information, data files, index files and metadata written into the second shard cluster recorded in the generated recovery statistics as a reference. If the configuration information, data files, index files and metadata of the specified object recovered in the second shard cluster are consistent with the number recorded in the recovery statistics, the verification is successful, and the subsequent verification step can be executed. If the configuration information, data files, index files and metadata of the specified object recovered in the second shard cluster are inconsistent with the number recorded in the recovery statistics, the verification fails, and the current backup recovery task fails.

[0112] The second verification step is to perform an access test on the specified object recovered in the second shard cluster, so as to verify the access correctness of the specified object recovered in the second shard cluster. For example, an access request for the specified object is set, and it is verified whether the access request is responded and whether the response result is correct. If the access request is responded and the response result is correct, the verification is successful, and the subsequent verification step can be executed. If the access request is not responded or the response result is incorrect, the verification fails, and the current backup recovery task fails.

[0113] The third verification step is to perform an access test on the first object in the second shard cluster, so as to verify the access correctness of the second shard cluster, the first object being any database or data table in the second shard cluster. For example, an access request (such as deleting data, changing data, inserting data, etc.) for a random object (the first object) is set, and it is verified whether the access request is responded and whether the response result is correct. If the access request is responded and the response result is correct, the verification is successful, and the current backup recovery task succeeds. If the access request is not responded or the response result is incorrect, the verification fails, and the current backup recovery task fails.

[0114] The first embodiment provides a data backup and recovery method, which is a combination of the high efficiency of the physical backup scheme and the fine granularity of the logical backup scheme, and can perform physical backup and recovery at the database / table level on a sharded cluster. It should be noted that the examples in the first embodiment are only for the explanation of the method of the present application, and are not limited to the actual use. The data backup and recovery method provided by the present application includes but is not limited to the method described in the first embodiment.

[0115] The second embodiment of the present application provides a data backup and recovery method, which exemplarily illustrates the method of the first embodiment of the present application with specific examples. In this embodiment, the method is applied to a sharded cluster MongoDB, and the first sharded cluster is defined as a source cluster, and the second sharded cluster is defined as a target cluster.

[0116] The data backup and recovery method provided by the present embodiment is based on the WiredTiger storage engine and can support multiple sharded clusters without being limited to a single replica set. It can support the behavior of specifying a data source and a specified database / table, and does not necessarily apply to objects under the MongoDB process startup. It can ensure that the database / table metadata obtained by backup is normally present, without the need to create shard keys (Shard Key), indexes, etc. again. It can ensure the integrity of data backup / recovery.

[0117] Figure 3 FIG. 1 is a schematic diagram of the data backup and recovery method provided by the present embodiment.

[0118] As shown in Figure 3 , the method provided by the present embodiment processes data files and configuration information separately to achieve synchronous backup and recovery of data files and configuration information. Specifically, it can include the following steps:

[0119] In step S311, for data files, the metadata is parsed according to the database / table name specified by the user end, the storage path of the specified database / table in the source cluster is obtained, and then the data files and index files of the specified database / table are extracted from the source cluster according to the storage path, to realize backup of data files and index files.

[0120] In step S312, for configuration information, there is only one configuration shard and multiple metadata-related files in MongoDB, such as collection (table-related metadata file), database (database-related metadata file), shard (shard-related metadata file), and chunks (data block-related metadata file). In this embodiment, the metadata file corresponding to the source cluster is parsed through the WiredTiger engine interface, and the configuration information of the specified database / table is extracted therefrom to realize backup of configuration information.

[0121] Step S313, according to the number of shards of the target cluster and the source cluster and the library / table condition, the appropriate shard is selected for data recovery, and the data file, index file and configuration information are recovered.

[0122] It should be noted that the above-mentioned data file and configuration information backup and recovery process of the specified library / table, and the verification of the data integrity and correctness after the recovery are involved, which ensures the integrity and correctness of the recovered data in the target cluster.

[0123] The above process is described in detail as follows:

[0124] The above-mentioned data backup and recovery method provided by the embodiment mainly involves five tools, which are wtdump (data backup tool), wtrestore (data recovery tool), configdump (configuration backup tool), configrestore (configuration recovery tool) and restorecheck (verification tool). The main function of wtdump is to realize the physical backup of the library / table granularity data file of the specified source cluster, support the cold backup of a single library of the specified source cluster to the specified output directory, support the table level granularity backup, backup the index file associated with the table, generate the metadata related to the backup file to the output directory, and verify the integrity of the backup file during the backup process. The main function of wtrestore is to realize the physical recovery of the library / table granularity data file of the specified target cluster, recover the data backup file generated by wtdump to the target library of the target cluster, support the table level granularity recovery, modify the metadata related to the table to be recovered in the target library, so as to correctly associate the recovered table, verify the integrity of the recovery file before and after the recovery, output the recovery statistics after the recovery. The main function of configdump is to realize the physical backup of the library / table granularity configuration information of the specified source cluster, and cold backup the configuration information of the specified library / table in the configuration information of the source cluster to the output path specified by the user. The main function of restorecheck is to realize the physical recovery of the library / table granularity configuration information of the specified target cluster, and support the recovery of the configuration backup data generated by configdump to the target cluster. Specifically, the MongoDB Driver (MongoDB driver program) is connected to the specified target cluster, and the configuration information of the library / table to be recovered is correctly set to the configuration shard of the target cluster. The main function of restorecheck is to realize the verification of the target cluster after the recovery, which mainly involves the verification of the number of libraries, the number of tables, the number of documents, the number of indexes, the number of shard keys, the index information, the shard key information, the database information, etc.

[0125] Based on this, the above-mentioned data backup and recovery process provided by the embodiment can further include the following steps S321 to S327:

[0126] Step S321, wtdump is executed to obtain the parameters of the library / table to be backed up, data file physical backup is performed, the data files and index files related to the library / table to be backed up are extracted from the shard backup directory, and backup metadata is generated. In this process, the mongod process or mongos process of MongoDB does not need to be started.

[0127] Step S322, configdump is executed to obtain the parameters of the library / table to be backed up, and configuration information physical backup is performed. The configuration information related to the library / table to be backed up is extracted from the source cluster configuration shard backup directory.

[0128] Step S323, configrestore is executed to obtain the parameters of the library / table to be restored and the access address of the target cluster, and the configuration information of the library / table to be restored is written to the target cluster. The mapping relationship between the source cluster shard and the target cluster shard is generated.

[0129] Step S324, wtrestore is executed to transmit the backup data files and backup metadata backed up from the source cluster shards to the corresponding shards of the target cluster according to the mapping relationship between the source cluster shard and the target cluster shard.

[0130] Step S325, for each shard of the target cluster, mongod is executed to copy the data files and index files related to the library / table to be restored to the library directory of the target cluster, and necessary metadata is written to complete physical restoration. At the same time, the recovery benchmark data is generated.

[0131] Step S326, restorecheck is executed to check the integrity and correctness of the target cluster after recovery according to the address transmitted to the target cluster and the recovery benchmark data of the library / table to be checked, mainly including the related statistics and accessibility of the restored library / table.

[0132] Step S327, the physical backup recovery of the shard cluster is completed after the verification is passed.

[0133] Data backup and recovery mainly involves two important aspects, namely, data structure and implementation process. Among them, the data structure defines how the data is organized, stored and finally recovered. The selection and design of the data structure directly affect the efficiency and effect of the backup and recovery, and the implementation process defines the specific operation steps and technical means, which ensures the smooth operation of the backup and recovery. Therefore, the backup and recovery of data files and the backup and recovery of configuration information are described in detail from the aspects of data structure and implementation process.

[0134] For the backup and recovery of data files:

[0135] In this embodiment, according to the designated namespace (db.collection), all data files and index files associated with the table to be backed up are found. Specifically, MongoDB encapsulates a special table metadata _mdb_catalog, which stores the correspondence between the table and its data files and index files. wtdump and wtrestore read and record the _mdb_catalog under the process execution directory through the WiredTiger engine, and find the data files and index files associated with the table to be backed up from it.

[0136] In this embodiment, in order to let the target cluster correctly identify and load the table to be restored, the wtdump / wtrestore backup and restore the MongoDB cluster metadata and WiredTiger engine metadata related to the specified table as shown in Table 1:

[0137] Table 1 Metadata for backup and restoration

[0138]

[0139] As can be seen from Table 1, the wtdump / wtrestore tool backs up and restores 5 pieces of metadata for a single table, including 3 pieces of WiredTiger engine metadata required for indexing.

[0140] Optionally, the wtdump / wtrestore tool encapsulates the above metadata corresponding to a single table into a data structure NSSMetadata (metadata document). NSSMetadata consists of four parts as shown in Table 2:

[0141] Table 2 Components of data structure NSSMetadata

[0142] Components Corresponding metadata metaContent _mdb_catalog metadata dataMetaInfo WiredTiger.wt metadata sizeMeta sizeStorer.wt metadata indexMetaInfos Index metadata

[0143] As can be seen from the above, when performing library / table level physical backup, wtdump not only backs up the data files and index files corresponding to the specified table, but also generates the data structure NSSMetadata corresponding to the table. When performing library / table level physical restoration, wtrestore not only restores the data files and index files of the specified table, but also sets the correct NSSMetadata metadata document for the table under the specified library according to the specific configuration of the target cluster, so as to ensure that the target cluster can be normally accessed after restoration.

[0144] Figure 4 is the flowchart of the data file physical backup provided in this embodiment.

[0145] As Figure 4As shown, the physical backup process of the data file may include the following steps S401 to S403:

[0146] In step S401, when wtdump is initialized, its internally encapsulated metadata parser uses the WiredTiger engine interface to open a WT_CONNECTION connection to the specified library of the source cluster and create a WT_SESSION; loads WiredTiger.wt as a meta file; uses WT_CURSOR to scan and load metadata associated with all tables under the specified library of the source cluster (such as app meta, colgroup meta, and file meta); opens sizeStorer.wt to scan and load capacity information of all tables; scans the MongoDB special metadata table _mdb_catalog; loads data files and index files of all tables, and associates and binds the above-obtained metadata.

[0147] Step S402: When wtdump performs a physical backup, it searches for the paths of the metadata, data files, and index files corresponding to the tables to be backed up one by one based on the directory of the tables to be backed up obtained by parsing the input parameters, and generates multiple physical backups and subtasks with tables as the granularity; it starts a physical backup thread pool including multiple threads, and adds all backup subtasks to the thread pool work queue to wait for scheduling and execution.

[0148] Step S403: After the backup subtask corresponding to each table is completed, the data file corresponding to the table will be copied to the specified output directory, and the metadata document corresponding to the table will be written into the metadata bson file wtdump.meta in the specified output directory.

[0149] Figure 5 This is a flowchart of physical recovery of data files provided by this embodiment.

[0150] like Figure 5 As shown, the physical recovery process of the data file may include the following steps S501 to S502:

[0151] Step S501, encapsulate the physical recovery task as WTRestoreTask, after parsing the metadata of the source cluster and the target cluster and analyzing the recovery parameters, generate a table-level recovery subtask collectionRestoreTask list, start a physical recovery thread pool including multiple threads, and add the recovery subtask collectionRestoreTask list to the thread pool to wait for scheduling execution.

[0152] Step S502, each recovery sub-task will copy the table corresponding data file and index file to the target cluster corresponding directory, and based on the backup metadata loaded from the source cluster, set the metadata associated with the table in table:_mdb_catalog.wt, file:WireTiger.wt, table:sizeStorer.wt of the target cluster.

[0153] Backup and recovery of configuration information:

[0154] configdump backs up the configuration information of the shard cluster related to the library / table to be backed up and recovered, and organizes the configuration information into a data structure CompleteConfigMetaInfo (configuration information document), which is injected into the target cluster after certain adjustment. In order to ensure the access correctness of the target cluster after recovery, CompleteConfigMetaInfo is at least composed of four parts of data shown in Table 3:

[0155] Table 3 backup and recovery configuration data

[0156] Data Role config.shards Sharding information of the cluster config.databases Database information of the cluster, including master shard, whether to open shard, etc. config.collections Configuration information of the cluster, including shard key, etc. config.chunks Database distribution information of the cluster, used to guide routing

[0157] configrestore performs physical recovery on the source cluster in the granularity of shards, and needs to find a corresponding target shard in the target cluster for each source shard of the source cluster during the recovery process, and establish a mapping relationship between the source shard in the source cluster and the target shard in the target cluster, which is the basis for the recovery of data files and index files.

[0158] Optionally, the embodiment provides a shard mapping algorithm for the source cluster and the target cluster, which follows the following principles: first, only find the mapping relationship between the source shard and the target shard related to the specified library / table; do not modify the primary shard configuration of the existing library in the target cluster configuration node; second, when the to-be-recovered table does not exist, different source shards are allowed to be mapped to the same target shard; third, when the to-be-recovered table exists, the number of source shards needs to be verified first, and different shards of the source cluster are not allowed to be mapped to the same shard of the target cluster; fourth, the mapping of the source shard and the target shard is as evenly distributed as possible under the allowed conditions, reducing the read-write load pressure of the single primary shard of the target cluster.

[0159] Figure 6 is the flowchart of the physical backup of the configuration information provided by the embodiment.

[0160] As shown in Figure 6 , the physical backup process of the configuration information can include the following steps S601 to S602:

[0161] Step S601, load the config.databases, config.collections, config.chunks, config.shards configuration information tables without starting the configuration shards of the source cluster.

[0162] Step S602, extract the configuration information of the specified database / table according to the backup parameters, convert it into a data structure CompleteConfigMetaInfo, and write it in the form of a bson file to the specified backup directory.

[0163] Figure 7 is the flowchart of the configuration information physical recovery provided in this embodiment.

[0164] As shown in Figure 7 , the configuration information physical recovery flow can include the following steps S701 to S704:

[0165] Step S701, extract and parse the configuration information of the to-be-recovered database / table from the recovery parameters.

[0166] Step S702, obtain the configuration information table in the target cluster.

[0167] Step S703, calculate the mapping relationship between the source shards and the target shards of the to-be-recovered database / table.

[0168] Step S704, adjust the extracted configuration information, and insert the configuration information related to the to-be-recovered database / table into the target cluster.

[0169] In this embodiment, a verification scheme is also provided to ensure the security of the backup recovery process, which mainly verifies the integrity of the data physical backup, the integrity before and after the data physical recovery, and the correctness and integrity of the target cluster after the data physical recovery.

[0170] The integrity verification of the data physical backup is mainly implemented based on wtdump when performing physical file backup. Wtdump verifies the size and file attribute information of the physical files of each source shard before and after backup against the physical files in the target directory, and the backup task fails if the verification fails. The physical backup output records the size and file attribute metadata of each backup physical file.

[0171] The integrity check before and after the data physical recovery is mainly implemented by the wtrestore when performing the physical file recovery. First, the wtrestore tool parses the metadata in the input backup before the recovery, and compares the file size recorded in the metadata with the actual file size in the input directory. If the comparison fails, the recovery task fails. Second, the wtrestore checks the physical file size in the target cluster directory after the recovery. If the physical file size is inconsistent with that in the source cluster, the recovery task fails.

[0172] The correctness and integrity of the target cluster after the data physical recovery are mainly implemented by the restorecheck tool in the checking phase. Figure 8 is a flowchart of checking the target cluster provided by the embodiment.

[0173] As shown in Figure 8 , the correctness and integrity checking process of the target cluster can include the following steps S801 to S802:

[0174] In step S801, since the wtrestore generates recovery statistics about the recovered library / table after the recovery, specifically, how many indexes are recovered for each table, how many rows of records are recovered, etc., in the checking phase, the wtrestore captures the recovery statistics in cooperation with the automatic process, and the restorecheck checks whether the number of recovered indexes, the number of records, the shard key, etc. in the target cluster are equal to the benchmark recovery data based on the recovery statistics of each table. In addition, for each table with records, a record is used to check whether it can be normally accessed.

[0175] In step S802, a series of CRUD (Create, Read, Update, and Delete) operations are performed in the target cluster to verify whether the recovered cluster can work normally, and whether the data is complete and accessible.

[0176] The second embodiment provides an optional data backup and recovery method, which realizes the library / table level physical cold backup and recovery on the MongoDB sharded cluster. The method can simultaneously backup and restore the index file and configuration information while backing up and restoring the data file, which is a one-step backup and recovery scheme. The method not only solves the problems of long backup and recovery process, low efficiency, inability to backup and restore index files and configuration information, and the need for manual creation in the existing logical backup scheme, but also solves the problem of large backup and recovery granularity in the existing physical backup and recovery scheme.

[0177] The method provided in this embodiment can be applied to various businesses and scenarios. Taking the online game business as an example, the method provided in this embodiment is applicable to the following data backup and recovery scenarios: multi-server same cluster project, during maintenance, it is desired to back up the data of a certain cluster in it, rather than the data of the entire server; a single cluster of the project has data from multiple different servers, and different servers store data in separate libraries or tables. During project maintenance, the data of a certain server is abnormal and needs to be rolled back to the data of a certain server (corresponding to a certain library or table) (for example, a sudden abnormality in the server merger); daily operation of the project requires confirmation of the data status of a certain table through backup and then review of player data; after the project data is restored offline, some required library and table data needs to be restored to an online test cluster to start the game to confirm the data status or conduct related gameplay testing; and designated data is restored to the office environment from time to time for testing.

[0178] Verified in online gaming, the method provided by this embodiment not only supports more flexible granular management, but also offers usage analysis and performance testing for selected online projects, compared to traditional solutions that restore to an offline cluster and then use mongodump / mongorestore for logical backup and recovery. It also boasts backup and recovery efficiency that is 3-6 times greater than traditional solutions. Furthermore, the lower the proportion of data in the entire cluster that the database or table being restored is, the more significant the improvement in backup and recovery efficiency. Traditional solutions can complete a backup and recovery process that takes several hours, while the method provided by this embodiment can complete it in just minutes, significantly improving business rollback efficiency in extreme situations.

[0179] It should be noted that the examples in the second embodiment are only for explaining the method described in this application and are not intended to be limiting for actual use. The data backup and recovery method provided in this application includes but is not limited to the method described in the second embodiment.

[0180] The third embodiment of the present application provides a data backup and recovery tool. Figure 9 Schematic diagram of the data backup and recovery tool provided in this embodiment.

[0181] like Figure 9 As shown, the data backup and recovery tool provided in this embodiment includes: a data backup module 901 , a configuration backup module 902 , a data recovery module 903 , and a configuration recovery module 904 .

[0182] The data backup module 901 is configured to acquire metadata corresponding to a specified object from a metadata list of a first shard cluster according to an object parameter of the specified object, and extract data files and index files of the specified object from a first data shard of the first shard cluster according to the metadata corresponding to the specified object; wherein the specified object includes a database or a data table in the first shard cluster, the object parameter is acquired in response to a backup recovery instruction and is used to indicate the first shard cluster to which the specified object belongs and a namespace of the specified object, the metadata at least records a corresponding relationship between the specified object and the data files and the index files of the specified object, and the first data shard is at least one data shard of a plurality of data shards included in the first shard cluster and storing the data files and the index files of the specified object.

[0183] The configuration backup module 902 is configured to extract configuration information of the specified object from a configuration shard of the first shard cluster according to the object parameter, and establish a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in a second shard cluster according to the configuration information; wherein the second shard cluster is specified by the backup recovery instruction, and the first number is the number of the first data shard.

[0184] The data recovery module 903 is configured to write the data files and the index files of the specified object into corresponding data shards of the second shard cluster based on the mapping relationship, and write the metadata corresponding to the specified object into a metadata list of the second shard cluster.

[0185] The configuration recovery module 904 is configured to write the configuration information of the specified object into a configuration shard of the second shard cluster.

[0186] Optionally, the data backup recovery tool provided in the embodiment further includes a verification module.

[0187] The verification module is configured to verify the integrity of the specified object restored in the second shard cluster, the access correctness of the specified object restored in the second shard cluster, and the access correctness of the second shard cluster.

[0188] The fourth embodiment of the present application provides a data backup recovery device. Figure 10 FIG. 1 is a structural schematic diagram of the data backup recovery device provided in the embodiment.

[0189] As shown in FIG. 1, the data backup recovery device provided in the embodiment includes an acquisition unit 1001, a first extraction unit 1002, a second extraction unit 1003, a first writing unit 1004, and a second writing unit 1005. Figure 10 The data backup recovery device provided in the embodiment includes an acquisition unit 1001, a first extraction unit 1002, a second extraction unit 1003, a first writing unit 1004, and a second writing unit 1005.

[0190] The acquisition unit 1001 is configured to acquire, in response to a backup recovery instruction, object parameters of a specified object, the object parameters being used to indicate a first sharded cluster to which the specified object belongs and a namespace of the specified object, and the specified object including a database or a data table in the first sharded cluster.

[0191] The first extraction unit 1002 is configured to acquire, according to the object parameters, metadata corresponding to the specified object from a metadata list of the first sharded cluster, and extract configuration information of the specified object from a configuration shard of the first sharded cluster, and the metadata at least records a correspondence between the specified object and data files and index files of the specified object.

[0192] Optionally, the acquiring, according to the object parameters, of the metadata corresponding to the specified object from the metadata list of the first sharded cluster includes:

[0193] parsing the metadata list of the first sharded cluster;

[0194] extracting, according to the namespace of the specified object, the metadata corresponding to the specified object from the metadata list, and the metadata at least includes first metadata used to record the correspondence between the specified object and data files and index files of the specified object, second metadata used to record file states of the data files and the index files of the specified object, third metadata used to record creation information of the first sharded cluster, fourth metadata used to record file information of the first sharded cluster, and fifth metadata used to record checkpoint information of the data files and the index files of the specified object in the first sharded cluster:

[0195] processing and packaging the metadata corresponding to the specified object into a metadata document, and the metadata document at least includes a first sub-data document corresponding to the first metadata, a second sub-data document corresponding to the second metadata, a third sub-data document corresponding to the third metadata, the fourth metadata, and the fifth metadata, and a fourth sub-data document corresponding to index metadata of the specified object, the index metadata being metadata related to the index in the metadata corresponding to the specified object;

[0196] Optionally, the extracting, from the configuration shard of the first sharded cluster, of the configuration information of the specified object includes:

[0197] parsing a configuration information list stored in the configuration shard of the first sharded cluster, and at least extracting, from the configuration information list, a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table;

[0198] extracting configuration information corresponding to the specified object from the extracted slice configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table according to a namespace of the specified object;

[0199] processing and encapsulating the extracted configuration information corresponding to the specified object into a configuration information document.

[0200] The second extraction unit 1003 is configured to extract data files and index files of the specified object from a first data slice of the first slice cluster according to metadata corresponding to the specified object, the first data slice being at least one data slice of a plurality of data slices included in the first slice cluster and storing the data files and index files of the specified object.

[0201] Optionally, the extracting of the data files and index files of the specified object from the first data slice of the first slice cluster according to the metadata corresponding to the specified object comprises:

[0202] determining a storage path of the data files and index files of the specified object in the first slice cluster according to a corresponding relationship between the specified object and the data files and index files of the specified object recorded in the metadata;

[0203] determining the first data slice storing the data files and index files of the specified object according to the storage path;

[0204] extracting the data files and index files of the specified object from the first data slice.

[0205] The first writing unit 1004 is configured to establish a mapping relationship between the first data slice in the first slice cluster and a first number of data slices in a second slice cluster according to the configuration information, and write the configuration information of the specified object into a configuration slice of the second slice cluster, the second slice cluster being specified by the backup and recovery instruction, and the first number being a number of the first data slice.

[0206] Optionally, the establishing of the mapping relationship between the first data slice in the first slice cluster and the first number of data slices in the second slice cluster according to the configuration information comprises:

[0207] parsing a configuration information list stored in the configuration slice of the second slice cluster, and extracting at least a slice configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list;

[0208] According to the configuration information corresponding to the specified object in the extracted shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table of the second shard cluster, a mapping relationship between the first data shard in the first shard cluster and a data shard in the second shard cluster is determined.

[0209] Optionally, the writing of the configuration information of the specified object into the configuration shard of the second shard cluster comprises:

[0210] According to the configuration information document, the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the configuration shard of the second shard cluster are changed;

[0211] The shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table in the configuration information list stored in the configuration shard of the second shard cluster are deleted, and the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table after the change according to the configuration information document are inserted into the configuration information list stored in the configuration shard of the second shard cluster.

[0212] The second writing unit 1005 is configured to write the data file and the index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and write the metadata corresponding to the specified object into the metadata list of the second shard cluster.

[0213] Optionally, the writing of the metadata corresponding to the specified object into the metadata list of the second shard cluster comprises:

[0214] The metadata document is written into the metadata list of the second shard cluster.

[0215] Optionally, the apparatus further comprises a checking unit.

[0216] After the step of extracting the data file and the index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object, the checking unit is configured to:

[0217] The data file and the index file of the specified object extracted from the first data shard of the first shard cluster are compared with the data file and the index file of the specified object stored in the first data shard of the first shard cluster in file state, so as to check the integrity of the extracted data file and index file of the specified object.

[0218] After the step of writing the data file and the index file of the specified object into the corresponding data shard of the second shard cluster and writing the metadata corresponding to the specified object into the metadata list of the second shard cluster based on the mapping relationship, the checking unit is configured to:

[0219] perform data state comparison on the metadata corresponding to the specified object written into the metadata list of the second shard cluster and the metadata corresponding to the specified object obtained from the metadata list of the first shard cluster, to check the integrity of the metadata corresponding to the specified object written into the metadata list of the second shard cluster.

[0220] perform file state comparison on the data file and the index file of the specified object written into the corresponding data shard of the second shard cluster and the data file and the index file of the specified object extracted from the first data shard of the first shard cluster, to check the integrity of the data file and the index file of the specified object written into the corresponding data shard of the second shard cluster.

[0221] generate recovery statistics, the recovery statistics being used to record the number of the configuration information, the data file, the index file and the metadata written into the second shard cluster.

[0222] The checking unit is configured to:

[0223] perform number comparison on the configuration information, the data file, the index file and the metadata of the specified object recovered in the second shard cluster and the configuration information, the data file, the index file and the metadata recorded in the recovery statistics and written into the second shard cluster, to check the integrity of the specified object recovered in the second shard cluster.

[0224] perform access test on the specified object recovered in the second shard cluster, to check the access correctness of the specified object recovered in the second shard cluster.

[0225] perform access test on a first object in the second shard cluster, to check the access correctness of the second shard cluster, the first object being any database or data table in the second shard cluster.

[0226] The fifth embodiment of the present application provides an electronic device, Figure 11 is a structural schematic diagram of the electronic device provided by the embodiment.

[0227] As Figure 8 shown, the electronic device provided by the embodiment includes a memory 1101 and a processor 1102.

[0228] The memory 1101 is configured to store computer instructions for executing the data backup recovery method.

[0229] The processor 1102 is configured to execute the computer instructions stored in the memory 1101 to perform the following operations:

[0230] In response to a backup recovery instruction, an object parameter of a specified object is obtained, the object parameter being used to indicate a first sharded cluster to which the specified object belongs and a namespace of the specified object, the specified object including a database or a data table in the first sharded cluster;

[0231] According to the object parameter, metadata corresponding to the specified object is obtained from a metadata list of the first sharded cluster, and configuration information of the specified object is extracted from a configuration shard of the first sharded cluster, the metadata at least recording a correspondence between the specified object and data files and index files of the specified object;

[0232] According to the metadata corresponding to the specified object, data files and index files of the specified object are extracted from a first data shard of the first sharded cluster, the first data shard being at least one data shard of a plurality of data shards included in the first sharded cluster and storing the data files and the index files of the specified object;

[0233] According to the configuration information, a mapping relationship between the first data shard in the first sharded cluster and a first number of data shards in a second sharded cluster is established, and the configuration information of the specified object is written into a configuration shard of the second sharded cluster, the second sharded cluster being specified by the backup recovery instruction, and the first number being a number of the first data shard;

[0234] Based on the mapping relationship, the data files and the index files of the specified object are written into corresponding data shards of the second sharded cluster, and the metadata corresponding to the specified object is written into a metadata list of the second sharded cluster.

[0235] Optionally, after the step of extracting the data files and the index files of the specified object from the first data shard of the first sharded cluster according to the metadata corresponding to the specified object, the following step is further performed:

[0236] The data files and the index files of the specified object extracted from the first data shard of the first sharded cluster are compared with the data files and the index files of the specified object stored in the first data shard of the first sharded cluster in a file state to verify the integrity of the extracted data files and index files of the specified object.

[0237] Optionally, after the step of writing the data file and the index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and writing the metadata corresponding to the specified object into the metadata list of the second shard cluster, the method further comprises:

[0238] performing data state comparison on the metadata corresponding to the specified object written into the metadata list of the second shard cluster and the metadata corresponding to the specified object obtained from the metadata list of the first shard cluster, to verify the integrity of the metadata corresponding to the specified object written into the metadata list of the second shard cluster;

[0239] performing file state comparison on the data file and the index file of the specified object written into the corresponding data shard of the second shard cluster and the data file and the index file of the specified object extracted from the first data shard of the first shard cluster, to verify the integrity of the data file and the index file of the specified object written into the corresponding data shard of the second shard cluster;

[0240] generating recovery statistics, the recovery statistics being used to record the number of the configuration information, the data file, the index file and the metadata written into the second shard cluster.

[0241] Optionally, the method further comprises:

[0242] performing number comparison on the configuration information, the data file, the index file and the metadata of the specified object recovered in the second shard cluster and the configuration information, the data file, the index file and the metadata recorded in the recovery statistics and written into the second shard cluster, to verify the integrity of the specified object recovered in the second shard cluster;

[0243] performing access test on the specified object recovered in the second shard cluster, to verify the access correctness of the specified object recovered in the second shard cluster;

[0244] performing access test on a first object in the second shard cluster, to verify the access correctness of the second shard cluster, the first object being any database or data table in the second shard cluster.

[0245] Optionally, the step of obtaining the metadata corresponding to the specified object from the metadata list of the first shard cluster according to the object parameter comprises:

[0246] parsing the metadata list of the first shard cluster;

[0247] extracting metadata corresponding to the specified object from the metadata list according to a namespace of the specified object, the metadata at least including first metadata for recording a correspondence relationship between the specified object and data files and index files of the specified object, second metadata for recording file states of the data files and the index files of the specified object, third metadata for recording creation information of the first shard cluster, fourth metadata for recording file information of the first shard cluster, and fifth metadata for recording checkpoint information of the data files and the index files of the specified object in the first shard cluster:

[0248] processing and packaging the metadata corresponding to the specified object into a metadata document, the metadata document at least including a first sub-data document corresponding to the first metadata, a second sub-data document corresponding to the second metadata, a third sub-data document corresponding to the third metadata, the fourth metadata, and the fifth metadata, and a fourth sub-data document corresponding to index metadata of the specified object, the index metadata being metadata related to indexes in the metadata corresponding to the specified object;

[0249] the writing of the metadata corresponding to the specified object into the metadata list of the second shard cluster includes:

[0250] writing the metadata document into the metadata list of the second shard cluster.

[0251] Optionally, the extracting of the data files and the index files of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object includes:

[0252] determining storage paths of the data files and the index files of the specified object in the first shard cluster according to the correspondence relationship between the specified object and the data files and the index files of the specified object recorded by the metadata corresponding to the specified object;

[0253] determining the first data shard storing the data files and the index files of the specified object according to the storage paths;

[0254] extracting the data files and the index files of the specified object from the first data shard.

[0255] Optionally, the extracting of the configuration information of the specified object from the configuration shard of the first shard cluster includes:

[0256] parsing a configuration information list stored in the configuration shard of the first shard cluster, and extracting at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list;

[0257] extracting configuration information corresponding to the specified object from the extracted shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table according to a namespace of the specified object;

[0258] processing and encapsulating the extracted configuration information corresponding to the specified object into a configuration information document.

[0259] Optionally, the establishing, according to the configuration information, a mapping relationship between the first data shard in the first shard cluster and a first quantity of data shards in the second shard cluster comprises:

[0260] parsing a configuration information list stored in a configuration shard of the second shard cluster, and extracting at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list;

[0261] determining a mapping relationship between the first data shard in the first shard cluster and a data shard in the second shard cluster according to the extracted shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table of the second shard cluster and the configuration information corresponding to the specified object.

[0262] Optionally, the writing the configuration information of the specified object into the configuration shard of the second shard cluster comprises:

[0263] changing the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the configuration shard of the second shard cluster according to the configuration information document;

[0264] deleting the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table in the configuration information list stored in the configuration shard of the second shard cluster, and inserting the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table changed according to the configuration information document into the configuration information list stored in the configuration shard of the second shard cluster.

[0265] The sixth embodiment of the present application provides a computer readable storage medium, which comprises computer instructions for implementing the method described in the embodiments of the present application when executed by a processor.

[0266] It is to be noted that the terms "first", "second", and the like in this text are used only to distinguish one entity or action from another, do not require or imply any actual relationship or sequence between these entities or actions. In addition, the terms "include", "have", "contain", and "comprise" and other similar forms are the same in meaning, and the end of any one or more items after any one of the above terms is open, and any one of the above terms does not mean that the one or more items have been listed exhaustively or are limited to these listed one or more items.

[0267] As used herein, unless otherwise expressly specified, the term "or" includes all possible combinations of the items listed, except where such combinations are mutually exclusive. For example, if a database is said to include A or B, then the database can include A, or B, or A and B, unless otherwise indicated or mutually exclusive. As a second example, if a database is said to include A, B, or C, then the database can include A, or B, or C, or A and B, or A and C, or B and C, or A and B and C, unless otherwise indicated or mutually exclusive.

[0268] It is to be noted that the above embodiments can be realized by hardware or software (program code), or a combination of hardware and software. If realized by software, it can be stored in the above computer readable medium. The software, when executed by a processor, can perform the above disclosed method. The computing units and other functional units described in the disclosure can be realized by hardware or software, or a combination of hardware and software. Those skilled in the art will also understand that the above multiple modules / units can be combined into one module / unit, and each of the above modules / units can be further divided into multiple sub-modules / sub-units.

[0269] In the above detailed description, the embodiments have been described with reference to many specific details, which can vary from implementation to implementation. Certain adaptations and modifications of the described embodiments can occur to those skilled in the art. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The true scope and spirit of the application is indicated by the claims, and the specification and examples are merely illustrative. The order of the steps shown in the figures is also merely for illustrative purposes, and is not meant to imply any specific order or sequence. Therefore, those skilled in the art will recognize that the steps can be performed in different orders, or performed concurrently.

[0270] In the figures and detailed description of the application, exemplary embodiments are disclosed. However, many variations and modifications can be made to these embodiments. Accordingly, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. A data backup and recovery method, characterized in that: The method comprises: In response to the backup and restore instruction, obtain object parameters of a specified object, where the object parameters are used to indicate a first shard cluster to which the specified object belongs and a namespace of the specified object, where the specified object includes a database or a data table in the first shard cluster; Obtaining metadata corresponding to the specified object from a metadata list of the first shard cluster according to the object parameters, and extracting configuration information of the specified object from a configuration shard of the first shard cluster, wherein the metadata at least records a correspondence between the specified object and the data file and index file of the specified object; Extracting, based on the metadata corresponding to the specified object, a data file and an index file of the specified object from a first data shard of the first shard cluster, where the first data shard is at least one data shard among the plurality of data shards included in the first shard cluster that stores the data file and the index file of the specified object; Establishing a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in the second shard cluster according to the configuration information, and writing the configuration information of the specified object into the configuration shard of the second shard cluster, where the second shard cluster is specified by the backup and restore instruction, and the first number is the number of the first data shards; Based on the mapping relationship, the data file and index file of the specified object are written into the corresponding data shard of the second shard cluster, and the metadata corresponding to the specified object is written into the metadata list of the second shard cluster.

2. The method according to claim 1, characterized in that After the step of extracting the data file and index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object, the method further includes: Perform a file status comparison on the data files and index files of the specified object extracted from the first data shard of the first shard cluster with the data files and index files of the specified object stored in the first data shard of the first shard cluster to verify the integrity of the extracted data files and index files of the specified object.

3. The method according to claim 2, characterized in that After the step of writing the data file and index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and writing the metadata corresponding to the specified object into the metadata list of the second shard cluster, the method further includes: Performing a data status comparison on the metadata corresponding to the specified object written in the metadata list of the second sharded cluster and the metadata corresponding to the specified object obtained from the metadata list of the first sharded cluster to verify the integrity of the metadata corresponding to the specified object written in the metadata list of the second sharded cluster; Performing a file status comparison on the data file and index file of the specified object written to the corresponding data shard of the second shard cluster with the data file and index file of the specified object extracted from the first data shard of the first shard cluster to verify the integrity of the data file and index file of the specified object written to the corresponding data shard of the second shard cluster; Generate recovery statistics, where the recovery statistics are used to record the amount of the configuration information, the data files, the index files, and the metadata written to the second shard cluster.

4. The method according to claim 3, characterized in that The method further comprises: Comparing the configuration information, data files, index files, and metadata of the specified object restored in the second sharded cluster with the configuration information, data files, index files, and metadata written to the second sharded cluster recorded in the recovery statistics data to verify the integrity of the specified object restored in the second sharded cluster; Performing an access test on the specified object restored in the second shard cluster to verify correct access to the specified object restored in the second shard cluster; An access test is performed on a first object in the second shard cluster to verify correct access to the second shard cluster, where the first object is any database or data table in the second shard cluster.

5. The method according to claim 1, wherein The acquiring metadata corresponding to the specified object from the metadata list of the first shard cluster according to the object parameters includes: Parsing the metadata list of the first shard cluster; Extracting metadata corresponding to the specified object from the metadata list according to the namespace of the specified object, the metadata including at least first metadata for recording a correspondence between the specified object and the data files and index files of the specified object, second metadata for recording file statuses of the data files and index files of the specified object, third metadata for recording creation information of the first shard cluster, fourth metadata for recording file information of the first shard cluster, and fifth metadata for recording checkpoint information of the data files and index files of the specified object in the first shard cluster: Encapsulating metadata corresponding to the specified object into a metadata document, the metadata document at least including a first sub-data document corresponding to the first metadata, a second sub-data document corresponding to the second metadata, third sub-data documents corresponding to the third metadata, the fourth metadata, and the fifth metadata, and a fourth sub-data document corresponding to index metadata of the specified object, where the index metadata is metadata related to an index in the metadata corresponding to the specified object; Writing the metadata corresponding to the specified object into the metadata list of the second shard cluster includes: The metadata document is written into the metadata list of the second shard cluster.

6. The method according to claim 1, characterized in that The extracting, according to the metadata corresponding to the specified object, the data file and the index file of the specified object from the first data shard of the first shard cluster includes: Determining a storage path of the data file and the index file of the specified object in the first shard cluster according to a correspondence between the specified object recorded in the metadata corresponding to the specified object and the data file and the index file of the specified object; Determine, according to the storage path, the first data shard storing the data file and index file of the specified object; The data file and index file of the specified object are extracted from the first data shard.

7. The method according to claim 1, characterized in that The extracting the configuration information of the specified object from the configuration shard of the first shard cluster includes: Parsing a configuration information list stored in a configuration shard of the first shard cluster, and extracting at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list; Extracting configuration information corresponding to the specified object from the extracted shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table according to the namespace of the specified object; The extracted configuration information corresponding to the specified object is processed and encapsulated into a configuration information document.

8. The method according to claim 7, characterized in that The establishing, according to the configuration information, a mapping relationship between the first data shard in the first shard cluster and the first number of data shards in the second shard cluster includes: Parsing a configuration information list stored in a configuration shard of the second shard cluster, and extracting at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list; Determine a mapping relationship between the first data shard in the first shard cluster and the data shard in the second shard cluster based on the extracted shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table of the second shard cluster and the configuration information corresponding to the specified object.

9. The method according to claim 8, characterized in that Writing the configuration information of the specified object into the configuration shard of the second shard cluster includes: Modify the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the configuration shard of the second shard cluster according to the configuration information document; Delete the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table in the configuration information list stored in the configuration shard of the second shard cluster, and insert the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table after being changed according to the configuration information document into the configuration information list stored in the configuration shard of the second shard cluster.

10. A data backup and recovery tool, characterized in that: The tool includes a data backup module, a configuration backup module, a data recovery module, and a configuration recovery module; The data backup module is configured to obtain metadata corresponding to a specified object from a metadata list of a first shard cluster based on object parameters of the specified object, and extract data files and index files of the specified object from a first data shard of the first shard cluster based on the metadata corresponding to the specified object; wherein the specified object comprises a database or a data table in the first shard cluster, the object parameters are obtained in response to a backup and restore instruction and are used to indicate the first shard cluster to which the specified object belongs and a namespace of the specified object, the metadata at least records a correspondence between the specified object and the data files and index files of the specified object, and the first data shard is at least one data shard among the multiple data shards included in the first shard cluster that stores the data files and index files of the specified object; The configuration backup module is configured to extract configuration information of the specified object from the configuration shard of the first shard cluster according to the object parameters, and establish a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in a second shard cluster according to the configuration information; wherein the second shard cluster is specified by the backup and restore instruction, and the first number is the number of the first data shards; The data recovery module is configured to write the data file and index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and write the metadata corresponding to the specified object into the metadata list of the second shard cluster; The configuration recovery module is configured to write the configuration information of the specified object into the configuration shard of the second shard cluster.

11. The tool according to claim 10, characterized in that The tool also includes a verification module; The verification module is used to verify the integrity of the specified object restored in the second shard cluster, the correctness of access to the specified object restored in the second shard cluster, and the correctness of access to the second shard cluster.

12. A data backup and recovery device, characterized in that: The device comprises: an acquisition unit, a first extraction unit, a second extraction unit, a first writing unit, and a second writing unit; The acquiring unit is configured to acquire object parameters of a specified object in response to a backup and restore instruction, where the object parameters are used to indicate a first shard cluster to which the specified object belongs and a namespace of the specified object, where the specified object includes a database or a data table in the first shard cluster; The first extraction unit is configured to obtain metadata corresponding to the specified object from a metadata list of the first shard cluster according to the object parameters, and extract configuration information of the specified object from the configuration shard of the first shard cluster, wherein the metadata at least records a correspondence between the specified object and the data file and index file of the specified object; the second extraction unit is configured to extract, based on the metadata corresponding to the specified object, the data file and the index file of the specified object from a first data shard of the first shard cluster, where the first data shard is at least one data shard among the plurality of data shards included in the first shard cluster that stores the data file and the index file of the specified object; The first writing unit is configured to establish a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in a second shard cluster according to the configuration information, and write the configuration information of the specified object into the configuration shard of the second shard cluster, where the second shard cluster is specified by the backup and restore instruction, and the first number is the number of the first data shards; The second writing unit is configured to write the data file and index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and write the metadata corresponding to the specified object into the metadata list of the second shard cluster.

13. An electronic device, characterized in that: include: Memory, processor; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to implement the method according to any one of claims 1 to 9.

14. A computer-readable storage medium having one or more computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the method according to any one of claims 1 to 9 is performed.

Images