Data backup recovery method and device and electronic equipment

Abstract

The invention discloses a data backup recovery method, tool and device, electronic equipment and a computer readable storage medium. The method comprises the steps of obtaining object parameters of a specified object; obtaining metadata corresponding to the specified object from the first fragment cluster according to the object parameters, and extracting configuration information of the specified object; extracting a data file and an index file of the specified object from the first fragment cluster according to metadata corresponding to the specified object; establishing a mapping relationship between a first data fragment in the first fragment cluster and a first number of data fragments in the second fragment cluster according to the configuration information, and writing the configuration information into the second fragment cluster; and based on the mapping relationship, writing the data file and the index file of the specified object into the second fragment cluster, and writing the metadata into the second fragment cluster. The method solves the technical problems of large backup recovery granularity, high efficiency and the like in the prior art.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data backup and recovery method, tool, device, electronic device, and computer-readable storage medium. Background Art

[0002] Document-based distributed databases are widely used in complex data storage businesses, such as storing online game data, due to their high data storage flexibility, easy scalability, and automatic load balancing. Among them, shard clusters are distributed architectures that guarantee high storage performance and scalability of document-based distributed databases.

[0003] Data backup and recovery is a strong guarantee for business data security. At present, data backup and recovery under sharded clusters mainly include two solutions: physical backup and recovery solution and logical backup and recovery solution. Among them, the physical backup and recovery solution is mainly based on file system snapshots, which has the advantage of high backup and recovery efficiency, but it can only perform backup and recovery at the overall cluster level, and cannot achieve backup and recovery at the library / table level in the cluster. Therefore, backup and recovery not only requires a large-capacity snapshot space, but also has the problem of synchronously restoring libraries / tables that do not need to be restored. The logical backup and recovery solution is mainly based on the command line tool provided by the document-based distributed database. It has the advantages of small backup and recovery granularity, can perform backup and recovery at the library / table level, and does not need to back up and restore the entire cluster. However, due to its long backup and recovery process and the inability to back up and restore the configuration information and index files of the sharded cluster, it has the problems of low backup and recovery efficiency and the need for manual creation at a later stage.

[0004] Therefore, there is an urgent need for a method that can implement physical backup and recovery at the library / table level to solve the technical problems of large backup and recovery granularity and low efficiency in the prior art. Summary of the invention

[0005] The present application provides a data backup and recovery method, tool, device, electronic device, and computer-readable storage medium to at least partially solve the technical problems of large backup and recovery granularity and low efficiency in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a data backup and recovery method, the method comprising: in response to a backup and recovery instruction, obtaining object parameters of a specified object, the object parameters being used to indicate a first shard cluster to which the specified object belongs and a namespace of the specified object, the specified object comprising a database or a data table in the first shard cluster; obtaining metadata corresponding to the specified object from a metadata list of the first shard cluster according to the object parameters, and extracting configuration information of the specified object from a configuration shard of the first shard cluster, the metadata at least recording a correspondence between the specified object and a data file and an index file of the specified object; extracting the specified object from a first data shard of the first shard cluster according to the metadata corresponding to the specified object; The method comprises the steps of: establishing a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in the second shard cluster according to the configuration information, and writing the configuration information of the specified object into the configuration shard of the second shard cluster, the second shard cluster is specified by the backup and recovery instruction, and the first number is the number of the first data shards; based on the mapping relationship, writing the data file and index file of the specified object into the corresponding data shard of the second shard cluster, and writing the metadata corresponding to the specified object into the metadata list of the second shard cluster.

[0007] In a second aspect, an embodiment of the present application provides a data backup and recovery tool, the tool comprising a data backup module, a configuration backup module, a data recovery module, and a configuration recovery module; the data backup module is used to obtain metadata corresponding to a specified object from a metadata list of a first shard cluster according to object parameters of the specified object, and extract data files and index files of the specified object from a first data shard of the first shard cluster according to the metadata corresponding to the specified object; wherein the specified object comprises a database or a data table in the first shard cluster, the object parameters are obtained in response to a backup and recovery instruction, and are used to indicate the first shard cluster to which the specified object belongs and a namespace of the specified object, the metadata at least records the correspondence between the specified object and the data files and index files of the specified object, and the first data shard is a plurality of shards included in the first shard cluster. at least one data shard storing the data files and index files of the specified object in the data shards; the configuration backup module is used to extract the configuration information of the specified object from the configuration shard of the first shard cluster according to the object parameters, and establish a mapping relationship between the first data shard in the first shard cluster and the first number of data shards in the second shard cluster according to the configuration information; wherein the second shard cluster is specified by the backup and recovery instruction, and the first number is the number of the first data shards; the data recovery module is used to write the data files and index files of the specified object into the corresponding data shards of the second shard cluster based on the mapping relationship, and write the metadata corresponding to the specified object into the metadata list of the second shard cluster; the configuration recovery module is used to write the configuration information of the specified object into the configuration shard of the second shard cluster.

[0008] In a third aspect, an embodiment of the present application provides a data backup and recovery device, the device comprising: an acquisition unit, a first extraction unit, a second extraction unit, a first writing unit, and a second writing unit; the acquisition unit is used to obtain object parameters of a specified object in response to a backup and recovery instruction, the object parameters are used to indicate the first shard cluster to which the specified object belongs and the namespace of the specified object, the specified object including a database or a data table in the first shard cluster; the first extraction unit is used to obtain metadata corresponding to the specified object from a metadata list of the first shard cluster according to the object parameters, and extract configuration information of the specified object from a configuration shard of the first shard cluster, the metadata at least recording the correspondence between the specified object and the data file and index file of the specified object; the second extraction unit is used to extract configuration information of the specified object from a metadata list of the first shard cluster according to the metadata corresponding to the specified object The data file and index file of the specified object are extracted from the first data shard of the first shard cluster, and the first data shard is at least one data shard that stores the data file and index file of the specified object among the multiple data shards included in the first shard cluster; the first writing unit is used to establish a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in the second shard cluster according to the configuration information, and write the configuration information of the specified object into the configuration shard of the second shard cluster, the second shard cluster is specified by the backup and recovery instruction, and the first number is the number of the first data shards; the second writing unit is used to write the data file and index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and write the metadata corresponding to the specified object into the metadata list of the second shard cluster.

[0009] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to implement the above method.

[0010] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having one or more computer instructions stored thereon, which, when executed by a processor, executes the above method.

[0011] Compared with the prior art, the data backup and recovery method provided by the present application includes: in response to a backup and recovery instruction, obtaining object parameters of a specified object, the object parameters are used to indicate the first shard cluster to which the specified object belongs and the namespace of the specified object, and the specified object includes a database or a data table in the first shard cluster; obtaining metadata corresponding to the specified object from a metadata list of the first shard cluster according to the object parameters, and extracting configuration information of the specified object from a configuration shard of the first shard cluster, the metadata at least recording the correspondence between the specified object and the data file and index file of the specified object; according to the metadata corresponding to the specified object, extracting the data file and index file of the specified object from the first data shard of the first shard cluster, the first data shard being at least one data shard storing the data file and index file of the specified object among the multiple data shards included in the first shard cluster; establishing a mapping relationship between the first data shard in the first shard cluster and the first number of data shards in the second shard cluster according to the configuration information, and writing the configuration information of the specified object into the configuration shard of the second shard cluster; based on the mapping relationship, writing the data file and index file of the specified object into the corresponding data shard of the second shard cluster, and writing the metadata corresponding to the specified object into the metadata list of the second shard cluster. This method is a solution that combines the high efficiency advantages of physical backup solutions and the fine-grained advantages of logical backup solutions, and can perform physical backup and recovery at the library / table level on a sharded cluster. First, compared with existing physical backup and recovery solutions, this method is based on the use of metadata to achieve backup and recovery at the library / table level, greatly reducing the backup and recovery granularity; second, compared with existing logical backup and recovery solutions, this method is a physical backup performed at the library / table level, which greatly shortens the backup and recovery process and has a high backup and recovery efficiency; third, this method backs up and restores configuration information and index files while backing up and restoring data files, avoiding the step of manual participation in the creation of configuration information and index files in the later stage. Therefore, the data backup and recovery method provided by the present application at least partially solves the technical problems of large backup and recovery granularity and efficiency in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is an application system diagram of the data backup and recovery method provided in the embodiment of the present application;

[0013] Figure 2 is a flow chart of the data backup and recovery method provided by the first embodiment of the present application;

[0014] Figure 3 is a schematic diagram of a data backup and recovery method provided in the second embodiment of the present application;

[0015] Figure 4 This is a flow chart of physical backup of data files provided by the second embodiment of the present application;

[0016] Figure 5 is a flow chart of physical recovery of data files provided by the second embodiment of the present application;

[0017] Figure 6 is a flowchart of physical backup of configuration information provided by the second embodiment of the present application;

[0018] Figure 7 is a flowchart of physical recovery of configuration information provided by the second embodiment of the present application;

[0019] Figure 8 is a flow chart of verifying a target cluster provided by the second embodiment of the present application;

[0020] Fig. 9 is a schematic diagram of a data backup and recovery tool provided in the third embodiment of the present application;

[0021] Fig.10 It is a structural schematic diagram of a data backup and recovery device provided in a fourth embodiment of the present application;

[0022] Fig.11 It is a schematic diagram of the structure of an electronic device provided in the fifth embodiment of the present application. DETAILED DESCRIPTION

[0023] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0024] With the development of information technology and the Internet, data and information have shown explosive growth. Document-based distributed databases are widely used in complex data storage businesses, such as storing online game data, due to their high data storage flexibility, easy scalability, and automatic load balancing. Among them, shard clusters are distributed architectures that guarantee high storage performance and scalability of document-based distributed databases.

[0025] The backup and recovery of data in document-based distributed databases is a powerful guarantee for the security of business data. An efficient backup and recovery mechanism can build a data security line for the business. For example, after the public beta of a game project ends, after game maintenance, or in business anomalies, the data and indexes of the specified library / table in the database need to be rolled back to a specific time point of a backup. The efficiency and security of data recovery are the core requirements of data rollback. It is necessary to roll back the data and indexes of the specified library / table to the specified time point in the shortest possible time without affecting the online database service as much as possible. In this process, the preferred implementation scheme should include the following requirements: First, only roll back the library / table that needs to restore the data, rather than restoring the entire cluster, to avoid rolling back some data that needs to be retained. Second, the data backup and recovery process is efficient. During the data backup and recovery period, the cluster is usually still running externally. The efficient backup and recovery process can greatly reduce the impact on the normal operation of the cluster, such as processing query delays. Third, the data backup and recovery process is safe and verifiable to ensure the correctness of data rollback. However, existing data backup and recovery methods for sharded clusters cannot simultaneously achieve efficient and secure data backup and recovery at the library / table level.

[0026] Currently, data backup and recovery methods mainly include two solutions: physical backup and recovery solution and logical backup and recovery solution.

[0027] The physical backup and recovery solution is mainly based on file system snapshots or sample snapshots. When backing up, the file system snapshots of each data shard and configuration shard of the entire cluster (source cluster) will be saved; when restoring, a target cluster with the same number of data shards and configuration shards as the source cluster will be created and started, and each shard of the target cluster will correspond to each shard of the source cluster one by one. The physical backup of each shard of the source cluster is downloaded on each shard of the target cluster to realize the backup and recovery of the entire cluster to the target cluster. Since the physical backup solution uniformly backs up and restores data files, index files, and metadata, it has the advantage of high backup and recovery efficiency. However, the physical backup and recovery solution can only perform backup and recovery at the overall cluster level, and does not support backup and recovery at the database / table level in the cluster. The backup and recovery granularity is too large, which requires not only a large-capacity snapshot space during backup and recovery, but also the database / table that does not need to be restored is also restored synchronously, resulting in recovery redundancy (e.g., only 1% of the data in the cluster needs to be restored, but the other 99% of the data will still be restored synchronously) or data rollback errors (e.g., some data that needs to be retained is also rolled back).

[0028] The logical backup and recovery solution is mainly implemented based on the command line tools provided by the document-based distributed database, such as mongodump and mongorestore in the MongoDB document-based distributed database. Among them, mongodump is responsible for logical backup of the source cluster. Specifically, the relevant information of the source cluster (such as the host name, port, user name, password, etc. of the source cluster) and the library / table to be backed up are specified through command line parameters. Mongodump extracts data from the specified library / table and converts it into BSON format data. The exported BSON format data is written to the local file system. The generated files are usually stored in a directory, and each table corresponds to a .bson file. In addition, mongodump will also generate a .metadata.json file in the local file system to store the index definition and other metadata of the table, but mongodump itself does not support the backup of the index file of the table. mongorestore is responsible for restoring the data backed up by mongodump to the target cluster. Specifically, the relevant information of the target cluster (such as the host name, port, user name, password, etc. of the target cluster) and the backup directory are specified through the command line parameters. mongorestore connects to the target cluster, reads the stored .bson files and .metadata.json files from the backup directory specified by the command line, and then inserts the BSON format data in the read .bson files into the corresponding library / table of the target cluster. Finally, according to the index definition contained in the .metadata.json file, the index file is recreated to achieve the backup and recovery of the library / table level. As can be seen from the above, the logical backup and recovery solution has the advantages of small backup and recovery granularity, which can be backed up and restored at the library / table level without backing up and restoring the entire cluster. However, its backup and recovery process is relatively long overall, and the data backup and recovery efficiency is low, which often affects the normal processing of requests by the database during data backup and recovery. In addition, the logical backup and recovery solution cannot back up and restore configuration information such as sharding keys and index files, and requires manual participation in creation and recovery.

[0029] Taking the backup and recovery of online game data as an example, the current game data backup and recovery usually includes the following steps: first, obtain the full backup file of the entire cluster from the source cluster; second, perform a full physical recovery in the offline environment; third, use the logical backup and recovery solution to restore the specified library / table to the online target cluster; fourth, restore the normal operation of the online game and business access to the target cluster.

[0030] In summary, there is an urgent need for a sharded cluster backup and recovery method that can achieve efficient and secure backup and recovery at the library / table level to solve the above-mentioned technical problems existing in the existing backup and recovery solutions in the data backup and recovery of sharded clusters.

[0031] In view of this, the present application provides a data backup and recovery method, which is a physical backup and recovery solution that combines the high efficiency advantages of physical backup solutions and the fine-grained advantages of logical backup solutions, and can perform physical backup and recovery solutions at the library / table level on a sharded cluster. First, compared with existing physical backup and recovery solutions, this method is based on the use of metadata, realizes backup and recovery at the library / table level, and greatly reduces the backup and recovery granularity; second, compared with existing logical backup and recovery solutions, this method is a physical backup at the library / table level, which greatly shortens the backup and recovery process and has a higher backup and recovery efficiency; third, this method backs up and restores configuration information and index files while backing up and restoring data files, avoiding the steps of manual participation in the creation of configuration information and index files in the later stage.

[0032] Before introducing the present application in detail, some technical terms involved in the present application are first introduced to facilitate a better understanding of the technical solution of the present application.

[0033] 1. Document-based distributed database

[0034] Document-based distributed databases are different from traditional relational databases. Document-based distributed databases do not use tables to store data, but rather use documents in JSON or BSON format to store data. These documents can contain multiple types of data structures, such as strings, integers, objects, arrays, etc., making document-based distributed databases very suitable for processing complex data types. MongoDB is an open source document-based distributed database that has the characteristics of flexible storage document structure, high performance, high availability, and easy scalability. It is particularly suitable for application scenarios that need to process large amounts of unstructured or semi-structured data, such as content management systems, Internet of Things applications, and big data processing.

[0035] 2. WiredTiger

[0036] WiredTiger is a high-performance, scalable database storage engine and the default storage engine for MongoDB document-based distributed databases. WiredTiger is designed to take full advantage of the performance of modern hardware (e.g., multi-core CPUs, large memory, high-speed disks) to provide efficient data storage and retrieval capabilities.

[0037] 3. Sharded Cluster

[0038] A sharded cluster is a distributed architecture that improves the performance and scalability of a system by horizontally splitting (sharding) data and distributing it across multiple nodes. This architecture is particularly suitable for processing very large data sets and high-throughput operations. The core idea of ​​a sharded cluster is to reduce the pressure on a single node by distributing data across multiple nodes, thereby achieving higher performance and better resource utilization. The main components of a sharded cluster include: (1) Shards, which are the actual storage units of data. A sharded cluster usually includes a configuration shard and multiple data shards. The configuration shard is used to store configuration information, and the data shard is used to store data files and index files; (2) Configuration servers, which store metadata information about the sharded cluster, such as shard lists, shard ranges, shard keys, etc.; (3) Routing processes (mongos), which are the front-end interfaces for clients to interact with sharded clusters. They can route client requests to the correct shards and merge the results from multiple shards and return them to the client.

[0039] 4. mongos

[0040] Mongos (routing process) is a stateless node in the sharded cluster that provides direct access to the outside world. It is mainly responsible for request routing, query distribution, data distribution management, aggregation, load balancing and other operations.

[0041] 5. mongod

[0042] mongod (daemon) is the service node that actually executes user requests within the sharded cluster, performs data storage, replication, and synchronization, receives user requests distributed by mongos, and replies to mongos after processing.

[0043] 6. Namespace

[0044] A namespace is a canonical name for a collection or index in a sharded cluster. It is a combination of the database name and the collection or index name. It is used to uniquely identify a collection or index, ensuring that different collections or indexes in the same database can be distinguished by unique names without name conflicts.

[0045] 7. Shard Server

[0046] Shard Server is a data shard service node in a shard cluster, responsible for data storage, replication and synchronization. Each Shard Server is responsible for storing a part of the data, distributing the load by horizontally splitting the data, and improving the performance and scalability of the system. Shard Server is composed of multiple mongod instances. In a shard cluster, each Shard Server is usually a replica set, containing multiple mongod instances. These mongod instances together constitute a Shard Server, which is responsible for storing and managing a part of the data.

[0047] 8. Shard Key

[0048] Shard Key is a field or combination of fields used to shard and distribute data to multiple shards. The value of the shard key determines how data chunks are distributed among shards, ensuring even distribution of data, optimizing query performance, and supporting data migration.

[0049] 9. Config Server

[0050] Config Server is a service node in a sharded cluster that manages cluster configuration data. It is used to store metadata about the sharded cluster, including shard lists, shard key ranges, data block locations, and more.

[0051] 10. Replica Set

[0052] A Replica Set is composed of one or more nodes in a single data shard / configuration shard. These nodes synchronize data through a replication protocol to ensure high availability and fault tolerance of data. Each node in a replica set replicates the data of the primary node to ensure that there are multiple copies of the data, so that even if a node fails, the data is still available.

[0053] 11.db

[0054] db (database) is a single database in a sharded cluster. db is a logical container used to organize and store related data. Each database has its own independent storage space and namespace.

[0055] 11. collection

[0056] A collection is a table in a sharded cluster. A collection is a logical container for storing a set of documents. Each collection has a unique name and belongs to a specific db.

[0057] 12. BSON

[0058] BSON is a binary format similar to JSON, used to represent structured data. It is a binary encoding form of JSON and provides richer data types and more efficient storage performance than JSON. It aims to provide more efficient data storage and transmission.

[0059] The data backup and recovery method, tool, device, electronic device, and computer-readable storage medium described in the present application are further described in detail below in conjunction with specific embodiments and drawings.

[0060] Figure 1 : is an application system diagram of the data backup and recovery method provided in the embodiment of the present application. Figure 1 As shown, the application system includes a user terminal 101 and a server terminal 102, and the user terminal 101 can be any device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant (PDA), etc. The server terminal 102 can be an independent server, deploying the data backup and recovery method provided by the present application, or it can be a server group composed of multiple servers, each of which deploys a module in the data backup and recovery method provided by the present application. Of course, the server terminal 102 can also be a cloud server, and the data backup and recovery method provided by the present application is deployed on the cloud server. The server terminal 102 receives the data backup and recovery request sent by the user terminal 101, and implements efficient and secure data physical backup and recovery at the library / table level based on this method.

[0061] The first embodiment of the present application provides a data backup and recovery method, which can be deployed in Figure 1 The server 102 is shown.

[0062] Figure 2 Flowchart of the data backup and recovery method provided by this embodiment. Figure 2 The data backup and recovery method provided by this embodiment is described in detail. The embodiments described below are used to explain the technical solution of this application and are not intended to be used as limitations for actual use.

[0063] like Figure 2 As shown, the data backup and recovery method provided in this embodiment includes the following steps S210 to S250:

[0064] Step S210, in response to the backup and restore instruction, obtain object parameters of the specified object, the object parameters are used to indicate the first shard cluster to which the specified object belongs and the namespace of the specified object, and the specified object includes a database or a data table in the first shard cluster.

[0065] The backup and recovery instructions can be understood as information sent by the user end to the server end that requires backup and recovery of the specified object. The backup and recovery instructions can be information generated by the user end in response to user operations, or can be information automatically generated by the user end based on business events (such as business exceptions, etc.). Specifically, the backup and recovery instructions carry object parameters of the specified object (such as the backup and recovery object specified by the user, the object that causes the business exception, etc.), and the object parameters indicate the namespace of the specified object and the shard cluster to which the specified object belongs. In this embodiment, the shard cluster to which the specified object belongs is defined as the first shard cluster, and the first shard cluster can also be understood as the source cluster.

[0066] In this embodiment, the specified object may be one or more databases or data tables in the first shard cluster, so as to implement backup and recovery at the database / table level in this embodiment.

[0067] Step S220, obtain metadata corresponding to the specified object from the metadata list of the first shard cluster according to the object parameters, and extract configuration information of the specified object from the configuration shard of the first shard cluster, the metadata at least records the correspondence between the specified object and the data file and index file of the specified object.

[0068] The metadata can be understood as data that describes the data, that is, information about the data. Source data provides information about the context, structure, content, quality, purpose, etc. of the data, and plays an important role in data management, data exchange, data analysis and other fields. Metadata mainly includes descriptive metadata (used to describe the content of the data, such as creation date, etc.), structural metadata (used to describe the organization and structure of the data, such as file format, table and field definition, etc.), and management metadata (used to manage and maintain data, such as access rights, data source, etc.). Sharding clusters involve multiple types of metadata. In this embodiment, these metadata are grouped together and referred to as a metadata list.

[0069] Since the namespace of the specified object is indicated in the object parameter, the metadata describing the specified object can be located in the metadata list based on the namespace, and then the metadata of the specified object can be extracted from the metadata list of the first shard cluster. Specifically, the extracted metadata may include multiple data types, which at least record the correspondence between the specified object and its corresponding data files and index files. In addition, in order for the target cluster to correctly identify and load the specified object backed up and restored from the source cluster, the metadata also needs to include other data types.

[0070] In an optional implementation, obtaining metadata corresponding to a specified object from a metadata list of the first shard cluster according to the object parameters may specifically include the following steps S221-1 to S221-3:

[0071] Step S221-1, parse the metadata list of the first shard cluster.

[0072] Step S221-2, according to the namespace of the specified object, extract the metadata corresponding to the specified object from the metadata list, the metadata including at least first metadata for recording the correspondence between the specified object and the data file and index file of the specified object, second metadata for recording the file status of the data file and index file of the specified object, third metadata for recording the creation information of the first shard cluster, fourth metadata for recording the file information of the first shard cluster, and fifth metadata for recording the checkpoint information of the data file and index file of the specified object in the first shard cluster.

[0073] Step S221-3, encapsulates the metadata processing corresponding to the specified object into a metadata document, which includes at least a first sub-data document corresponding to the first metadata, a second sub-data document corresponding to the second metadata, a third sub-data document corresponding to the third metadata, the fourth metadata, and the fifth metadata, and a fourth sub-data document corresponding to the index metadata of the specified object. The index metadata is the metadata related to the index in the metadata corresponding to the specified object.

[0074] In summary, the metadata of the specified object includes five parts of content, among which the first metadata records the correspondence between the specified object and its corresponding data files and index files. The second metadata records the file status of the data files and index files, such as the file size of the data files and index files. The third metadata records the creation information of the first shard cluster, such as the creation time and name of each collection of the database in the first shard cluster. The fourth metadata records the file information of the first shard cluster, such as the physical location and size of each file in the first shard cluster. The fifth metadata records the checkpoint information of the data files and index files in the first shard cluster, such as the timestamp and status of the checkpoint.

[0075] In this embodiment, after the above-mentioned metadata of the specified object is extracted from the metadata list, the above-mentioned metadata will be processed and packaged. Specifically, the first metadata is processed into a first sub-data document, the second metadata is processed into a second sub-data document, the third, fourth, and fifth metadata are processed into a third sub-data document, and the metadata related to the index in the first, second, third, fourth, and fifth metadata is processed into a fourth sub-data document, and then the above-mentioned sub-data documents are packaged into metadata documents corresponding to the specified object.

[0076] In addition, in this step, configuration information for the specified object is extracted from the configuration shard of the first shard cluster based on the namespace of the specified object. In an optional implementation, the following steps S222-1 to S222-3 may be included:

[0077] Step S222-1, parse the configuration information list stored in the configuration shard of the first shard cluster, and extract at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list.

[0078] Step S222-2, according to the namespace of the specified object, extract the configuration information corresponding to the specified object from the extracted shard configuration information table, database configuration information table, data table configuration information table, and data block configuration information table.

[0079] Step S222-3: Process and encapsulate the extracted configuration information corresponding to the specified object into a configuration information document.

[0080] A variety of configuration information is stored in the configuration shard. In this embodiment, these configuration information are grouped together and referred to as a configuration information list. In order to ensure that the target cluster can be correctly accessed after the specified object is backed up and restored to the target cluster, it is necessary to extract the shard configuration information table (used to store the shard configuration information of the shard cluster), the database configuration information table (used to store the database configuration information of the shard cluster, such as the primary shard, whether to enable sharding, etc.), the data table configuration information table (used to store the table configuration information of the shard cluster, such as the shard key, etc.), and the data block configuration information table (used to store the data block distribution information of the shard cluster, used for reasons) from the parsed configuration information list. After extracting the above configuration information table, multiple configuration information related to the specified object can be extracted from the above configuration information table based on the namespace of the specified object, and these configuration information can be processed and encapsulated into a configuration information document.

[0081] Step S230: extracting the data file and index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object, where the first data shard is at least one data shard storing the data file and index file of the specified object among the multiple data shards included in the first shard cluster.

[0082] The metadata records the correspondence between the specified object and its corresponding data files and index files, and specifically records the correspondence between the specified object and the data shards storing the corresponding data files and index files. Therefore, based on the correspondence recorded in the metadata, the data shards storing the data files and index files of the specified object can be determined, and then the data files and index files of the specified object can be extracted therefrom. In this embodiment, the data shard storing the specified corresponding data is defined as the first data shard. Specifically, the first data shard can be one data shard or multiple data shards, depending on whether the data files and index files of the specified object are stored in shards.

[0083] Based on this, in an optional implementation, according to the metadata corresponding to the specified object, extracting the data file and the index file of the specified object from the first data shard of the first shard cluster may specifically include the following steps S231 to S233:

[0084] Step S231, determining the storage path of the data file and index file of the specified object in the first shard cluster according to the correspondence between the specified object recorded in the metadata corresponding to the specified object and the data file and index file of the specified object.

[0085] Step S232: Determine the first data segment storing the data file and index file of the specified object according to the storage path.

[0086] Step S233: extracting the data file and index file of the specified object from the first data slice.

[0087] Step S240, establish a mapping relationship between the first data shard in the first shard cluster and the first number of data shards in the second shard cluster according to the configuration information, and write the configuration information of the specified object into the configuration shard of the second shard cluster, the second shard cluster is specified by the backup and recovery instructions, and the first number is the number of the first data shards.

[0088] After extracting the configuration information corresponding to the specified object from the first shard cluster, a mapping relationship between the first data shard storing the data files and index files of the specified object in the first shard cluster and the data shards in the target shard cluster can be established based on the configuration information, so as to restore the data files and index files of the specified object extracted from the first data shard of the first shard cluster to the corresponding data shards of the target shard cluster. In this embodiment, the target shard cluster for pre-recovering the specified object is defined as the second shard cluster. Since there may be multiple first data shards storing the specified corresponding data files and index files in the first shard cluster, it is necessary to establish a mapping relationship between the same number of data shards in the second shard cluster and the first data shard.

[0089] In an optional implementation, establishing a mapping relationship between a first data shard in the first shard cluster and a first number of data shards in the second shard cluster according to the configuration information may specifically include the following steps S241-1 to S241-2:

[0090] Step S241-1, parse the configuration information list stored in the configuration shard of the second shard cluster, and extract at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list.

[0091] Step S241-2, determine the mapping relationship between the first data shard in the first shard cluster and the data shard in the second shard cluster according to the extracted shard configuration information table, database configuration information table, data table configuration information table, and data block configuration information table of the second shard cluster and the configuration information corresponding to the specified object.

[0092] The configuration shard of the second shard cluster also stores a configuration information list. In this embodiment, configuration information that is consistent with the configuration information extracted from the configuration information list of the first shard cluster can be extracted from the configuration information list, that is, the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table. Thus, a mapping relationship between the first data shard in the first shard cluster and the corresponding data shard in the second shard cluster is established based on the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the first shard cluster and the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the second shard cluster.

[0093] In this embodiment, after the mapping relationship between the first shard cluster and the second shard cluster is established based on the configuration information, the configuration information is written into the configuration shard of the second shard cluster. In an optional implementation, the configuration information of the specified object is written into the configuration shard of the second shard cluster, which may specifically include the following steps S242-1 to S242-2:

[0094] Step S242-1, according to the configuration information document, the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the configuration shard of the second shard cluster are modified.

[0095] Step S242-2, delete the shard configuration information table, database configuration information table, data table configuration information table, and data block configuration information table in the configuration information list stored in the configuration shard of the second shard cluster, and insert the shard configuration information table, database configuration information table, data table configuration information table, and data block configuration information table after the change according to the configuration information document into the configuration information list stored in the configuration shard of the second shard cluster.

[0096] Step S250: based on the mapping relationship, write the data file and index file of the specified object into the corresponding data shard of the second shard cluster, and write the metadata corresponding to the specified object into the metadata list of the second shard cluster.

[0097] After the mapping relationship between the first data shard in the first shard cluster and the corresponding data shard in the second shard cluster is established, the designated corresponding data files and index files can be written into the second shard cluster to achieve the recovery of the data files and index files in the second shard cluster. Optionally, if there are data files and index files in the corresponding data shard of the second shard cluster, the data files and index files in the corresponding data shard can be cleared first, and then the data files and index files of the designated object can be written into the corresponding data shard after being cleared.

[0098] In this embodiment, the metadata of the specified object also needs to be written into the metadata list of the second shard cluster. Optionally, after the data file and the index file are restored in the second shard cluster, the metadata document is written into the metadata list of the second shard cluster.

[0099] Accordingly, in the above backup and recovery process, not only the data files of the specified object are backed up and restored, but also the index files, configuration information, and metadata of the specified object are backed up and restored.

[0100] In an optional implementation provided in this embodiment, the method provided in this embodiment not only realizes the physical backup and recovery of the specified object, but also realizes the verification of the backup and recovery process to ensure the security of the backup and recovery process, the integrity of the backup and recovery data, and the accessibility of the second shard cluster.

[0101] In a specific implementation, after the step of extracting the data files and index files of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object, the method provided in this embodiment also includes: performing a file status comparison between the data files and index files of the specified object extracted from the first data shard of the first shard cluster and the data files and index files of the specified object stored in the first data shard of the first shard cluster to verify the integrity of the extracted data files and index files of the specified object.

[0102] The purpose of this verification is to verify the integrity of the physical backup, and to compare the data files and index files of the specified object extracted from the first shard cluster with the data files and index files of the specified object stored in the first data shard of the first shard cluster. Specifically, the file size and file attributes are compared to verify whether the data backed up from the first shard cluster is complete. If it is complete, the verification is successful and the subsequent steps can be executed. If it is incomplete, the verification fails and the backup task fails.

[0103] In a specific implementation, after the step of writing the data file and index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and writing the metadata corresponding to the specified object into the metadata list of the second shard cluster, the method provided by this embodiment further includes the following execution content:

[0104] Execution content one: perform a data status comparison on the metadata corresponding to the specified object in the metadata list written to the second shard cluster and the metadata corresponding to the specified object obtained from the metadata list of the first shard cluster to verify the integrity of the metadata corresponding to the specified object in the metadata list written to the second shard cluster.

[0105] The purpose of this verification is to verify the integrity of the metadata before and after physical recovery. The metadata corresponding to the specified object in the metadata list written to the second shard cluster is compared with the metadata corresponding to the specified object obtained from the metadata list of the first shard cluster. Specifically, the data size is compared to verify whether the metadata is complete before and after recovery. If it is complete, the verification is successful and the subsequent steps can be executed. If it is incomplete, the verification fails and the current recovery task fails.

[0106] Execution content two: perform a file status comparison on the data files and index files of the specified object written into the corresponding data shard of the second shard cluster with the data files and index files of the specified object extracted from the first data shard of the first shard cluster to verify the integrity of the data files and index files of the specified object written into the corresponding data shard of the second shard cluster.

[0107] The purpose of this verification is to verify the integrity of the data files and index files before and after physical recovery. The data files and index files of the specified object written into the corresponding data shard of the second shard cluster are compared with the data files and index files of the specified object extracted from the first data shard of the first shard cluster. Specifically, the file size, file attributes, etc. are compared to verify whether the data files and index files are complete before and after recovery. If they are complete, the verification is successful and the subsequent steps can be executed. If they are incomplete, the verification fails and the current recovery task fails.

[0108] Execution content three: Generate recovery statistics, which are used to record the number of configuration information, data files, index files, and metadata written to the second shard cluster. For example, how many index files and data files are restored in the second shard cluster for the specified object.

[0109] In a specific implementation, the method provided in this embodiment may further include the following verification steps:

[0110] Verification step one: compare the configuration information, data files, index files, and metadata of the specified object restored in the second shard cluster with the configuration information, data files, index files, and metadata written to the second shard cluster recorded in the recovery statistics to verify the integrity of the specified object restored in the second shard cluster.

[0111] The purpose of this verification is to verify the integrity of the specified object restored in the second shard cluster, mainly based on the number of configuration information, data files, index files, and metadata written to the second shard cluster recorded in the generated recovery statistics. If the configuration information, data files, index files, and metadata of the specified object restored in the second shard cluster are consistent with the number recorded in the recovery statistics, the verification is successful and the subsequent verification steps can be executed. If the configuration information, data files, index files, and metadata of the specified object restored in the second shard cluster are inconsistent with the number recorded in the recovery statistics, the verification fails and the backup and recovery task fails.

[0112] Verification step 2: Perform an access test on the specified object restored in the second shard cluster to verify the correctness of access to the specified object restored in the second shard cluster. For example, set an access request for the specified object to verify whether it responds and whether the response result is correct. If the response is correct and the response result is correct, the verification is successful and the subsequent verification steps can be performed. If the response is incorrect or the response result is incorrect, the verification fails and the backup and recovery task fails.

[0113] Verification step three: perform an access test on the first object in the second shard cluster to verify the correctness of access to the second shard cluster. The first object is any database or data table in the second shard cluster. For example, set an access request for a random object (the first object) (such as deleting data, changing data, inserting data, etc.) to verify whether there is a response and whether the response result is correct. If the response is correct and the response result is correct, the verification is successful and the backup and recovery task is successful. If the response is incorrect or the response result is incorrect, the verification fails and the backup and recovery task fails.

[0114] The first embodiment described above provides a data backup and recovery method, which is a solution that combines the high efficiency advantage of the physical backup solution and the fine-grained advantage of the logical backup solution, and can perform physical backup and recovery at the library / table level on a sharded cluster. It should be noted that the examples in the first embodiment are only for explaining the method described in this application, and are not intended to be used as a limitation for actual use. The data backup and recovery method provided in this application includes but is not limited to the method described in the first embodiment.

[0115] The second embodiment of the present application provides a data backup and recovery method, which uses a specific example to exemplify the method described in the first embodiment of the present application. In this embodiment, the method is applied to a sharded cluster MongoDB, and the first sharded cluster is defined as a source cluster, and the second sharded cluster is defined as a target cluster.

[0116] The data backup and recovery method provided in this embodiment is implemented based on the WiredTiger storage engine, and can support multi-shard clusters instead of being limited to a single replica set; it can support the behavior of specifying data sources and specifying libraries / tables, but not necessarily objects started by the MongoDB process; it can ensure that the backed-up library / table metadata exists normally, without the need to re-create the shard key (Shard Key), index (index), etc.; it can ensure the integrity of data backup / recovery.

[0117] Figure 3 It is a schematic diagram of the data backup and recovery method provided in this embodiment.

[0118] like Figure 3 As shown, the method provided in this embodiment processes the data files and configuration information separately to achieve synchronous backup and recovery of the data files and configuration information. Specifically, the following steps may be included:

[0119] Step S311, for data files, parse metadata according to the library / table name specified by the user, obtain the storage path of the specified library / table in the source cluster, and then extract the data files and index files of the specified library / table from the source cluster according to the storage path to realize data file and index file backup.

[0120] Step S312, for configuration information, there is only one configuration shard and multiple metadata-related files in MongoDB, such as collection (file of table-related metadata), database (file of database-related metadata), shard (file of shard-related metadata), and chunks (file of data block-related metadata). In this embodiment, the metadata file corresponding to the source cluster is parsed through the WiredTiger engine interface, and the configuration information of the specified library / table is extracted from it to realize configuration information backup.

[0121] Step S313, according to the number of shards and the library / table conditions of the target cluster and the source cluster, select the appropriate shard for data recovery to achieve the recovery of data files, index files and configuration information.

[0122] It should be noted that the data integrity and correctness are verified during the backup and recovery process of the data files and configuration information of the specified library / table, as well as after the recovery is completed, to ensure the integrity and correctness of the restored data in the target cluster.

[0123] The following is a detailed description of the above process:

[0124] The above-mentioned data backup and recovery method provided in this embodiment mainly involves 5 tools, namely wtdump (data backup tool), wtrestore (data recovery tool), configdump (configuration backup tool), configrestore (configuration recovery tool), and restorecheck (verification tool). Among them, the main function of wtdump is to realize the physical backup of the library / table granularity data files of the specified source cluster, support the cold backup of a single library of the specified source cluster to the specified output directory, support the backup of the table level granularity, and the index file associated with the backup table, generate metadata related to the backup file to the output directory, and verify the integrity of the backup file during the backup process. The main function of wtrestore is to realize the physical recovery of the library / table granularity data files of the specified target cluster, restore the data backup file generated by wtdump to the target library specified by the target cluster, support the recovery of the table level granularity, and modify the metadata related to the table to be restored in the target library so that it is correctly associated with the restored table, and verify the integrity of the restored file before and after the recovery, and output the restored statistical data after the recovery is completed. The main function of configdump is to implement physical backup of the library / table granularity configuration information of the specified source cluster, and to cold backup the configuration information of the specified library / table in the source cluster configuration information to the output path specified by the user. The main function of restorecheck is to implement physical recovery of the library / table granularity configuration information of the specified target cluster, and to support restoring the configuration backup data generated by configdump to the target cluster. Specifically, the specified target cluster is connected through MongoDB Driver (MongoDB driver), and the library / table configuration information to be restored is correctly set to the configuration shard of the target cluster. The main function of restorecheck is to verify the target cluster after recovery, which mainly involves verification of the number of libraries, tables, documents, indexes, shard keys, index information, shard key information, database information, etc.

[0125] Based on this, the data backup and recovery process provided in this embodiment may further include the following steps S321 to S327:

[0126] Step S321, execute wtdump, obtain the parameters of the library / table to be backed up, perform physical backup of data files, extract data files and index files related to the library / table to be backed up from each shard backup directory, and generate backup metadata. In this process, it is not necessary to start the mongod process or mongos process of MongoDB.

[0127] Step S322, execute configdump, obtain the parameters of the library / table to be backed up, perform physical backup of configuration information, and extract configuration information related to the library / table to be backed up from the source cluster configuration shard backup directory.

[0128] Step S323, execute configrestore, obtain the parameters of the library / table to be restored and the target cluster access address, write the configuration information of the library / table to be restored to the target cluster, and generate a mapping relationship between the source cluster shard and the target cluster shard.

[0129] Step S324, executing wtrestore, and transferring the backup data files and backup metadata backed up by wtdump from each shard of the source cluster to the corresponding shard of the target cluster according to the mapping relationship between the shard of the source cluster and the shard of the target cluster.

[0130] Step S325: Execute mongod for each shard of the target cluster, copy the data files and index files related to the database / table to be restored to the database directory of the target cluster, and write necessary metadata to complete the physical restoration. At the same time, generate the restoration benchmark data.

[0131] Step S326, execute restorecheck, and verify the integrity and correctness of the target cluster after recovery based on the address of the target cluster and the recovery benchmark data of the library / table to be verified, mainly including the relevant statistics and accessibility of the restored library / table.

[0132] Step S327, after verification, complete the physical backup and recovery of the sharded cluster.

[0133] Data backup and recovery mainly involves two important aspects, namely, data structure and implementation process. Among them, the data structure defines how data is organized, stored, and ultimately restored. The selection and design of the data structure directly affects the efficiency and effect of backup and recovery. The implementation process defines the specific operating steps and technical means to ensure the smooth progress of backup and recovery operations. Therefore, the following details the backup and recovery of data files and the backup and recovery of configuration information from the two aspects of data structure and implementation process.

[0134] For data file backup and recovery:

[0135] In this embodiment, all data files and index files associated with the table to be backed up are searched according to the specified namespace (db.collection). Specifically, MongoDB encapsulates the special table metadata _mdb_catalog, which stores the correspondence between the table and its data files and index files. wtdump and wtrestore read and record the _mdb_catalog in the process execution directory through the WiredTiger engine, and search for data files and index files associated with the table to be backed up.

[0136] In this embodiment, in order to allow the target cluster to correctly identify and load the table to be restored, wtdump / wtrestore backs up and restores the MongoDB cluster metadata and WiredTiger engine metadata related to the specified table as shown in Table 1:

[0137] Table 1 Metadata for backup and recovery

[0138]

[0139] As shown in Table 1, the wtdump / wtrestore tool can restore five metadata items for a single table backup, including three WiredTiger engine metadata items required for indexing.

[0140] Optionally, the wtdump / wtrestore tool encapsulates the above metadata corresponding to a single table into a data structure NSSMetadata (metadata document). NSSMetadata consists of four parts as shown in Table 2:

[0141] Table 2 Components of the data structure NSSMetadata

[0142] Components Corresponding metadata metaContent _mdb_catalog metadata dataMetaInfo WiredTiger.wt Metadata sizeMeta sizeStorer.wt metadata indexMetaInfos Index metadata

[0143] As can be seen above, when wtdump performs physical backup at the library / table level, it not only backs up the data files and index files corresponding to the specified table, but also generates the data structure NSSMetadata corresponding to the table. When wtrestore performs physical recovery at the library / table level, it not only recovers the data files and index files of the specified table, but also sets the correct NSSMetadata metadata document of the table under the specified library according to the specific configuration of the target cluster, so as to ensure that the target cluster can be accessed normally after recovery.

[0144] Figure 4 This is a flow chart of the physical backup of data files provided by this embodiment.

[0145] like Figure 4As shown, the physical backup process of the data file may include the following steps S401 to S403:

[0146] Step S401, when wtdump is initialized, the metadata parser encapsulated inside it uses the interface of the WiredTiger engine to open a WT_CONNECTION to connect to the specified library of the source cluster and create a WT_SESSION; loads WiredTiger.wt in the form of a meta file; uses WT_CURSOR to scan and load the metadata associated with all tables under the specified library of the source cluster (such as app meta, colgroup meta, and file meta); opens sizeStorer.wt to scan and load the capacity information of all tables; scans the MongoDB special metadata table _mdb_catalog; loads the data files and index files of all tables, and associates and binds the metadata obtained above.

[0147] Step S402, when wtdump performs physical backup, it searches for the paths of metadata, data files, and index files corresponding to the table to be backed up one by one according to the directory of the table to be backed up obtained by parsing the input parameters, and generates multiple physical backups and subtasks with table as the granularity; starts a physical backup thread pool including multiple threads, and adds all backup subtasks to the thread pool work queue to wait for scheduling and execution.

[0148] Step S403, after the backup subtask corresponding to each table is executed, the data file corresponding to the table will be copied to the specified output directory, and the metadata document corresponding to the table will be written into the wtdump.meta of the metadata bson file in the specified output directory.

[0149] Figure 5 This is a flow chart of physical recovery of data files provided by this embodiment.

[0150] like Figure 5 As shown, the physical recovery process of the data file may include the following steps S501 to S502:

[0151] Step S501, encapsulate the physical recovery task as WTRestoreTask, after parsing the metadata of the source cluster and the target cluster and analyzing the recovery parameters, generate a table-level recovery subtask collectionRestoreTask list, start a physical recovery thread pool including multiple threads, and add the recovery subtask collectionRestoreTask list to the thread pool waiting for scheduling and execution.

[0152] In step S502, each recovery subtask will copy the data files and index files corresponding to the table to the directory corresponding to the target cluster, and set the metadata associated with the table in table:_mdb_catalog.wt, file:WireTiger.wt, and table:sizeStorer.wt of the target cluster based on the backup metadata loaded from the source cluster.

[0153] To back up and restore configuration information:

[0154] Configdump backs up the configuration information of the database / table to be backed up and restored in the sharded cluster, organizes the configuration information into a data structure CompleteConfigMetaInfo (configuration information document), and injects it into the target cluster after some adjustments. In order to ensure the correct access to the target cluster after restoration, CompleteConfigMetaInfo consists of at least four parts of data as shown in Table 3:

[0155] Table 3 Configuration data for backup and restoration

[0156] data effect config.shards Cluster sharding information config.databases The database information of the cluster, including the primary shard, whether sharding is enabled, etc. config.collections Cluster configuration information, including sharding keys, etc. config.chunks Database distribution information of the cluster, used to guide routing

[0157] configrestore physically restores the source cluster at the shard level. During the recovery process, it is necessary to find a corresponding target shard in the target cluster for each source shard in the source cluster and establish a mapping relationship between the source shard in the source cluster and the target shard in the target cluster. This mapping relationship is the basis for restoring data files and index files.

[0158] Optionally, this embodiment provides a shard mapping algorithm for the source cluster and the target cluster, which follows the following principles: first, only the mapping relationship between the source shards involved in the specified library / table and the target shards is found; the configuration of the primary shards of the existing library in the target cluster configuration node cannot be modified; second, when there are no shards in the table to be recovered, different source shards are allowed to be mapped to the same target shard; third, when there are shards in the table to be recovered, it is necessary to first verify the number of source shards and the number of target shards, and different shards of the source cluster are not allowed to be mapped to the same shard of the target cluster; fourth, the mapping of source shards and target shards is distributed as evenly as possible under permitted conditions to reduce the read and write load pressure of a single primary shard of the target cluster.

[0159] Figure 6 This is a flow chart of physical backup of configuration information provided in this embodiment.

[0160] like Figure 6 As shown, the physical backup process of the configuration information may include the following steps S601 to S602:

[0161] Step S601, loading the config.databases, config.collections, config.chunks, and config.shards configuration information tables without starting the configuration sharding of the source cluster.

[0162] Step S602: extract the configuration information of the specified library / table according to the backup parameters, convert it into the data structure CompleteConfigMetaInfo, and write it into the specified backup directory in the form of a bson file.

[0163] Figure 7 This is a flowchart of physical recovery of configuration information provided in this embodiment.

[0164] like Figure 7 As shown, the physical recovery process of the configuration information may include the following steps S701 to S704:

[0165] Step S701: extract and parse the configuration information of the library / table to be restored from the restoration parameters.

[0166] Step S702: Obtain a configuration information table in the target cluster.

[0167] Step S703: Calculate the mapping relationship between the source shard and the target shard of the library / table to be restored.

[0168] Step S704: adjust the extracted configuration information, and insert the configuration information related to the library / table to be restored into the target cluster.

[0169] In this embodiment, a verification scheme is also provided to ensure the security of backup and recovery during the data backup and recovery process, which mainly verifies from three aspects: the integrity of the physical backup of the data, the integrity of the data before and after physical recovery, and the correctness and integrity of the target cluster after physical recovery of the data.

[0170] The integrity check of physical data backup is mainly based on wtdump when performing physical file backup. wtdump compares the size and file attribute information of the physical files of each source shard and the physical files in the target directory before and after the backup. If the check fails, the backup task fails. The physical backup output records metadata such as the size and file attributes of each backed up physical file.

[0171] The integrity check before and after physical data recovery is mainly implemented based on wtrestore when performing physical file recovery. First, the wtrestore tool parses the metadata in the input backup before recovery, and compares the file size recorded in the metadata with the actual file size in the input directory. If it fails, the recovery task fails. Secondly, wtrestore verifies the physical file size in the target cluster directory after recovery. If it is inconsistent with the physical file size in the source cluster, the recovery task fails.

[0172] The correctness and integrity of the target cluster after physical data recovery are mainly achieved during the verification phase using the restorecheck tool. Figure 8 This is a flow chart of verifying a target cluster provided in this embodiment.

[0173] like Figure 8 As shown, the correctness and integrity verification process of the target cluster may include the following steps S801 to S802:

[0174] Step S801: After wtrestore is restored, it will generate recovery statistics about the recovery library / table, specifically corresponding to how many indexes and rows of records are restored for each table. Therefore, in the verification phase, wtrestore cooperates with the automated process to capture the recovery statistics. Restorecheck checks whether the number of indexes, records, shard keys, etc. restored in the target cluster are equal to the baseline recovery data based on the recovery statistics of each table. In addition, for each table with records, one record is used to check whether it can be accessed normally.

[0175] Step S802, perform a series of CRUD (Create, Read, Update, Delete) operations in the target cluster to verify whether the restored cluster can work normally and whether the data is complete and accessible.

[0176] The second embodiment provides an optional data backup and recovery method, which implements physical cold backup and recovery at the library / table level on a MongoDB shard cluster. This method can simultaneously backup and restore index files and configuration information while backing up and restoring data files, and is a one-step backup and recovery solution. This method not only solves the problems of long and inefficient backup and recovery processes and the inability to backup and restore index files and configuration information that require manual creation in existing logical backup solutions, but also solves the problem of large backup and recovery granularity in existing physical backup and recovery solutions.

[0177] The method provided in this embodiment can be applied in various business scenarios. Taking the online game business as an example, the method provided in this embodiment is applicable to the following data backup and recovery scenarios: a multi-server same cluster project, during maintenance, you want to back up the data of a certain cluster in it, rather than the data of the entire server; a single cluster of the project has data from multiple servers, and different servers are stored in separate libraries or tables. During project maintenance, the data of a certain server is abnormal and needs to be rolled back to the data of a certain server (corresponding to a certain library or table) (for example, a sudden abnormality in the merged server); in daily project operations, it is necessary to confirm the data status of a certain table through backup and then review the player data; after the project data is restored offline, it is necessary to restore some required library and table data to an online test cluster to start the game to confirm the data status or conduct related gameplay tests; and from time to time, designated data is restored to the office environment for testing.

[0178] After verification in online gaming business, the method provided by this embodiment not only supports more flexible granular management, but also the use analysis and performance testing of some online projects, and its backup and recovery efficiency is more than 3-6 times that of the traditional solution, compared with the traditional solution of restoring to the offline cluster and then using mongodump / mongorestore for logical backup and recovery. In addition, the lower the proportion of data in the database / table to be restored in the entire cluster, the more obvious the backup and recovery efficiency of this solution is. The backup and recovery process based on the traditional solution requires several hours to execute, while the method provided by this embodiment can be completed in just a few minutes, greatly improving the efficiency of business rollback in extreme cases.

[0179] It should be noted that the examples in the second embodiment are only for explaining the method described in this application and are not intended to be limiting for actual use. The data backup and recovery method provided in this application includes but is not limited to the method described in the second embodiment.

[0180] The third embodiment of the present application provides a data backup and recovery tool. Fig. 9 It is a schematic diagram of the data backup and recovery tool provided in this embodiment.

[0181] like Fig. 9 As shown, the data backup and recovery tool provided in this embodiment includes: a data backup module 901 , a configuration backup module 902 , a data recovery module 903 , and a configuration recovery module 904 .

[0182] The data backup module 901 is used to obtain metadata corresponding to the specified object from the metadata list of the first shard cluster according to the object parameters of the specified object, and extract the data file and index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object; wherein the specified object includes a database or a data table in the first shard cluster, the object parameter is obtained in response to a backup and recovery instruction, and is used to indicate the first shard cluster to which the specified object belongs and the namespace of the specified object, the metadata at least records the correspondence between the specified object and the data file and index file of the specified object, and the first data shard is at least one data shard that stores the data file and index file of the specified object among the multiple data shards included in the first shard cluster.

[0183] The configuration backup module 902 is used to extract the configuration information of the specified object from the configuration shard of the first shard cluster according to the object parameters, and establish a mapping relationship between the first data shard in the first shard cluster and the first number of data shards in the second shard cluster according to the configuration information; wherein the second shard cluster is specified by the backup and recovery instruction, and the first number is the number of the first data shards.

[0184] The data recovery module 903 is used to write the data file and index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and write the metadata corresponding to the specified object into the metadata list of the second shard cluster.

[0185] The configuration recovery module 904 is used to write the configuration information of the specified object into the configuration shard of the second shard cluster.

[0186] Optionally, the data backup and recovery tool provided in this embodiment further includes: a verification module;

[0187] The verification module is used to verify the integrity of the specified object restored in the second shard cluster, the correctness of access to the specified object restored in the second shard cluster, and the correctness of access to the second shard cluster.

[0188] The fourth embodiment of the present application provides a data backup and recovery device. Fig.10 It is a structural diagram of the data backup and recovery device provided in this embodiment.

[0189] like Fig.10 As shown, the data backup and recovery device provided by this embodiment includes: an acquisition unit 1001, a first extraction unit 1002, a second extraction unit 1003, a first writing unit 1004, and a second writing unit 1005.

[0190] The acquisition unit 1001 is used to obtain object parameters of a specified object in response to a backup and recovery instruction, where the object parameters are used to indicate a first shard cluster to which the specified object belongs and a namespace of the specified object, and the specified object includes a database or a data table in the first shard cluster.

[0191] The first extraction unit 1002 is used to obtain the metadata corresponding to the specified object from the metadata list of the first shard cluster according to the object parameters, and extract the configuration information of the specified object from the configuration shard of the first shard cluster, and the metadata at least records the correspondence between the specified object and the data file and index file of the specified object.

[0192] Optionally, acquiring metadata corresponding to the specified object from a metadata list of the first shard cluster according to the object parameter includes:

[0193] Parsing the metadata list of the first shard cluster;

[0194] According to the namespace of the specified object, metadata corresponding to the specified object is extracted from the metadata list, where the metadata includes at least first metadata for recording a correspondence between the specified object and a data file and an index file of the specified object, second metadata for recording a file status of the data file and the index file of the specified object, third metadata for recording creation information of the first shard cluster, fourth metadata for recording file information of the first shard cluster, and fifth metadata for recording checkpoint information of the data file and the index file of the specified object in the first shard cluster:

[0195] encapsulating the metadata corresponding to the designated object into a metadata document, wherein the metadata document at least includes a first sub-data document corresponding to the first metadata, a second sub-data document corresponding to the second metadata, a third sub-data document corresponding to the third metadata, the fourth metadata, and the fifth metadata, and a fourth sub-data document corresponding to index metadata of the designated object, wherein the index metadata is metadata related to the index in the metadata corresponding to the designated object;

[0196] Optionally, extracting the configuration information of the specified object from the configuration shard of the first shard cluster includes:

[0197] Parsing a configuration information list stored in a configuration shard of the first shard cluster, and extracting at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list;

[0198] Extracting configuration information corresponding to the specified object from the extracted shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table according to the namespace of the specified object;

[0199] The extracted configuration information corresponding to the specified object is processed and encapsulated into a configuration information document.

[0200] The second extraction unit 1003 is used to extract the data file and index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object. The first data shard is at least one data shard that stores the data file and index file of the specified object among the multiple data shards included in the first shard cluster.

[0201] Optionally, extracting the data file and the index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object includes:

[0202] Determine a storage path of the data file and the index file of the specified object in the first shard cluster according to a correspondence between the specified object recorded in the metadata corresponding to the specified object and the data file and the index file of the specified object;

[0203] Determine the first data shard storing the data file and the index file of the specified object according to the storage path;

[0204] The data file and index file of the specified object are extracted from the first data slice.

[0205] The first writing unit 1004 is used to establish a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in the second shard cluster according to the configuration information, and write the configuration information of the specified object into the configuration shard of the second shard cluster, where the second shard cluster is specified by the backup and recovery instruction, and the first number is the number of the first data shards.

[0206] Optionally, establishing a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in the second shard cluster according to the configuration information includes:

[0207] Parsing the configuration information list stored in the configuration shard of the second shard cluster, and extracting at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list;

[0208] Determine a mapping relationship between the first data shard in the first shard cluster and the data shard in the second shard cluster based on the extracted shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table and the configuration information corresponding to the specified object.

[0209] Optionally, writing the configuration information of the specified object into the configuration shard of the second shard cluster includes:

[0210] Modify the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the configuration shard of the second shard cluster according to the configuration information document;

[0211] Delete the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table in the configuration information list stored in the configuration shard of the second shard cluster, and insert the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table that have been changed according to the configuration information document into the configuration information list stored in the configuration shard of the second shard cluster.

[0212] The second writing unit 1005 is used to write the data file and index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and write the metadata corresponding to the specified object into the metadata list of the second shard cluster.

[0213] Optionally, writing the metadata corresponding to the specified object into the metadata list of the second shard cluster includes:

[0214] The metadata document is written into the metadata list of the second shard cluster.

[0215] Optionally, the device further comprises a verification unit;

[0216] After the step of extracting the data file and the index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object, the verification unit is used to:

[0217] Perform a file status comparison on the data file and index file of the specified object extracted from the first data shard of the first shard cluster with the data file and index file of the specified object stored in the first data shard of the first shard cluster to verify the integrity of the extracted data file and index file of the specified object.

[0218] After the step of writing the data file and index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and writing the metadata corresponding to the specified object into the metadata list of the second shard cluster, the verification unit is used to:

[0219] Performing a data status comparison on the metadata corresponding to the specified object written in the metadata list of the second shard cluster and the metadata corresponding to the specified object obtained from the metadata list of the first shard cluster to verify the integrity of the metadata corresponding to the specified object written in the metadata list of the second shard cluster;

[0220] Performing a file status comparison on the data file and index file of the specified object written in the corresponding data shard of the second shard cluster with the data file and index file of the specified object extracted from the first data shard of the first shard cluster to verify the integrity of the data file and index file of the specified object written in the corresponding data shard of the second shard cluster;

[0221] Generate recovery statistics, where the recovery statistics are used to record the number of the configuration information, the data files, the index files, and the metadata written to the second shard cluster.

[0222] The verification unit is used for:

[0223] Performing a quantitative comparison between the configuration information, data files, index files, and metadata of the specified object restored in the second shard cluster and the configuration information, data files, index files, and metadata written to the second shard cluster recorded in the restored statistical data, so as to verify the integrity of the specified object restored in the second shard cluster;

[0224] Performing an access test on the designated object restored in the second shard cluster to verify correctness of access to the designated object restored in the second shard cluster;

[0225] An access test is performed on a first object in the second shard cluster to verify correctness of access to the second shard cluster, where the first object is any database or data table in the second shard cluster.

[0226] A fifth embodiment of the present application provides an electronic device, Fig.11 It is a schematic diagram of the structure of the electronic device provided in this embodiment.

[0227] like Figure 8 As shown, the electronic device provided in this embodiment includes: a memory 1101 and a processor 1102;

[0228] The memory 1101 is used to store computer instructions for executing the data backup and recovery method;

[0229] The processor 1102 is configured to execute computer instructions stored in the memory 1101 to perform the following operations:

[0230] In response to the backup and restore instruction, obtain object parameters of a specified object, where the object parameters are used to indicate a first shard cluster to which the specified object belongs and a namespace of the specified object, where the specified object includes a database or a data table in the first shard cluster;

[0231] Acquire metadata corresponding to the specified object from a metadata list of the first shard cluster according to the object parameters, and extract configuration information of the specified object from a configuration shard of the first shard cluster, wherein the metadata at least records a correspondence between the specified object and a data file and an index file of the specified object;

[0232] Extracting, according to the metadata corresponding to the specified object, a data file and an index file of the specified object from a first data shard of the first shard cluster, where the first data shard is at least one data shard storing the data file and the index file of the specified object among the multiple data shards included in the first shard cluster;

[0233] Establishing a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in the second shard cluster according to the configuration information, and writing the configuration information of the specified object into the configuration shard of the second shard cluster, where the second shard cluster is specified by the backup and restore instruction, and the first number is the number of the first data shards;

[0234] Based on the mapping relationship, the data file and the index file of the specified object are written into the corresponding data shard of the second shard cluster, and the metadata corresponding to the specified object is written into the metadata list of the second shard cluster.

[0235] Optionally, after the step of extracting the data file and the index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object, the following is further performed:

[0236] Perform a file status comparison on the data file and index file of the specified object extracted from the first data shard of the first shard cluster with the data file and index file of the specified object stored in the first data shard of the first shard cluster to verify the integrity of the extracted data file and index file of the specified object.

[0237] Optionally, after the step of writing the data file and the index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and writing the metadata corresponding to the specified object into the metadata list of the second shard cluster, the following is further performed:

[0238] Performing a data status comparison on the metadata corresponding to the specified object written in the metadata list of the second shard cluster and the metadata corresponding to the specified object obtained from the metadata list of the first shard cluster to verify the integrity of the metadata corresponding to the specified object written in the metadata list of the second shard cluster;

[0239] Performing a file status comparison on the data file and index file of the specified object written in the corresponding data shard of the second shard cluster with the data file and index file of the specified object extracted from the first data shard of the first shard cluster to verify the integrity of the data file and index file of the specified object written in the corresponding data shard of the second shard cluster;

[0240] Generate recovery statistics, where the recovery statistics are used to record the number of the configuration information, the data files, the index files, and the metadata written to the second shard cluster.

[0241] Optionally, also execute:

[0242] Performing a quantitative comparison between the configuration information, data files, index files, and metadata of the specified object restored in the second shard cluster and the configuration information, data files, index files, and metadata written to the second shard cluster recorded in the restored statistical data, so as to verify the integrity of the specified object restored in the second shard cluster;

[0243] Performing an access test on the designated object restored in the second shard cluster to verify correctness of access to the designated object restored in the second shard cluster;

[0244] An access test is performed on a first object in the second shard cluster to verify correctness of access to the second shard cluster, where the first object is any database or data table in the second shard cluster.

[0245] Optionally, acquiring metadata corresponding to the specified object from a metadata list of the first shard cluster according to the object parameter includes:

[0246] Parsing the metadata list of the first shard cluster;

[0247] According to the namespace of the specified object, metadata corresponding to the specified object is extracted from the metadata list, where the metadata includes at least first metadata for recording a correspondence between the specified object and a data file and an index file of the specified object, second metadata for recording a file status of the data file and the index file of the specified object, third metadata for recording creation information of the first shard cluster, fourth metadata for recording file information of the first shard cluster, and fifth metadata for recording checkpoint information of the data file and the index file of the specified object in the first shard cluster:

[0248] encapsulating the metadata corresponding to the designated object into a metadata document, wherein the metadata document at least includes a first sub-data document corresponding to the first metadata, a second sub-data document corresponding to the second metadata, a third sub-data document corresponding to the third metadata, the fourth metadata, and the fifth metadata, and a fourth sub-data document corresponding to index metadata of the designated object, wherein the index metadata is metadata related to the index in the metadata corresponding to the designated object;

[0249] The step of writing the metadata corresponding to the specified object into the metadata list of the second shard cluster includes:

[0250] The metadata document is written into the metadata list of the second shard cluster.

[0251] Optionally, extracting the data file and the index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object includes:

[0252] Determine a storage path of the data file and the index file of the specified object in the first shard cluster according to a correspondence between the specified object recorded in the metadata corresponding to the specified object and the data file and the index file of the specified object;

[0253] Determine the first data shard storing the data file and the index file of the specified object according to the storage path;

[0254] The data file and index file of the specified object are extracted from the first data slice.

[0255] Optionally, extracting the configuration information of the specified object from the configuration shard of the first shard cluster includes:

[0256] Parsing a configuration information list stored in a configuration shard of the first shard cluster, and extracting at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list;

[0257] Extracting configuration information corresponding to the specified object from the extracted shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table according to the namespace of the specified object;

[0258] The extracted configuration information corresponding to the specified object is processed and encapsulated into a configuration information document.

[0259] Optionally, establishing a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in the second shard cluster according to the configuration information includes:

[0260] Parsing the configuration information list stored in the configuration shard of the second shard cluster, and extracting at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list;

[0261] Determine a mapping relationship between the first data shard in the first shard cluster and the data shard in the second shard cluster based on the extracted shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table and the configuration information corresponding to the specified object.

[0262] Optionally, writing the configuration information of the specified object into the configuration shard of the second shard cluster includes:

[0263] Modify the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the configuration shard of the second shard cluster according to the configuration information document;

[0264] Delete the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table in the configuration information list stored in the configuration shard of the second shard cluster, and insert the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table that have been changed according to the configuration information document into the configuration information list stored in the configuration shard of the second shard cluster.

[0265] A sixth embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes computer instructions, and when the computer instructions are executed by a processor, they are used to implement the methods described in the embodiments of the present application.

[0266] It should be noted that the relational terms such as "first" and "second" in this article are only used to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or order between these entities or operations. In addition, the words "include", "have", "include" and "includes" and other similar forms are the same in meaning, and the end of any one or more items after any of the above words is open-ended, and any of the above nouns does not mean that the one or more items have been exhaustively listed or are limited to the one or more items listed.

[0267] As used herein, unless expressly stated otherwise, the term "or" includes all possible combinations, except those that are not feasible. For example, if it is expressed that a database may include A or B, then unless otherwise specifically stated or not feasible, it may include database A, or B, or A and B. As a second example, if it is expressed that a database may include A, B, or C, then unless otherwise specifically stated or not feasible, the database may include database A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0268] It is worth noting that the above embodiments can be implemented by hardware or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above-mentioned computer-readable medium. When the software is executed by a processor, it can execute the above-mentioned disclosed method. The computing unit and other functional units described in the present disclosure can be implemented by hardware or software, or a combination of hardware and software. Those of ordinary skill in the art will also understand that the above-mentioned multiple modules / units can be combined into one module / unit, and each of the above-mentioned modules / units can be further divided into multiple sub-modules / sub-units.

[0269] In the above detailed description, the embodiments have been described with reference to many specific details, which may vary from implementation to implementation. Certain adaptations and modifications may be made to the embodiments. For those skilled in the art, other embodiments may be readily apparent from the specific embodiments disclosed in the present application. This specification and examples are for exemplary purposes only, and the true scope and essence of the present application are described by the claims. The sequence of steps shown in the diagrams is also for the purpose of explanation only and is not intended to be limited to any particular steps or sequence. Therefore, those skilled in the art will appreciate that these steps may be performed in different orders when implementing the same method.

[0270] In the drawings and detailed description of the present application, exemplary embodiments are disclosed. However, many variations and modifications may be made to these embodiments. Accordingly, although specific terms are used, these terms are only general and descriptive and not for the purpose of limitation.

Claims

1. A data backup and recovery method, characterized in that: The method comprises: In response to the backup and restore instruction, obtain object parameters of a specified object, where the object parameters are used to indicate a first shard cluster to which the specified object belongs and a namespace of the specified object, where the specified object includes a database or a data table in the first shard cluster; Acquire metadata corresponding to the specified object from a metadata list of the first shard cluster according to the object parameters, and extract configuration information of the specified object from a configuration shard of the first shard cluster, wherein the metadata at least records a correspondence between the specified object and a data file and an index file of the specified object; Extracting, according to the metadata corresponding to the specified object, a data file and an index file of the specified object from a first data shard of the first shard cluster, where the first data shard is at least one data shard storing the data file and the index file of the specified object among the multiple data shards included in the first shard cluster; Establishing a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in the second shard cluster according to the configuration information, and writing the configuration information of the specified object into the configuration shard of the second shard cluster, where the second shard cluster is specified by the backup and restore instruction, and the first number is the number of the first data shards; Based on the mapping relationship, the data file and the index file of the specified object are written into the corresponding data shard of the second shard cluster, and the metadata corresponding to the specified object is written into the metadata list of the second shard cluster.

2. The method according to claim 1, characterized in that After the step of extracting the data file and the index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object, the method further includes: Perform a file status comparison on the data file and index file of the specified object extracted from the first data shard of the first shard cluster with the data file and index file of the specified object stored in the first data shard of the first shard cluster to verify the integrity of the extracted data file and index file of the specified object.

3. The method according to claim 2, characterized in that After the step of writing the data file and the index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and writing the metadata corresponding to the specified object into the metadata list of the second shard cluster, the method further includes: Performing a data status comparison on the metadata corresponding to the specified object written in the metadata list of the second shard cluster and the metadata corresponding to the specified object obtained from the metadata list of the first shard cluster to verify the integrity of the metadata corresponding to the specified object written in the metadata list of the second shard cluster; Performing a file status comparison on the data file and index file of the specified object written in the corresponding data shard of the second shard cluster with the data file and index file of the specified object extracted from the first data shard of the first shard cluster to verify the integrity of the data file and index file of the specified object written in the corresponding data shard of the second shard cluster; Generate recovery statistics, where the recovery statistics are used to record the number of the configuration information, the data files, the index files, and the metadata written to the second shard cluster.

4. The method according to claim 3, characterized in that The method further comprises: Performing a quantitative comparison between the configuration information, data files, index files, and metadata of the specified object restored in the second shard cluster and the configuration information, data files, index files, and metadata written to the second shard cluster recorded in the restored statistical data, so as to verify the integrity of the specified object restored in the second shard cluster; Performing an access test on the designated object restored in the second shard cluster to verify correctness of access to the designated object restored in the second shard cluster; An access test is performed on a first object in the second shard cluster to verify correctness of access to the second shard cluster, where the first object is any database or data table in the second shard cluster.

5. The method according to claim 1, characterized in that: The acquiring metadata corresponding to the specified object from the metadata list of the first shard cluster according to the object parameter includes: Parsing the metadata list of the first shard cluster; According to the namespace of the specified object, metadata corresponding to the specified object is extracted from the metadata list, where the metadata includes at least first metadata for recording a correspondence between the specified object and a data file and an index file of the specified object, second metadata for recording a file status of the data file and the index file of the specified object, third metadata for recording creation information of the first shard cluster, fourth metadata for recording file information of the first shard cluster, and fifth metadata for recording checkpoint information of the data file and the index file of the specified object in the first shard cluster: encapsulating the metadata corresponding to the designated object into a metadata document, wherein the metadata document at least includes a first sub-data document corresponding to the first metadata, a second sub-data document corresponding to the second metadata, a third sub-data document corresponding to the third metadata, the fourth metadata, and the fifth metadata, and a fourth sub-data document corresponding to index metadata of the designated object, wherein the index metadata is metadata related to the index in the metadata corresponding to the designated object; The step of writing the metadata corresponding to the specified object into the metadata list of the second shard cluster includes: The metadata document is written into the metadata list of the second shard cluster.

6. The method according to claim 1, characterized in that The step of extracting the data file and the index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object includes: Determine a storage path of the data file and the index file of the specified object in the first shard cluster according to a correspondence between the specified object recorded in the metadata corresponding to the specified object and the data file and the index file of the specified object; Determine the first data shard storing the data file and the index file of the specified object according to the storage path; The data file and index file of the specified object are extracted from the first data slice.

7. The method according to claim 1, characterized in that The extracting the configuration information of the specified object from the configuration shard of the first shard cluster includes: Parsing a configuration information list stored in a configuration shard of the first shard cluster, and extracting at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list; Extracting configuration information corresponding to the specified object from the extracted shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table according to the namespace of the specified object; The extracted configuration information corresponding to the specified object is processed and encapsulated into a configuration information document.

8. The method according to claim 7, characterized in that The establishing, according to the configuration information, a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in the second shard cluster includes: Parsing the configuration information list stored in the configuration shard of the second shard cluster, and extracting at least a shard configuration information table, a database configuration information table, a data table configuration information table, and a data block configuration information table from the configuration information list; Determine a mapping relationship between the first data shard in the first shard cluster and the data shard in the second shard cluster based on the extracted shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table and the configuration information corresponding to the specified object.

9. The method according to claim 8, characterized in that Writing the configuration information of the specified object into the configuration shard of the second shard cluster includes: Modify the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table extracted from the configuration shard of the second shard cluster according to the configuration information document; Delete the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table in the configuration information list stored in the configuration shard of the second shard cluster, and insert the shard configuration information table, the database configuration information table, the data table configuration information table, and the data block configuration information table that have been changed according to the configuration information document into the configuration information list stored in the configuration shard of the second shard cluster.

10. A data backup and recovery tool, characterized in that: The tool includes a data backup module, a configuration backup module, a data recovery module, and a configuration recovery module; The data backup module is configured to obtain metadata corresponding to a specified object from a metadata list of a first shard cluster according to an object parameter of the specified object, and extract data files and index files of the specified object from a first data shard of the first shard cluster according to the metadata corresponding to the specified object; wherein the specified object includes a database or a data table in the first shard cluster, the object parameter is obtained in response to a backup and recovery instruction, and is used to indicate the first shard cluster to which the specified object belongs and a namespace of the specified object, the metadata at least records a correspondence between the specified object and the data files and index files of the specified object, and the first data shard is at least one data shard storing the data files and index files of the specified object among a plurality of data shards included in the first shard cluster; The configuration backup module is used to extract the configuration information of the specified object from the configuration shard of the first shard cluster according to the object parameters, and establish a mapping relationship between the first data shard in the first shard cluster and the first number of data shards in the second shard cluster according to the configuration information; wherein the second shard cluster is specified by the backup and recovery instruction, and the first number is the number of the first data shards; The data recovery module is used to write the data file and index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and write the metadata corresponding to the specified object into the metadata list of the second shard cluster; The configuration recovery module is used to write the configuration information of the specified object into the configuration shard of the second shard cluster.

11. The tool according to claim 10, characterized in that The tool also includes a verification module; The verification module is used to verify the integrity of the specified object restored in the second shard cluster, the correctness of access to the specified object restored in the second shard cluster, and the correctness of access to the second shard cluster.

12. A data backup and recovery device, characterized in that: The device comprises: an acquisition unit, a first extraction unit, a second extraction unit, a first writing unit, and a second writing unit; The acquisition unit is used to acquire object parameters of a specified object in response to a backup and recovery instruction, where the object parameters are used to indicate a first shard cluster to which the specified object belongs and a namespace of the specified object, where the specified object includes a database or a data table in the first shard cluster; The first extraction unit is used to obtain metadata corresponding to the specified object from the metadata list of the first shard cluster according to the object parameters, and extract configuration information of the specified object from the configuration shard of the first shard cluster, wherein the metadata at least records the corresponding relationship between the specified object and the data file and index file of the specified object; The second extraction unit is used to extract the data file and the index file of the specified object from the first data shard of the first shard cluster according to the metadata corresponding to the specified object, where the first data shard is at least one data shard storing the data file and the index file of the specified object among the multiple data shards included in the first shard cluster; The first writing unit is used to establish a mapping relationship between the first data shard in the first shard cluster and a first number of data shards in the second shard cluster according to the configuration information, and write the configuration information of the specified object into the configuration shard of the second shard cluster, where the second shard cluster is specified by the backup and recovery instruction, and the first number is the number of the first data shards; The second writing unit is used to write the data file and index file of the specified object into the corresponding data shard of the second shard cluster based on the mapping relationship, and write the metadata corresponding to the specified object into the metadata list of the second shard cluster.

13. An electronic device, characterized in that: include: Memory, processor; The memory is used to store one or more computer instructions; The processor is used to execute the one or more computer instructions to implement the method according to any one of claims 1-9.

14. A computer-readable storage medium having one or more computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the method according to any one of claims 1 to 9 is performed.

Images