A sharding information backup method and device of a distributed database
By acquiring and parsing the data distribution information of the distributed database, the problem of the inability to back up shard information in existing technologies is solved, enabling efficient backup of data table shard information and ensuring business continuity and data recovery for financial institutions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WEBANK (CHINA)
- Filing Date
- 2021-06-23
- Publication Date
- 2026-06-05
Smart Images
Figure CN113407378B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the database field within the Fintech sector, and more particularly to a method and apparatus for sharded information backup of a distributed database. Background Technology
[0002] With the development of computer technology, more and more technologies are being applied in the financial field, and the traditional financial industry is gradually transforming into Fintech. However, due to the security and real-time requirements of the financial industry, higher demands are being placed on technology. Currently, financial data has high fault tolerance requirements; therefore, distributed databases (such as MongoDB) are often used to store financial data. This means that the data in a distributed database is distributed across multiple database instances (e.g., one database instance per host), and each database instance stores only a portion of the data from the distributed database. In some cases, it is necessary to back up the distributed database.
[0003] However, current backups of distributed databases only back up the data records themselves. The specific database instance where each table's data records are stored depends on the table's sharding information (one or more column names). Furthermore, distributed databases do not provide interfaces or commands to retrieve this sharding information, so backing up the sharding information of tables in a distributed database is currently not possible. This is a problem that urgently needs to be solved. Summary of the Invention
[0004] This invention provides a method and apparatus for backing up sharded information of a distributed database, which solves the problem in the prior art that sharded information of data tables in a distributed database cannot be backed up.
[0005] In a first aspect, the present invention provides a method for backing up sharded information in a distributed database, comprising:
[0006] Obtain data distribution information of a distributed database; the data distribution information indicates the distribution of data records of each data table in the distributed database across multiple database instances;
[0007] The data distribution information is converted into information with a preset structure.
[0008] For any data table in the aforementioned data tables, locate the sub-information of the data table from the information in the preset structure; parse the sub-information to extract the fragmentation information of the data table;
[0009] Back up the sharding information of each data table.
[0010] In the above method, after converting the data distribution information into information with a preset structure, the sub-information of the data table can be located from the information with the preset structure. Then, the sub-information can be parsed to extract the sharding information of the data table. Based on the data distribution information, the sharding information of the data table can be directly extracted through data processing and transformation, thereby enabling the backup of the sharding information of the data table.
[0011] Optionally, converting the data distribution information into information with a preset structure includes:
[0012] The data distribution information is converted into a binary data input stream, and the data input stream is converted into at least one string according to a preset format;
[0013] The at least one string is used as information in the preset structure.
[0014] In the above method, the data distribution information is converted into a binary data input stream, and the data input stream is converted into at least one string according to a preset format, thereby standardizing and structuring the data distribution information. The information obtained by the preset structure can be more convenient for data processing and for extracting the fragmentation information of each data table.
[0015] Optionally, locating the sub-information of the data table from the information of the preset structure includes: for a first string, where the first string is any one of the at least one strings, if the first string is a string of database attributes, and the database variable in the first string indicates the distributed database, and the data table variable in the first string indicates the data table, then the strings within a preset position range corresponding to the first string are determined to be the sub-information of the data table, and the strings within the preset position range include the first string.
[0016] In the above method, the sub-information of the data table is located through database attributes, database variables, data table variables, etc. Since the distributed database can be identified by the database attributes, and the database and data table are unique when they are created, the sub-information of the data table to be parsed can be located more efficiently and accurately from the information of the preset structure.
[0017] Optionally, parsing the sub-information and extracting the sharding information of the data table includes:
[0018] For any string within the preset position range, if the string is the first string, then the database name of the distributed database and the table name of the data table are parsed from the string; if the string includes a sharding identifier, then the sharding column name is parsed from the column name corresponding to the sharding identifier in the string; the database name, the table name, and the sharding column name are used as the sharding information of the data table.
[0019] In the above method, the first string is parsed to obtain the database name of the distributed database and the table name of the data table. The string including the sharding identifier is parsed to obtain the sharding column name. Thus, the database name, the table name, and the sharding column name are all obtained. Therefore, the sharding information of the data table can clearly represent the database table information and the basis for the distribution on the corresponding database instance.
[0020] Optionally, obtaining the data distribution information of the distributed database includes: querying the system table of the distributed database to obtain data block storage information in the plurality of database instances; and determining the data distribution information based on the data block storage information in the plurality of database instances.
[0021] In the above manner, by querying the system table of the distributed database, the data block storage information of the multiple database instances can be obtained, and the data distribution information can be obtained directly from the physical distribution, which is faster and more direct.
[0022] Optional, also includes:
[0023] Obtain the table index objects of each data table in the distributed database;
[0024] For any of the data tables, determine the index fields and index attributes of the data table based on the table index object of the data table;
[0025] Based on the index fields and index attributes of the data table, an index creation command is generated for the data table; the index creation command is used to back up the index information of the data table.
[0026] In the above method, after determining the index fields and index attributes of the data table, an index creation command for the data table is generated based on the index fields and index attributes of the data table. This saves the index of the data table in the form of an index creation command, thereby facilitating the recovery of the distributed database.
[0027] Optionally, determining the index fields and index attributes of the data table based on the table index object of the data table includes:
[0028] Fields whose field names match the index identifier in the index object are used as index fields of the data table;
[0029] Based on the value of the first field in the index object, determine whether the index of the data table is a unique index;
[0030] The expiration time of the index of the data table is determined based on the value of the second field in the index object.
[0031] In the above method, by matching index identifiers and the values of the first and second fields, the index fields and index attributes of the data table can be determined more accurately.
[0032] Optional, also includes:
[0033] For any one of the data tables, read the data records of that data table;
[0034] For any data record in the data table, obtain the default primary key in the data record. If the default primary key is a primary key of a preset data type, then convert the data record into a JSON string.
[0035] The JSON string is formatted according to the Structured Query Language (SQL) to generate an SQL command for the data record; the SQL command is used to back up the data record.
[0036] In the above method, for any data record, if the default primary key is a primary key of a preset data type, the data record can be converted into a JSON string, and then an SQL command for the data record can be generated. Since the primary key is unique in the data table, the basic information of the data record can be accurately preserved. Moreover, the storage space occupied by the JSON string is small, which can save the data record more directly and accurately, and generate an SQL command that can be executed directly, thereby facilitating the recovery of the data record.
[0037] Secondly, the present invention provides a sharding information backup device for a distributed database, comprising: a backup module for acquiring data distribution information of a distributed database; the data distribution information indicating the distribution of data records of each data table of the distributed database across multiple database instances; and converting the data distribution information into information of a preset structure.
[0038] The parsing module is used to locate the sub-information of any data table from the information of the preset structure for any data table in the data tables; parse the sub-information and extract the fragmentation information of the data table;
[0039] The backup module is also used to back up the sharding information of each data table.
[0040] Optionally, the parsing module is specifically used for:
[0041] The data distribution information is converted into a binary data input stream, and the data input stream is converted into at least one string according to a preset format;
[0042] The at least one string is used as information in the preset structure.
[0043] Optionally, the parsing module is specifically used for:
[0044] For the first string, which is any one of the at least one strings, if the first string is a database attribute string, and the database variable in the first string indicates the distributed database, and the data table variable in the first string indicates the data table, then the strings within a preset position range corresponding to the first string are determined to be sub-information of the data table, and the strings within the preset position range include the first string.
[0045] Optionally, the parsing module is specifically used for:
[0046] For any string within the preset position range, if the string is the first string, then the database name of the distributed database and the table name of the data table are parsed from the string.
[0047] If the string contains a sharding identifier, then the sharding column name is parsed out based on the column name corresponding to the sharding identifier in the string;
[0048] The database name, the table name, and the sharding column name are used as the sharding information for the data table.
[0049] Optionally, the parsing module is specifically used to: query the system table of the distributed database to obtain data block storage information in the multiple database instances;
[0050] Based on the data block storage information in the multiple database instances, the data distribution information is determined by querying the system table of the distributed database to obtain the data block storage information of the multiple database instances;
[0051] The data distribution information is determined based on the data block storage information in the multiple database instances.
[0052] Optionally, the backup module is further configured to:
[0053] Obtain the table index objects of each data table in the distributed database;
[0054] For any of the data tables, determine the index fields and index attributes of the data table based on the table index object of the data table;
[0055] Based on the index fields and index attributes of the data table, an index creation command is generated for the data table; the index creation command is used to back up the index information of the data table.
[0056] Optionally, the backup module is specifically used for:
[0057] Fields whose field names match the index identifier in the index object are used as index fields of the data table;
[0058] Based on the value of the first field in the index object, determine whether the index of the data table is a unique index;
[0059] The expiration time of the index of the data table is determined based on the value of the second field in the index object.
[0060] Optionally, the backup module is further configured to:
[0061] For any one of the data tables, read the data records of that data table;
[0062] For any data record in the data table, obtain the default primary key in the data record. If the default primary key is a primary key of a preset data type, then convert the data record into a JSON string.
[0063] The JSON string is formatted according to the Structured Query Language (SQL) to generate an SQL command for the data record; the SQL command is used to back up the data record.
[0064] The beneficial effects of the second aspect and the various optional devices described above can be referred to the beneficial effects of the first aspect and the various optional methods described above, and will not be repeated here.
[0065] Thirdly, the present invention provides a computer device including a program or instructions, which, when executed, are used to perform the first aspect and the various optional methods of the first aspect.
[0066] Fourthly, the present invention provides a storage medium including a program or instructions, which, when executed, perform the first aspect and the various optional methods of the first aspect.
[0067] These or other aspects of the invention will become more apparent from the following description of the embodiments. Attached Figure Description
[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0069] Figure 1 This is a flowchart illustrating a method for backing up sharded information in a distributed database, as provided in an embodiment of the present invention.
[0070] Figure 2 This is a schematic diagram of the preset structure information in a distributed database sharding information backup method provided in an embodiment of the present invention;
[0071] Figure 3 A timing diagram corresponding to a distributed database sharding information backup method provided in an embodiment of the present invention;
[0072] Figure 4 This is a schematic diagram of the structure of a distributed database sharding information backup device provided in an embodiment of the present invention. Detailed Implementation
[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0074] In the operation of financial institutions (banks, insurance companies, or securities firms) (such as bank loan and deposit businesses), the data in a distributed database is stored across multiple database instances (e.g., one database instance per host). In some cases, it is necessary to back up the distributed database. Currently, backups of distributed databases are performed on the entire dataset, but not on the sharding information across multiple database instances. This does not meet the needs of banks and other financial institutions and cannot guarantee the efficient operation of their various business processes.
[0075] like Figure 1 As shown, the present invention provides a method for sharding information backup of a distributed database.
[0076] Step 101: Obtain the data distribution information of the distributed database.
[0077] Step 102: Convert the data distribution information into information with a preset structure.
[0078] Step 103: For any data table in the data tables, locate the sub-information of the data table from the information in the preset structure; parse the sub-information and extract the fragmentation information of the data table.
[0079] Step 104: Back up the sharding information of each data table.
[0080] For example, the distributed database mentioned above could be MongoDB. MongoDB is a database based on distributed file storage, a NoSQL database, and also a NoSQL database product. Sharding refers to the distribution of data across each instance in a distributed database. For example, MongoDB can distribute data across different MongoDB instances; each instance is a shard. For instance, table A has 30 million rows of data, and the database has 3 shards (3 machines), each shard can store 10 million rows of data. When querying a shard, only 1 / 3 of the original data range needs to be retrieved to find the desired data, greatly improving query efficiency. Sharding information refers to the column names specified to distribute data across different machines; these column names can also be called shard keys.
[0081] It should be noted that in current practical applications, backing up a complete distributed database also requires backing up the index information of each data table in the distributed database. The index information is used to restore the indexes of each data table in the distributed database. Based on the restored indexes of each data table in the distributed database, the data records to be backed up in each data table can be restored according to the indexes to obtain the data records of each data table. Then, based on the data distribution information, the sharding information of each data table can be parsed to obtain the data records of each data table. Therefore, backup can be performed based on the distributed database, each data table, and the sharding information of each data table.
[0082] It should be noted that in the actual software implementation, the database backup configuration file can be modified before step 101 to specify the distributed database to be connected to, and to specify which tables in the distributed database need to export data records. You can choose to back up part of the data in the distributed database. The data in each table can be all the data in the distributed database or part of the data. Each step can be encapsulated in a shell script.
[0083] Specifically, the shell script can encapsulate the `backup` and `restore` functions. The `backup` function executes steps 101 to 104. It can be encapsulated within a JAR file command to simplify user operations and reduce the probability of errors. The specific command is `backup.sh backup`. The `restore` function connects to the database and sequentially restores the indexes, data records, and shard information of each table. The specific command is `backup.sh restore`.
[0084] In one possible implementation, the steps for backing up the index information of each table in the distributed database can specifically be as follows:
[0085] Obtain the table index objects of each data table in the distributed database;
[0086] For any of the data tables, determine the index fields and index attributes of the data table based on the table index object of the data table;
[0087] Based on the index fields and index attributes of the data table, an index creation command is generated for the data table; the index creation command is used to back up the index information of the data table.
[0088] It should be noted that the table index object of each of the above data tables records the index fields and index attributes of that data table. There are several ways to determine the index fields and index attributes of a data table based on its table index object. For example, you can define variables in the table index object and record the index fields and index attributes by assigning values to the variables. Alternatively, you can define a function and record the index fields and index attributes by the result of the function's execution.
[0089] For example, the index creation command is: db.collection.createIndex(keys,options). db.collection.createIndex is the name of the index creation command, and keys and options are the parameters.
[0090] The `keys` parameter defines which fields comprise the index and can also include other information, such as the sorting order of the index fields, for example, `{acctNo:1,mntSysTime:1}`. The `options` parameter defines the index attributes, including whether to create the index in the background, whether it is unique, the index name, and the data expiration time.
[0091] In the above method, after determining the index fields and index attributes of the data table, an index creation command for the data table is generated based on the index fields and index attributes of the data table. This saves the index of the data table in the form of an index creation command, thereby facilitating the recovery of the distributed database.
[0092] In the above implementation, the specific process of determining the index fields and index attributes of the data table based on the table index object can be as follows:
[0093] Fields whose field names match the index identifier in the index object are used as index fields of the data table;
[0094] Based on the value of the first field in the index object, determine whether the index of the data table is a unique index;
[0095] The expiration time of the index of the data table is determined based on the value of the second field in the index object.
[0096] For example, the specific steps are as follows:
[0097] The process involves reading the names of each data table to facilitate retrieving the table's index and backing up data; traversing the data tables to obtain their table index objects; and parsing the index fields and attributes based on these objects. Simultaneously, it generates index creation commands based on the parsed index fields and attributes for easy index recovery later. A detailed example of the steps is as follows:
[0098] Assemble the keys parameter: retrieve the key field of the table index object, obtain the index field object, and convert the object into a string.
[0099] Assemble the options parameters:
[0100] Retrieve the background field from the table index object to determine whether the index needs to be created in the background, and whether the field can be null. It needs to be compatible with null values, which can be represented by the value of the background field.
[0101] The index name is obtained by retrieving the name field from the table index object.
[0102] Retrieve the unique field (first field) from the table index object to determine if the index is unique. This field may be null, and we need to accommodate null values.
[0103] Retrieve the `expireAfterSeconds` field (the second field) from the table index object to obtain the data expiration time. This field may be empty; an empty field indicates that the data will never expire. It's necessary to determine whether the expiration time is an integer or a floating-point number, and then obtain the appropriate data type value based on the data type.
[0104] The executable command for the index is obtained by concatenating strings and written to a file. Example: `db.table_name.createIndex({acctNo:1,mntSysTime:1},{name:idx_xxxx,background:xxx,expireAfterSeconds:xxx,unique:xxxx})`. Here, `createIndex` represents the index execution command, `{acctNo:1,mntSysTime:1}` represents the indexed field, and `{name:idx_xxxx,background:xxx,expireAfterSeconds:xxx,unique:xxxx})` represents the indexed attribute. `name`, `background`, `expireAfterSeconds`, and `unique` are all indexed attributes.
[0105] In the above method, by matching index identifiers and the values of the first and second fields, the index fields and index attributes of the data table can be determined more accurately.
[0106] In one possible implementation, the specific steps for backing up the data records to be backed up in each data table can be:
[0107] For any of the data tables, read the data records of that data table; for any data record in the data table, obtain the default primary key in the data record; if the default primary key is a primary key of a preset data type, convert the data record into a JSON string; generate an SQL command for the data record from the JSON string according to the format of Structured Query Language (SQL); the SQL command is used to back up the data record. The specific steps can be as follows:
[0108] Configure the data for each table to be backed up according to the configuration file; generate executable commands from the read data tables to facilitate subsequent data recovery. The specific steps are as follows:
[0109] Based on the table names, check if the current table needs data backup; if not, skip the backup. Read all data from the table and iterate through it; you can also retrieve the default primary key field, such as the "_id" field. This field is the default primary key; if not filled in during data insertion, it will be automatically generated. This step needs to check whether the returned primary key is a business-meaning primary key generated by the distributed database system (a primary key of a preset data type) or a meaningless primary key automatically generated by the database. If it is a business-meaning primary key, it needs to be retained; otherwise, it should be ignored when generating executable commands. This is determined by checking the data type of the returned "_id".
[0110] The `toJson` method can be used to convert the data records to be backed up (i.e., the data in each table) into a string that can be executed from the command line, concatenate the data insertion command, and write the command to a file. An example of the final concatenated command is: `db.table_name.insert({fieldA: value of fieldA, fieldB: value of fieldB})`, which is a SQL command in SQL format. It should be noted that the SQL command may or may not be an SQL statement. SQL commands and SQL statements can be mapped using preset rules, thereby reducing the storage space required for SQL commands.
[0111] In the above method, for any data record, if the default primary key is a primary key of a preset data type, the data record can be converted into a JSON string, and then an SQL command for the data record can be generated. Since the primary key is unique in the data table, the basic information of the data record can be accurately preserved. Moreover, the storage space occupied by the JSON string is small, which can save the data record more directly and accurately, and generate an SQL command that can be executed directly, thereby facilitating the recovery of the data record.
[0112] In one possible implementation, the specific steps for obtaining the data distribution information of the distributed database across multiple database instances in step 101 can be as follows:
[0113] Execute the status query command to obtain data distribution information of the distributed database.
[0114] Step 102 can be specifically described as follows:
[0115] The data distribution information is converted into a binary data input stream, and the data input stream is converted into at least one string according to a preset format; the at least one string is used as information of the preset structure.
[0116] Specifically, it can be as follows:
[0117] The `sh.status` command (status query command) is used to return data distribution information for all databases in a distributed database cluster (such as MongoDB) with multiple database instances. This includes block information for each shard. However, this command is not a direct data query; it returns unstructured, messy text information with poor readability. Unlike data query commands that return results directly, `sh.status` does not directly output results; it simply outputs the results to the console (multiple lines), and the return value is empty. Therefore, you can obtain an input stream by calling the Java underlying method `Runtime.getRuntime().exec(command)`, then read the binary data from the stream line by line, and convert each line into a string and store it in a collection as information with a pre-defined structure. A code example can be found below. Figure 2 As shown, for subsequent analysis.
[0118] In the above method, the data distribution information is converted into a binary data input stream, and the data input stream is converted into at least one string according to a preset format, thereby standardizing and structuring the data distribution information. The information obtained by the preset structure can be more convenient for data processing and for extracting the fragmentation information of each data table.
[0119] Specifically, the process of parsing the data distribution information to obtain the sharding information of each data table can be as follows:
[0120] For the first string, which is any one of the at least one strings, if the first string is a database attribute string, and the database variable in the first string indicates the distributed database, and the data table variable in the first string indicates the data table, then the strings within a preset position range corresponding to the first string are determined to be sub-information of the data table, and the strings within the preset position range include the first string.
[0121] The specific steps for parsing the sub-information and extracting the sharding information of the data table are as follows:
[0122] For any string within the preset position range, if the string is the first string, then the database name of the distributed database and the table name of the data table are parsed from the string; if the string includes a sharding identifier, then the sharding column name is parsed from the column name corresponding to the sharding identifier in the string; the database name, the table name, and the sharding column name are used as the sharding information of the data table.
[0123] In the above method, the first string is parsed to obtain the database name of the distributed database and the table name of the data table. The string including the sharding identifier is parsed to obtain the sharding column name. Thus, the database name, the table name, and the sharding column name are all obtained. Therefore, the sharding information of the data table can clearly represent the database table information and the basis for the distribution on the corresponding database instance.
[0124] The specific steps can be as follows:
[0125] Check if the string begins with "databases:" (whether it is a database attribute string). If so, set the value of the gotDatabaseNode variable to true. If gotDatabaseNode (database variable) is true (default value), and the string is parsed as "{\"_id\":\""+distributed database name+"\",", set gotDbnameNode to true.
[0126] If gotDbnameNode is true and the string begins with the database name of the distributed database, then the string can be parsed to reveal the database name of the distributed database and the table name of the data table. Meanwhile, the next line (a string within a preset position range) must include the sharding identifier, which can be parsed to reveal the sharding column name.
[0127] The process of parsing the sharded column names is as follows:
[0128] Get the next line of the string after the first string. If the line starts with "shardkey:", then extract the string after "shardkey:", and the extracted string will be the shard column name. Example: shard key:{"custacctNo":1,"entityCode":1}.
[0129] The results from the two steps are then concatenated to form an executable sharding command for the database. Example: `db.runCommand({shardCollection:"database name.table name",key:{"field A":1.0,"field B":1.0,"field C":1.0}})`. The results are also written to a file.
[0130] Each string is traversed using the steps described above until all strings have been traversed. It should be noted that, for ease of parsing, each string can be set to a separate line.
[0131] The above steps have completed the backup of the table structure, data, and sharding information. If recovery is not required, the following steps can be skipped. This step modifies which database the backed-up data should be restored to.
[0132] In another possible implementation, the specific steps for obtaining the data distribution information of the distributed database across multiple database instances in step 101 can be:
[0133] Query the system table of the distributed database to obtain the data block storage information in the multiple database instances;
[0134] The data distribution information is determined based on the data block storage information in the multiple database instances.
[0135] In relational databases, all table information, including indexes, fields, data types, primary keys, partitions, etc., can be found in the metadata database. Similarly, in distributed databases (such as MongoDB), the `chunks` table is found. This is a MongoDB system table that stores information about data blocks within the MongoDB cluster. This data block storage information can be further parsed to obtain sharding information and generate sharding commands.
[0136] In the above manner, by querying the system table of the distributed database, the data block storage information of the multiple database instances can be obtained, and the data distribution information can be obtained directly from the physical distribution, which is faster and more direct.
[0137] In the methods of steps 101 to 104, after converting the data distribution information into information of a preset structure, the sub-information of the data table can be located from the information of the preset structure. The sub-information is then parsed to extract the sharding information of the data table. Based on the data distribution information, the sharding information of the data table can be directly extracted through data processing and conversion, thereby enabling the backup of the sharding information of the data table.
[0138] The following is combined Figure 3 The present invention further describes in detail a method for backing up sharded information in a distributed database.
[0139] Step 1: Modify the database connection information and specify which tables need to be backed up.
[0140] Step 2: Execute the backup.sh backup command.
[0141] Step 3: Obtain the data tables of the distributed database ( Figure 3 (The example below shows all data tables).
[0142] Step 4: Traverse each data table in the distributed database and obtain the index object (index information object) of each data table.
[0143] Step 5: Parse the returned index object, filter index-related information (index fields and index attributes), and generate an executable command (index creation command).
[0144] Step 6: Read the data records (basic data) of each data table according to the configuration file.
[0145] Step 7: Generate executable commands from the data records of each data table.
[0146] Step 8: Execute sh.status.
[0147] Step 9: Parse the returned text (information with a preset structure) and generate fragmented information executable commands.
[0148] Step 10: Execute the database information that needs to be restored.
[0149] Step 11: Execute backup.sh restore to restore the indexes and sharding information of each data table.
[0150] Step 12: Execute the index creation command from step 5.
[0151] Step 13: Execute the basic data recovery command from step 7.
[0152] Step 14: Execute the command from Step 10 to create a shard.
[0153] It should be noted that, based on Figure 3 The method shown involves executing the `backup.sh backup` command to back up the distributed database's indexes, data records, and shard information after each version is released. Therefore, when recovery is needed, any version can be restored. This eliminates the need to manually integrate and sequentially organize the database scripts for each historical version. Furthermore, the method filters out data tables that don't require backup, backing up only the specified tables, resulting in high efficiency. Data recovery simply requires executing `backup.sh restore`, a simple and highly operable command.
[0154] like Figure 4 As shown, the present invention provides a sharding information backup device for a distributed database, comprising: a backup module 401, used to acquire data distribution information of the distributed database; the data distribution information indicates the distribution of data records of each data table of the distributed database across multiple database instances; and converting the data distribution information into information with a preset structure.
[0155] The parsing module 402 is used to locate the sub-information of any data table from the information of the preset structure for any data table in the data tables; parse the sub-information and extract the fragmentation information of the data table;
[0156] The backup module 401 is also used to back up the sharding information of each data table.
[0157] Optionally, the parsing module 402 is specifically used for:
[0158] The data distribution information is converted into a binary data input stream, and the data input stream is converted into at least one string according to a preset format;
[0159] The at least one string is used as information in the preset structure.
[0160] Optionally, the parsing module 402 is specifically used for:
[0161] For the first string, which is any one of the at least one strings, if the first string is a database attribute string, and the database variable in the first string indicates the distributed database, and the data table variable in the first string indicates the data table, then the strings within a preset position range corresponding to the first string are determined to be sub-information of the data table, and the strings within the preset position range include the first string.
[0162] Optionally, the parsing module 402 is specifically used for:
[0163] For any string within the preset position range, if the string is the first string, then the database name of the distributed database and the table name of the data table are parsed from the string.
[0164] If the string contains a sharding identifier, then the sharding column name is parsed out based on the column name corresponding to the sharding identifier in the string;
[0165] The database name, the table name, and the sharding column name are used as the sharding information for the data table.
[0166] Optionally, the parsing module 402 is specifically used to: query the system table of the distributed database to obtain data block storage information in the multiple database instances;
[0167] Based on the data block storage information in the multiple database instances, the data distribution information is determined by querying the system table of the distributed database to obtain the data block storage information of the multiple database instances;
[0168] The data distribution information is determined based on the data block storage information in the multiple database instances.
[0169] Optionally, the backup module 401 is further configured to:
[0170] Obtain the table index objects of each data table in the distributed database;
[0171] For any of the data tables, determine the index fields and index attributes of the data table based on the table index object of the data table;
[0172] Based on the index fields and index attributes of the data table, an index creation command is generated for the data table; the index creation command is used to back up the index information of the data table.
[0173] Optionally, the backup module 401 is specifically used for:
[0174] Fields whose field names match the index identifier in the index object are used as index fields of the data table;
[0175] Based on the value of the first field in the index object, determine whether the index of the data table is a unique index;
[0176] The expiration time of the index of the data table is determined based on the value of the second field in the index object.
[0177] Optionally, the backup module 401 is further configured to:
[0178] For any one of the data tables, read the data records of that data table;
[0179] For any data record in the data table, obtain the default primary key in the data record. If the default primary key is a primary key of a preset data type, then convert the data record into a JSON string.
[0180] The JSON string is formatted according to the Structured Query Language (SQL) to generate an SQL command for the data record; the SQL command is used to back up the data record.
[0181] Based on the same inventive concept, embodiments of the present invention also provide a computer device, including a program or instructions, which, when the program or instructions are executed, execute the distributed database sharding information backup method and any optional method provided in embodiments of the present invention.
[0182] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, including a program or instructions, which, when executed, can be used to perform methods such as the distributed database sharding information backup method and any optional method provided in embodiments of the present invention.
[0183] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0184] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0185] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0186] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0187] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0188] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A method for sharded information backup in a distributed database, characterized in that, include: Query the system tables of the distributed database to obtain data block storage information in multiple database instances; Based on the data block storage information in the multiple database instances, data distribution information is determined; the data distribution information indicates the distribution of data records of each data table in the distributed database across the multiple database instances. The data distribution information is converted into a binary data input stream, and the data input stream is converted into at least one string according to a preset format, and the at least one string is used as information of a preset structure; For any of the data tables, if the first string is a database attribute string, and the database variable in the first string indicates the distributed database, and the data table variable in the first string indicates the data table, then the strings within a preset position range corresponding to the first string are determined to be sub-information of the data table, and the strings within the preset position range include the first string; The first string is any one of the at least one strings; Parsing the sub-information to extract the sharding information of the data table includes: for any string within the preset position range, if the string is the first string, then parsing the database name of the distributed database and the table name of the data table based on the string; if the string includes a sharding identifier, then parsing the sharding column name based on the column name corresponding to the sharding identifier in the string; and using the database name, the table name, and the sharding column name as the sharding information of the data table. Back up the sharding information of each data table.
2. The method as described in claim 1, characterized in that, Also includes: Obtain the table index objects of each data table in the distributed database; For any of the data tables, determine the index fields and index attributes of the data table based on the table index object of the data table; Based on the index fields and index attributes of the data table, an index creation command is generated for the data table; the index creation command is used to back up the index information of the data table.
3. The method as described in claim 2, characterized in that, The step of determining the index fields and index attributes of the data table based on the table index object of the data table includes: Fields whose field names match the index identifier in the index object are used as index fields of the data table; Based on the value of the first field in the index object, determine whether the index of the data table is a unique index; The expiration time of the index of the data table is determined based on the value of the second field in the index object.
4. The method as described in claim 1, characterized in that, Also includes: For any one of the data tables, read the data records of that data table; For any data record in the data table, obtain the default primary key in the data record. If the default primary key is a primary key of a preset data type, then convert the data record into a JSON string. The SQL command is generated from the JSON string according to the format of the Structured Query Language (SQL). The SQL command is used to back up the data records.
5. A sharded information backup device for a distributed database, characterized in that, include: The backup module is used to query the system tables of the distributed database to obtain data block storage information from multiple database instances; Based on the data block storage information in the multiple database instances, data distribution information is determined; the data distribution information indicates the distribution of data records of each data table in the distributed database across the multiple database instances; the data distribution information is converted into a binary data input stream, and the data input stream is converted into at least one string according to a preset format, and the at least one string is used as information of a preset structure; The parsing module is used to determine, for any data table in the data tables, if the first string is a database attribute string, and the database variable in the first string indicates the distributed database, and the data table variable in the first string indicates the data table, then the strings within a preset position range corresponding to the first string are sub-information of the data table, and the strings within the preset position range include the first string; The first string is any one of the at least one strings; Parsing the sub-information to extract the sharding information of the data table includes: for any string within the preset position range, if the string is the first string, then parsing the database name of the distributed database and the table name of the data table based on the string; if the string includes a sharding identifier, then parsing the sharding column name based on the column name corresponding to the sharding identifier in the string; and using the database name, the table name, and the sharding column name as the sharding information of the data table. The backup module is also used to back up the sharding information of each data table.
6. A computer device, characterized in that, Includes a program or instruction, which, when executed, performs the method as described in any one of claims 1 to 4.