Data processing method, electronic device and storage medium of distributed database

Abstract

The present application provides a data processing method, electronic device and storage medium for a distributed database, and relates to the field of electronic digital data processing technology. The method includes: obtaining error information; the error information includes: error type and error node; determining nodes with the same sequence number as voting nodes according to the sequence number of the error node to determine abnormal nodes and available nodes; when the error type corresponding to the error information is inconsistent table structure, determining the voting result according to the table structure information; when the error type corresponding to the error information is inconsistent metadata, determining the voting result according to the target reference information; the target reference information includes one or more of availability, number of rows and SCN; the voting result includes abnormal nodes and available nodes; according to the sharding information corresponding to the available node in the voting result, the corresponding node data is recovered. The present application can adapt to data recovery processing of two types of errors, namely adaptive metadata and table structure, and improve data recovery efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of electronic digital data processing, in particular to the field of database data processing, and more particularly to a distributed database data processing method, electronic equipment and storage medium. Background Art

[0002] A distributed database consists of multiple nodes, and data is organized in the form of data tables, including table structure and metadata. When storing a data table, the data table is divided into n shards according to a specific algorithm, and each shard stores m backup shards on m nodes with the same sequence number. The m backup shards correspond to the m nodes with the same sequence number one by one. In the same distributed database, the distribution position of each shard is fixed.

[0003] During the operation of a distributed data cluster, emergencies such as a database system restart or unexpected power outage may occur, resulting in the last batch of data not being correctly written to the disk, causing metadata or table structure errors. In addition, if a physical machine using RAID technology is damaged, or a RAID card lacks batteries or fails for too long, metadata or table structure errors may occur in the event of a power outage. The related technology lacks a data recovery method that is adaptive to both metadata and table structure errors, and the efficiency of data recovery processing needs to be improved. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, an electronic device, and a storage medium for a distributed database to solve the problem that the efficiency of data recovery processing needs to be improved.

[0005] In a first aspect, an embodiment of the present application provides a data processing method for a distributed database, wherein a data table is divided into n shards; each shard stores m backup shards on m nodes with the same sequence number; the m backup shards correspond one-to-one to the m nodes with the same sequence number; the method comprises:

[0006] Obtain error information; wherein the error information includes: error type and error node;

[0007] According to the sequence number of the error-reporting node, nodes with the same sequence number are determined as voting nodes to determine abnormal nodes and available nodes;

[0008] When the error type corresponding to the error information is inconsistent table structure, obtain the table structure information of the shards corresponding to each voting node, and determine the voting result according to the table structure information; when the error type corresponding to the error information is inconsistent metadata, obtain the target reference information of the shards corresponding to each voting node, and determine the voting result according to the target reference information; wherein the target reference information includes one or more of availability, number of rows and SCN; the voting result includes abnormal nodes and available nodes;

[0009] According to the shard information corresponding to the available nodes in the voting results, data recovery is performed on the corresponding node data.

[0010] In a possible implementation, determining, according to the sequence number of the error-reporting node, a node with the same sequence number as a voting node includes:

[0011] According to the sequence number of the error-reporting node, h nodes with the same sequence number are randomly selected as voting nodes; where h is an odd number; 3≤h<m; or,

[0012] According to the sequence number of the error-reporting node, all nodes with the same sequence number are determined as voting nodes.

[0013] In a possible implementation, the obtaining of table structure information of shards corresponding to each voting node includes:

[0014] According to the serial number of each voting node, issue a command to view the table creation process and obtain the table structure information fed back by each voting node;

[0015] The table structure information includes one or more of the following: table name, column, column type, table comment, data table volume, and default value.

[0016] In a possible implementation, determining the voting result according to the table structure information includes:

[0017] According to the majority voting system, the nodes with the same table structure information and the largest number are selected as available nodes.

[0018] In a possible implementation, obtaining target reference information of a shard corresponding to each voting node and determining a voting result according to the target reference information includes:

[0019] Get the availability of the shards corresponding to each voting node;

[0020] When only one voting node's corresponding shard is available, the corresponding node is used as an available node, and when all voting nodes' corresponding shards are available, the number of rows of each voting node's corresponding shard is obtained;

[0021] When the number of rows of the shards corresponding to each voting node is different, the voting node corresponding to the largest number of rows is the available node; otherwise, the SCN of the shard corresponding to each voting node is obtained;

[0022] The voting nodes with the same SCN and the largest number are considered available nodes; otherwise, all are considered available nodes.

[0023] In a possible implementation, obtaining target reference information of a shard corresponding to each voting node and determining a voting result according to the target reference information includes:

[0024] Obtain the availability, number of rows, and SCN of the corresponding shards of each voting node, and determine the voting results according to the priority order based on the majority voting mechanism;

[0025] Among them, the availability priority is higher than the row number; the row number priority is higher than the SCN.

[0026] In a possible implementation manner, before obtaining the target reference information of the shards corresponding to each voting node, the method further includes:

[0027] Execute a rollback request or a revert request.

[0028] In a possible implementation manner, after determining the voting result according to the table structure information, or after determining the voting result according to the target reference information, the method further includes:

[0029] A recovery log is recorded according to the voting result, so as to facilitate data recovery according to the recovery log.

[0030] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect or any possible implementation of the first aspect are implemented.

[0031] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method described in the first aspect or any possible implementation of the first aspect.

[0032] The embodiment of the present application provides a data processing method, electronic device and storage medium of a distributed database, which obtains error types and error nodes, and sets different abnormal node and available node identification schemes according to different error types. Among them, nodes with the same sequence number are determined as voting nodes according to the sequence number of the error node. When the error type corresponding to the error information is inconsistent table structure, the table structure information of the shards corresponding to each voting node is obtained, and the voting result is determined according to the table structure information to distinguish the difference between the abnormal node and the available node table structure information. When the error type corresponding to the error information is inconsistent metadata, the target reference information of the shards corresponding to each voting node is obtained, and the voting result is determined according to the target reference information to distinguish the difference between the abnormal node and the available node metadata when the table structure is the same. After the voting is completed, the corresponding node data is restored according to the shard information corresponding to the available node in the voting result. Among them, the target reference information includes one or more of availability, number of rows and SCN. By comprehensively comparing and voting on table structure information and metadata, the reference data during data recovery is determined to improve the accuracy of data recovery. Ensure that the data table can be used normally after an abnormality occurs to avoid repeated errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0034] Figure 1 It is a flow chart of the implementation of the data processing method of the distributed database provided by one embodiment of the present application;

[0035] Figure 2 This is a data table sharding distribution method provided in an embodiment of the present application;

[0036] Figure 3 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0037] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0038] The terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present application described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0039] Unless otherwise specified, the term "plurality" means two or more. The character " / " indicates that the objects before and after are in an "or" relationship. For example, A / B means: A or B. The term "and / or" is a description of the association relationship between objects, indicating that there can be three relationships. For example, A and / or B means: A or B, or, A and B.

[0040] The words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or including (comprising) refer to the existence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups of these. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method or device including the elements.

[0041] In this application, each embodiment may focus on the differences from other embodiments, and the same or similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts may refer to the description of the method part.

[0042] When a system or storage device encounters an emergency, the following error message is displayed when inserting or loading data: source table and destination table are not same: src: (l).

[0043] However, after executing the write operation before the system or storage device encounters an emergency, the cluster layer determines that the write has been successful. In fact, the data has not been written to the disk when the emergency occurs, that is, the primary shard or the backup shard has missed one or more operations. In the specific implementation process, when the type of operation is different, the inconsistency of the primary shard or the backup shard is different.

[0044] In one possible implementation, when one or more Data Definition Language (DDL) operations are missing, the metadata modifications are not written to the disk, which results in inconsistent column numbers and column attributes. This is an abnormal table structure.

[0045] In other possible implementations, when one or more Data Manipulation Language (DML) operations are missing, the changes to the metadata and data are not written to the disk, which is manifested as inconsistent metadata between the primary and backup shards in subsequent checks, such as inconsistent no_obj and savepos_loc. This is a metadata anomaly.

[0046] Later, the cluster layer will find "master-slave inconsistency error" when performing commit, load, insert select operations, but cannot determine which shard has an abnormality, so it takes rollback or revert to roll back the data and reports an error for the result of this operation. However, rollback or revert cannot solve the inconsistency problem, and the external manifestation is that an error is reported every time the table is written, making the table unusable.

[0047] The present application aims to provide a method for determining the specific information of the abnormal shard and updating the data when the data of the primary and backup shards are found to be inconsistent, so as to ensure that the data table can continue to be used.

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0049] Figure 1 is a flow chart of a distributed database data processing method provided by an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0050] S101, obtaining error information, wherein the error information includes: error type and error node.

[0051] like Figure 2The data table shard distribution method is shown in an exemplary manner. When storing a data table, according to a specific algorithm, the data table is divided into n shards (shown as n1, n2 and n3 in the figure, including but not limited to 3), and each shard stores m backup shards on m nodes with the same sequence number (shown as 3 backup shards in the figure). The m backup shards correspond to the m nodes with the same sequence number one by one (that is, the n1 shard is stored on three m1 nodes respectively).

[0052] The execution subject of each embodiment of the present application can be a server, a processor, a microprocessor or other device with data processing function. In the actual implementation process, the execution subject is specifically a cluster layer.

[0053] In some optional implementations, after a problem occurs, when a data insertion or loading operation is performed, the specific form of the error message is: source table and destination table are not same: src: (l).

[0054] S102, determining nodes with the same sequence number as the voting nodes according to the sequence number of the error-reporting node, so as to determine abnormal nodes and available nodes.

[0055] According to the above introduction to the data table storage method, nodes with the same serial number store the same backup shards. When data addition, data deletion, data modification, column addition, column deletion, or column modification is performed, a device abnormality occurs, which will cause some backup shards to fail to synchronize successfully. After the device is restored, when the data of each node is used, there are nodes with available data and nodes with data errors. Voting can effectively distinguish between abnormal nodes and available nodes, and data recovery can be performed based on the available node shard data.

[0056] S103, when the error type corresponding to the error information is inconsistent table structure, obtain the table structure information of the shards corresponding to each voting node, and determine the voting result according to the table structure information; when the error type corresponding to the error information is inconsistent metadata, obtain the target reference information of the shards corresponding to each voting node, and determine the voting result according to the target reference information; wherein the target reference information includes one or more of availability, number of rows and SCN; the voting result includes abnormal nodes and available nodes.

[0057] When processing table data, if a device anomaly occurs during the process of deleting columns, adding columns, or modifying the data type and default value of a column, it may cause inconsistencies in the table structure. Inconsistent table structure means that there are differences in the entire data of a column. On the other hand, metadata inconsistency usually manifests as inconsistent data in a row or a cell, but the overall column type and number of columns remain consistent.

[0058] In order to improve the efficiency of identification and data recovery, clear voting schemes are formulated for table structure inconsistencies and metadata inconsistencies. Based on this, it is helpful to improve the speed of identifying table structure inconsistencies and metadata inconsistencies, thereby improving the efficiency of data recovery.

[0059] S104, recovering the corresponding node data according to the shard information corresponding to the available nodes in the voting results.

[0060] Among them, the shard information corresponding to the available nodes can avoid repeated errors during the use of the table. Based on this, data is restored for nodes with the same sequence number according to the shard information corresponding to the available nodes, thereby improving the stability and reliability of database operation.

[0061] In this embodiment, by obtaining the error type and the error node, different abnormal node and available node identification schemes are set according to different error types. Among them, the node with the same sequence number is determined as the voting node according to the sequence number of the error node. When the error type corresponding to the error information is inconsistent table structure, the table structure information of the shard corresponding to each voting node is obtained, and the voting result is determined according to the table structure information to distinguish the difference between the abnormal node and the available node table structure information. When the error type corresponding to the error information is inconsistent metadata, the target reference information of the shard corresponding to each voting node is obtained, and the voting result is determined according to the target reference information to distinguish the difference between the abnormal node and the available node metadata when the table structure is the same. After the voting is completed, the corresponding node data is restored according to the shard information corresponding to the available node in the voting result. Among them, the target reference information includes one or more of availability, number of rows and SCN. By comprehensively comparing the table structure information and metadata, the reference data for data recovery is determined to improve the accuracy of data recovery. Ensure that the data table can be used normally after an abnormality occurs to avoid repeated errors.

[0062] In different embodiments, in step S102, the manner of selecting voting nodes is different.

[0063] In a possible implementation, determining nodes with the same sequence number as the voting nodes according to the sequence number of the error-reporting node includes: randomly selecting h nodes with the same sequence number as the voting nodes according to the sequence number of the error-reporting node; wherein h is an odd number; and 3≤h<m.

[0064] In different embodiments, when m backup shards are stored for each shard, the number of backups is different. When the number of backups is large, all nodes participate in voting, which will increase the voting time and reduce the data recovery efficiency. Therefore, some nodes with the same sequence number as the error reporting node are selected as voting nodes and participate in the voting.

[0065] In addition, since the data of each backup shard is consistent with that of the primary shard at the beginning, the voting is performed according to the majority voting system. Selecting an odd number of voting nodes can avoid the situation where the number of votes of abnormal nodes and available nodes in the voting results is equal, ensuring the validity of the voting results.

[0066] In a specific embodiment, considering that there may be many voting nodes, the voting process is carried out in a manner of gradually expanding the scope, that is, when node n1 has other copies, the voting results of other copies of n1 are given priority. Once one party obtains the majority of votes, the judgment is terminated. This process can shorten the voting time, especially for the case where the number of backup shards is large.

[0067] For example, the cluster layer determines abnormal node shards by voting by letting the node shards that are "temporarily considered normal" as voters, and the one with fewer votes is abnormal. "Temporarily considered normal" means the node shards that are not involved in this error message. Assume that there are three data nodes, namely gnode1, gnode2, and gnode3, and adopt a 1-active-1-standby deployment mode, that is, there are n1 and n2 shards on gnode1, n2 and n3 shards on gnode2, and n3 and n1 shards on gnode3. The shards with the same sequence number are each other's active and standby shards; if the active and standby shards of n1 are inconsistent, it means that gnode1 and gnode3 may have abnormal power failures, so the n2 shards on gnode1 and the n3 shards on gnode3 cannot participate in the vote. Therefore, only the n2 and n3 shards on gnode2 can be voters. Assume that there are three data nodes, namely gnode1, gnode2, and gnode3, and a 3-active-2-backup deployment mode is adopted, that is, gnode1, gnode2, and gnode3 all have n1, n2, and n3 shards, or the n1 shard on gnode1 and gnode2 has a master-slave inconsistency. According to the previous algorithm, it is determined that only n1, n2, and n3 on gnode3 can be voters, and at this time, the voting results of other backup shards with the same serial number are given priority, that is, the voting result of the n1 shard on gnode3 is given priority.

[0068] In a possible implementation, determining nodes with the same sequence number as the error-reporting node as voting nodes according to the sequence number of the error-reporting node includes: determining all nodes with the same sequence number as the voting nodes according to the sequence number of the error-reporting node.

[0069] This implementation is suitable for storing m backup shards for each shard, when the number of backups is small, and can avoid reducing voting efficiency due to too many voting nodes, and avoid affecting data recovery efficiency.

[0070] In the specific implementation process, there are not only multiple ways to determine the voting nodes, but also different implementation methods when determining the table structure information and target reference information of the shards corresponding to each voting node.

[0071] In a possible implementation, in step S103, obtaining the table structure information of the shards corresponding to each voting node includes:

[0072] According to the serial number of each voting node, issue a command to view the table creation process and obtain the table structure information fed back by each voting node;

[0073] The table structure information includes one or more of the following: table name, column, column type, table comment, data table volume, and default value.

[0074] In a specific embodiment, the table structure information also includes data type, whether it is empty, etc.

[0075] Optionally, the command for viewing the table creation process is a show create table command. Among them, show createtable is a commonly used command for viewing the table creation process in the MySQL database. In this embodiment, the command for viewing the table creation process is intended to obtain the table structure of the shard information corresponding to each voting node. To achieve the same purpose, in other embodiments, other instructions for obtaining the table structure of the shard information can be selected in addition to the show create table command.

[0076] In a possible implementation, determining the voting result according to the table structure information includes:

[0077] According to the majority voting system, the nodes with the same table structure information and the largest number are selected as available nodes.

[0078] The voting process is roughly divided into two steps. First, the table structure information is analyzed and compared. The goal of this step is to find nodes with the same table structure information and group them. Secondly, the number of nodes with the same table structure information needs to be counted to meet the requirements of the majority voting system, that is, the nodes with the largest number are selected as available nodes. Finally, according to the principle of majority voting, the voting results are determined and the optimal solution is found among multiple optional solutions, thereby providing data support for data recovery.

[0079] In this embodiment, the nodes with the same table structure information and the largest number are selected as available nodes, and data recovery is performed on a smaller number of abnormal nodes to improve the overall data recovery efficiency.

[0080] The above is an introduction to the implementation method of determining the table structure information of the shard corresponding to each voting node. The following is an introduction to the implementation method of determining the metadata information of the shard corresponding to each voting node.

[0081] In a possible implementation, obtaining target reference information of a shard corresponding to each voting node and determining a voting result according to the target reference information includes:

[0082] Get the availability of the shards corresponding to each voting node;

[0083] When only one voting node's corresponding shard is available, the corresponding node is used as an available node, and when all voting nodes' corresponding shards are available, the number of rows of each voting node's corresponding shard is obtained;

[0084] When the number of rows of the shards corresponding to each voting node is different, the voting node corresponding to the largest number of rows is the available node; otherwise, the SCN of the shard corresponding to each voting node is obtained;

[0085] The voting nodes with the same SCN and the largest number are considered available nodes; otherwise, all are considered available nodes.

[0086] In some optional implementations, when obtaining the availability of the shards corresponding to each voting node, the cluster layer sends a select * from t where rowid%65536=1 instruction to the data layer, and determines the availability of the abnormal shard based on the feedback from the data layer. When the feedback from the data layer is received correctly, the corresponding shard is available, otherwise, the corresponding shard is unavailable.

[0087] When all shards are available, further judgment is made on the number of rows of each shard. When judging based on the number of rows, the voting node corresponding to the maximum number of rows is used as the available node to ensure data integrity. It should be noted that the shard data corresponding to the maximum number of rows is not the final processing result before the device abnormality. For example, when a device abnormality occurs when a deletion operation is performed on a row or multiple rows of data but the final confirmation is not completed, the shard data of some nodes has not been updated to the deleted state. After the device is restored, if the restored data is the deleted data, data loss will result. At this time, restoring the data with the largest number of rows can better ensure the maximum data volume and avoid data omissions compared to restoring the deleted data.

[0088] When all shards are available and the number of rows is consistent, further judgment is made based on the SCN of the shard. The larger the SCN, the closer the corresponding shard data update time is to the current time. When all shards are available and the number of rows is consistent, the voting nodes with the same SCN and the largest number are the available nodes, and the corresponding available node data consistency is higher.

[0089] When the SCNs of all voting nodes are consistent, it is determined that the shard data of each node is consistent, and data recovery can be performed using the shard data of any node.

[0090] In this implementation, there are three voting stages, which are based on the availability, number of rows and SCN of the shards to vote on available nodes and abnormal nodes. After the available nodes and abnormal nodes are clearly distinguished at a certain stage, the voting division process ends, reducing the voting time and improving data recovery efficiency. This method is suitable for the case of a large number of backup shards, which can reduce the amount of data obtained before the voting stage, reduce voting pressure, and improve voting efficiency.

[0091] In a possible implementation, obtaining target reference information of a shard corresponding to each voting node and determining a voting result according to the target reference information includes:

[0092] Obtain the availability, number of rows, and SCN of the corresponding shards of each voting node, and determine the voting results according to the priority order based on the majority voting mechanism;

[0093] Among them, availability takes precedence over the number of rows; the number of rows takes precedence over SCN.

[0094] In this implementation, there are three voting stages, which are based on the availability, number of rows and SCN of the shards to vote on available nodes and abnormal nodes. After the available nodes and abnormal nodes are clearly distinguished at a certain stage, the voting division process ends, reducing the voting time and improving data recovery efficiency. This method is suitable for the case where the number of backup shards is small. It can reduce the number of times to obtain data before voting during multiple voting processes, and achieve one-time acquisition of target reference information at each voting stage. The pressure of data acquisition has little impact on voting efficiency.

[0095] In a possible implementation, before obtaining the target reference information of the shards corresponding to each voting node, the following is further included:

[0096] Execute a rollback request or a revert request.

[0097] When metadata anomalies occur, usually due to the loss of one or more DML operations, the cluster layer executes a rollback request or a revert request to undo the changes that have been executed, ensuring that the database returns to the state before the transaction started, thereby maintaining data consistency. However, a rollback request or a revert request cannot completely guarantee data consistency on all shard nodes. Therefore, performing a vote after executing a rollback request or a revert request can improve the efficiency of distinguishing abnormal nodes from available nodes.

[0098] In a possible implementation, after determining the voting result according to the table structure information, or after determining the voting result according to the target reference information, the method further includes:

[0099] A recovery log is recorded according to the voting result, so as to facilitate data recovery according to the recovery log.

[0100] In some optional implementations, dmlstorageevent is recorded according to the voting result, so as to perform data recovery based on dmlstorageevent.

[0101] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0102] In a specific embodiment, a data processing method of a distributed database includes the following steps:

[0103] 1) Data consistency check: When data inconsistency occurs and the error message source table and destination table are not same:src:(l) is obtained, check the error immediately to determine which nodes and shards have inconsistencies between the primary and backup shards.

[0104] 2) Voting algorithm: determine abnormal node shards;

[0105] 1. Handling inconsistent table structures: The cluster layer issues the show create table command to let each node shard return its own table definition. The cluster layer compares the returned table structure information to find the node with the majority vote, thereby determining the abnormal node shard and recording the dmlstorageevent.

[0106] 2. Dealing with metadata inconsistencies

[0107] After the cluster layer executes rollback or revert when inconsistency is found, it starts to determine which shard has an abnormality. The determination process is as follows:

[0108] (a) Determine the availability of the two abnormal shards by selecting * from t where rowid%65536=1. If one of the shards is unavailable, the available shard is used as the available node. The determination ends and the available shard is used as the available node.

[0109] (b) Compare the number of data rows of the two abnormal shards. If the number of rows is different, the node with more rows is used as the available node, and the judgment ends;

[0110] (c) The SCN is judged by voting. If one party's SCN obtains the majority of votes, the judgment ends and the node with the majority of votes is regarded as an available node.

[0111] (d) When the SCNs of all node shards are consistent, one is randomly selected and forced to take it as the standard.

[0112] (e) Since power outage may cause damage to metadata files and data files, a dmlstorageevent is recorded for the abnormal shard.

[0113] 3) Data recovery: Based on the abnormal node record dmlstorageevent, the abnormal node is restored. The data writing operation to the table can be executed successfully, and the table can continue to be used normally.

[0114] The following is an embodiment of the device of the present application. For details not described in detail, please refer to the corresponding method embodiment described above.

[0115] Figure 3 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. When the processor 30 executes the computer program 32, the steps in the above-mentioned distributed database data processing method embodiments are implemented, such as Figure 1 The steps shown.

[0116] Exemplarily, the computer program 32 may be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 30 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 32 in the electronic device 3.

[0117] The electronic device 3 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device 3 may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of the electronic device 3 and does not constitute a limitation of the electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0118] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0119] The memory 31 may be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 31 may also be an external storage device of the electronic device 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 3. Further, the memory 31 may also include both an internal storage unit and an external storage device of the electronic device 3. The memory 31 is used to store the computer program and other programs and data required by the electronic device. The memory 31 may also be used to temporarily store data that has been output or is to be output.

[0120] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0121] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0122] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0123] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0124] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0126] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned distributed database data processing method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium.

[0127] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data processing method for a distributed database, characterized in that: The data table is divided into n shards; each shard stores m backup shards on m nodes with the same sequence number; There is a one-to-one correspondence between m backup shards and m nodes with the same sequence number; The method includes: Obtain error information; wherein the error information includes: error type and error node; According to the sequence number of the error-reporting node, nodes with the same sequence number are determined as voting nodes to determine abnormal nodes and available nodes; When the error type corresponding to the error information is inconsistent table structure, obtain the table structure information of the shards corresponding to each voting node, and determine the voting result according to the table structure information; when the error type corresponding to the error information is inconsistent metadata, obtain the target reference information of the shards corresponding to each voting node, and determine the voting result according to the target reference information; wherein the target reference information includes one or more of availability, number of rows and SCN; the voting result includes abnormal nodes and available nodes; According to the shard information corresponding to the available nodes in the voting results, data recovery is performed on the corresponding node data; Determining the voting result according to the table structure information includes: According to the majority voting system, the nodes with the same table structure information and the largest number are selected as available nodes; The obtaining target reference information of the shards corresponding to each voting node and determining the voting result according to the target reference information includes: Get the availability of the shards corresponding to each voting node; When only one voting node's corresponding shard is available, the corresponding node is used as an available node, and when all voting nodes' corresponding shards are available, the number of rows of each voting node's corresponding shard is obtained; When the number of rows of the shards corresponding to each voting node is different, the voting node corresponding to the largest number of rows is the available node; otherwise, the SCN of the shard corresponding to each voting node is obtained; The voting nodes with the same SCN and the largest number are considered available nodes; otherwise, all are considered available nodes.

2. The method according to claim 1, characterized in that The step of determining, according to the sequence number of the error-reporting node, a node with the same sequence number as a voting node includes: According to the sequence number of the error-reporting node, h nodes with the same sequence number are randomly selected as voting nodes; where h is an odd number; 3≤h<m; or, According to the sequence number of the error-reporting node, all nodes with the same sequence number are determined as voting nodes.

3. The method according to claim 1, characterized in that The obtaining of the table structure information of the shards corresponding to each voting node includes: According to the serial number of each voting node, issue a command to view the table creation process and obtain the table structure information fed back by each voting node; The table structure information includes one or more of the following: table name, column, column type, table comment, data table volume, and default value.

4. The method according to claim 1, characterized in that: The obtaining target reference information of the shards corresponding to each voting node and determining the voting result according to the target reference information includes: Obtain the availability, number of rows, and SCN of the corresponding shards of each voting node, and determine the voting results according to the priority order based on the majority voting mechanism; Among them, the availability priority is higher than the row number; the row number priority is higher than the SCN.

5. The method according to claim 1, characterized in that: Before obtaining the target reference information of the shards corresponding to each voting node, the method further includes: Execute a rollback request or a revert request.

6. The method according to claim 1, characterized in that After determining the voting result according to the table structure information, or after determining the voting result according to the target reference information, the method further includes: A recovery log is recorded according to the voting result, so as to facilitate data recovery according to the recovery log.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images