Data processing method of distributed database, electronic equipment and storage medium

Abstract

The invention provides a data processing method of a distributed database, electronic equipment and a storage medium, and relates to the technical field of electric digital data processing. The method comprises the following steps: acquiring error reporting information; the error reporting information comprises an error reporting type and an error reporting node; determining nodes with the same serial number as voting nodes according to the serial number of the error reporting node so as to determine abnormal nodes and available nodes; when the error report type corresponding to the error report information is that the table structure is inconsistent, determining a voting result according to the table structure information; when the error report type corresponding to the error report information is metadata inconsistency, determining a voting result according to the target reference information; the target reference information comprises one or more of availability, row number and SCN; the voting result comprises abnormal nodes and available nodes; and performing data recovery on the corresponding node data according to the fragment information corresponding to the available nodes in the voting result. The method and the device can adapt to data recovery processing of two types of errors of adaptive metadata and table structures, and the data recovery efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing technology, particularly to the field of database data processing, and especially to a data processing method, electronic device and storage medium for a distributed database. Background Technology

[0002] A distributed database consists of multiple nodes, and the data is organized in the form of tables, including table structure and metadata. When storing data tables, according to a specific algorithm, the table is divided into n shards, and each shard stores m backup shards on m nodes with the same sequence number. There is a one-to-one correspondence between the m backup shards and the m nodes with the same sequence number. Within the same distributed database, the distribution location of each shard is fixed.

[0003] During the operation of a distributed data cluster, unexpected events such as database system restarts or power outages may occur, causing the last batch of data to fail to be written to disk correctly, resulting in metadata or table structure errors. Additionally, damage to physical machines using RAID technology, or RAID cards lacking batteries or failing due to prolonged use, can also cause metadata or table structure errors during power outages. Currently, there is a lack of adaptive data recovery methods for both metadata and table structure errors, and data recovery efficiency needs improvement. Summary of the Invention

[0004] This application provides a data processing method, electronic device, and storage medium for a distributed database to address the problem of insufficient efficiency in data recovery processing.

[0005] In a first aspect, embodiments of this application provide a data processing method for a distributed database, wherein a data table is divided into n shards; each shard stores m backup shards on m nodes with the same sequence number; the m backup shards and the m nodes with the same sequence number correspond one-to-one; the method includes: Obtain error information; wherein, the error information includes: error type and error node; Based on the sequence number of the error-reporting node, nodes with the same sequence number are identified as voting nodes to determine the abnormal nodes and available nodes; When the error type corresponding to the error message is inconsistent table structure, the table structure information of the shards corresponding to each voting node is obtained, and the voting result is determined based on the table structure information; when the error type corresponding to the error message is inconsistent metadata, the target reference information of the shards corresponding to each voting node is obtained, and the voting result is determined based on the target reference information; wherein, the target reference information includes one or more of availability, number of rows, and SCN; the voting result includes abnormal nodes and available nodes; Based on the sharding information corresponding to the available nodes in the voting results, data recovery is performed on the corresponding node data.

[0006] In one possible implementation, determining nodes with the same sequence number as voting nodes based on the sequence number of the error-reporting node includes: Based on the sequence number of the error-reporting node, randomly select h nodes with the same sequence number as voting nodes; where h is an odd number; 3 ≤ h < m; or, Based on the sequence number of the error-reporting node, all nodes with the same sequence number are identified as voting nodes.

[0007] In one possible implementation, obtaining the table structure information of the shards corresponding to each voting node includes: Based on the sequence number of each voting node, issue a command to view the table creation process and obtain the table structure information returned by each voting node; The table structure information includes one or more of the following: table name, columns, column types, table comments, number of data items, and default values.

[0008] In one possible implementation, determining the voting result based on the table structure information includes: Based on the majority voting system, the node with the largest number of identical table structure information is selected as the available node.

[0009] In one possible implementation, obtaining the target reference information of the shard corresponding to each voting node and determining the voting result based on the target reference information includes: Obtain the availability of the shards corresponding to each voting node; When only one voting node's corresponding shard is available, the corresponding node is used as the available node, and when all voting nodes' corresponding shards are available, the number of rows in each voting node's corresponding shard is obtained; If the number of rows in the shards corresponding to each voting node is different, the voting node with the largest number of rows is the available node; otherwise, the SCN of the shards corresponding to each voting node is obtained. The node with the most votes corresponding to the same SCN is considered a usable node; otherwise, all nodes are considered usable nodes.

[0010] In one possible implementation, obtaining the target reference information of the shard corresponding to each voting node and determining the voting result based on the target reference information includes: Obtain the availability, number of rows, and SCN of the shard corresponding to each voting node, and determine the voting results according to the majority voting mechanism and priority order; Wherein, availability has a higher priority than the number of rows; the number of rows has a higher priority than the SCN.

[0011] In one possible implementation, before obtaining the target reference information for each voting node's corresponding shard, the method further includes: Execute a rollback request or a revert request.

[0012] In one possible implementation, after determining the voting result based on the table structure information, or after determining the voting result based on the target reference information, the method further includes: The recovery log is recorded based on the voting results to facilitate data recovery.

[0013] In a second aspect, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect or any possible implementation of the first aspect above.

[0014] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect or any possible implementation of the first aspect.

[0015] This application provides a data processing method, electronic device, and storage medium for a distributed database. It obtains error types and error nodes, and sets different identification schemes for abnormal and available nodes based on different error types. Specifically, nodes with the same sequence number are identified as voting nodes. When the error type is inconsistent table structure, the table structure information of the shards corresponding to each voting node is obtained, and the voting result is determined based on the table structure information to distinguish the differences in table structure information between abnormal and available nodes. When the error type is inconsistent metadata, the target reference information of the shards corresponding to each voting node is obtained, and the voting result is determined based on the target reference information to distinguish the differences in metadata between abnormal and available nodes when the table structures are the same. After voting, data recovery is performed on the corresponding node data based on the shard information corresponding to the available nodes in the voting results. The target reference information includes one or more of availability, row count, and SCN. By comprehensively comparing and voting on table structure information and metadata from multiple aspects, reference data for data recovery is determined, improving the accuracy of data recovery. This ensures that the data table can be used normally after an anomaly occurs, avoiding repeated error reporting. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the implementation of a data processing method for a distributed database according to an embodiment of this application. Figure 2 This is a data table sharding distribution method provided in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0019] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0020] Unless otherwise stated, the term "multiple" means two or more. The character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B. The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0021] The terms used in this application are for describing embodiments only and are not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, the term “and / or” as used herein means including one or more of the associated listed elements and all possible combinations thereof. Additionally, when used in this application, the terms “comprise” and its variations “comprises” and / or “comprising” refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitation, an element defined by the phrase “comprising an…” does not exclude the presence of additional identical elements in the process, method, or apparatus that includes said element.

[0022] In this application, each embodiment focuses on describing the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, then the relevant parts can be referred to the description of the method section.

[0023] When performing data insertion or loading operations after a sudden failure of the system or storage device, the error message is: source table and destination table are not the same:src:(l).

[0024] However, if a write operation is performed before a system or storage device experiences a sudden failure, the cluster layer may determine that the write was successful. In reality, the data was not written to disk during the failure, meaning that the primary or backup shard is missing one or more operations. In practice, different types of operations can lead to different inconsistencies in the primary or backup shards.

[0025] In one possible implementation, when one or more Data Definition Language (DDL) operations are missing, the modifications to the metadata are not persisted to disk, which manifests as inconsistent column counts and inconsistent column attributes in subsequent checks. This is a table structure anomaly. In other possible implementations, if one or more Data Manipulation Language (DML) operations are missing, the modifications to metadata and data are not persisted to disk. This manifests as inconsistencies in the metadata of the primary and backup shards, such as inconsistencies in no_obj and savepos_loc, which is a metadata anomaly.

[0026] Subsequently, when the cluster layer performs commit, load, insert, and select operations, it encounters a "master-slave inconsistency error," but cannot determine which specific shard is experiencing the anomaly. Therefore, it attempts to roll back the data using rollback or revert, and reports an error for the current operation. However, rollback or revert does not resolve the inconsistency issue; externally, it results in an error every time the table is written, rendering the table unusable.

[0027] This application aims to provide a method to determine the specific information of the abnormal shards and update the data when inconsistency is found between the primary and backup shards, so as to ensure that the data table can continue to be used.

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0029] Figure 1 This is a flowchart illustrating the implementation of a data processing method for a distributed database according to an embodiment of this application. Figure 1 As shown, the method includes the following steps: S101, Obtain error information. The error information includes: error type and error node.

[0030] like Figure 2 This example illustrates a data table sharding distribution method. When storing a data table, according to a specific algorithm, the data table is divided into n shards (shown as n1, n2, and n3 in the diagram, but not limited to 3). Each shard stores m backup shards on m nodes with the same sequence number (shown as 3 backup shards in the diagram). The m backup shards and the m nodes with the same sequence number correspond one-to-one (i.e., the n1 shard is stored on three m1 nodes).

[0031] The execution entity of each embodiment of this application can be a server, processor, microprocessor, or other device with data processing capabilities. In actual implementation, the execution entity is specifically a cluster layer.

[0032] In some optional implementations, when a problem occurs and data insertion or loading operations are performed, the specific error message is in the form of: source table and destination table are not the same:src:(l).

[0033] S102, determine the nodes with the same sequence number as the error reporting nodes as voting nodes, in order to identify the abnormal nodes and the available nodes.

[0034] As described above regarding data table storage methods, nodes with the same sequence number store the same backup shards. When data is added, deleted, modified, or columns are added, deleted, or modified, and a device malfunction occurs, some backup shards may fail to synchronize successfully. Therefore, after device recovery, when data is used on each node, there will be nodes with available data and nodes with incorrect data. Voting can effectively distinguish between malfunctioning and available nodes, and data recovery can be performed based on the shard data from available nodes.

[0035] S103, when the error type corresponding to the error message is inconsistent table structure, obtain the table structure information of the shard corresponding to each voting node, and determine the voting result based on the table structure information; when the error type corresponding to the error message is inconsistent metadata, obtain the target reference information of the shard corresponding to each voting node, and determine the voting result based on the target reference information; wherein, the target reference information includes one or more of availability, number of rows and SCN; the voting result includes abnormal nodes and available nodes.

[0036] When processing tabular data, device malfunctions during the deletion, addition, or modification of column data types and default values ​​can lead to inconsistencies in the table structure. Table structure inconsistencies indicate differences in all data within a single column. Conversely, metadata inconsistencies typically manifest as discrepancies in a single row or cell, while the overall column type and number remain consistent.

[0037] To improve identification and data recovery efficiency, specific voting schemes were developed for both table structure inconsistencies and metadata inconsistencies. This helps to accelerate the identification of these inconsistencies, thereby improving data recovery efficiency.

[0038] S104. Based on the sharding information corresponding to the available nodes in the voting results, perform data recovery on the corresponding node data.

[0039] The sharding information corresponding to available nodes can prevent repeated errors during table usage. Based on this, data recovery is performed on nodes with the same sequence number according to the sharding information corresponding to available nodes, thereby improving the stability and reliability of database operation.

[0040] In this embodiment, by acquiring the error type and error node, different identification schemes for abnormal and available nodes are set according to different error types. Specifically, nodes with the same sequence number are identified as voting nodes. When the error type corresponding to the error message is inconsistent table structure, the table structure information of the shards corresponding to each voting node is acquired, and the voting result is determined based on the table structure information to distinguish the differences in table structure information between abnormal and available nodes. When the error type corresponding to the error message is inconsistent metadata, the target reference information of the shards corresponding to each voting node is acquired, and the voting result is determined based on the target reference information to distinguish the differences in metadata between abnormal and available nodes when the table structure is the same. After voting, data recovery is performed on the corresponding node data based on the shard information corresponding to the available nodes in the voting results. The target reference information includes one or more of availability, row count, and SCN. By comprehensively comparing and voting on table structure information and metadata from multiple aspects, reference data for data recovery is determined, improving the accuracy of data recovery. This ensures that the data table can be used normally after an anomaly occurs, avoiding repeated error reports.

[0041] In different embodiments, the method of selecting voting nodes in step S102 is different.

[0042] In one possible implementation, determining nodes with the same sequence number as voting nodes based on the sequence number of the error-reporting nodes includes: randomly selecting h nodes with the same sequence number as voting nodes based on the sequence number of the error-reporting nodes; where h is an odd number; 3 ≤ h < m.

[0043] In different embodiments, the number of backups varies when storing m backup shards for each shard. When the number of backups is large, all nodes participating in the vote will increase the voting time and reduce data recovery efficiency. Therefore, some nodes with the same sequence number as the node reporting the error are selected as voting nodes and participate in the vote.

[0044] Furthermore, since the data of each backup shard is initially consistent with that of the primary shard, the voting process follows a majority voting system. Selecting an odd number of voting nodes avoids situations where the number of votes from abnormal nodes and available nodes is equal, ensuring the validity of the voting results.

[0045] In a specific implementation, considering that there may be many voting nodes, the voting process is carried out in a way that gradually expands the scope. That is, when node n1 has other replicas, the voting results of other replicas of n1 are given priority. Once one side gets a majority of votes, the process ends. This process can shorten the voting time, especially for cases with a large number of backup shards.

[0046] For example, the cluster layer uses shards that are "temporarily considered normal" as voters to determine abnormal shards. The shard with the fewest votes is considered abnormal. "Temporarily considered normal" means a shard not mentioned in the current error message. Assume there are three data nodes: gnode1, gnode2, and gnode3, deployed in a 1-master-1-slave configuration. gnode1 has shards n1 and n2, gnode2 has shards n2 and n3, and gnode3 has shards n3 and n1. Shards with the same sequence number act as master and slave shards to each other. If shard n1 experiences a master-slave inconsistency, it means both gnode1 and gnode3 may have experienced abnormal power loss. Therefore, shards n2 on gnode1 and n3 on gnode3 cannot participate in the vote. Only shards n2 and n3 on gnode2 can act as voters. Assuming there are three data nodes, gnode1, gnode2, and gnode3, deployed in a 3-master, 2-slave configuration, meaning each of gnode1, gnode2, and gnode3 has shards n1, n2, and n3 respectively, and a master-slave inconsistency occurs between the n1 shard on gnode1 and gnode2, the algorithm determines that only the n1, n2, and n3 shards on gnode3 can act as voters. In this case, the voting result of other backup shards with the same sequence number is prioritized, i.e., the voting result of the n1 shard on gnode3 is prioritized.

[0047] In one possible implementation, determining nodes with the same sequence number as the error-reporting nodes as voting nodes includes: determining all nodes with the same sequence number as voting nodes based on the error-reporting nodes' sequence numbers.

[0048] This implementation is suitable for situations where m backup shards are stored for each shard, and the number of backups is relatively small. It avoids reducing voting efficiency due to an excessive number of voting nodes, and also avoids affecting data recovery efficiency.

[0049] In the actual implementation process, there are not only multiple ways to determine voting nodes, but also different implementation methods for determining the table structure information and target reference information of the shards corresponding to each voting node.

[0050] In one possible implementation, step S103 involves obtaining the table structure information of the shards corresponding to each voting node, including: Based on the sequence number of each voting node, issue a command to view the table creation process and obtain the table structure information returned by each voting node; The table structure information includes one or more of the following: table name, columns, column types, table comments, number of data items, and default values.

[0051] In a specific embodiment, the table structure information also includes data type, whether it is empty, etc.

[0052] Optionally, the command to view the table creation process is the `show create table` command. `show create table` is a commonly used command in MySQL databases for viewing table creation processes. In this embodiment, the command aims to obtain the table structure corresponding to the sharding information of each voting node. To achieve the same purpose, in other embodiments, other commands besides `show create table` can be selected to obtain the table structure of sharding information.

[0053] In one possible implementation, the voting result is determined based on the table structure information, including: Based on the majority voting system, the node with the largest number of identical table structure information is selected as the available node.

[0054] The voting process generally consists of two steps. First, the table structure information is analyzed and compared. The goal of this step is to find nodes with the same table structure information and group them together. Second, the number of nodes with the same table structure information needs to be counted to meet the requirements of majority voting, i.e., the node with the most nodes is selected as the available node. Finally, according to the principle of majority voting, the voting result is determined, and the optimal solution is found among multiple options, thereby providing data support for data recovery.

[0055] In this embodiment, the nodes with the most identical table structure information are selected as available nodes, and data recovery is performed on a smaller number of abnormal nodes, thereby improving the overall data recovery efficiency.

[0056] The above is an introduction to the implementation method of determining the table structure information of the shard corresponding to each voting node. The following is an introduction to the implementation method of determining the metadata information of the shard corresponding to each voting node.

[0057] In one possible implementation, the target reference information for each voting node's corresponding shard is obtained, and the voting result is determined based on the target reference information, including: Obtain the availability of the shards corresponding to each voting node; When only one voting node's corresponding shard is available, the corresponding node is used as the available node, and when all voting nodes' corresponding shards are available, the number of rows in each voting node's corresponding shard is obtained; If the number of rows in the shards corresponding to each voting node is different, the voting node with the largest number of rows is the available node; otherwise, the SCN of the shards corresponding to each voting node is obtained. The node with the most votes corresponding to the same SCN is considered a usable node; otherwise, all nodes are considered usable nodes.

[0058] In some optional implementations, when obtaining the availability of shards corresponding to each voting node, the cluster layer sends a `select * from t where rowid%65536=1` command to the data layer, and determines the availability of abnormal shards based on the feedback from the data layer. If the data layer's feedback is received correctly, the corresponding shard is available; otherwise, the corresponding shard is unavailable.

[0059] If all shards are available, further evaluation is performed on the number of rows in each shard. When determining availability based on row count, the voting node corresponding to the largest number of rows is selected as the available node to ensure data integrity. It's important to note that the shard data corresponding to the largest number of rows is not necessarily the final processing result before the device malfunctioned. For example, if a device malfunction occurs while deleting one or more rows of data but before final confirmation, some node shards may not yet be updated to the deleted state. If the recovered data is the deleted data, it will result in data loss. In this case, recovering the data with the largest number of rows, compared to recovering the deleted data, better ensures maximum data volume and avoids data omission.

[0060] When all shards are available and have the same number of rows, further judgment is made based on the shard's SCN. The larger the SCN, the closer the corresponding shard's data update time is to the current time. When all shards are available and have the same number of rows, the voting node with the largest number of identical SCNs is considered the available node, thus ensuring higher data consistency among the available nodes.

[0061] If the SCNs of all voting nodes are consistent, then the data shards of each node are considered consistent, and data recovery can be performed using the data shards of any node.

[0062] This implementation consists of three voting phases, each based on shard availability, row count, and SCN to determine which nodes are available and which are not. Once a node is clearly distinguished as available or not, the voting process ends, reducing voting time and improving data recovery efficiency. This method is suitable for scenarios with a large number of backup shards, reducing the amount of data acquired before the voting phase, lowering voting pressure, and improving voting efficiency.

[0063] In one possible implementation, the target reference information for each voting node's corresponding shard is obtained, and the voting result is determined based on the target reference information, including: Obtain the availability, number of rows, and SCN of the shard corresponding to each voting node, and determine the voting results according to the majority voting mechanism and priority order; Among these, availability takes precedence over number of rows; and number of rows takes precedence over SCN.

[0064] This implementation consists of three voting phases. Each phase divides the voting process into available and abnormal nodes based on shard availability, row count, and SCN. Once the available and abnormal nodes are clearly distinguished at a given phase, the voting process ends, reducing voting time and improving data recovery efficiency. This method is suitable for scenarios with a small number of backup shards. It reduces the number of times data is retrieved before voting in a multi-voting process, enabling the one-time retrieval of target reference information for each voting phase, and minimizing the impact of data retrieval pressure on voting efficiency.

[0065] In one possible implementation, before obtaining the target reference information for each voting node's corresponding shard, the following is also included: Execute a rollback request or a revert request.

[0066] When metadata anomalies occur, it's usually due to the absence of one or more DML operations. In this case, the cluster layer executes a rollback or revert request to undo the changes already made, ensuring the database returns to its state before the transaction began, thus maintaining data consistency. However, rollback or revert requests cannot completely guarantee data consistency across all shard nodes. Therefore, performing a vote after executing a rollback or revert request can improve the efficiency of distinguishing between abnormal and available nodes.

[0067] In one possible implementation, after determining the voting result based on the table structure information, or after determining the voting result based on the target reference information, the following is also included: The recovery log is recorded based on the voting results to facilitate data recovery.

[0068] In some alternative implementations, dmlstorageevent is recorded based on the voting results for data recovery.

[0069] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0070] In one specific embodiment, the data processing method for a distributed database includes the following steps: 1) Data consistency check: When data inconsistency occurs and the error message "source table and destination table are not the same: src:(l)" is obtained, check the error message as soon as possible to determine which nodes and which shards have inconsistencies between the primary and backup. 2) Voting algorithm: Determine the sharding of abnormal nodes; 1. Handling inconsistent table structures: The cluster layer issues the show create table command, which allows each node shard to return its own table definition. The cluster layer compares the returned table structure information to find the node with the majority vote, thereby determining the shard of the abnormal node and recording dmlstorageevent. 2. Handling inconsistent metadata After detecting inconsistencies and executing a rollback or revert at the cluster layer, the process begins to determine which shard is experiencing the anomaly. The determination process is as follows: (a) Determine the availability of two abnormal shards by selecting * from t where rowid%65536=1. If one is unavailable, the available one shall be used as the standard, and the judgment ends. The available one shall be the available node. (b) Compare the number of data rows in the two abnormal partitions. If the number of rows is different, the one with more rows is taken as the usable node, and the judgment ends. (c) Use a voting method to determine the SCN. If one side's SCN receives a majority of votes, the determination ends, and the node with the majority of votes for its SCN is taken as the available node.

[0071] (d) When the SCNs of all node shards are consistent, randomly select one and force it to be the standard.

[0072] (e) Because power outages may cause damage to metadata files and data files, abnormal fragment records are recorded as dmlstorageevent.

[0073] 3) Data Recovery: Based on the abnormal node record dmlstorageevent, the abnormal node is recovered. Data write operations to the table can then be executed successfully, and the table can continue to be used normally.

[0074] The following are device embodiments of this application. For details not described in detail, please refer to the corresponding method embodiments described above.

[0075] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 3 in this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. When the processor 30 executes the computer program 32, it implements the steps in the data processing method embodiments of the various distributed databases described above, for example... Figure 1 The steps shown.

[0076] For example, the computer program 32 may be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 30 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 32 in the electronic device 3.

[0077] The electronic device 3 can be a desktop computer, laptop, handheld computer, or cloud server, etc. The electronic device 3 may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0078] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0079] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or memory. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store the computer program and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0080] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0081] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0082] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0083] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0086] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the data processing methods of the various distributed databases described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0087] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data processing method for a distributed database, characterized in that: The data table is divided into n shards; each shard stores m backup shards on m nodes with the same sequence number; There is a one-to-one correspondence between m backup shards and m nodes with the same sequence number; The method includes: Obtain error information; wherein the error information includes: error type and error node; According to the sequence number of the error-reporting node, nodes with the same sequence number are determined as voting nodes to determine abnormal nodes and available nodes; When the error type corresponding to the error information is inconsistent table structure, obtain the table structure information of the shards corresponding to each voting node, and determine the voting result according to the table structure information; when the error type corresponding to the error information is inconsistent metadata, obtain the target reference information of the shards corresponding to each voting node, and determine the voting result according to the target reference information; wherein the target reference information includes one or more of availability, number of rows and SCN; the voting result includes abnormal nodes and available nodes; According to the shard information corresponding to the available nodes in the voting results, data recovery is performed on the corresponding node data.

2. The method according to claim 1, characterized in that: The step of determining, according to the sequence number of the error-reporting node, a node with the same sequence number as a voting node includes: According to the sequence number of the error-reporting node, randomly select h nodes with the same sequence number as voting nodes; where h is an odd number; 3≤h<m; or, According to the sequence number of the error-reporting node, all nodes with the same sequence number are determined as voting nodes.

3. The method according to claim 1, characterized in that The obtaining of the table structure information of the shards corresponding to each voting node includes: According to the serial number of each voting node, issue a command to view the table creation process and obtain the table structure information fed back by each voting node; The table structure information includes one or more of the following: table name, column, column type, table comment, data table volume, and default value.

4. The method according to claim 3, characterized in that Determining the voting result according to the table structure information includes: According to the majority voting system, the nodes with the same table structure information and the largest number are selected as available nodes.

5. The method according to claim 1, characterized in that The obtaining target reference information of the shards corresponding to each voting node and determining the voting result according to the target reference information includes: Get the availability of the shards corresponding to each voting node; When only one voting node's corresponding shard is available, the corresponding node is used as an available node, and when all voting nodes' corresponding shards are available, the number of rows of each voting node's corresponding shard is obtained; When the number of rows of the shards corresponding to each voting node is different, the voting node corresponding to the largest number of rows is the available node; otherwise, the SCN of the shard corresponding to each voting node is obtained; The voting nodes with the same SCN and the largest number are considered available nodes; otherwise, all are considered available nodes.

6. The method according to claim 1, characterized in that The obtaining target reference information of the shards corresponding to each voting node and determining the voting result according to the target reference information includes: Obtain the availability, number of rows, and SCN of the corresponding shards of each voting node, and determine the voting results according to the priority order based on the majority voting mechanism; Among them, the availability priority is higher than the row number; the row number priority is higher than the SCN.

7. The method according to claim 1, characterized in that Before obtaining the target reference information of the shards corresponding to each voting node, the method further includes: Execute a rollback request or a revert request.

8. The method according to claim 1, characterized in that After determining the voting result according to the table structure information, or after determining the voting result according to the target reference information, the method further includes: A recovery log is recorded according to the voting result, so as to facilitate data recovery according to the recovery log.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images