Distributed database processing methods, devices, electronic equipment, and readable media

Abstract

This application relates to a method, apparatus, electronic device, and readable medium for processing distributed databases. The method includes: when a first transaction is executed on a first data table and a first shared lock has been generated for the first data table, and a second transaction is awaiting execution, obtaining the target statement of the second transaction; determining whether the target statement meets preset conditions; and if the target statement meets the preset conditions, downgrading the exclusive lock of the target statement to a second shared lock to avoid mutual exclusion between the exclusive lock and the first shared lock. By downgrading the exclusive lock of a statement that meets preset conditions to a shared lock when a shared lock already exists, mutual exclusion between exclusive locks and shared locks is avoided, thus solving the session blocking problem.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a method, apparatus, electronic device, and readable medium for processing a distributed database. Background Technology

[0002] With the rapid development of computer technology and the national economy, traditional centralized databases are increasingly unable to meet the demands of storing and processing massive amounts of data. Distributed databases, with their flexible architecture, high availability, and scalability, can effectively resolve the contradiction between the explosive growth of data volume and the limited capacity and performance of centralized databases. However, during parallel online DDL operations, brief exclusive locks can lead to session blocking, potentially resulting in distributed transaction deadlocks. Once session blocking causes a distributed transaction deadlock, the user must wait for the metadata lock timeout, which has a long-term impact on business operations.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and readable medium for processing distributed databases to solve the aforementioned technical problem of "session blocking".

[0005] According to one aspect of the embodiments of this application, this application provides a method for processing a distributed database, including: when a first shared lock is generated for a first data table during the execution of a first transaction and a second transaction is waiting to be executed, obtaining the target statement of the second transaction; determining whether the target statement meets a preset condition; and if the target statement meets the preset condition, downgrading the exclusive lock of the target statement to a second shared lock to avoid mutual exclusion between the exclusive lock and the first shared lock.

[0006] Optionally, determining whether a target statement meets preset conditions includes: obtaining a preset statement table, wherein the statements in the preset statement table do not undergo actual data changes during execution; searching the preset statement table for the existence of a target statement; if the target statement exists in the preset statement table, then the target statement is determined to meet the preset conditions; if the target statement does not exist in the preset statement table, then the target statement is determined not to meet the preset conditions.

[0007] Optionally, after downgrading the exclusive lock of the target statement to a second shared lock, the method further includes updating the first data table as follows: if target data is detected to be added to the first data table, the first data table is copied to obtain a data table copy; the target data is added to the data table copy to obtain a second data table; and the link referencing the first data table is changed to the second data table so that the data read when a read operation is performed is the data in the second data table.

[0008] Optionally, the method further includes: counting the amount of metadata in the first data table and the amount of metadata in the second data table using atomic variables; if the count value of the atomic variables is zero, then releasing the data in the first data table.

[0009] According to another aspect of the embodiments of this application, this application provides a processing apparatus for a distributed database, including: an acquisition module, configured to acquire a target statement of a second transaction when a first shared lock is generated for the first data table during the execution of a first transaction and a second transaction is waiting to be executed; a determination module, configured to determine whether the target statement meets preset conditions; and a downgrade module, configured to downgrade the exclusive lock of the target statement to a second shared lock when the target statement meets the preset conditions, so as to avoid mutual exclusion between the exclusive lock and the first shared lock.

[0010] Optionally, the determining module includes: an acquisition unit for acquiring a preset statement table, wherein the statements in the preset statement table do not undergo actual data changes during execution; a search unit for searching whether a target statement exists in the preset statement table; and a determining unit for determining that if a target statement exists in the preset statement table, the target statement satisfies the preset conditions, and if a target statement does not exist in the preset statement table, the target statement does not satisfy the preset conditions.

[0011] Optionally, the apparatus further includes: an update module, used to update the first data table after downgrading the exclusive lock of the target statement to a second shared lock. The update module includes: a copy unit, used to copy the first data table to obtain a data table copy if target data to be added to the first data table is detected; an add unit, used to add the target data to the data table copy to obtain a second data table; and a change unit, used to change the links referencing the first data table to the second data table so that the data read when performing a read operation is data from the second data table.

[0012] Optionally, the update module further includes: a counting unit, used to count the number of metadata in the first data table and the number of metadata in the second data table using atomic variables; and a release unit, used to release the data in the first data table if the count value of the atomic variable is zero.

[0013] According to another aspect of the embodiments of this application, this application provides an electronic device, including a memory, a processor, a communication interface and a communication bus. The memory stores a computer program that can run on the processor. The memory and the processor communicate with each other through the communication bus and the communication interface. When the processor executes the computer program, it implements the steps of the above method.

[0014] According to another aspect of the embodiments of this application, this application also provides a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the above-described method.

[0015] Compared with related technologies, the technical solutions provided in this application have the following advantages:

[0016] This application provides a distributed database processing method, comprising: when a first shared lock is generated for a first data table during the execution of a first transaction, and a second transaction is awaiting execution, obtaining the target statement of the second transaction; determining whether the target statement meets preset conditions; and if the target statement meets the preset conditions, downgrading the exclusive lock of the target statement to a second shared lock to avoid mutual exclusion between the exclusive lock and the first shared lock. By downgrading the exclusive lock of a statement that meets preset conditions to a shared lock when a shared lock already exists, mutual exclusion between exclusive locks and shared locks is avoided, thus solving the session blocking problem. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of an optional distributed database processing method provided according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram illustrating a scenario of session blocking in an optional distributed database according to an embodiment of this application.

[0021] Figure 3 This is a block diagram of an optional distributed database processing apparatus according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of an optional electronic device structure provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustration and has no specific meaning in itself. Therefore, "module" and "part" may be used interchangeably.

[0025] With the rapid development of computer technology and the national economy, traditional centralized databases are increasingly unable to meet the demands of storing and processing massive amounts of data. Distributed databases, with their flexible architecture, high availability, and scalability, can effectively resolve the contradiction between the explosive growth of data volume and the limited capacity and performance of centralized databases. However, as business processes iterate and change, the table structure of databases often changes as well. Operations such as DDL (Data Definition Language) typically involve an MDL (Meta Data Lock) locking process. During this process, even if there are active transactions on the table, write operations on metadata are not allowed. Therefore, many online service upgrades are scheduled to avoid peak traffic periods to prevent impacting online business, and sometimes, when service update cycles are long, service may be suspended altogether.

[0026] In a single-machine database, the online DDL operation process consists of three phases: Prepare, DDL execution, and Commit. In the Prepare and Commit phases, exclusive metadata locks exist, prohibiting the reading and writing of DML (Data Manipulation Language) statements. Only in the DDL execution phase is this lock downgraded to a shared metadata lock, allowing DML statement reading and writing. This approach is clearly unfeasible in distributed databases. Currently, distributed databases can support parallel DDL across shards, although the Prepare and Commit phases in a single-machine database are significantly shorter than the DDL execution phase.

[0027] However, during parallel online DDL, even a brief exclusive lock can cause session blocking, which may lead to distributed transaction deadlock. Once a distributed transaction deadlock occurs, it is necessary to wait for the timeout period of the metadata lock, which has a long impact on business.

[0028] To address the problems mentioned in the background art, according to one aspect of an embodiment of this application, a method for processing a distributed database is provided, such as... Figure 1 As shown, it includes:

[0029] Step 101: When a first shared lock is generated for the first data table during the execution of the first transaction and the second transaction is waiting to be executed, obtain the target statement of the second transaction;

[0030] Step 103: Determine whether the target statement meets the preset conditions;

[0031] Step 105: If the target statement meets the preset conditions, downgrade the exclusive lock of the target statement to the second shared lock to avoid mutual exclusion between the exclusive lock and the first shared lock.

[0032] Deadlock refers to a situation where two or more processes are waiting for each other due to competing for resources. Without external intervention, they will be unable to proceed. At this point, the system is said to be in a deadlock state or a deadlock has occurred. These processes that are perpetually waiting for each other are called deadlocked processes.

[0033] To avoid deadlocks in distributed transactions, the GoldenDB distributed database reduces the exclusive MDL lock of DDL statements that do not actually operate on data to a shared lock, thus preventing the blocking of DDL statements and subsequent DML statements.

[0034] The solution provided in this application solves the problem of session blocking caused by executing DDL statements without actual data operations in distributed transaction scenarios, thereby improving the database's response speed and high availability.

[0035] As an optional embodiment, determining whether a target statement meets preset conditions includes: obtaining a preset statement table, wherein the statements in the preset statement table do not undergo actual data changes during execution; searching the preset statement table to see if a target statement exists; if a target statement exists in the preset statement table, then the target statement is determined to meet the preset conditions; if a target statement does not exist in the preset statement table, then the target statement is determined not to meet the preset conditions.

[0036] The solution provided in this application is designed to prevent DML operations from being blocked during DDL processes without actual data operations, thereby improving the database's response speed and high availability. Therefore, the default condition is that the target statement does not involve any actual data operations.

[0037] You can first store statements that do not actually operate on data into a preset statement table, and then check whether the target statement exists in the preset statement table to determine whether the preset conditions are met.

[0038] Specifically, the statements in the preset statement table include, but are not limited to: CREATE INDEX (for creating an index in a table); DROP INDEX (for deleting an index); ALTER TABLE ADD INDEX (for adding an index using the ALTER TABLE statement); ALTER TABLE DROP INDEX (for deleting an index using the ALTER TABLE statement); partition operations ADD / DROP / COALESCE / TRUNCATE / REMOVE / REORGANIZE PARTITION; adding columns without specifying position information (FIRST / AFTERCOLUMN) and NOT NULL (no default value specified) are considered offline DDL operations; adding multiple columns without specifying FIRST / AFTER is considered online DDL; modifying fields with COMMENT; and extending the length of VARCHAR type fields.

[0039] For example, Figure 2 The original DDL statements in the distributed database system provided in this application may encounter deadlock scenarios in distributed transactions, as shown in the figure. With three sessions open, sessions one and two execute SELECT statements on shards one and two respectively, adding a shared lock to table T. Then, in session three, an ALTER statement is executed, which adds an exclusive lock to table T. This causes the DDL statements in session three to be blocked. This is because when the SELECT statement is executed, a metadata shared lock is added, which blocks the metadata exclusive lock; they are mutually exclusive. Similarly, when sessions one and two execute DML statements, they will also be blocked by the exclusive lock existing in session three. In this scenario, once a distributed deadlock occurs, the user needs to wait for the metadata lock, resulting in a long time impact on business operations and affecting database response speed and high availability.

[0040] The solution provided in this application can effectively avoid deadlock caused by parallel DDL, and can greatly shorten batch processing time compared to serial DDL.

[0041] According to the solution provided in this application, the above situation arises from the mutual exclusion between exclusive locks and shared locks, leading to subsequent session blocking and ultimately a distributed transaction deadlock. Therefore, the DDL statements in Session 3 can be differentiated: for DDL statements without actual data operations, their metadata exclusive lock can be downgraded to a second shared lock. This avoids the mutual exclusion between the two types of locks that occur when a transaction has a shared lock, allowing the DDL statements to execute normally without session blocking.

[0042] As an optional embodiment, after downgrading the exclusive lock of the target statement to a second shared lock, the method further includes updating the first data table in the following manner: if target data is detected to be added to the first data table, the first data table is copied to obtain a data table copy; the target data is added to the data table copy to obtain a second data table; and the link referencing the first data table is changed to the second data table so that the data read when a read operation is performed is the data in the second data table.

[0043] For example, when new data is added to a table, it is not added directly to the current table. Instead, the current table is copied to create a new table, and then the data is added to the new table. After the data is added, the reference of the original table is set to the new table. When a read operation is performed, the data is read from the current old table.

[0044] This application utilizes the concept of read-write separation, allowing reads and writes to be performed in different tables. Concurrent reads can be performed on the current table without the need for locking, because no new data is added to the current table. Once the write operation is complete and the reference points to the new table, data can be read directly from the new table.

[0045] As an optional embodiment, the method further includes: counting the amount of metadata in the first data table and the amount of metadata in the second data table using atomic variables; if the count value of the atomic variables is zero, then releasing the data in the first data table.

[0046] For example, to ensure the integrity of metadata, an atomic variable is used to count the metadata before and after the copy during the creation of new metadata (i.e., the number of metadata in the table structure before copying and the number of metadata in the table structure after copying). When the value of the atomic variable is zero, it means that the metadata before and after is consistent, and the original data can be released.

[0047] This application creates new metadata by copy-on-write and uses atomic variables to count the metadata before and after it, thus solving the problem of releasing the original metadata.

[0048] This application provides a distributed database processing method, comprising: when a first shared lock is generated for a first data table during the execution of a first transaction, and a second transaction is awaiting execution, obtaining the target statement of the second transaction; determining whether the target statement meets preset conditions; and if the target statement meets the preset conditions, downgrading the exclusive lock of the target statement to a second shared lock to avoid mutual exclusion between the exclusive lock and the first shared lock. By downgrading the exclusive lock of a statement that meets preset conditions to a shared lock when a shared lock already exists, mutual exclusion between exclusive locks and shared locks is avoided, thus solving the session blocking problem.

[0049] According to another aspect of the embodiments of this application, this application provides a processing apparatus for a distributed database, such as... Figure 3 As shown, it includes:

[0050] The acquisition module 302 is used to acquire the target statement of the second transaction when a first shared lock is generated for the first data table during the execution of the first transaction and the second transaction is waiting to be executed.

[0051] Module 304 is used to determine whether the target statement meets preset conditions.

[0052] The downgrade module 306 is used to downgrade the exclusive lock of the target statement to a second shared lock when the target statement meets the preset conditions, so as to avoid the exclusive lock and the first shared lock being mutually exclusive.

[0053] It should be noted that the acquisition module 302 in this embodiment can be used to execute step 101 in this application embodiment, the determination module 304 in this embodiment can be used to execute step 103 in this application embodiment, and the downgrade module 306 in this embodiment can be used to execute step 105 in this application embodiment.

[0054] Optionally, the determining module 304 includes: an acquisition unit for acquiring a preset statement table, wherein the statements in the preset statement table do not undergo actual data changes during execution; a search unit for searching whether a target statement exists in the preset statement table; and a determining unit for determining that if a target statement exists in the preset statement table, the target statement satisfies the preset conditions, and if a target statement does not exist in the preset statement table, the target statement does not satisfy the preset conditions.

[0055] Optionally, the apparatus further includes: an update module, used to update the first data table after downgrading the exclusive lock of the target statement to a second shared lock. The update module includes: a copy unit, used to copy the first data table to obtain a data table copy if target data to be added to the first data table is detected; an add unit, used to add the target data to the data table copy to obtain a second data table; and a change unit, used to change the links referencing the first data table to the second data table so that the data read when performing a read operation is data from the second data table.

[0056] Optionally, the update module further includes: a counting unit, used to count the number of metadata in the first data table and the number of metadata in the second data table using atomic variables; and a release unit, used to release the data in the first data table if the count value of the atomic variable is zero.

[0057] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments.

[0058] According to another aspect of the embodiments of this application, this application provides an electronic device, such as... Figure 4 As shown, the system includes a memory 401, a processor 403, a communication interface 405, and a communication bus 407. The memory 401 stores a computer program that can run on the processor 403. The memory 401 and the processor 403 communicate through the communication interface 405 and the communication bus 407. When the processor 403 executes the computer program, it implements the steps of the above method.

[0059] The memory and processor in the aforementioned electronic devices communicate with each other via a communication bus and a communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0060] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0061] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0062] According to another aspect of the embodiments of this application, a computer-readable medium having processor-executable non-volatile program code is also provided.

[0063] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0064] In specific implementation, the embodiments of this application can be referred to the above embodiments and have corresponding technical effects.

[0065] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0066] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0067] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0068] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0069] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0072] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks. It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0073] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for processing a distributed database, characterized in that, include: If a first shared lock is generated for the first data table during the execution of the first transaction, and the second transaction is waiting to be executed, then obtain the target statement of the second transaction; Determine whether the target statement meets the preset conditions; If the target statement meets the preset conditions, the exclusive lock of the target statement is downgraded to the second shared lock to avoid the exclusive lock and the first shared lock being mutually exclusive. Determining whether the target statement meets the preset conditions includes: Obtain a preset statement table, wherein the statements in the preset statement table do not undergo actual data changes during execution; Search the preset statement table to see if the target statement exists; If the target statement exists in the preset statement table, then the target statement is determined to satisfy the preset condition; if the target statement does not exist in the preset statement table, then the target statement is determined not to satisfy the preset condition.

2. The method according to claim 1, characterized in that, After downgrading the exclusive lock of the target statement to a second shared lock, the method further includes updating the first data table in the following manner: If target data is detected to be added to the first data table, the first data table is copied to obtain a copy of the data table; The target data is added to the copy of the data table to obtain a second data table; Change the link referencing the first data table to the second data table so that the data read during a read operation is data from the second data table.

3. The method according to claim 2, characterized in that, The method further includes: The number of metadata entries in the first data table and the number of metadata entries in the second data table are counted using atomic variables. If the count value of the atomic variable is zero, then the data in the first data table is released.

4. A processing device for a distributed database, characterized in that, include: The acquisition module is used to acquire the target statement of the second transaction when a first shared lock is generated for the first data table during the execution of the first transaction and the second transaction is waiting to be executed. The determination module is used to determine whether the target statement meets preset conditions; The downgrade module is used to downgrade the exclusive lock of the target statement to a second shared lock when the target statement meets the preset conditions, so as to avoid the exclusive lock and the first shared lock being mutually exclusive; The determining module includes: The acquisition unit is used to acquire a preset statement table, wherein the statements in the preset statement table do not undergo actual data changes during execution; A search unit is used to search the preset statement table to determine whether the target statement exists. The determining unit is configured to determine that the target statement satisfies the preset condition if the target statement exists in the preset statement table, and to determine that the target statement does not satisfy the preset condition if the target statement does not exist in the preset statement table.

5. The apparatus according to claim 4, characterized in that, The apparatus further includes: an update module, configured to update the first data table after downgrading the exclusive lock of the target statement to a second shared lock, the update module comprising: The copying unit is used to copy the first data table to obtain a copy of the data table if it detects that target data is to be added to the first data table. An adding unit is used to add the target data to the copy of the data table to obtain a second data table; The modification unit is used to change the link referencing the first data table to the second data table, so that the data read when performing a read operation is the data in the second data table.

6. The apparatus according to claim 5, characterized in that, The update module also includes: A counting unit is used to count the amount of metadata in the first data table and the amount of metadata in the second data table using atomic variables; A release unit is used to release data in the first data table if the count value of the atomic variable is zero.

7. An electronic device comprising a memory, a processor, a communication interface, and a communication bus, wherein the memory stores a computer program executable on the processor, and the memory and the processor communicate via the communication bus and the communication interface, characterized in that... When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

8. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the method of any one of claims 1 to 3.

Images