Database processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a database processing method and device, electronic equipment and a storage medium. When a first transaction comprising at least one first data language statement is received, a first partition metadata lock is added to a target partition table corresponding to the first transaction; adding a target partition metadata lock for a first target partition corresponding to the first transaction; when the target partition metadata lock is added, releasing a first partition metadata lock corresponding to the target partition table; based on the target partition metadata lock, executing at least one first data language statement corresponding to the first transaction, so as to execute at least one second data language statement corresponding to the target partition table when a second transaction comprising at least one second data language statement for the target partition table is received in the process of executing the first transaction. And under the condition that the second target partition corresponding to the second transaction is inconsistent with the first target partition, the first transaction and the second transaction are executed in parallel, so that the concurrency of the partition table in the database is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a database processing method, device, electronic equipment and storage medium. Background Art

[0002] In mainstream database systems, Data Definition Language (DDL) statements are SQL statements used to define database schemas. They are used to create, modify, and delete tables, views, indexes, and other objects in a database. Generally, DDL execution involves the modification of metadata and data, so DDL and other statements cannot be executed in parallel, otherwise some unknown problems will arise. Partitioned tables are a database technology that divides large tables into smaller, more manageable parts. Each part is called a partition, which is usually divided according to the value of a column (such as date or ID). By dividing large tables into smaller partitions, data can be queried and updated faster. Statements involving partitioned tables, such as Data Manipulation Language (DML) and Data Query Language (DQL), are very common in daily database operations.

[0003] Currently, for a partitioned table, during the execution of statements or DDL statements involving the partitioned table, the entire table is locked, and the table cannot be accessed by other sessions during the entire transaction involving statements or DDL statements involving the partitioned table. It can be seen that the existing database processing method has the problem that DML statements and DDL statements that modify different partitions of the partitioned table cannot be executed in parallel, which seriously limits the concurrency of the partitioned table in the database. Summary of the invention

[0004] The present invention provides a database processing method, device, electronic device and storage medium to solve the problem that different statement transactions for modifying different partitions of a partition table cannot be processed in parallel, thereby improving the concurrency of the partition table in the database.

[0005] In a first aspect, an embodiment of the present invention provides a database processing method, the method comprising:

[0006] When a first transaction including at least one first data language statement is received, after adding a first partition metadata lock to a target partition table corresponding to the first transaction, a target partition metadata lock is added to a first target partition corresponding to the first transaction;

[0007] When the target partition metadata lock is added, releasing the first partition metadata lock corresponding to the target partition table;

[0008] Based on the target partition metadata lock, the at least one first data language statement corresponding to the first transaction is executed, so that during the execution of the first transaction, when a second transaction including at least one second data language statement is received for the target partition table, the first transaction and the second transaction are executed in parallel if a second target partition corresponding to the second transaction is inconsistent with the first target partition.

[0009] In a second aspect, an embodiment of the present invention further provides a database processing device, the device comprising:

[0010] a target lock adding module, configured to, when receiving a first transaction including at least one first data language statement, add a first partition metadata lock to a target partition table corresponding to the first transaction, and then add a target partition metadata lock to a first target partition corresponding to the first transaction;

[0011] A first lock releasing module, configured to release the first partition metadata lock corresponding to the target partition table when the target partition metadata lock is added;

[0012] A transaction parallel execution module is used to execute the at least one first data language statement corresponding to the first transaction based on the target partition metadata lock, so that during the execution of the first transaction, when a second transaction including at least one second data language statement is received for the target partition table, the first transaction and the second transaction are executed in parallel if the second target partition corresponding to the second transaction is inconsistent with the first target partition.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] The storage device is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the database processing method as any one of the embodiments of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute a database processing method as described in any one of the embodiments of the present invention.

[0017] The technical solution of the embodiment of the present invention is that when a first transaction including at least one first data language statement is received, after adding a first partition metadata lock to the target partition table corresponding to the first transaction, a target partition metadata lock is added to the first target partition corresponding to the first transaction, and then, when the target partition metadata lock is added, the first partition metadata lock corresponding to the target partition table is released, so that, based on the target partition metadata lock, at least one first data language statement corresponding to the first transaction is executed, so that in the process of executing the first transaction, when a second transaction including at least one second data language statement for the target partition table is received, the first transaction and the second transaction are executed in parallel when the second target partition corresponding to the second transaction is inconsistent with the first target partition. The technical solution provided by this embodiment solves the problem that different statement transactions involving modifications to different partitions of a partition table cannot be executed in parallel, reduces the lock granularity of the database, enables different statement transactions involving different partitions of the database partition table to be executed concurrently, and improves the concurrency of the partition table in the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required to describe the embodiments. Obviously, the drawings introduced are only drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a database processing method provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of an implementation flow of a database processing method involved in this embodiment;

[0021] Figure 3 A schematic diagram of the implementation flow of another database processing method involved in this embodiment;

[0022] Figure 4 A flowchart of a database processing method provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of an implementation flow of another database processing method involved in this embodiment;

[0024] Figure 6 A schematic diagram of the structure of a database processing device provided by an embodiment of the present invention;

[0025] Figure 7 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Embodiment 1

[0027] Figure 1 A flowchart of a database processing method provided in an embodiment of the present invention is provided. This embodiment is applicable to a scenario in which transactions of different statement types are executed in parallel in different partitions of a database partition table. The method can be executed by a database processing device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal, a PC, or a server.

[0028] like Figure 1 As shown, the database processing method includes:

[0029] S110. When a first transaction including at least one first data language statement is received, after adding a first partition metadata lock to a target partition table corresponding to the first transaction, a target partition metadata lock is added to a first target partition corresponding to the first transaction.

[0030] The first transaction is a database transaction consisting of one or more first data language statements. A database transaction refers to a set of operations on a database, which are either all executed or not executed, and has four major characteristics: atomicity, consistency, isolation, and persistence. A database may include multiple partition tables, and the target partition table is the partition table involved in the first transaction. The partition table includes multiple partitions, and the first target partition refers to the partition involved in the first transaction.

[0031] Among them, the first data language statement can be a data definition language statement (DDL) or a statement involving a partition table (DML, DQL, etc.). The data definition language (DDL) is used to define and modify the structure of a database and database objects. DDL includes operations of creating, modifying, and deleting a database or table. In this embodiment, if the first data language statement is a data definition language statement (DDL), the first data language statement includes but is not limited to statements for adding partitions, deleting partitions, TRUNCATE partition statements (clearing data in a partition), compressing partitions, and rebuilding partitions for the target partition table. The data manipulation language (DML) is used to add, delete, and modify data in a database. In this embodiment, if the first data language statement is a data manipulation language (DML), the first data language statement may include statements for adding, deleting, modifying, and checking data in the first target partition.

[0032] Among them, the first partition metadata lock is a partition metadata lock of the first type, for example, the first type is a statement-level type, that is, the first partition metadata lock is a statement-level partition metadata lock. The target partition metadata lock is a partition metadata lock of the second type, for example, the first type is a transaction-level type, that is, the target partition metadata lock is a transaction-level partition metadata lock. More specifically, the metadata lock is used to control concurrent access to database metadata. Metadata refers to data that describes data, such as the structure of a table (column name, data type, etc.). When a transaction attempts to modify the structure of a database (for example, add or delete a column), it needs to first obtain a metadata lock for the corresponding object. Before this transaction is committed or rolled back, other transactions that attempt to access or modify the object must wait, that is, be blocked, thereby preventing data inconsistency or errors. A partition metadata lock refers to a global lock that protects a partition. It can be set to a statement level or a transaction level, and can be divided into types such as read, write, upgradeable, and exclusive locks. In this embodiment, when a DML / DDL statement is executed, the partition information that the current SQL will modify is obtained, and a partition metadata lock is added to the corresponding partition. In other words, a partition metadata lock is used to lock the metadata of a partition in a partition table to ensure data consistency when modifying the partition table structure. Statement-level locks and transaction-level locks in databases are two different lock mechanisms, which differ in locking scope and effect. Specifically, statement-level locks refer to locking data access during the execution of a single statement. When a statement is executed, it locks the data rows or objects involved until the statement is executed. This lock has a smaller scope and is limited to the data involved in the currently executed statement. For example, when a SELECT statement in a transaction is executed, it only locks the data rows involved in the query without affecting the execution of other statements. Statement-level locks are usually used to ensure data consistency for a single operation, but do not provide guarantees for the integrity of the entire transaction. Transaction-level locks refer to locking all operations in the entire transaction. When a transaction starts, all operations involved are executed within a locked scope. Transaction-level locks ensure that all operations within the transaction are either all successful or all failed, and there will be no partial execution. The locking scope of transaction-level locks is larger, covering all statements in the transaction, from the beginning to the commit or rollback. The main purpose of transaction-level locks is to ensure the atomicity, consistency, isolation, and durability of transactions, and to ensure data consistency and integrity.

[0033] Specifically, when the first data language statement is a DDL statement or a DML statement, the specific implementation process of S110 is consistent. For the convenience of explanation, the first data language statement can be taken as an example of a DDL statement. When a first transaction including one or more DDL statements is received, the target partition table involved in the first transaction can be determined. Thus, a first partition metadata lock can be added to each partition of each target partition table. The purpose of adding a first partition metadata lock to each partition is that the first target partition involved in the first transaction is not yet determined, so all partitions can only be locked separately. Furthermore, after the first partition metadata lock is added to each partition, the relevant information of the first target partition involved in the first transaction is obtained, so as to add a target partition metadata lock for the first target partition.

[0034] On the basis of the above embodiment, optionally, when a first transaction including at least one first data language statement is received, after adding a first partition metadata lock for the target partition table corresponding to the first transaction, a specific implementation method of adding a target partition metadata lock for the first target partition corresponding to the first transaction may include: when a first transaction including at least one first data language statement is received, determining the target partition table corresponding to the first transaction, and adding a statement-level metadata lock for the target partition table; obtaining metadata information of the target partition table to determine all partitions included in the target partition table according to the metadata information; after adding a statement-level partition metadata lock for each partition included in the target partition table, releasing the statement-level metadata lock corresponding to the target partition table; after the statement-level metadata lock is released, obtaining at least one first target partition corresponding to the first transaction, and adding a transaction-level partition metadata lock for the at least one first target partition.

[0035] Among them, metadata lock refers to a global lock that protects the metadata of the partition table. It can be set to statement level or transaction level, and can be divided into read, write, upgradeable and exclusive locks. By adding metadata locks to the database, the problem of concurrent metadata reading / modification caused by concurrent execution of DDL / DML can be avoided, ensuring the security and robustness of database operation. Statement-level metadata locks are statement-type metadata locks, and the metadata locks here can protect the metadata of the entire target partition table. For example, a statement-level metadata lock can specifically be a metadata shared read lock at the table statement level.

[0036] Metadata information refers to the data stored in the database that describes the structure and properties of the target partition table. For example, the metadata information of a partition table includes the following aspects: Partition column: records the partition column information of the partition table, that is, the field used to define the partition. Partition value: records the specific value of each partition column, which is used to determine the specific partition of the data. Partition path: records the storage path of the partition to help locate the data storage location. Partition attribute: records other attribute information of the partition, such as the creation time and modification time of the partition.

[0037] In this embodiment, when a first transaction including one or more first data language statements is received, the target partition table corresponding to the first transaction can be determined based on the additional information carried in the first transaction. At this time, in order to ensure that the target partition table will not be accessed by other sessions, a statement-level metadata lock can be added to the target partition table. Further, metadata information of the target partition table can be obtained, and which partitions the target partition table includes can be determined based on the metadata information. Further, a statement-level partition metadata lock can be added to each partition included in the target partition table. Further, after the first partition metadata lock is added to each partition, the statement-level metadata lock previously added to the target partition table is released. Further, after the statement-level metadata lock on the target partition table is released, the first target partitions involved in the first transaction are obtained, and transaction-level partition metadata locks are added to these first target partitions.

[0038] The technical solution provided by the embodiment of the present invention designs a partition metadata lock. During the execution of a DML transaction or a DDL transaction, a partition metadata lock is added to the partition targeted by the DML transaction or the DDL transaction, while other partitions can still be accessed by other sessions, so as to improve the concurrency of the database and the user experience.

[0039] S120. When the target partition metadata lock is added, release the first partition metadata lock corresponding to the target partition table.

[0040] In this embodiment, when the target partition metadata lock on the first target partition is added completely, the first partition metadata lock previously added to each partition corresponding to the target partition table may be released.

[0041] S130. Based on the target partition metadata lock, execute at least one first data language statement corresponding to the first transaction, so that during the execution of the first transaction, when a second transaction including at least one second data language statement is received for the target partition table, if the second target partition corresponding to the second transaction is inconsistent with the first target partition, the first transaction and the second transaction are executed in parallel.

[0042] Among them, the second transaction is a database transaction composed of one or more second data language statements. It should be particularly noted that, when the language type corresponding to the first data language statement contained in the first transaction is a data manipulation language statement (DML) or a data query language statement (DQL), the language type corresponding to the second data language statement is a data definition language statement (DDL). When the language type corresponding to the first data language statement is a data definition language statement (DDL), the language type corresponding to the second data language statement is a data manipulation language statement (DML) or a data query language statement (DQL). The second target partition refers to the partition involved in the second transaction.

[0043] In this embodiment, one or more first data language statements corresponding to the first transaction can be executed when the target partition metadata lock is added to the first target partition. In this way, since the first target partition involved in the first transaction has added the target partition metadata lock, the first target partition cannot execute the second transaction in parallel during the execution of the first transaction; since other partitions corresponding to the target partition table have released the first partition metadata lock, these partitions can still execute the second transaction during the subsequent execution of the first transaction. Therefore, during the execution of the first transaction, when the second transaction for the target partition table is received, when the second target partition corresponding to the second transaction is obtained, it can be further determined whether the second target partition is consistent with the first target partition; if not, the first transaction and the second transaction can be executed in parallel.

[0044] Exemplarily, the database processing method provided in this embodiment is described by taking a first transaction including multiple DML statements as an example. For a schematic diagram of a specific implementation flow of the database processing method, see Figure 2 .like Figure 2 As shown, for a first transaction of the DML type (DML transaction for short), the implementation process of the DML transaction may include: a transaction parsing phase, a preprocessing phase, an optimization phase, an execution phase, and a commit phase.

[0045] Transaction parsing phase: When a DML transaction including multiple DML statements is received, it enters the transaction parsing phase (Parse phase).

[0046] Preprocessing phase: After the transaction parsing phase is completed, the preprocessing phase (Prepare phase) is entered. In the preprocessing phase, first, the target partition table corresponding to the DML transaction is opened, and a statement-level metadata lock is added to the target partition table; further, the metadata information of the target partition table is obtained to determine all table partitions contained in the target partition table based on the metadata information, thereby adding a statement-level partition metadata lock to each table partition contained in the target partition table; finally, the statement-level metadata lock corresponding to the target partition table is released.

[0047] Optimization phase: After the preprocessing phase is completed, the optimization phase (Optimize phase) is entered. First, at least one first target table partition corresponding to the DML transaction is obtained, and then transaction-level partition metadata locks are added to these first target table partitions. When the target partition metadata locks are added, the first partition metadata locks of all table partitions of the target partition table are released.

[0048] Execution phase: After the optimization phase is completed, the execution phase (Execute phase) is entered. During the execution of the DML transaction, the target partition metadata lock on the first target table partition takes effect, which is mutually exclusive with other DDLs that use these partitions. That is, during the execution of the DML transaction, when the second transaction (referred to as the DDL transaction) of the DDL statement for the target partitioned table is received, if the second target table partition corresponding to the DDL transaction is consistent with the first target table partition, the DDL transaction will be blocked; if the second target table partition is inconsistent with the first target table partition, the DML transaction and the DDL transaction can be executed in parallel.

[0049] Commit stage: Finally, the commit stage is entered. When the processing result corresponding to the DML transaction is obtained, the transaction-level partition metadata lock corresponding to the first target table partition is released.

[0050] For further example, the database processing method provided in this embodiment is described by taking the first transaction including multiple DDL statements as an example. For a specific implementation flow diagram of the database processing method, see Figure 3 .like Figure 3 As shown, for the first transaction of the DDL type (referred to as DDL transaction), the implementation process of the DDL transaction may include: transaction parsing stage, preprocessing stage, execution stage and submission stage. It should be noted that the optimization stage is mainly query optimization, and the DDL transaction does not involve data query, so the DDL transaction does not consider the optimization stage.

[0051] Transaction parsing phase: When a DDL transaction including multiple DDL statements is received, the transaction parsing phase (Parse phase) is entered.

[0052] Preprocessing phase: After the transaction parsing phase is completed, it enters the preprocessing phase (Prepare phase). The processing process of the preprocessing phase is consistent with the process of the previous example and will not be repeated here.

[0053] Execution phase: After the preprocessing phase is completed, enter the execution phase (Execute phase). First, obtain at least one first target table partition corresponding to the DDL transaction, and then add transaction-level partition metadata locks for these first target table partitions. When the target partition metadata locks are added, release the first partition metadata locks of all table partitions of the target partition table. Furthermore, during the execution of the DDL transaction, the target partition metadata lock on the first target table partition takes effect and is mutually exclusive with other DMLs that use these partitions. That is, during the execution of the DDL transaction, when a second transaction (DML transaction) of a DML statement for the target partitioned table is received, if the second target table partition corresponding to the DML transaction is consistent with the first target table partition, the DML transaction will be blocked; if the second target table partition is inconsistent with the first target table partition, the DDL transaction and the DML transaction can be executed in parallel.

[0054] Commit stage: Finally, the commit stage is entered. When the processing result corresponding to the DML transaction is obtained, the transaction-level partition metadata lock corresponding to the first target table partition is released.

[0055] Based on the above embodiment, optionally, during the process of executing the first transaction and the second transaction in parallel, when deadlock warning information of the first transaction and the second transaction is received, a target transaction whose language type is a data definition language statement is determined from the first transaction and the second transaction; the thread corresponding to the target transaction is closed, and when the first transaction or the second transaction outside the target transaction is executed, the thread corresponding to the target transaction is restarted.

[0056] Among them, deadlock warning information refers to the warning information generated when multiple threads enter a deadlock state. Deadlock state means that during the operation of the system, two or more processes or threads are detected to be in a state of waiting for each other due to competition for resources or communication with each other, which makes them unable to continue to execute. This state is called deadlock. Without external intervention, these processes will never be able to move forward.

[0057] In a high-concurrency actual environment, partition metadata locks and metadata locks may cause deadlock problems, which may reduce database performance. In response to the above problems, this embodiment optimizes the deadlock detection process and preferentially kills the DDL transaction holding the lock.

[0058] Specifically, the partition metadata lock is used to protect the partition data from being accessed and modified by other DML statements; the metadata lock is used to protect the metadata security of the target partition table. The two locks will definitely lead to deadlocks due to their locking logic and order caused by code design. This problem is a deadlock that is inevitably generated by the design. The DDL statement and the DML statement block each other until the deadlock alarm is reported, which is in line with the expected blocking. In this embodiment, in the process of executing the first transaction and the second transaction in parallel, when the deadlock alarm information of the first transaction and the second transaction is received, the DDL transaction is determined as the target transaction from the first transaction and the second transaction, so that the thread corresponding to the target transaction is closed, and when the DML transaction is executed, the thread corresponding to the target transaction is restarted. The purpose of this design is that in the actual production environment, the priority of the DML statement is higher than that of the DDL statement. When encountering such a deadlock, that is, when the two locks are detected to be deadlocked, the DDL transaction thread holding the lock can be killed, and the DML statement business execution is given priority. At this time, the DDL statement can be terminated with an error, and can be repeatedly executed later to ensure the security and robustness of the database operation.

[0059] The technical solution of the embodiment of the present invention is that when a first transaction including at least one first data language statement is received, after adding a first partition metadata lock to the target partition table corresponding to the first transaction, a target partition metadata lock is added to the first target partition corresponding to the first transaction, and then, when the target partition metadata lock is added, the first partition metadata lock corresponding to the target partition table is released, so that, based on the target partition metadata lock, at least one first data language statement corresponding to the first transaction is executed, so that in the process of executing the first transaction, when a second transaction including at least one second data language statement for the target partition table is received, the first transaction and the second transaction are executed in parallel when the second target partition corresponding to the second transaction is inconsistent with the first target partition. The technical solution provided by this embodiment solves the problem that different statement transactions involving modifications to different partitions of a partition table cannot be executed in parallel, reduces the lock granularity of the database, enables different statement transactions involving different partitions of the database partition table to be executed concurrently, and improves the concurrency of the partition table in the database.

[0060] Embodiment 2

[0061] Figure 4 This is a flowchart of a database processing method provided by Embodiment 2 of the present invention. On the basis of the above-mentioned embodiment, when the first data language statement contained in the first transaction is a data definition language statement, the database processing method provided by this embodiment can add a transaction-level upgradeable partition metadata lock to the first target partition corresponding to the first transaction. The specific implementation method can refer to the detailed description of the embodiment of the present disclosure. Among them, the technical features that are the same or similar to the above-mentioned embodiments are not repeated here.

[0062] like Figure 4 As shown, the database processing method includes:

[0063] S210. When a first transaction including at least one first data language statement is received, after adding a first partition metadata lock to a target partition table corresponding to the first transaction, a transaction-level upgradeable partition metadata lock is added to a first target partition corresponding to the first transaction.

[0064] The transaction-level upgradeable partition metadata lock is a transaction-level partition metadata lock that can be upgraded or downgraded during the execution of the first transaction.

[0065] S220. When the transaction-level upgradeable partition metadata lock is added, release the first partition metadata lock corresponding to the target partition table.

[0066] S230. Execute at least one first data language statement corresponding to the first transaction based on the transaction-level scalable partition metadata lock.

[0067] Accordingly, based on the transaction-level scalable partition metadata lock, a specific implementation method of executing at least one first data language statement corresponding to the first transaction may include:

[0068] S2301. In a process of executing a first transaction based on a transaction-level scalable partition metadata lock, for at least one first data language statement corresponding to the first transaction, determine whether the current first data language statement needs to protect data in a first target partition.

[0069] The current first data language statement refers to the first data language statement currently being processed.

[0070] In this embodiment, in the process of executing the first transaction (i.e., DDL transaction) based on the transaction-level scalable partition metadata lock, the DDL transaction may include multiple intermediate processing flows, each of which may correspond to one or more first data language statements. When a certain intermediate processing flow is specifically executed, it is determined whether the current first data language statement corresponding to the intermediate processing flow needs to protect the data in the first target partition based on the additional information of the intermediate processing flow. If yes, S2302 is executed; if no, S2303 is executed.

[0071] S2302. If the current first data language statement needs to protect the data in the first target partition, the transaction-level upgradeable partition metadata lock is upgraded so that when a second transaction for the target partition table is received and the second target partition corresponding to the second transaction is inconsistent with the first target partition, the first transaction and the second transaction are executed in parallel.

[0072] In this embodiment, if the current first data language statement corresponding to the current intermediate processing flow needs to protect the data in the first target partition, at this time, the transaction-level upgradeable partition metadata lock can be upgraded. In this case, during the execution of the current intermediate processing flow, if a second transaction (i.e., a DML transaction) for the target partition table is received, it can be determined whether the second target partition corresponding to the DML transaction is consistent with the first target partition; if they are inconsistent, the DDL transaction and the DML transaction can be executed in parallel. In particular, if they are consistent, the DML transaction will be blocked.

[0073] S2303: If the current first data language statement does not need to protect the data in the first target partition, downgrade the transaction-level upgradeable partition metadata lock so that when a second transaction for the target partition table is received, the first transaction and the second transaction are executed in parallel.

[0074] In this embodiment, if the current first data language statement corresponding to the current intermediate processing flow does not need to protect the data in the first target partition, at this time, the transaction-level upgradeable partition metadata lock can be downgraded. In this case, during the execution of the current intermediate processing flow, when a second transaction (i.e., a DML transaction) for the target partition table is received, even if the second target partition corresponding to the DML transaction is consistent with the first target partition, the DML transaction and the DDL transaction can be executed in parallel.

[0075] For example, in the case where the first data language statement included in the first transaction is a DDL statement, the specific implementation flow diagram of the database processing method provided in this embodiment is shown in FIG. Figure 5 .like Figure 5 As shown in the figure, the processing of the transaction parsing phase (Parse phase) and preprocessing phase (Prepare phase) is the same as Figure 3 The processing process is the same and will not be repeated here.

[0076] After entering the execution phase, first, at least one first target table partition corresponding to the DDL transaction is obtained; then, transaction-level upgradeable partition metadata locks are added to these first target table partitions. At this time, the partition metadata lock level on the first target table partition is relatively low, and any other DDL transactions are blocked, but DML transactions can be executed in parallel. In the specific execution process of the DDL transaction, a semantic-level metadata lock can be added to the target partition table, the purpose of which is to protect the metadata write and other SQL read security. Then, when executing a certain intermediate process, it can be determined whether the current first data language statement corresponding to the intermediate processing process needs to protect the data in the first target partition; if so, the transaction-level upgradeable partition metadata lock is upgraded to an exclusive lock. At this time, the DML transaction for the first target partition cannot be executed in parallel with the currently executing DDL transaction; if not, and the transaction-level upgradeable partition metadata lock corresponding to the first target partition is in an exclusive lock state, then the lock is downgraded. At this time, the DML transaction for the first target partition can be executed in parallel with the currently executing DDL transaction. It should be noted that after the current intermediate processing flow is executed, the transaction-level upgradeable partition metadata lock can be downgraded to reduce the lock scope. Each intermediate processing flow corresponding to the DDL transaction is processed based on the same processing method.

[0077] Finally, when the handle access count of the target partition table is equal to one, that is, when the current thread is exclusive, the commit phase is entered. Thus, when the processing result corresponding to the DDL transaction is obtained, all locks are released.

[0078] In the technical solution provided in this embodiment, when the first data language statement included in the first transaction is a data definition language statement, after adding the first partition metadata lock for each partition included in the target partition table corresponding to the first transaction, a transaction-level upgradeable partition metadata lock can be added for the first target partition corresponding to the first transaction. When the target partition metadata lock is added, the first partition metadata locks of all partitions of the target partition table are released, and at least one first data language statement corresponding to the first transaction is executed based on the transaction-level upgradeable partition metadata lock. The technical solution of this embodiment, by adding a transaction-level upgradeable partition metadata lock for the first transaction involving the data definition language statement, during the execution of the first transaction, when the target partitions of the first transaction and the second transaction are consistent, the first transaction and the second transaction can also be executed concurrently in a part of the time period, further reducing the locking scope and improving the concurrency performance of the database.

[0079] Embodiment 3

[0080] Figure 6A schematic diagram of the structure of a database processing device provided in an embodiment of the present invention, wherein the distributed database includes multiple storage nodes, and the device includes: a detection task generation module 310, a task feedback monitoring module 320, a result task generation module 330 and a result task processing module 340.

[0081] The target lock adding module 310 is used to add a target partition metadata lock for a first target partition corresponding to the first transaction after adding a first partition metadata lock for a target partition table corresponding to the first transaction when receiving a first transaction including at least one first data language statement;

[0082] A first lock releasing module 320, configured to release the first partition metadata lock corresponding to the target partition table when the target partition metadata lock is added;

[0083] The transaction parallel execution module 330 is used to execute the at least one first data language statement corresponding to the first transaction based on the target partition metadata lock, so that during the execution of the first transaction, when a second transaction including at least one second data language statement is received for the target partition table, the first transaction and the second transaction are executed in parallel if the second target partition corresponding to the second transaction is inconsistent with the first target partition.

[0084] Based on the above device, optionally, the first partition metadata lock is a statement-level partition metadata lock, and the target partition metadata lock is a transaction-level partition metadata lock.

[0085] Based on the above device, optionally, the target lock adding module 310 includes:

[0086] A metadata lock adding and generating unit, configured to, when receiving a first transaction including at least one first data language statement, determine a target partition table corresponding to the first transaction, and add a statement-level metadata lock to the target partition table;

[0087] A partition determination unit, used to obtain metadata information of a target partition table, so as to determine all partitions included in the target partition table according to the metadata information;

[0088] A partition metadata lock adding unit is used to release the statement-level metadata lock corresponding to the target partition table after adding a statement-level partition metadata lock to each partition included in the target partition table;

[0089] The transaction partition lock task generation unit is used to obtain at least one first target partition corresponding to the first transaction after the statement-level metadata lock is released, and add a transaction-level partition metadata lock for the at least one first target partition.

[0090] Based on the above device, optionally, when the language type corresponding to the first data language statement is a data operation language statement or a data query language statement, the language type corresponding to the second data language statement is a data definition language statement.

[0091] Based on the above device, optionally, when the language type corresponding to the first data language statement is a data definition language statement, the language type corresponding to the second data language statement is a data operation language statement or a data query language statement.

[0092] Based on the above device, optionally, the target partition metadata lock is a transaction-level upgradeable partition metadata lock, and the target lock adding module 310 is further used to add a transaction-level upgradeable partition metadata lock to the first target partition corresponding to the first transaction after adding a first partition metadata lock to each partition included in the target partition table corresponding to the first transaction when receiving a first transaction including at least one first data language statement;

[0093] Correspondingly, the transaction parallel execution module 330 is also used to determine whether the current first data language statement corresponding to at least one first data language statement of the first transaction needs to protect the data in the first target partition during the process of executing the first transaction based on the transaction-level upgradeable partition metadata lock; if so, the transaction-level upgradeable partition metadata lock is upgraded so that when a second transaction for the target partition table is received and the second target partition corresponding to the second transaction is inconsistent with the first target partition, the first transaction and the second transaction are executed in parallel; if not, the transaction-level upgradeable partition metadata lock is downgraded so that when a second transaction for the target partition table is received, the first transaction and the second transaction are executed in parallel.

[0094] Based on the above device, optionally, the database processing device further includes: a deadlock processing module;

[0095] A deadlock processing module is used to determine a target transaction whose language type is a data definition language statement from the first transaction and the second transaction when receiving deadlock warning information of the first transaction and the second transaction during the process of executing the first transaction and the second transaction in parallel; close the thread corresponding to the target transaction, and restart the thread corresponding to the target transaction when the first transaction or the second transaction outside the target transaction is executed.

[0096] The technical solution of the embodiment of the present invention is that when a first transaction including at least one first data language statement is received, after adding a first partition metadata lock to the target partition table corresponding to the first transaction, a target partition metadata lock is added to the first target partition corresponding to the first transaction, and then, when the target partition metadata lock is added, the first partition metadata lock corresponding to the target partition table is released, so that, based on the target partition metadata lock, at least one first data language statement corresponding to the first transaction is executed, so that in the process of executing the first transaction, when a second transaction including at least one second data language statement for the target partition table is received, the first transaction and the second transaction are executed in parallel when the second target partition corresponding to the second transaction is inconsistent with the first target partition. The technical solution provided by this embodiment solves the problem that different statement transactions involving modifications to different partitions of a partition table cannot be executed in parallel, reduces the lock granularity of the database, enables different statement transactions involving different partitions of the database partition table to be executed concurrently, and improves the concurrency of the partition table in the database.

[0097] The database processing device provided in the embodiment of the present invention can execute the database processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0098] It is worth noting that the various units and modules included in the above system are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0099] Embodiment 4

[0100] Figure 7 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 7 A block diagram of an exemplary electronic device 40 suitable for implementing exemplary embodiments of the present invention is shown. Figure 7 The electronic device 40 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0101] like Figure 7 As shown, the electronic device 40 is in the form of a general computing device. The components of the electronic device 40 may include but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 connecting different system components (including the system memory 402 and the processing unit 401).

[0102] Bus 403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0103] The electronic device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 40, including volatile and non-volatile media, removable and non-removable media.

[0104] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. Electronic device 40 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 406 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 403 via one or more data medium interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.

[0105] A program / utility 408 having a set (at least one) of program modules 407 may be stored, for example, in memory 402, such program modules 407 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 407 generally perform the functions and / or methods of the embodiments described herein.

[0106] The electronic device 40 may also communicate with one or more external devices 409 (e.g., keyboards, pointing devices, displays 810, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 40, and / or may communicate with any device that enables the electronic device 40 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 411. Furthermore, the electronic device 40 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 412. As shown, the network adapter 412 communicates with other modules of the electronic device 40 via the bus 403. It should be understood that although Figure 7 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 40, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0107] The processing unit 401 executes various functional applications and page processing by running the programs stored in the system memory 402, for example, implementing the database processing method provided in the embodiment of the present invention.

[0108] Embodiment 5

[0109] An embodiment of the present invention further provides a storage medium containing computer executable instructions, which are used to execute a database processing method when executed by a computer processor. The method includes:

[0110] When a first transaction including at least one first data language statement is received, after adding a first partition metadata lock for each partition included in the target partition table corresponding to the first transaction, a target partition metadata lock is added for the first target partition corresponding to the first transaction;

[0111] When the target partition metadata lock is added, the first partition metadata lock of all partitions of the target partition table is released;

[0112] Based on the target partition metadata lock, at least one first data language statement corresponding to the first transaction is executed, so that during the execution of the first transaction, when a second transaction including at least one second data language statement is received for the target partition table, the first transaction and the second transaction are executed in parallel if the second target partition corresponding to the second transaction is inconsistent with the first target partition.

[0113] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0114] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0115] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0116] Computer program code for performing the operation of embodiments of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0117] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A database processing method, characterized in that: include: When a first transaction including at least one first data language statement is received, after adding a first partition metadata lock to a target partition table corresponding to the first transaction, a target partition metadata lock is added to a first target partition corresponding to the first transaction; When the target partition metadata lock is added, releasing the first partition metadata lock corresponding to the target partition table; Based on the target partition metadata lock, the at least one first data language statement corresponding to the first transaction is executed, so that during the execution of the first transaction, when a second transaction including at least one second data language statement is received for the target partition table, the first transaction and the second transaction are executed in parallel if a second target partition corresponding to the second transaction is inconsistent with the first target partition.

2. The method according to claim 1, characterized in that The first partition metadata lock is a statement-level partition metadata lock, and the target partition metadata lock is a transaction-level partition metadata lock.

3. The method according to claim 2, characterized in that The method of adding a target partition metadata lock to a first target partition corresponding to the first transaction after adding a first partition metadata lock to a target partition table corresponding to the first transaction upon receiving a first transaction including at least one first data language statement comprises: When a first transaction including at least one first data language statement is received, determining a target partition table corresponding to the first transaction, and adding a statement-level metadata lock to the target partition table; Obtaining metadata information of a target partition table, so as to determine all partitions included in the target partition table according to the metadata information; After adding a statement-level partition metadata lock for each partition included in the target partition table, releasing the statement-level metadata lock corresponding to the target partition table; After the statement-level metadata lock is released, at least one first target partition corresponding to the first transaction is acquired, and a transaction-level partition metadata lock is added to the at least one first target partition.

4. The method according to claim 1, characterized in that In the case where the language type corresponding to the first data language statement is a data operation language statement or a data query language statement, the language type corresponding to the second data language statement is a data definition language statement.

5. The method according to claim 1, characterized in that In the case where the language type corresponding to the first data language statement is a data definition language statement, the language type corresponding to the second data language statement is a data operation language statement or a data query language statement.

6. The method according to claim 5, characterized in that The target partition metadata lock is a transaction-level upgradeable partition metadata lock, and adding a target partition metadata lock to the first target partition corresponding to the first transaction includes: Adding the transaction-level scalable partition metadata lock to the first target partition corresponding to the first transaction; Accordingly, executing the at least one first data language statement corresponding to the first transaction based on the target partition metadata lock includes: In the process of executing the first transaction based on the transaction-level scalable partition metadata lock, determining, for the at least one first data language statement corresponding to the first transaction, whether the current first data language statement needs to protect data in the first target partition; If yes, performing lock upgrade processing on the transaction-level upgradeable partition metadata lock, so that when a second transaction for the target partition table is received, the first transaction and the second transaction are executed in parallel when a second target partition corresponding to the second transaction is inconsistent with the first target partition; If not, lock downgrade processing is performed on the transaction-level upgradeable partition metadata lock, so that when the second transaction for the target partition table is received, the first transaction and the second transaction are executed in parallel.

7. The method according to claim 1, characterized in that The method also includes: In the process of executing the first transaction and the second transaction in parallel, when receiving deadlock warning information of the first transaction and the second transaction, determining a target transaction whose language type is a data definition language statement from the first transaction and the second transaction; The thread corresponding to the target transaction is closed, and when the first transaction or the second transaction other than the target transaction is completed, the thread corresponding to the target transaction is restarted.

8. A database processing device, characterized in that: The device includes: a target lock adding module, configured to, when receiving a first transaction including at least one first data language statement, add a first partition metadata lock to a target partition table corresponding to the first transaction, and then add a target partition metadata lock to a first target partition corresponding to the first transaction; A first lock releasing module, configured to release the first partition metadata lock corresponding to the target partition table when the target partition metadata lock is added; A transaction parallel execution module is used to execute the at least one first data language statement corresponding to the first transaction based on the target partition metadata lock, so that during the execution of the first transaction, when a second transaction including at least one second data language statement is received for the target partition table, the first transaction and the second transaction are executed in parallel if the second target partition corresponding to the second transaction is inconsistent with the first target partition.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor so that the at least one processor can execute the database processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the database processing method according to any one of claims 1 to 7 when executed.

Images