Metadata management method, device, equipment and medium based on distributed transactions

Abstract

An embodiment of the present disclosure provides a metadata management method based on distributed transactions, which is applied to a transaction manager on any target database node executing a distributed transaction in a distributed database. The method comprises: receiving a transaction registration request sent by a database management program on any database node in the distributed database; wherein the database management program is distributedly installed on each database node in the distributed database; the transaction registration request includes operation metadata; in response to the transaction registration request, creating a distributed transaction for the database management program to manage unstructured operation metadata, and executing the distributed transaction; and in response to successful execution of the distributed transaction, initiating a callback to the database management program on the target database node using the unstructured operation metadata as a callback parameter, thereby triggering the database management program to perform a corresponding database management operation on the target database node based on the operation metadata.

Description

Technical Field

[0001] This specification relates to the field of database technology, and in particular to a metadata management method, apparatus, device, and medium based on distributed transactions. Background Art

[0002] A distributed system is one whose components are distributed across a network and communicate and coordinate actions through message passing. Its core concept is to enable multiple servers to work together to complete tasks that a single server cannot handle, especially those with high concurrency or large data volumes.

[0003] With the development of technology, the amount of data generated by various industries has exploded, often reaching hundreds of TB or PB, which has far exceeded the processing capacity of traditional stand-alone databases. Therefore, distributed databases came into being.

[0004] A distributed database consists of multiple interconnected databases managed by a distributed database management system (DBMS). The database nodes in a distributed database are physically located in different locations and connected via a network, but logically function as a single entity, centrally managed by the DMS. Nodes can be distributed across different data centers, cities, or even countries. Summary of the Invention

[0005] In view of this, one or more embodiments of this specification provide a metadata management method, apparatus, device, and medium based on distributed transactions to solve problems existing in related technologies.

[0006] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0007] According to a first aspect of an embodiment of this specification, a metadata management method based on a distributed transaction is provided, which is applied to a transaction manager on any target database node executing the distributed transaction in a distributed database, the method comprising:

[0008] receiving a transaction registration request sent by a database management program on any database node in the distributed database; wherein the database management program is distributedly installed on each database node in the distributed database; and the transaction registration request includes unstructured operation metadata corresponding to a database management operation to be performed by the database management program on multiple database nodes in the distributed database;

[0009] In response to the transaction registration request, creating a distributed transaction for the database management program for managing the unstructured operational metadata, and executing the distributed transaction; wherein the distributed transaction includes a transaction executed in a distributed manner on multiple database nodes in the distributed database;

[0010] In response to the successful execution of the distributed transaction, a callback is initiated for the database management program on the target database node using the unstructured operation metadata as a callback parameter to trigger the database management program to perform a corresponding database management operation on the target database node based on the operation metadata.

[0011] According to a second aspect of an embodiment of this specification, a metadata management device based on a distributed transaction is provided, which is applied to a transaction manager on any target database node executing the distributed transaction in a distributed database, the device comprising:

[0012] a receiving module configured to receive a transaction registration request sent by a database management program on any database node in the distributed database; wherein the database management program is distributedly installed on each database node in the distributed database; and the transaction registration request includes unstructured operation metadata corresponding to database management operations to be performed by the database management program on multiple database nodes in the distributed database;

[0013] a transaction module, in response to the transaction registration request, creating a distributed transaction for the database management program for managing the unstructured operational metadata, and executing the distributed transaction; wherein the distributed transaction includes a transaction executed in a distributed manner on multiple database nodes in the distributed database;

[0014] A callback module, in response to the successful execution of the distributed transaction, uses the unstructured operation metadata as a callback parameter to initiate a callback to a database management program on the target database node, so as to trigger the database management program to perform a corresponding database management operation on the target database node based on the operation metadata.

[0015] According to a third aspect of the embodiments of this specification, there is provided an electronic device, comprising a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus;

[0016] The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0017] According to a fourth aspect of the embodiments of this specification, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-readable instructions, and the machine-readable instructions implement the above method when called and executed by a processor.

[0018] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0019] Through the above technical solution, a distributed transaction for managing the unstructured operational metadata in a transaction registration request sent to a database management program distributedly installed in a distributed database can be created. When the distributed transaction is successfully executed, a callback is made to the database management program on the target database node to trigger the database management program to perform database management operations based on the operational metadata. In the above process, managing the unstructured operational metadata generated by the data management program within the distributed database through distributed transactions can reduce the difficulty of the data management program in managing complex and diverse unstructured operational metadata, and avoid the impact on the performance of the distributed database caused by the data management program directly managing complex and diverse unstructured operational metadata. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a metadata management method based on distributed transactions provided as an exemplary embodiment of this specification;

[0021] Figure 2 A schematic diagram of a metadata management method based on distributed transactions provided as an exemplary embodiment of this specification;

[0022] Figure 3 A multi-party interaction diagram of a metadata management method based on distributed transactions provided by an exemplary embodiment of this specification;

[0023] Figure 4 A schematic structural diagram of an electronic device in which a metadata management device based on distributed transactions is located, provided as an exemplary embodiment of this specification;

[0024] Figure 5 A block diagram of a metadata management device based on distributed transactions is provided as an exemplary embodiment of this specification. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0026] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0027] A transaction is an indivisible logical unit of work. It is a sequence of tasks executed on a database in a specific logical order. Simply put, it unifies the operations to be executed. Either all operations succeed or none of them succeed. If an operation fails during execution, all previous operations are rolled back to the state before the series of operations was executed, that is, rolled back to the original state.

[0028] Among them, transactions usually have four properties: atomicity, consistency, isolation, and durability, referred to as ACID.

[0029] Atomicity means that either all operations included in a transaction are performed or none of them are performed.

[0030] Consistency means that the consistency constraints of the database are not violated before and after the transaction is executed, that is, there will be no intermediate state data.

[0031] Isolation means that the execution of a transaction cannot be interfered with by other transactions, and the data within the transaction is isolated from other transactions.

[0032] Persistence means that after a transaction is completed, the changes made to the data by the transaction will persist in the database and will not be rolled back.

[0033] Transactions can ensure that a certain business logic operates correctly on the database. For example, if an SQL operation is not completed or an error occurs in the program, it will be rolled back according to the specified transaction rules to ensure the security and integrity of our data.

[0034] A distributed transaction is one in which the transaction participants, transaction-supporting servers, resource servers, and transaction managers are located on different nodes in different distributed systems. Simply put, this is the implementation of transactions within a distributed system. Because distributed systems split an application into multiple independently deployable services, remote collaboration between services is required to complete transactions. This type of distributed system environment, where transactions are completed remotely across the network between different services, is called a distributed transaction. Examples include user registration bonus points, order creation inventory reduction, and bank transfer transactions.

[0035] Taking cross-database instances as an example, user information and order information are stored in two MySQL instances respectively. When the user management system deletes user information, it is necessary to delete the user information and the user's order information separately. Since the data is distributed in different data instances, it is necessary to operate the data through different database links. At this time, since the single system needs to access multiple database instances, distributed transactions are generated.

[0036] Among them, in distributed transaction scenarios, the three characteristics of consistency (Consistency), availability (Availability) and partition tolerance (Partition tolerance) will not be met at the same time, referred to as CAP. At most, only two of the three items of consistency, availability and partition tolerance can be met at the same time.

[0037] Consistency means that the read operation after the write operation can read the latest data status. When data is distributed on multiple nodes, the data read from any node is in the latest status.

[0038] Availability means that any transaction operation can get a response result without response timeout or response error.

[0039] Partition tolerance means that services can still be provided to the outside world even when communication between nodes in different network partitions in a distributed system fails due to network problems.

[0040] In addition to processing and storing user data, databases typically generate various metadata during operation. These metadata can be categorized into three types based on the objects they describe: business metadata, technical metadata, and operational metadata. Business metadata describes the business meaning and rules of the data; technical metadata structures the data, facilitating its identification, storage, transmission, and exchange by computers or databases; and operational metadata describes the operational attributes of the data, including the management department and responsible individuals.

[0041] It is worth noting that the above metadata may be structured data or unstructured data.

[0042] Structured data refers to data logically expressed and implemented using a two-dimensional table structure, strictly adhering to data format and length specifications. This data is also known as row data. Unstructured data, on the other hand, refers to data with an irregular or incomplete structure, lacking any predefined data model, and not easily represented using a two-dimensional logical table. Examples include images, HTML, video, and audio.

[0043] For structured metadata, the management method in the database is similar to that of user data. It can be stored by creating a clearly structured table corresponding to the structured metadata and supporting SQL statement access. Therefore, the execution of structured metadata can be performed through distributed transactions.

[0044] Unstructured metadata, however, cannot be described using table structures and therefore requires different management methods. The management methods for different structured metadata vary widely, leading to increasingly high management costs. Furthermore, as the types and amount of metadata increase, the complexity and diversity of unstructured metadata present numerous challenges for management systems. For example, since each module within the database needs to access unstructured metadata through interfaces, this complicates interactions between modules and poses challenges to system stability.

[0045] In view of this, this specification provides a technical solution for managing the unstructured operation metadata generated by the data management program within the distributed database through distributed transactions, which can reduce the difficulty of the data management program in managing complex and diverse unstructured operation metadata, and avoid the impact on the performance of the distributed database when the data management program directly manages complex and diverse unstructured operation metadata.

[0046] During implementation, a transaction registration request may be received from a database management program on any database node in the distributed database; wherein the database management program is distributedly installed on each database node in the distributed database; and the transaction registration request includes unstructured operation metadata corresponding to database management operations to be performed by the database management program on multiple database nodes in the distributed database.

[0047] Then, in response to the transaction registration request, a distributed transaction for managing the unstructured operational metadata may be created for the database management program, and the distributed transaction may be executed; wherein the distributed transaction includes a transaction that is distributedly executed on multiple database nodes in the distributed database;

[0048] Afterwards, in response to the successful execution of the distributed transaction, the unstructured operation metadata can be used as a callback parameter to initiate a callback to the database management program on the target database node, so as to trigger the database management program to perform the corresponding database management operation on the target database node based on the operation metadata.

[0049] Through the above technical solution, a distributed transaction for managing the unstructured operational metadata in a transaction registration request sent to a database management program distributedly installed in a distributed database can be created. When the distributed transaction is successfully executed, a callback is made to the database management program on the target database node to trigger the database management program to perform database management operations based on the operational metadata. In the above process, managing the unstructured operational metadata generated by the data management program within the distributed database through distributed transactions can reduce the difficulty of the data management program in managing complex and diverse unstructured operational metadata, and avoid the impact on the performance of the distributed database caused by the data management program directly managing complex and diverse unstructured operational metadata.

[0050] The metadata management method based on distributed transactions in this specification is described in detail below with reference to the accompanying drawings.

[0051] See Figure 1 , Figure 1 This is a flowchart of a metadata management method based on distributed transactions provided by an exemplary embodiment of this specification, which is applied to a transaction manager on any target database node in a distributed database that executes the distributed transaction. Figure 1 As shown, the method includes the following execution steps:

[0052] Step 101: receiving a transaction registration request sent by a database management program on any database node in the distributed database;

[0053] The database management program is distributedly installed on each database node in the distributed database; the transaction registration request includes unstructured operation metadata corresponding to the database management operation to be executed by the database management program on the multiple database nodes in the distributed database;

[0054] Step 102: In response to the transaction registration request, create a distributed transaction for the database management program for managing the unstructured operational metadata, and execute the distributed transaction;

[0055] Wherein, the distributed transaction includes a transaction executed in a distributed manner on multiple database nodes in the distributed data;

[0056] Step 103: In response to the successful execution of the distributed transaction, a callback is initiated for the database management program on the target database node using the unstructured operation metadata as a callback parameter to trigger the database management program to perform a corresponding database management operation on the target database node based on the operation metadata.

[0057] See Figure 2 , Figure 2 This is a schematic diagram of a metadata management method based on distributed transactions provided as an exemplary embodiment of this specification. Figure 2 As shown, the distributed database may be composed of database nodes 1 to 4. Each database node locally includes a local transaction manager and a local database management program. For example, database node 1 includes transaction manager 1 and database management program 1.

[0058] It should be noted that the database management program is distributedly installed on each database node in the distributed database. The above database management program can be the same program, which is isolated from each other by being distributedly installed on each database node in the distributed database, and each performs corresponding database management operations.

[0059] In one example, the distributed database may be an OceanBase database.

[0060] In this embodiment, the transaction manager on the target database node may receive a transaction registration request sent by a database management program on any database node in the distributed database.

[0061] For example, if Figure 2 As shown, database node 1 may initiate a transaction registration request, and database nodes 2 to 4 may serve as target database nodes and receive the transaction registration request through a local transaction manager.

[0062] According to different actual requirements, there can be one or more target database nodes, which is not limited in this specification.

[0063] It is worth noting that the transaction registration request may include unstructured operation metadata corresponding to the database management operation to be executed by the database management program on multiple database nodes in the distributed database.

[0064] In one example, a database management program of each database node may be used to execute a database management operation to be executed, and the database management program needs to execute the database management operation according to unstructured operation metadata carried in the transaction registration request.

[0065] In another example, the above-mentioned unstructured operation metadata can be abstracted into a byte stream through serialization and carried in the transaction registration request, so that the byte stream can be cached in the memory structure of the transaction context information based on the transaction write, thereby unifying the storage form of the unstructured operation metadata and facilitating the storage of the unstructured operation metadata.

[0066] It is worth noting that, in the subsequent step 103, the original unstructured operation metadata can be obtained by deserializing the byte stream.

[0067] In this embodiment, in response to the transaction registration request, a distributed transaction for managing the unstructured operation metadata may be created for the database management program, and the distributed transaction may be executed.

[0068] For example, Figure 2 For example, the transaction manager 2 may create a distributed transaction for the database management program 1 for managing unstructured operation metadata in response to a transaction registration request sent by the database management program 1, and execute the distributed transaction.

[0069] It should be noted that, since distributed transactions include transactions that are distributedly executed on multiple database nodes in distributed data, transaction managers 3 and 4, as transaction managers on the target database nodes, will also respond to the transaction registration request sent by database management program 1, create a distributed transaction for database management program 1 to manage unstructured operation metadata, and execute the distributed transaction.

[0070] It is worth noting that since operational metadata is unstructured, operations on structured metadata cannot be performed directly through distributed transactions like structured metadata. Therefore, distributed transactions can be orchestrated for unstructured operational metadata to manage unstructured operational metadata.

[0071] In one example, a corresponding distributed transaction can be designed for unstructured operational metadata, and by dividing the distributed transaction into different stages, the transaction manager can drive the database management program on the target database node to perform database management operations on the operational metadata corresponding to the transaction stage based on the different stages of the distributed transaction when executing the transaction.

[0072] In this embodiment, in response to the successful execution of the distributed transaction, the unstructured operation metadata can be used as a callback parameter to initiate a callback to the database management program on the target database node, so as to trigger the database management program to perform the corresponding database management operation on the target database node based on the operation metadata.

[0073] For example, for Figure 2 For the transaction manager on each target database node in the distributed transaction, the local transaction manager can, in response to the successful execution of the distributed transaction, use the unstructured operation metadata as a callback parameter to initiate a callback for the local database management program, so as to trigger the local database management program to perform the corresponding database management operation on the local database node based on the operation metadata.

[0074] In one embodiment shown, the distributed transaction includes multiple transaction execution phases; the database management program includes multiple callback functions corresponding to the multiple transaction execution phases; different transaction execution phases correspond to different callback functions; and different callback functions are associated with different database management operations.

[0075] Furthermore, in response to the successful execution of any target transaction execution phase in the distributed transaction, the unstructured operation metadata can be used as a callback parameter to initiate a callback of a target callback function corresponding to the target transaction execution phase in the database management program on the target database node, so as to trigger the database management program to perform the database management operation associated with the target callback function on the target database node based on the operation metadata.

[0076] For example, a distributed transaction can be divided into multiple transaction execution stages. Different transaction execution stages correspond to different callback functions included in the database management program. These different callback functions are associated with different database management operations. Therefore, in different transaction execution stages, the database management operations associated with the callback function can be executed according to the callback function corresponding to the current stage.

[0077] Continuing with the example, Figure 2 Taking the transaction manager 2 in the example, the transaction manager 2 can respond to the successful execution of any target transaction execution phase in the distributed transaction, use the unstructured operation metadata as a callback parameter, and initiate a callback of the target callback function corresponding to the target transaction execution phase in the database management program 2 on the local database node, that is, the database node 2, thereby triggering the database management program 2 to execute the database management operation associated with the target callback function on the database node 2 based on the operation metadata.

[0078] See next Figure 3 , Figure 3 This is a multi-party interaction diagram of a metadata management method based on distributed transactions provided by an exemplary embodiment of this specification. Figure 3 As shown, the distributed transaction can be divided into multiple transaction execution phases, including: transaction registration phase, redo log generation phase and transaction submission phase. Figure 3 In the example, the target database nodes are target database node A and target database node B, and target database node A also has a replica node A1. Of course, there can be multiple target database nodes and replica nodes. Figure 3 The following are for illustration only.

[0079] In one embodiment shown, the distributed transaction includes a transaction registration phase; the database management operation associated with the first callback function corresponding to the transaction registration phase includes a basic management operation corresponding to the operation metadata;

[0080] Furthermore, the transaction operation corresponding to the transaction registration phase of the distributed transaction includes: establishing a memory structure for caching transaction context information corresponding to the distributed transaction, and caching the unstructured operation metadata in the memory structure;

[0081] Then, when executing the above step 103, in response to successfully caching the unstructured operation metadata in the memory structure, the unstructured operation metadata cached in the memory structure can be used as a callback parameter to initiate a callback of a first target callback function corresponding to the transaction registration phase in the database management program on the target database node, thereby triggering the database management program to perform basic management operations associated with the first callback function on the target database node based on the operation metadata.

[0082] For example, if Figure 3 As shown, taking the target database node A as an example, the transaction registration phase includes the following execution steps:

[0083] S301: Receive a transaction registration request.

[0084] The transaction registration request includes unstructured operation metadata corresponding to the database management operation to be executed by the database management program on multiple database nodes in the distributed database.

[0085] It is worth noting that the processing flow for target database node B is similar to that for target database node A, and by referring to the relevant description of target database node A, the processing flow for target database node B can be clearly understood. Therefore, the processing flow for target database node B is not described in Figure 3 Shown in.

[0086] S302: Create a distributed transaction for the database management program in response to the transaction registration request.

[0087] S303: Execute the transaction operation corresponding to the transaction registration phase. The transaction manager A may establish a memory structure for caching transaction context information corresponding to the distributed transaction and cache unstructured operation metadata in the memory structure.

[0088] S304, in response to successfully caching the unstructured operation metadata into the memory structure, the transaction manager A may initiate a callback to the first target callback function corresponding to the transaction registration phase in the database management program A using the unstructured operation metadata cached in the memory structure as a callback parameter.

[0089] In one embodiment shown, the memory address of the unstructured operation metadata in the memory structure may be used as a callback parameter to initiate a callback to the database management program on the target database node.

[0090] For example, the memory address of the unstructured operation metadata in the memory structure can be used as a callback parameter to initiate a callback to the database management program on the target database node, so that the database management program does not need to store the unstructured operation metadata. On the one hand, the database management program does not need to bear the responsibility of storing and recovering the unstructured operation metadata, avoiding increasing the complexity of the system. On the other hand, by passing the memory address, resource overhead can be saved.

[0091] S305 : Database management program A executes the basic management operation associated with the first callback function on target database node A based on the operation metadata.

[0092] In one illustrated embodiment, the operational metadata includes operational metadata corresponding to a data management operation of creating a data table on a plurality of database nodes in the distributed database;

[0093] The basic management operations include: creating a data table in the distributed database;

[0094] The first database management operation includes: adding identification information of the generated redo log to the created data table;

[0095] The second database management operation includes: switching the state of the created data table to a visible state.

[0096] For example, taking the creation of a data table as an example, hereinafter referred to as creating a table, the above-mentioned operation metadata can be operation metadata corresponding to the data management operation of creating a data table on multiple database nodes in a distributed database. The above-mentioned data management operation of creating a data table can be executed by the database management program on the target database node. For example, the database management program can be a DDL module (Data Definition Language, database model definition language).

[0097] In the above step S305, the above first target callback function can be a function for creating a data table. The transaction manager A can initiate a callback for the first target callback function, triggering the database management program A to perform the operation of creating a data table associated with the first callback function based on the operation metadata, thereby creating a data table in the target database node A.

[0098] S306 , the data management program A returns the execution result of the basic management operation associated with the first callback function to the transaction manager A.

[0099] If the execution is completed, the transaction manager may continue to execute the phases after the transaction registration phase.

[0100] In one embodiment shown, the distributed transaction further includes a redo log generation phase executed after the transaction registration phase; the database management operation associated with the second callback function corresponding to the redo log generation phase includes a first database management operation of associating and saving identification information of the generated redo log with an operation result obtained by executing the basic management operation;

[0101] Furthermore, the transaction operation corresponding to the transaction registration phase of the distributed transaction includes: writing the unstructured operation metadata cached in the memory structure into a redo log generated for the distributed transaction;

[0102] Then, when executing the above step 103, in response to the unstructured operation metadata cached in the memory structure being successfully written into the redo log generated for the distributed transaction, a callback of a second target callback function corresponding to the redo log generation phase in the database management program on the target database node can be initiated using the unstructured operation metadata cached in the memory structure as a callback parameter, so as to trigger the database management program to execute the first database management operation associated with the second callback function on the target database node based on the operation metadata.

[0103] For example, if Figure 3 As shown, taking transaction manager A as an example, the redo log generation phase includes the following execution steps:

[0104] S307 , in response to the completion of the transaction registration phase, transaction manager A may execute the transaction operation corresponding to the redo log generation phase. Transaction manager A may write the unstructured operation metadata cached in the memory structure into the redo log generated for the distributed transaction.

[0105] In one embodiment shown, the target database node includes at least one replica device;

[0106] It is worth noting that the target database node may have multiple copies. Figure 3 For example, even if the target of table creation is target database node A and target database node B, since the target database node has replica node A1, the table creation is required to be completed on replica node A1 as well.

[0107] Furthermore, the transaction operations corresponding to the transaction registration phase of the distributed transaction include:

[0108] The unstructured operation metadata cached in the memory structure is written into a redo log generated for the distributed transaction, and the generated redo log is synchronized to other replica devices using a distributed consistency algorithm.

[0109] Then, distributed transactions also need to be executed on the replica devices. When executing the above step 103, each replica device can obtain the unstructured operation metadata written in the redo log in response to the redo log synchronized to the local device, and use the operation metadata as a callback parameter to initiate a callback of the first target callback function corresponding to the transaction registration phase in the database management program on each replica device, thereby triggering the database management program to perform basic management operations associated with the first callback function on the replica device based on the operation metadata.

[0110] For example, if Figure 3 As shown, taking transaction manager A and transaction manager A1 as an example, the replica device of the target database node, that is, the replica node A1 of the target database node A, includes the following steps:

[0111] S308, transaction manager A synchronizes the redo log to transaction manager A1 using a distributed consistency algorithm;

[0112] In one embodiment shown, the distributed consensus algorithm includes a Paxos algorithm.

[0113] Of course, those skilled in the art may also use other consistency algorithms to perform log synchronization.

[0114] S309 , transaction manager A1 obtains the unstructured operation metadata written in the redo log in response to the redo log synchronized to the local computer.

[0115] S310, the transaction manager A1 uses the operation metadata as a callback parameter to initiate a callback of the first target callback function corresponding to the transaction registration phase in the database management program A1;

[0116] S311, the database management program A1 executes the basic management operation associated with the first callback function on the replica node A1 based on the operation metadata;

[0117] S312 , the data management program A1 returns the execution result of the basic management operation associated with the first callback function to the transaction manager A1 .

[0118] It should be noted that after transaction manager A1 synchronizes and writes redo locally, the subsequent processes of different transaction execution stages of the distributed transaction can refer to the subsequent description of S313-S320 of the target database node A, which will not be repeated here.

[0119] In addition, the above step S308 occurs after step S307, but step S313 does not need to wait for the completion of steps S308-312. In the target database node A, step S313 can be continued after step S307 is completed.

[0120] S313, in response to the successful writing of the operation metadata into the redo log, transaction manager A may use the unstructured operation metadata cached in the memory structure as a callback parameter to initiate a callback of a second target callback function corresponding to the redo log generation phase in database management program A on target database node A.

[0121] Continuing with the aforementioned creation of a data table as an example, in the above step S313, the above second target callback function may be a function for adding the identification information of the generated redo log to the created data table, and transaction manager A may initiate a callback for the second target callback function.

[0122] S314 , database management program A executes the first database management operation associated with the second callback function on target database node A based on the operation metadata.

[0123] Continuing with the aforementioned creation of a data table as an example, in the above step S314, the database management program A can execute the operation associated with the second callback function based on the operation metadata to add the identification information of the generated redo log to the created data table, thereby filling in the identification information of the redo log in the created data table.

[0124] In the above process, by adding the identification information of the redo log to the created table, the identification information of the log and the execution results of the basic management operations can be associated and saved, which facilitates log recycling and prevents the generation of excessive logs, which leads to waste of resources.

[0125] S315 , the data management program A returns the execution result of the first database management operation associated with the second callback function to the transaction manager A.

[0126] If the execution is complete, the transaction manager may continue to execute the phase after the redo log generation phase.

[0127] In one embodiment shown, the distributed transaction further includes a transaction commit phase executed after the redo log generation phase; the database management operation associated with the third callback function corresponding to the transaction commit phase includes a second database management operation that updates the status of an operation result obtained by executing the basic management operation;

[0128] Furthermore, the transaction operation corresponding to the transaction commit phase of the distributed transaction includes: committing the distributed transaction in response to a transaction commit instruction of a global transaction manager selected from the transaction managers on the respective database nodes in the distributed database;

[0129] Then, when executing the above step 103, in response to the successful submission of the distributed transaction, the unstructured operation metadata cached in the memory structure can be used as a callback parameter to initiate a callback of a third target callback function corresponding to the transaction submission phase in the database management program on the target database node, so as to trigger the database management program to perform the second database management operation associated with the third callback function on the target database node based on the operation metadata.

[0130] For example, if Figure 3 As shown, taking transaction manager A as an example, the transaction commit phase includes the following execution steps:

[0131] S316: Receive a transaction commit instruction from a global transaction manager selected from the transaction managers on each database node in the distributed database.

[0132] In one example, the elected global transaction manager can serve as the coordinator, with other transaction managers acting as participants. The global transaction manager can use the two-phase commit protocol (2PC) to determine whether the other transaction managers meet the commit conditions. If so, the coordinator sends a transaction commit instruction to each participant to commit the distributed transaction. Of course, the global transaction manager can also use other commit protocols, which are not discussed here.

[0133] S317, commit the distributed transaction.

[0134] Among them, Figure 3 In the example, taking transaction manager A as the global transaction manager, transaction manager A can send a commit instruction to transaction manager B and transaction manager A1. When each transaction manager commits successfully, it can be determined that the distributed transaction is committed successfully.

[0135] S318 , in response to the successful submission of the distributed transaction, transaction manager A uses the unstructured operation metadata cached in the memory structure as a callback parameter to initiate a callback of a third target callback function corresponding to the transaction submission phase in the database management program on the target database node.

[0136] Continuing with the aforementioned creation of a data table as an example, in the above step S317, the above third target callback function may be a function for switching the state of the created data table to a visible state, and the transaction manager A may initiate a callback for the third target callback function.

[0137] S319 , database management program A executes the second database management operation associated with the third callback function on the target database node based on the operation metadata.

[0138] Continuing with the aforementioned creation of a data table as an example, in the above step S319, the database management program A can execute the operation of switching the state of the created data table to a visible state associated with the third callback function based on the operation metadata, thereby switching the state of the data table to a visible state.

[0139] S320 , the data management program A returns the execution result of the second database management operation associated with the third callback function to the transaction manager A.

[0140] At this point, the above-mentioned distributed transaction processing including multiple transaction execution phases is completed.

[0141] Based on the above Figure 3The technical solution shown can divide distributed transactions into different transaction execution phases, execute transactions in different phases through the transaction manager, and call the callback function corresponding to the current phase in the database management program during different transaction execution phases, driving the database management program to implement different database management operations associated with the callback function. In the above process, not only can the management difficulty of the complex and diverse unstructured operation metadata of the data management program be reduced, avoiding the impact on the performance of the distributed database when the complex and diverse unstructured operation metadata is directly managed by the data management program, but the unstructured metadata can also be updated atomically based on the callback of the distributed transaction, ensuring that the metadata takes effect atomically on multiple target database nodes, and standardizing the metadata management process.

[0142] In one embodiment shown, the unstructured metadata cached in the memory structure may be periodically read, and the read unstructured metadata may be persistently stored.

[0143] For example, unstructured metadata cached in the memory structure of transaction context information can be periodically read and stored in the transaction context table for persistent storage to alleviate memory pressure and release space in a timely manner.

[0144] In one embodiment shown, in response to an exception occurring in the execution of the transaction, the redo log generated in the redo log generation phase may be recovered based on the persistently stored unstructured metadata, and the distributed transaction may be re-executed based on the recovered redo log.

[0145] It is worth noting that the redo log is also called the redo log, which is used to ensure that transactions can be successfully executed even when a failure occurs. Therefore, transactions can be re-executed through the recovered redo log.

[0146] For example, if a transaction execution exception occurs, such as a target database node failure, the unstructured metadata persistently stored in the transaction context table can be used to recover the redo logs generated during the transaction's redo log generation phase. The distributed transaction can then be re-executed based on the recovered redo logs. Therefore, by persistently storing metadata in the transaction context table, metadata can be recovered from any state, simplifying system complexity.

[0147] Through the above technical solution, a distributed transaction for managing the unstructured operational metadata in a transaction registration request sent to a database management program distributedly installed in a distributed database can be created. When the distributed transaction is successfully executed, a callback is made to the database management program on the target database node to trigger the database management program to perform database management operations based on the operational metadata. In the above process, managing the unstructured operational metadata generated by the data management program within the distributed database through distributed transactions can reduce the difficulty of the data management program in managing complex and diverse unstructured operational metadata, and avoid the impact on the performance of the distributed database caused by the data management program directly managing complex and diverse unstructured operational metadata.

[0148] In an exemplary embodiment of this specification, a device capable of implementing the above method is also provided.

[0149] Figure 4 This is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410. Of course, it may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 409 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0150] Please refer to Figure 5 In a software implementation, a metadata management device 500 based on distributed transactions is provided, which is applied to a transaction manager on any target database node in a distributed database that executes the distributed transaction. Figure 5 As shown, the apparatus 500 includes:

[0151] A receiving module 501 receives a transaction registration request sent by a database management program on any database node in the distributed database; wherein the database management program is distributedly installed on each database node in the distributed database; and the transaction registration request includes unstructured operation metadata corresponding to database management operations to be performed by the database management program on multiple database nodes in the distributed database.

[0152] A transaction module 502, in response to the transaction registration request, creates a distributed transaction for the database management program for managing the unstructured operational metadata, and executes the distributed transaction; wherein the distributed transaction includes a transaction executed in a distributed manner on multiple database nodes in the distributed database;

[0153] The callback module 503, in response to the successful execution of the distributed transaction, uses the unstructured operation metadata as a callback parameter to initiate a callback to the database management program on the target database node, so as to trigger the database management program to perform the corresponding database management operation on the target database node based on the operation metadata.

[0154] Optionally, the distributed transaction includes multiple transaction execution phases; the database management program includes multiple callback functions corresponding to the multiple transaction execution phases; different transaction execution phases correspond to different callback functions; and different callback functions are associated with different database management operations.

[0155] The callback module 503 further:

[0156] In response to the successful execution of any target transaction execution phase in the distributed transaction, a callback is initiated for a target callback function corresponding to the target transaction execution phase in a database management program on the target database node using the unstructured operation metadata as a callback parameter, so as to trigger the database management program to execute a database management operation associated with the target callback function on the target database node based on the operation metadata.

[0157] Optionally, the distributed transaction includes a transaction registration phase; the database management operation associated with the first callback function corresponding to the transaction registration phase includes a basic management operation corresponding to the operation metadata;

[0158] Optionally, the transaction operations corresponding to the transaction registration phase of the distributed transaction include:

[0159] Establishing a memory structure for caching transaction context information corresponding to the distributed transaction, and caching the unstructured operation metadata in the memory structure;

[0160] The callback module 503 further:

[0161] In response to successfully caching the unstructured operation metadata in the memory structure, a callback is initiated for a first target callback function corresponding to the transaction registration phase in a database management program on the target database node using the unstructured operation metadata cached in the memory structure as a callback parameter, so as to trigger the database management program to perform basic management operations associated with the first callback function on the target database node based on the operation metadata.

[0162] Optionally, the distributed transaction further includes a redo log generation phase executed after the transaction registration phase; the database management operation associated with the second callback function corresponding to the redo log generation phase includes a first database management operation of associating and saving identification information of the generated redo log with an operation result obtained by executing the basic management operation;

[0163] Optionally, the transaction operations corresponding to the transaction registration phase of the distributed transaction include:

[0164] Writing the unstructured operation metadata cached in the memory structure into a redo log generated for the distributed transaction;

[0165] The callback module 503 further:

[0166] In response to successfully writing the unstructured operation metadata cached in the memory structure into the redo log generated for the distributed transaction, a callback is initiated for a second target callback function corresponding to the redo log generation phase in a database management program on the target database node, using the unstructured operation metadata cached in the memory structure as a callback parameter, so as to trigger the database management program to execute the first database management operation associated with the second callback function on the target database node based on the operation metadata.

[0167] Optionally, the target database node includes at least one replica device;

[0168] Optionally, the transaction operations corresponding to the transaction registration phase of the distributed transaction include:

[0169] The unstructured operation metadata cached in the memory structure is written into a redo log generated for the distributed transaction, and the generated redo log is synchronized to other replica devices using a distributed consistency algorithm, so as to trigger each replica device to obtain the unstructured operation metadata written in the redo log in response to the synchronized redo log, and to initiate a callback for a first target callback function corresponding to the transaction registration phase in a database management program on each replica device using the operation metadata as a callback parameter, so as to trigger the database management program to perform basic management operations associated with the first callback function on the replica device based on the operation metadata.

[0170] Optionally, the distributed consensus algorithm includes a Paxos algorithm.

[0171] Optionally, the distributed transaction further includes a transaction commit phase executed after the redo log generation phase; the database management operation associated with the third callback function corresponding to the transaction commit phase includes a second database management operation of updating the status of an operation result obtained by executing the basic management operation;

[0172] Optionally, the transaction operations corresponding to the transaction commit phase of the distributed transaction include:

[0173] Committing the distributed transaction in response to a transaction commit instruction from a global transaction manager selected from transaction managers on each database node in the distributed database;

[0174] The callback module 503 further:

[0175] In response to the distributed transaction being successfully committed, a callback is initiated for a third target callback function corresponding to the transaction commit phase in a database management program on the target database node, using the unstructured operation metadata cached in the memory structure as a callback parameter, so as to trigger the database management program to perform the second database management operation associated with the third callback function on the target database node based on the operation metadata.

[0176] Optionally, the apparatus 500 further includes:

[0177] A reading module 504 (not shown in the figure) periodically reads the unstructured metadata cached in the memory structure;

[0178] The storage module 505 (not shown in the figure) performs persistent storage on the unstructured metadata read.

[0179] Optionally, the apparatus 500 further includes:

[0180] The recovery module 506 (not shown in the figure) recovers the redo log generated in the redo log generation phase based on the persistently stored unstructured metadata in response to an exception in the execution of the transaction, and re-executes the distributed transaction based on the recovered redo log.

[0181] Optionally, the operation metadata includes operation metadata corresponding to a data management operation of creating a data table on a plurality of database nodes in the distributed database;

[0182] The basic management operations include: creating a data table in the distributed database;

[0183] The first database management operation includes: adding identification information of the generated redo log to the created data table;

[0184] The second database management operation includes: switching the state of the created data table to a visible state.

[0185] Optionally, the callback module 503 further:

[0186] The memory address of the unstructured operation metadata in the memory structure is used as a callback parameter to initiate a callback to the database management program on the target database node.

[0187] The implementation process of the functions and effects of each module in the above-mentioned device 500 is specifically detailed in the implementation process of the corresponding steps in the above-mentioned metadata management method based on distributed transactions. For relevant matters, please refer to the partial description of the method implementation method, which will not be repeated here.

[0188] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0189] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0190] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0191] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0192] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0193] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0194] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0195] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0196] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0197] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A metadata management method based on distributed transactions, applied to a transaction manager on any target database node executing the distributed transaction in a distributed database, the method comprising: receiving a transaction registration request sent by a database management program on any database node in the distributed database; wherein the database management program is distributedly installed on each database node in the distributed database; and the transaction registration request includes unstructured operation metadata corresponding to a database management operation to be performed by the database management program on multiple database nodes in the distributed database; In response to the transaction registration request, creating a distributed transaction for the database management program for managing the unstructured operational metadata, and executing the distributed transaction; wherein the distributed transaction includes a transaction executed in a distributed manner on multiple database nodes in the distributed database; In response to the successful execution of the distributed transaction, a callback is initiated for the database management program on the target database node using the unstructured operation metadata as a callback parameter to trigger the database management program to perform a corresponding database management operation on the target database node based on the operation metadata.

2. The method according to claim 1, wherein the distributed transaction includes multiple transaction execution phases; the database management program includes multiple callback functions corresponding to the multiple transaction execution phases; wherein, Different transaction execution stages correspond to different callback functions; different callback functions are associated with different database management operations; In response to the distributed transaction being successfully executed, initiating a callback to a database management program on the target database node using the unstructured operation metadata as a callback parameter to trigger the database management program to perform a corresponding database management operation on the target database node based on the operation metadata, including: In response to the successful execution of any target transaction execution phase in the distributed transaction, a callback is initiated for a target callback function corresponding to the target transaction execution phase in a database management program on the target database node using the unstructured operation metadata as a callback parameter, so as to trigger the database management program to execute a database management operation associated with the target callback function on the target database node based on the operation metadata.

3. The method according to claim 2, wherein the distributed transaction includes a transaction registration phase; the database management operation associated with the first callback function corresponding to the transaction registration phase includes a basic management operation corresponding to the operation metadata; The transaction operations corresponding to the transaction registration phase of the distributed transaction include: Establishing a memory structure for caching transaction context information corresponding to the distributed transaction, and caching the unstructured operation metadata in the memory structure; In response to the distributed transaction being successfully executed, initiating a callback to a database management program on the target database node using the unstructured operation metadata as a callback parameter to trigger the database management program to perform a corresponding database management operation on the target database node based on the operation metadata, including: In response to successfully caching the unstructured operation metadata in the memory structure, a callback is initiated for a first target callback function corresponding to the transaction registration phase in a database management program on the target database node using the unstructured operation metadata cached in the memory structure as a callback parameter, so as to trigger the database management program to perform basic management operations associated with the first callback function on the target database node based on the operation metadata.

4. The method according to claim 3, wherein the distributed transaction further comprises a redo log generation phase executed after the transaction registration phase; the database management operation associated with the second callback function corresponding to the redo log generation phase comprises a first database management operation of associating and saving identification information of the generated redo log with an operation result obtained by executing the basic management operation; The transaction operations corresponding to the transaction registration phase of the distributed transaction include: Writing the unstructured operation metadata cached in the memory structure into a redo log generated for the distributed transaction; In response to the distributed transaction being successfully executed, initiating a callback to a database management program on the target database node using the unstructured operation metadata as a callback parameter to trigger the database management program to perform a corresponding database management operation on the target database node based on the operation metadata, including: In response to successfully writing the unstructured operation metadata cached in the memory structure into the redo log generated for the distributed transaction, a callback is initiated for a second target callback function corresponding to the redo log generation phase in a database management program on the target database node, using the unstructured operation metadata cached in the memory structure as a callback parameter, so as to trigger the database management program to execute the first database management operation associated with the second callback function on the target database node based on the operation metadata.

5. The method according to claim 4, wherein the target database node comprises at least one replica device; The transaction operations corresponding to the transaction registration phase of the distributed transaction include: The unstructured operation metadata cached in the memory structure is written into a redo log generated for the distributed transaction, and the generated redo log is synchronized to other replica devices using a distributed consistency algorithm, so as to trigger each replica device to obtain the unstructured operation metadata written in the redo log in response to the synchronized redo log, and to initiate a callback for a first target callback function corresponding to the transaction registration phase in a database management program on each replica device using the operation metadata as a callback parameter, so as to trigger the database management program to perform basic management operations associated with the first callback function on the replica device based on the operation metadata.

6. The method according to claim 5, wherein the distributed consensus algorithm comprises a Paxos algorithm.

7. The method according to claim 5, wherein the distributed transaction further comprises a transaction commit phase executed after the redo log generation phase; the database management operation associated with the third callback function corresponding to the transaction commit phase comprises a second database management operation for updating the status of an operation result obtained by executing the basic management operation; The transaction operations corresponding to the transaction commit phase of the distributed transaction include: Committing the distributed transaction in response to a transaction commit instruction from a global transaction manager selected from transaction managers on each database node in the distributed database; In response to the distributed transaction being successfully executed, initiating a callback to a database management program on the target database node using the unstructured operation metadata as a callback parameter to trigger the database management program to perform a corresponding database management operation on the target database node based on the operation metadata, including: In response to the distributed transaction being successfully committed, a callback is initiated for a third target callback function corresponding to the transaction commit phase in a database management program on the target database node, using the unstructured operation metadata cached in the memory structure as a callback parameter, so as to trigger the database management program to perform the second database management operation associated with the third callback function on the target database node based on the operation metadata.

8. The method according to claim 7, further comprising: Periodically reading the unstructured metadata cached in the memory structure; The read unstructured metadata is persistently stored.

9. The method according to claim 8, further comprising: In response to an exception occurring in the execution of the transaction, the redo log generated in the redo log generation phase is restored based on the persistently stored unstructured metadata, and the distributed transaction is re-executed based on the restored redo log.

10. The method according to claim 7, wherein the operation metadata comprises operation metadata corresponding to a data management operation of creating data tables on a plurality of database nodes in the distributed database; The basic management operations include: creating a data table in the distributed database; The first database management operation includes: adding identification information of the generated redo log to the created data table; The second database management operation includes: switching the state of the created data table to a visible state.

11. The method according to claim 3, wherein the method uses the unstructured operation metadata as a callback parameter to initiate a callback to a database management program on the target database node, comprising: The memory address of the unstructured operation metadata in the memory structure is used as a callback parameter to initiate a callback to the database management program on the target database node.

12. A metadata management device based on distributed transactions, applied to a transaction manager on any target database node executing the distributed transaction in a distributed database, the device comprising: a receiving module configured to receive a transaction registration request sent by a database management program on any database node in the distributed database; wherein the database management program is distributedly installed on each database node in the distributed database; and the transaction registration request includes unstructured operation metadata corresponding to database management operations to be performed by the database management program on multiple database nodes in the distributed database; a transaction module, in response to the transaction registration request, creating a distributed transaction for the database management program for managing the unstructured operational metadata, and executing the distributed transaction; wherein the distributed transaction includes a transaction executed in a distributed manner on multiple database nodes in the distributed database; A callback module, in response to the successful execution of the distributed transaction, uses the unstructured operation metadata as a callback parameter to initiate a callback to a database management program on the target database node, so as to trigger the database management program to perform a corresponding database management operation on the target database node based on the operation metadata.

13. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 11 by running the executable instructions.

14. A machine-readable storage medium having machine-readable instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images