Data processing method, device, electronic device, and computer-readable storage medium

Abstract

The present application proposes a data processing method, device, electronic device and computer-readable storage medium. The method includes: in response to a first read transaction triggering a first data read operation on a first node in a distributed database, determining whether a predetermined message from the first node is received; wherein the predetermined message is sent when the first write transaction on the first node is in a prepared state; when the predetermined message from the first node is received, reading the table data in the first node, and updating the cache data based on the table data. The embodiment of the present application can maintain strong consistency between the table data and the cache data in the database while reducing the overhead of synchronization between the cache and the actual table data.

Description

Technical Field

[0001] The present application relates to the field of databases, and in particular to a data processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] To improve database performance, a cache is typically maintained within database nodes. This cache contains metadata such as table definitions, index definitions, constraints, and distributed database shard routing information. This data is frequently used and rarely modified. When needed, data is read directly from the cache, avoiding the overhead of scanning table data from disk. When table data is modified, the corresponding cache must be refreshed to prevent applications from reading outdated data.

[0003] In distributed databases, to address timing issues in distributed environments and ensure consistent visibility of distributed transactions across all database nodes, a clock-based protocol and a two-phase approach to transaction committing—Prepare and Commit—are proposed. However, during the period between the Prepare and Commit states of a write transaction, if a read transaction retrieves data directly from the cache, it may obtain outdated data. This compromises the strong consistency between table data and the cache in distributed databases. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method, apparatus, electronic device, and computer-readable storage medium to solve the problems existing in the related art. The technical solutions are as follows:

[0005] In a first aspect, an embodiment of the present application provides a data processing method, comprising:

[0006] In response to a first read transaction triggering a first data read operation on a first node, determining whether a predetermined message from the first node is received; wherein the predetermined message is sent when the first write transaction on the first node is in a prepared state;

[0007] When a predetermined message is received, table data in the first node is read, and cache data is updated based on the table data.

[0008] In a second aspect, an embodiment of the present application provides a data processing method, including:

[0009] When a first write transaction on a first node is in a prepared state, a predetermined message is sent to a first process connected to the first node; wherein the predetermined message is used to instruct the first process to read table data in the first node and update cache data of the first process based on the table data when a first read transaction triggers a first data read operation on the first node.

[0010] In a third aspect, an embodiment of the present application provides a data processing device, including:

[0011] a node determination module, configured to, in response to a first read transaction triggering execution of a first data read operation on a first node, determine whether a predetermined message from the first node has been received; wherein the predetermined message is sent when the first write transaction on the first node is in a prepared state;

[0012] The first reading module is configured to read the table data in the first node upon receiving a predetermined message, and update the cache data based on the table data.

[0013] In a fourth aspect, an embodiment of the present application provides a data processing device, including:

[0014] A message sending module is used to send a predetermined message to a first process connected to the first node when a first write transaction on the first node is in a prepared state; wherein the predetermined message is used to instruct the first process to read table data in the first node and update cache data of the first process based on the table data when a first read transaction triggers a first data read operation on the first node.

[0015] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method provided in any embodiment of the present application when executing the computer program.

[0016] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method provided in any embodiment of the present application is implemented.

[0017] According to the technical solution of the embodiments of the present application, when a write transaction is in the preparation state, the node participating in the write transaction sends a predetermined message to other processes connected to the node, notifying each process to access table data rather than cached data when performing data read operations. When reading data, the read transaction first checks whether there is a predetermined message. If so, it reads the table data and refreshes the cached data. This ensures that the cached data is refreshed in a timely manner, maintains strong consistency between the table data and the cache in the distributed database, and minimizes the overhead of synchronizing the cache with the actual table data.

[0018] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0020] Figure 1 This is a schematic diagram of the distributed transaction protocol in an embodiment of the present application.

[0021] Figure 2 A schematic diagram of an exemplary application scenario of the data processing method provided in an embodiment of the present application.

[0022] Figure 3 Flowchart of a data processing method according to an embodiment of the present application.

[0023] Figure 4 Flowchart of a data processing method according to another embodiment of the present application.

[0024] Figure 5 This is a schematic diagram of the processing flow of writing transactions in the application example of this application.

[0025] Figure 6 2 is a structural block diagram of a data processing device according to an embodiment of the present application.

[0026] Figure 7 2 is a structural block diagram of a data processing device according to another embodiment of the present application.

[0027] Figure 8 FIG. 4 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0029] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following relevant technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all fall within the protection scope of the embodiments of the present application.

[0030] In a standalone database, there are several ways to maintain strong consistency between table data and the corresponding cache. PostgreSQL maintains a cache of system tables in each database connection. When a DDL (Data Definition Language) operation modifies a system table, a data expiration message is written to shared memory when the transaction commits, indicating the change. When another connection executes a transaction, it first checks shared memory for data expiration messages. If so, it rereads the table data and refreshes the cache for that connection, ensuring that outdated data is not read from the cache.

[0031] However, in a distributed database, the above method cannot guarantee that outdated data will not be read from the cache. A distributed database, as used here, refers to a database that contains multiple data storage nodes. It achieves horizontal scalability of computing and storage by partitioning user table data across multiple nodes. Distributed databases use distributed transactions to achieve globally consistent ACID (Atomicity-Consistency-Isolation-Durability) properties, making the distributed database appear consistent with a stand-alone relational database.

[0032] The following first introduces the concurrency control mechanism and distributed transaction protocol of distributed databases.

[0033] Distributed databases can use MVCC (Multi-version Concurrency Control) to provide transaction isolation and ensure ACID isolation. Distributed clocks are often combined with single-node MVCC to provide consistent distributed transaction concurrency isolation.

[0034] Under the MVCC mechanism, the same data can have multiple different versions. When updating data, the original tuple is not modified, but a new version is added. This version becomes visible only when the associated transaction commits. A tuple refers to a row in a database table. Data read operations (also referred to as read operations) can concurrently read the latest version of the data visible to that transaction. This ensures that reads and writes do not affect each other. MVCC implements transaction isolation semantics through snapshot isolation. A transaction is assigned a start timestamp at the start of the transaction, and when a tuple is read, the latest visible version is read. When the transaction commits, a commit timestamp is assigned and stored in the database. Snapshot isolation (SI) is an isolation level in database transaction processing that ensures that read operations of a transaction see a consistent snapshot of the database. A snapshot, also referred to as a snapshot, refers to the data version visible to a transaction, determined based on the transaction's start timestamp.

[0035] Specifically, each transaction has a start timestamp and a commit timestamp. When the start timestamp of transaction T1 is greater than or equal to the commit timestamp of transaction T2, T1 can see the changes made by T2. In other words, distributed databases store the commit timestamp for each write transaction to support timestamp-based visibility. In practical applications, distributed databases can assign a start timestamp (startTS) and a commit timestamp (commitTS) to each transaction at the coordinating node.

[0036] The following combination Figure 1 The schematic diagram of the distributed transaction protocol shown and the specific examples explain the distributed transaction protocol.

[0037] like Figure 1 As shown in the figure, in a distributed cluster-based database, there are one or more coordination nodes (CN) and multiple data storage nodes (DN). Figure 1 The example uses a CN and two DNs. The CN, as the component that receives user requests, only stores metadata and data routing information, while the data is actually stored on the DNs. Upon receiving a user request, the CN calculates which DN the data is stored on based on the routing information, then requests the data from that DN. Finally, after collecting the required data, it returns it to the user. DDL operations, including SQL (Structured Query Language) operations that change database object definitions, such as creating tables, modifying table structures, and deleting tables, involve all CNs and DNs.

[0038] Taking DML (Data Manipulation Language) transactions as an example, Figure 1 Taking transaction T1 as an example, at the start of T1, the current clock (ClockCurrent) is assigned to it on the coordinating node as the start timestamp startTS, i.e., startTS = ClockCurrent. After startTS is distributed to each data storage node, it is compared with the local clock of the data storage node. If startTS is greater than the node's local clock, the node's local clock is updated to startTS. This step can be expressed as ClockUpdate(startTS).

[0039] When transaction T1 is prepared, the current local clock of each data storage node participating in transaction T1 will be increased by 1 to obtain the preparation timestamp prepareTS, that is, prepareTS = ClockAdvance. And prepareTS is returned to the coordination node. The coordination node will select the maximum value from the prepareTS sent by each data storage node as the commit timestamp commitTS of transaction T1, that is, commitTS = max{DN1.prepareTS, DN2.prepareTS,…}. If commitTS is larger than the local clock of the coordination node, commitTS is used to update the local clock of the coordination node. This step can be expressed as ClockUpdate(commitTS). The coordination node sends commitTS to each DN node to commit the transaction, and uses commitTS to advance the local clock of each data storage node participating in transaction T1.

[0040] As you can see, the distributed transaction protocol synchronizes the causal relationships between concurrent transactions through a two-phase commit involving prepare waits, resolving the issue of event ordering in a distributed environment and ensuring consistent visibility of distributed transactions across all database nodes. Through prepared waits, a distributed database can ensure that even though transaction T1 is committed at different physical times across multiple data storage nodes, if the start timestamp of transaction T2 (T2.startTS) is greater than or equal to the commit timestamp of T1 (T1.commitTS), T2 will consistently see T1's modifications across all nodes, thus ensuring snapshot isolation (SI).

[0041] The following describes the specific visibility judgment rules. Assume that there are two concurrent transactions T1 and T2 on any node, such as DN1. For T2, the visibility judgment rules for T1 are as follows for several situations in the timing between T1 and T2:

[0042] Case 1: If T1 has not been prepared and has no commit timestamp when T2 is scanning T1's modifications, T1 is considered invisible to T2.

[0043] Case 2: If T2 is scanning T1's modifications while T1 is in the prepared state but does not have the prepare timestamp prepareTS, T2 needs to wait for T1 to commit or abort. If T1 aborts, T1 is not visible to T2. If T1 commits, then T1 has the commit timestamp commitTS, and visibility is determined directly based on timestamp comparison. That is, if T2's start timestamp T2.startTS ≥ T1's commit timestamp T1.commitTS, then T1 is visible to T2. If T2.startTS < T1's commit timestamp T1.commitTS, then T1 is not visible to T2.

[0044] Case 3: If T1 is in the prepared state and has a prepare timestamp of prepareTS when T2 is scanning T1's modifications, T2's start timestamp, T2.startTS, is compared with T1's prepare timestamp, T1.prepareTS. If T2.startTS < T1.prepareTS, then it can be inferred that T2.startTS < T1.commitTS, and therefore T1 is not visible to T2. If T2.startTS ≥ T1.prepareTS, then T2 must wait for T1 to commit or abort. The handling of the waiting result can be referred to Case 2 above.

[0045] Case 4: If T2 is scanning T1 for modifications and T1 has a commit timestamp, visibility is determined directly by comparing the timestamps. That is, if T2's start timestamp, T2.startTS, ≥ T1's commit timestamp, T1.commitTS, then T1 is visible to T2. If T2.startTS < T1's commit timestamp, T1 is not visible to T2.

[0046] After in-depth research, the inventors of this application discovered that the reason a single-server database cache consistency algorithm cannot guarantee strong consistency between table data and the cache in a distributed database is that, when a single-server database commits a transaction, it writes a cache expiration message to shared memory, prompting other connections to read the table data and refresh the cache at the start of the transaction. However, distributed transactions commit in two phases. When a transaction is in the prepared state, other transactions must wait for the prepared transaction to commit to obtain commitTS for comparison. After commitTS is generated and the two-phase commit is completed, other database connections are notified of the need to refresh the cache. During the period between the write transaction being in the prepared state and the actual writing of commitTS, the visibility of other read transactions may vary depending on the write transaction's modification information. For example, the data may be determined to be visible or not yet determined to be invisible, but the cached data has not been refreshed. Therefore, if a read transaction accesses data directly from the cache, the data may be outdated and should not be retrieved. This undermines the strong consistency between the table data and the cache.

[0047] In related technologies, solutions for maintaining strong consistency between table data and cache in distributed databases often require sacrificing certain system performance, such as increasing synchronization overhead, or cannot effectively achieve strong consistency between table data and cache. Based on this, embodiments of the present application provide a data processing method for maintaining strong consistency between table data and cache data in a database without affecting database performance.

[0048] In order to more clearly demonstrate the data processing method provided in the embodiments of the present application, an application scenario that can be used to implement the method is first introduced. Figure 2 A schematic diagram of an exemplary application scenario is shown. Figure 2 As shown, multiple data storage nodes are set in the distributed database, such as node 1, node 2 and node 3, and multiple database connections can be created simultaneously in the distributed database. Here, the database connection can refer to the process used to process data interaction between the user terminal and the database, such as Figure 2 The second process and the first process are shown. A database connection can also be simply referred to as a connection. Specifically, for each user terminal or each user request, the database management backend can assign a process to connect to the user terminal or process the user request. These processes can be called database connections.

[0049] According to actual user needs, each process can handle one or more transactions. Figure 2 As shown, the second process processes the first write transaction, and the first process processes the first read transaction. In the embodiment of the present application, a write transaction may refer to a transaction including but not limited to a data write operation, and a read transaction may refer to a transaction including but not limited to a data read operation. Figure 2 As shown, the second process can write data to one or more data storage nodes such as node 1 and node 2 based on the first write transaction, wherein the second process can write data to one or more data storage nodes such as node 1 and node 2. Figure 1 The two-phase commit shown in FIG. 4 completes the transaction commit. Simultaneously, the first process reads data from one or more data storage nodes, such as Node 2, based on the first read transaction. In other words, the same data storage node, such as Node 2, can have multiple database connections simultaneously. Optionally, each database connection can interact with the data storage node through a coordinating node.

[0050] At the same time, each database connection can maintain a cache of part of the data in the table data (hereinafter referred to as cached data), that is, each database connection can each maintain a cache for quickly reading this part of the data. This part of the data can be some key data that is read more and written less, such as data sharding routing information, database table structure, index, constraint and other metadata information. For this part of the data, the first process can read directly from the cache. However, due to the two-phase commit of the first write transaction of the second process, the cached data may become outdated, thereby destroying the strong consistency of the table data and the cache, or destroying the consistency of the snapshot read. It can be understood that this consistency means that reading the cache and reading the table data have the same effect. In other words, due to the two-phase commit of the write transaction, data consistent with the table data cannot be read from the cache. The embodiment of the present application is mainly to solve this problem.

[0051] According to an embodiment of the present application, during the two-phase commit process of the first write transaction, the data storage node sends a predetermined message to other connections of the node, such as Figure 2 The first process is shown. When other connections need to perform data read operations on this node, they first check whether the scheduled message has been received. If the scheduled message is received, they read the table data from the node and update the cache based on the table data. This ensures that the cached data on each connection can be refreshed in a timely manner, maintaining strong consistency between the table data and the cache.

[0052] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application. Figure 3 A flow chart of a data processing method according to an embodiment of the present application is shown. The method may include:

[0053] Step S310: In response to a first read transaction triggering a first data read operation on a first node, determining whether a predetermined message from the first node is received; wherein the predetermined message is sent when a first write transaction on the first node is in a prepared state;

[0054] Step S320: When a predetermined message is received, read the table data in the first node, and update the cache data based on the table data.

[0055] The above method can be executed by the management backend of the distributed database. For example, it can be executed by the first process of the management backend, which refers to the database connection connected to the first node and used to process the first read transaction, for example Figure 1 The first process shown in, but not limited to this. Among them, the first node can be a data storage node (DN) in a distributed database, which is mainly used to store part of the data in the distributed database and support the horizontal scalability of distributed database computing and storage. In an embodiment of the present application, the first node is a node participating in the first read transaction or a participating node in the first read transaction. In addition, in actual applications, the first node may also be a participating node in the first write transaction. When the first node is a participating node in the first write transaction, if the first write transaction is in a preparatory state, the first node sends a predetermined message to the processing process of the first read transaction.

[0056] Exemplarily, the first process is a database connection for processing the first read transaction, that is, the first process is the processing process of the first read transaction. Among them, the first read transaction may include at least one data read operation, or in other words, at least one data read operation will be triggered in the process of executing the first read transaction. Exemplarily, the data processing operations in the transaction, such as data read operations or data write operations, can be described by SQL statements, that is, the first process can complete the first read transaction by executing at least one SQL statement. Specifically, in the above step S310, the first process can trigger the first data read operation in the process of processing the first read transaction, such as in the process of executing an SQL statement. Here, triggering the data read operation can be understood as starting the reading of data. It should be noted that in the embodiment of the present application, each read transaction reads a data version or snapshot that is visible to itself, and the data version is determined based on the start timestamp of the read transaction.

[0057] Optionally, in an embodiment of the present application, the first data read operation may include a read operation on specific data, wherein the specific data is data that can be read both from the database's data storage nodes and from the cache. For example, the specific data may be frequently used but rarely modified data, including metadata such as data shard routing information, database table structures, indexes, and constraints.

[0058] According to the above step S310, when the first data read operation in the first node is started, it is first checked whether a predetermined message from the first node is received. Exemplarily, the predetermined message is used by the first node to notify each connection on the first node that the first write transaction on the first node enters the prepared state. Here, the prepared state may refer to the prepared state in the two-phase commit of the distributed transaction in the aforementioned description, indicating that the first write transaction writes data in the first node, but it is not completely certain whether the data is visible to other transactions (such as transactions processed by other processes connected to the first node). At this time, the cache of the first process may be consistent with the table data in the first node, or it may be inconsistent.

[0059] According to the above-mentioned step S320, the first process reads the table data in the first node when receiving the predetermined message. Here, the table data refers to the database data stored in the data storage node, and by reading the table data, the latest data can be read. It should be noted that, in the embodiment of the present application, the first process reads the table data, which refers to snapshot reading of the table data, that is, reading the data version in the table data that is visible to the first read transaction. Specifically, it refers to reading the data visible to the first read transaction in accordance with the visibility judgment rule. Exemplarily, when reading the table data, if the modification information of other transactions is scanned, the visibility judgment rule in the above description can be used to determine whether the modification information is visible, so as to accurately read the information visible to the first read transaction. Furthermore, by updating the cache data according to the read table data, the cache data of the first process can be refreshed in time to be consistent with the data version visible to the first read transaction, thereby ensuring the consistency of the snapshot read, that is, the results of reading the cache data and the snapshot reading of the table data are consistent.

[0060] As can be seen from the above method, when a write transaction is in the prepared state, the node participating in the write transaction sends a scheduled message to other connections on the node, or other processes connected to the node, to notify each connection to access table data instead of cached data when performing data read operations. When reading data, the read transaction first checks whether there is a scheduled message. If so, it reads the table data and refreshes the cached data. This ensures timely refresh of the cached data, maintains strong consistency between table data and cache in the distributed database, and minimizes the overhead of synchronizing the cache with the actual table data.

[0061] Optionally, the data processing method further includes:

[0062] In the case where no predetermined message is received, the cached data is read.

[0063] That is, when the predetermined message is not received, the first process can directly read the cached data, that is, read the data in the cache, without having to read the table data and refresh the cached data. This can improve the data reading efficiency of the first process. For example, the cached data can include the specific data described above, such as metadata information.

[0064] Based on the above embodiments, as an optional implementation, the data processing method may further include:

[0065] When reading table data in the first node, if it is determined that modification information of the table data by the second write transaction on the first node is not in a visible state, a first flag is set; wherein the first flag is used to indicate that cached data is to be updated.

[0066] It should be noted that the second write transaction may include any transaction including a data write operation that the first node participates in. The second write transaction and the first write transaction may be the same transaction or different transactions.

[0067] As described above, in the process of reading the table data in the data storage node, each information in the table data, such as each tuple, can be scanned. When the modification information of other transactions, such as the second write transaction mentioned above, is scanned, it can be determined whether the modification information is visible based on the visibility judgment rule. Generally speaking, the judgment results can include visible, invisible, and waiting, wherein waiting refers to waiting for the write transaction to be committed and then performing a timestamp comparison. In an embodiment of the present application, the modification information of the table data by the second write transaction is not in a visible state, specifically referring to the modification information not being in a visible state to the first read transaction, including the judgment result being invisible or waiting. In this case, the first process sets the first mark. In this way, when the first process processes the subsequent data read operation, it can determine that the cache data is to be updated, and then choose to access the table data and refresh the cache data, thereby ensuring strong consistency between the table data and the cache. Here, the cache data to be updated can be understood as the cache data may be inconsistent with the table data for subsequent data read operations, that is, the results of reading the cache and reading the table data may be different, and the cache needs to be updated after reading the table data.

[0068] Optionally, the data processing method in the embodiment of the present application may further include a step of determining whether the modification information is in a visible state. Exemplarily, before setting the first mark, the data processing method may further include:

[0069] When the second write transaction is in the prepared state, or the start time of the first read transaction is less than the commit time of the second write transaction, it is determined that the modification information of the table data by the second write transaction is not in the visible state.

[0070] Specifically, if the second write transaction is in the prepared state, such as in Cases 2 and 3 described above, it can be determined that the table data modifications made by the second write transaction are not visible, or a timestamp comparison must be performed after the second write transaction commits. If the second write transaction already has a commit time and the start time of the first read transaction is later than the commit time of the second write transaction, it can be determined that the table data modifications made by the second write transaction are not visible. In these cases, the modifications may become visible to the read transaction processed by the first process in subsequent processing. Therefore, by setting the first flag, the first process can promptly read the visible information in subsequent processing.

[0071] Exemplarily, the above data processing method may further include:

[0072] When triggering a second data read operation on the first node, if the first flag is detected, the table data in the first node is read and the cache data is updated based on the table data; wherein the second data read operation is triggered after the first data read operation.

[0073] According to the above steps, when the second read operation is triggered after the first data read operation, the first process checks whether the first mark exists. If so, the table data in the first node is read to read the visible information in time, and the cache data is updated based on the table data to maintain strong consistency between the table data and the cache.

[0074] It can be understood that, in the case of reading table data in the first node, according to the above description, if the first process determines that the modification information of the table data by the second write transaction on the first node is not in a visible state, the first flag is set so that subsequent processing continues to read the table data and refresh the cache until there is no information that is determined to be not in a visible state, that is, the latest version of the data has been read, and the first flag may not be set.

[0075] In actual applications, different processing can be performed on read data for different isolation levels. Since each connection on the data storage node, that is, each process connected to the data storage node, is allocated to the user end or user request, the isolation level here can be selected by the user. Here, the characteristics of different isolation levels are first explained. As mentioned above, completing a transaction includes executing one or more SQL statements, that is, executing multiple processing operations. If the isolation level of a transaction is Repeatable Read, different processing operations or different SQL statements in the same transaction use the same snapshot, that is, they see the same data. If the isolation level of a transaction is Read Committed, each processing operation or each SQL statement in the same transaction uses a new snapshot, that is, they can see different data.

[0076] For example, when the isolation level is Repeatable Read, the second data read operation includes a data read operation triggered by a second read transaction after the first read transaction. That is to say, for the RepeatableRead isolation level, when other read transactions after the first read transaction, such as the next read transaction, trigger a data read operation of the first process, the table data will be read again according to the first mark and the cache will be refreshed. For other data read operations after the first data read operation in the first read transaction, the cached data can be read directly. Because the same snapshot is always used under the Repeatable Read isolation level, if a data read operation triggered by a certain SQL statement cannot read the latest data, the subsequent SQL statements of the transaction will not read the latest data when they are executed.

[0077] For example, when the isolation level is Read Committed, the second data read operation includes a data read operation triggered by the first read transaction after the first data read operation, or a data read operation triggered by the second read transaction after the first read transaction. That is, for the Read Committed isolation level, when the first process triggers other data read operations after the first data read operation in the first read transaction, the table data will be read again and the cache will be refreshed according to the first mark. If the first data read operation is the last data read in the first read transaction, then when other read transactions after the first read transaction, such as the next read transaction, trigger data read operations, the table data will be read again and the cache will be refreshed according to the first mark.

[0078] The above embodiments describe the technical ideas of the embodiments of the present application from the perspective of read transactions. The following provides another embodiment, which is described from the perspective of write transactions. Specifically, Figure 4 A data processing method according to another embodiment of the present application is shown, which may include:

[0079] Step S410: When a first write transaction on a first node is in a prepared state, a predetermined message is sent to a first process connected to the first node.

[0080] The predetermined message is used to instruct the first process to read the table data in the first node and update the cache data of the first process based on the table data when the first read transaction triggers a first data read operation on the first node.

[0081] Exemplarily, the above method can be executed by the first node. As can be seen, according to the above method, when a write transaction is in the prepared state, the node participating in the write transaction sends a predetermined message to other connections on the node, or other processes connected to the node, to notify each connection to access table data rather than cached data when performing data read operations. Therefore, when reading data, the read transaction can first check whether there is a predetermined message. If so, it reads the table data and refreshes the cached data. This ensures timely refresh of the cached data, maintains strong consistency between the table data and the cache in the distributed database, and minimizes the overhead of synchronizing the cache with the actual table data.

[0082] The technical details of each information in the above method can be implemented with reference to the above embodiments, and will not be described in detail here. For ease of understanding, a specific application example is provided below.

[0083] In this application example, based on the distributed transaction protocol, changes are made to both the distributed transaction submission process and the data reading process.

[0084] Specifically, for write transactions, refer to Figure 5 The complete process of transaction T1 is as follows:

[0085] 1. The start of the transaction is initiated on all participating nodes of T1. At the same time, the coordinating node also sends the local clock as startTS to all participating nodes. After receiving startTS, each participating node compares it with the local clock. If startTS is greater than the local clock, the local clock is updated to startTS.

[0086] 2. Update data on all participating nodes, such as updating table data, and then start the two-phase commit process.

[0087] 3. After receiving the prepare message, each participating node first marks transaction T1 as prepare status.

[0088] 4. Each participating node generates a cache invalidation message (i.e., a predetermined message in the embodiment of the present application) and sends it to other connections in the node.

[0089] 5. Each participating node adds 1 to its local clock as prepareTS, and then replies prepareTS to the coordinating node.

[0090] 6. After receiving the prepareTS responses from all participating nodes, the coordinating node takes the maximum value as commitTS and sends it to each participating node.

[0091] 7. Each participating node receives commitTS and commits it.

[0092] 8. End.

[0093] For read transactions, the system first checks whether it has received a cache invalidation message. If so, it directly reads the table data and refreshes the cache. If it does not receive a cache invalidation message (which is the behavior in most cases), it directly reads the cache. During the read process, whether the data is retrieved from the table or the cache, it must be a snapshot read, that is, the current transaction can only read tuples submitted before the startTS.

[0094] The specific process is as follows:

[0095] 1. When a transaction starts, there will be a start timestamp startTS, which is used for visibility judgment, so that the current transaction can only read tuples submitted before the start of startTS.

[0096] 2. When reading data, it first checks whether a cache invalidation message has been received. If not, the data is read directly from the cache. If a cache invalidation message is received, the table data is read as a snapshot and the cache is refreshed based on the read data. (Because of snapshot reading, the new version of the data being submitted cannot be seen at this time.)

[0097] 3. When reading table data, the timestamp visibility rules of distributed transactions are followed to determine whether the scanned tuple is visible to the current transaction. If a tuple version is not yet visible, a mark is set. The next time the data is accessed, the cache can be refreshed again until the connection reads the latest version of the data. Specifically:

[0098] If the transaction to which the tuple belongs is in the prepared state and the startTS of the read transaction is less than the prepareTS of the transaction to which the tuple belongs, or if the startTS of the read transaction is less than the commitTS of the transaction to which the tuple belongs, the table data will be read directly the next time data is read. If no tuples in the committing state are seen during the scan, it means that the latest version of the data has been read.

[0099] 4. When receiving a cache invalidation message or being marked as needing to refresh the cache again, the reading of data can be further optimized according to the isolation level selected by the user. Specifically:

[0100] When the user uses the Repeatable Read isolation level, if the current transaction receives a cache invalidation message or is marked as needing to refresh the cache again, and reads table data but does not read the latest data, then when the SQL statements subsequently executed in the current transaction need to access the data, they can directly use the data in the cache without having to query the data in the table again (because the same snapshot is always used under the Repeatable Read isolation level, if the latest data cannot be read this time, then the subsequent SQL statements of the transaction will not read the latest data either). At the end of the transaction, by setting a mark, the cache will be refreshed again at the beginning of the next transaction of this connection, that is, the table data will be directly read and the cache will be refreshed.

[0101] When the user uses the Read Committed isolation level, each SQL statement will retrieve a new snapshot. If the current transaction receives a cache invalidation message or is marked as needing to refresh the cache again, it will read the table data but the latest data is not read. In this case, in subsequent SQL statements executed in the current transaction, when accessing the data, it is necessary to continue to query the table data and update the cache until the latest data is read.

[0102] To summarize the entire read and write process, when a distributed transaction commits, each participating node marks the transaction as prepared and then sends a message to other connections on the node, notifying them to refresh their caches before accessing data. When reading data, a check is first performed to see if there are cache invalidation messages. If so, a snapshot read of the table data is performed and the cache is refreshed. Furthermore, if the latest data is not read during the snapshot read or data that is still being committed is seen, depending on the isolation level, the table data is read again according to the current snapshot in the next transaction (Repeatable Read) or the next SQL statement execution (Read Committed), and the cache is refreshed until the cache is updated with the latest data.

[0103] The above read and write process ensures strong consistency between the cache and actual table data, meeting snapshot read and isolation requirements. Furthermore, if the current snapshot read fails to retrieve the latest table data, the cache on all connections can be updated by directly reading the table data multiple times.

[0104] The above example is applicable to distributed databases. For critical data with high reads and low writes, such as data shard routing information, database table structure, indexes, constraints, and other metadata, this ensures strong consistency between the cache and actual table data while maintaining global data consistency. This ensures data consistency while minimizing the synchronization overhead between the cache and actual table data.

[0105] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a data processing device 600. Figure 6 , the apparatus 600 may include:

[0106] a node determination module 610 configured to, in response to a first read transaction triggering a first data read operation on a first node, determine whether a predetermined message from the first node has been received; wherein the predetermined message is sent when the first write transaction on the first node is in a prepared state;

[0107] The first reading module 620 is configured to read the table data in the first node upon receiving a predetermined message from the first node, and update the cache data based on the table data.

[0108] Optionally, the apparatus 600 may further include:

[0109] The second reading module is configured to read the cached data when no predetermined message is received.

[0110] Optionally, the apparatus 600 may further include:

[0111] The marking module is configured to set a first mark when reading table data in the first node and determining that modification information of the table data by the second write transaction on the first node is not in a visible state; wherein the first mark is used to indicate that cached data is to be updated.

[0112] Optionally, the apparatus 600 may further include:

[0113] The state determination module is used to determine that the modification information of the second write transaction on the table data is not in the visible state when the second write transaction is in the prepared state or the start time of the first read transaction is less than the commit time of the second write transaction.

[0114] Optionally, the apparatus 600 may further include:

[0115] The third reading module is used to read the table data in the first node and update the cache data based on the table data if the first mark is detected when the second data reading operation is triggered for the first node; wherein the second data reading operation is triggered after the first data reading operation.

[0116] Optionally, when the isolation level is repeatable read, the second data read operation includes a data read operation triggered by a second read transaction after the first read transaction.

[0117] Optionally, when the isolation level is Read Committed, the second data read operation includes a data read operation triggered by the first read transaction after the first data read operation, or a data read operation triggered by a second read transaction after the first read transaction.

[0118] The embodiment of the present application also provides a data processing device 700. Figure 7 , the apparatus 700 may include:

[0119] The message sending module 710 is configured to send a predetermined message to a first process connected to the first node when a first write transaction on the first node is in a prepared state; wherein the predetermined message is configured to instruct the first process to read table data in the first node and update cache data of the first process based on the table data when a first read transaction triggers a first data read operation on the first node.

[0120] The functions of each module in each device in the embodiment of the present application can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0121] An embodiment of the present application also provides an electronic device for implementing the above method. Figure 8 FIG. 1 shows a structural block diagram of an electronic device according to an embodiment of the present application. Figure 8 As shown, the electronic device includes: a memory 810 and a processor 820. The memory 810 stores a computer program that can be run on the processor 820. When the processor 820 executes the computer program, the data processing method in the above embodiment is implemented. The number of the memory 810 and the processor 820 can be one or more.

[0122] The electronic device also includes:

[0123] The communication interface 830 is used to communicate with external devices and perform data exchange transmission.

[0124] If the memory 810, the processor 820, and the communication interface 830 are implemented independently, the memory 810, the processor 820, and the communication interface 830 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0125] Optionally, in a specific implementation, if the memory 810, the processor 820 and the communication interface 830 are integrated on a chip, the memory 810, the processor 820 and the communication interface 830 can communicate with each other through an internal interface.

[0126] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in any embodiment of the present application.

[0127] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the method provided in any embodiment of the present application.

[0128] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided in the embodiment of the present application.

[0129] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0130] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0131] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus random access memory (DR RAM).

[0132] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0133] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0134] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0135] Any process or method description in a flow chart or otherwise described herein can be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present application includes additional implementations in which the functions may be performed in a different order than shown or discussed, including in a substantially simultaneous manner or in a reverse order depending on the functions involved.

[0136] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0137] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0138] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0139] The above are only specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data processing method, comprising: In response to a first read transaction triggering a first data read operation on a first node, determining whether a predetermined message from the first node is received; wherein the predetermined message is sent when a first write transaction on the first node is in a prepared state; Upon receiving the predetermined message, reading table data in the first node and updating cache data based on the table data; When reading table data in the first node, if it is determined that modification information of the table data by the second write transaction on the first node is not in a visible state, setting a first flag; wherein the first flag is used to indicate that the cached data is to be updated; In the case of triggering a second data read operation for the first node, if the first flag is detected, reading table data in the first node and updating the cache data based on the table data; wherein the second data read operation is triggered after the first data read operation; Wherein, when the isolation level is repeatable read, the second data read operation includes a data read operation triggered by a second read transaction after the first read transaction; In which, when the isolation level is read committed, the second data read operation includes a data read operation triggered by the first read transaction after the first data read operation, or a data read operation triggered by a second read transaction after the first read transaction.

2. The method according to claim 1, wherein The method further comprises: If the predetermined message is not received, the cached data is read.

3. The method according to claim 1, wherein The method further comprises: When the second write transaction is in a prepared state, or the start time of the first read transaction is less than the commit time of the second write transaction, it is determined that modification information of the table data by the second write transaction is not in a visible state.

4. A data processing device comprising: a node determination module, configured to, in response to a first read transaction triggering a first data read operation on a first node, determine whether a predetermined message from the first node has been received; wherein the predetermined message is sent when a first write transaction on the first node is in a prepared state; a first reading module, configured to read the table data in the first node upon receiving the predetermined message, and update cache data based on the table data; A second reading module is used to read the cached data when no predetermined message is received; The device may also include: a marking module configured to, when reading table data from the first node, set a first mark if it is determined that modification information of the table data by the second write transaction on the first node is not in a visible state; wherein the first mark is used to indicate that cached data is to be updated; a third reading module, configured to, when a second data reading operation is triggered for the first node, read the table data in the first node and update the cache data based on the table data if the first flag is detected, wherein the second data reading operation is triggered after the first data reading operation; When the isolation level is repeatable read, the second data read operation includes a data read operation triggered by a second read transaction after the first read transaction; When the isolation level is Read Committed, the second data read operation includes a data read operation triggered by the first read transaction after the first data read operation, or a data read operation triggered by a second read transaction after the first read transaction.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images