Data processing method, system, device, computer program product and storage medium

Abstract

This application provides a data processing method, system, device, computer program product, and storage medium. The method includes: when a data page does not meet the conditions for writing to disk, generating a copy page; wherein the copy page includes: a first sequence identifier of the log corresponding to the data page at a first time modification, a second sequence identifier of the log corresponding to the data page at a second time modification, and data information of the data page after multiple modifications between the first and second times; the first time is earlier than the second time; when the copy page meets the conditions for writing to disk, an operation is performed to update the first sequence identifier corresponding to the data page in the disk to the second sequence identifier. The solution provided by this application can accelerate the progress of read-write nodes writing data pages to disk (i.e., disk persistence), and correspondingly, read-only nodes can also quickly delete logs accumulated in memory, enabling them to receive new logs.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to data processing methods, systems, devices, computer program products, and storage media. Background Technology

[0002] With the development of cloud-native database technology, more and more users are participating in cloud technology. Users' requirements for database performance, such as data processing capacity and processing speed, are also gradually increasing.

[0003] If data on a data page is modified on a read / write node in a database, but the data page on disk is not updated in a timely manner, inconsistencies will occur between the data page on the read / write node and the data page on disk. This data page is called a dirty page. To ensure consistency between the data page on disk and the data page in the read / write node's memory, the data page needs to be flushed, which is the operation of persisting the modified data from the read / write node's memory to disk. However, in practical applications, various reasons can prevent the data page from being flushed, leading to thread blocking and directly impacting database performance. Summary of the Invention

[0004] To address or improve the problems existing in the prior art, various embodiments of this application provide data processing methods, systems, devices, computer program products, and storage media.

[0005] In one embodiment of this application, a data processing method is provided. This method, applicable to read / write nodes, specifically includes:

[0006] When a data page does not meet the conditions for writing to disk, a copy page is generated; wherein, the copy page includes: a first sequence identifier of the log corresponding to the data page at a first time, a second sequence identifier of the log corresponding to the data page at a second time, and data information of the data page after multiple modifications during the period from the first time to the second time; the first time is earlier than the second time;

[0007] If the copied page meets the conditions for writing to disk, the operation of updating the first sequence identifier corresponding to the data page in the disk to the second sequence identifier is performed.

[0008] In another embodiment of this application, a data processing method is provided. This method is applicable to read-only nodes and specifically includes:

[0009] Read logs from shared storage;

[0010] Store the read logs in memory;

[0011] After determining that the first sequence identifier corresponding to the data page has been updated to the second sequence identifier included in the copied page, the logs corresponding to the multiple modifications of the data page in the memory during the first time period to the second time period are deleted.

[0012] The copied page is generated when the data page does not meet the conditions for writing to disk. The copied page includes: a first sequence identifier of the log corresponding to the data page at a first time, a second sequence identifier of the log corresponding to the data page at a second time, and data information of the data page after multiple modifications during the period from the first time to the second time; the first time is earlier than the second time.

[0013] In another embodiment of this application, a database system is provided. The database system includes:

[0014] A read-write node is used to generate a copy page when a data page does not meet the conditions for writing to disk; wherein, the copy page includes: a first sequence identifier of the log corresponding to the data page at a first time, a second sequence identifier of the log corresponding to the data page at a second time, and data information of the data page after multiple modifications between the first time and the second time; the first time is earlier than the second time; the copy page meets the conditions for writing to disk, and the operation of updating the first sequence identifier corresponding to the data page in the disk to the second sequence identifier is performed;

[0015] Shared storage, including the aforementioned disk, is used to store data pages and logs;

[0016] A read-only node is used to read logs from shared storage; store the read logs in memory; and after determining that the first sequence identifier corresponding to the data page has been updated to the second sequence identifier included in the replicated page, delete the logs corresponding to the multiple modifications of the data page in memory during the first time period to the second time period.

[0017] In another embodiment of this application, an electronic device is provided. The electronic device includes a memory and a processor; wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the steps in the above-described data processing method.

[0018] In another embodiment of this application, a computer program product is provided. This computer program product includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is enabled to perform the steps in the data processing method described above.

[0019] In another embodiment of this application, a computer-readable storage medium storing a computer program is provided, which, when executed by a computer, can implement the steps in the above-described data processing method.

[0020] Typically, data pages that cannot meet the write conditions are frequently modified pages (i.e., hot pages). Read-write nodes frequently receive write requests for these hot pages. Because these write requests are constantly being modified, the data page can never meet the conditions for being written to disk (i.e., persisting to disk). Therefore, in the technical solution provided in this application embodiment, when a data page does not meet the conditions for being written to disk (i.e., flushing dirty data), a page replication mechanism is used. This replicated page is considered a page that is no longer being modified by the read-write nodes; that is, the second sequence identifier of the log corresponding to the second modification is fixed and no longer incremented. At this time, the replicated page will definitely meet the conditions for being written to disk, thus accelerating the progress of the read-write nodes writing the data page to disk (i.e., persisting to disk). Correspondingly, read-only nodes can also quickly delete the logs accumulated in memory, enabling them to receive new logs. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the structure of a database system provided in an embodiment of this application;

[0023] Figure 2 A schematic diagram illustrating another manifestation of the database system provided in one embodiment of this application;

[0024] Figure 3 A schematic flowchart illustrating a data processing method provided in an embodiment of this application;

[0025] Figure 4 A schematic flowchart illustrating a data processing method provided in another embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the structure of a data processing apparatus provided in another embodiment of this application;

[0028] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0030] In some processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. These operations may be executed out of order or in parallel. Operation numbers such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the terms "first," "second," etc., used herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. Moreover, the embodiments described below are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0031] Before introducing the various embodiments of this application, a brief explanation of the proper nouns and terms used herein will be provided.

[0032] Shared storage refers to the use of the same data storage on different nodes of the database. Under the shared storage architecture, it is possible to achieve better scaling of read request load without increasing disk storage. It is possible to quickly add and delete read-only nodes and perform real-time HA (High Availability) switching between read-only nodes and read-write nodes, which greatly improves the availability of the instance.

[0033] Physical replication refers to the synchronization of master and slave data in MySQL databases through Redolog. Unlike the original binlog master-slave synchronization mechanism, it has better synchronization performance and lower synchronization latency, and is the core technology of PolarDB.

[0034] Redolog, also known as physical log, records the physical modifications to data pages and is used to update data pages with the modified information. The redolog contains information such as the data page identifier, modified content, and write time (or timestamp).

[0035] Redolog application: The process of using Redolog to update data pages to reflect the modified data pages.

[0036] LSN: A continuously incrementing log sequence number, representing the total amount of logs generated from instance installation to the current time. The position of a log in the log can be calculated based on the LSN. When each data page is written to disk (or flushed), the LSN of its most recent modification is also recorded. This ensures that during crash recovery, logs before this LSN do not need to be applied.

[0037] Flushing dirty pages: This refers to the operation where read / write nodes in a database persist modified data pages from memory to disk. Generally, a background thread in the read / write node periodically flushes dirty data pages (Pages) in memory.

[0038] Hot pages: Pages in a database that are constantly modified frequently, indicating that there are continuous user requests for frequent updates to the same or consecutive data.

[0039] PolarDB uses a one-write-multiple-read architecture to serve users. As Figure 1 shown, the read / write node is responsible for read and write requests, also known as the Primary node, and generates the persistence of Redolog and data pages (Pages). The generated Redolog and persisted data pages are stored on the shared storage PFS. There can be multiple read-only nodes, which are only responsible for read requests and are also known as Replica nodes. Read-only nodes update (or replay) data pages by reading Redolog on the shared storage, updating the data pages in their buffer pool (Buffer Pool) to the latest modified data pages, so that subsequent read requests can access the latest data in a timely manner.

[0040] In the PolarDB architecture, to ensure data consistency when user threads access data pages on read-only nodes, when the read / write node (Primary node) flushes a dirty data page P₁, it needs to ensure that the LSN of the most recent modification of this data page P₁ does not exceed the minimum LSN of the Redolog applied on all read-only nodes, to prevent users from accessing overly new data on read-only nodes. Here, the applied Redolog is not necessarily the Redolog of this data P₁. For example, assume Figure 1 the database system shown contains read-only node 1, read-only node 2, and read-only node n. Read-only node 1 updated data page P₂ by applying Redolog with LSN = a; read-only node 2 updated data page P₃ by applying Redolog with LSN = b; read-only node n updated data page P₁ by applying Redolog with LSN = c. Among them, a < b < c. At this time, the read / write node needs to ensure that the LSN of the most recent modification of this data page P₁ does not exceed the LSN = a of the Redolog applied on read-only node 1.

[0041] Therefore, in order to ensure that disk data always remains in a consistent state, when a read / write node flushes a data page, it must consider the LSN of the Redo logs applied on all read-only nodes, and use the LSN of the Redo logs applied on the read-only nodes to complete the data write-to-disk process.

[0042] Define the smallest LSN of the Redo log applied on all read-only nodes as the safe LSN. When a read-write node flushes a data page P1, it must ensure that the Redo log LSN (new_modification_lsn) of the latest modification of data page P1 is less than the safe LSN. Otherwise, the data page P1 cannot be flushed to disk, which may result in data page P1 on the read-write node not being flushed in time and the oldest modification LSN (oldest_flush_lsn) not being advanced.

[0043] On read-only nodes, a runtime application mechanism has been added to accelerate the synchronization efficiency of physical replication. Runtime Apply means that when applying the redo log, if data page P1 is not in memory (such as a buffer pool), the corresponding redo log will not be applied to update (or replay) data page P1. This avoids the background Apply thread on the read-only node frequently reading data page P1 from shared storage. However, the parsed redo log corresponding to data page P1 still needs to be cached, that is, stored in memory (such as a parse buffer), so that when a subsequent user read request arrives, the corresponding data page P1 on shared storage is read, and all redo log modifications for data page P1 cached in memory (such as a parse buffer) are applied through Runtime Apply, ultimately returning the latest data page. Redo logs cached in memory (such as Parse Buffer) can only be cleaned up after the oldest_flush_lsn of the read / write node has been pushed forward. This means that once the data page P1 corresponding to the modification of this redo log has been written to disk, this redo log can be discarded.

[0044] Under this constraint, if hot page updates occur (i.e., new_modification_lsn is constantly being updated) or read / write nodes flush the redo log too slowly, a large amount of parsed redo log for that hot page will accumulate in the parse buffer of the read-only node. This will affect the performance of the read-only node's parse buffer and redo log apply, causing the read-only node's application of redo log LSN to advance too slowly. Conversely, this will make it even more difficult for read / write nodes to flush the redo log, ultimately affecting the write operations of user threads. If the speed at which read / write nodes apply the redo log becomes too slow, the gap between the speed of redo log application and the redo log generated by read / write nodes will widen, leading to a continuous increase in replication latency.

[0045] As mentioned above, when a read / write node flushes a data page, it requires that the LSN of the latest modified Redo log corresponding to the data page being written to disk cannot exceed the safe LSN. When a data page is frequently updated, the LSN (newest_modification_lsn) of the latest modified Redo log for this data page will be constantly updated, and it will never meet the flushing condition (i.e., it will always be greater than the safe LSN). This will prevent the data page from being written to the disk data file, thus preventing the flushing of the LSN from progressing. The end result is that the Redo log of the read-only node accumulates in memory (specifically, in the buffer pool), making it unable to buffer new write requests.

[0046] To address this issue, the technical solutions provided in the embodiments of this application introduce a copy page technique. A copy page is a temporary data copy generated when a data page, in an architecture similar to PolarDB (i.e., a write-many-read architecture), cannot be written to the disk data file in a timely manner due to not meeting the dirty flushing conditions. This copy page contains: all data in the data page, the oldest modified LSN, the most recently modified LSN, etc. This allows the data page to meet the dirty flushing conditions, thus advancing the dirty flushing process.

[0047] Figure 1 and Figure 2 A schematic diagram of the structure of a database system provided in an embodiment of this application is shown. As shown in the figure, the database system includes:

[0048] Read / write node 11 is used to generate a copy page when a data page does not meet the conditions for writing to disk; wherein, the copy page includes: a first sequence identifier of the log corresponding to the data page at a first time modification, a second sequence identifier of the log corresponding to the data page at a second time modification, and data information of the data page after multiple modifications during the period from the first time to the second time; the first time is earlier than the second time; when the copy page meets the conditions for writing to disk, the operation of updating the first sequence identifier corresponding to the data page in the disk to the second sequence identifier is performed;

[0049] Shared storage 13, having the disk, is used to store data pages and logs;

[0050] Read-only node 12 is used to read logs from shared storage; store the read logs in memory; and after determining that the first sequence identifier corresponding to the data page has been updated to the second sequence identifier included in the copied page, delete the logs corresponding to the multiple modifications of the data page in memory during the first time period to the second time period.

[0051] There can be multiple read-only nodes 12 to provide data reading services to multiple users. For example, in a specific implementation, the database system described in this embodiment may also include devices, equipment, or modules responsible for load balancing, read / write distribution, and other functions. For example, the database system described in this embodiment may also include an intermediate layer device. After receiving multiple user read requests and / or write requests, the intermediate layer device will classify and send the read and write requests, sending the write requests to read / write nodes and the read requests to read-only nodes. If there are multiple read requests, they can be distributed to different read-only nodes according to the load balancing principle.

[0052] The database node described in this embodiment can be a database system based on the PolarDB architecture. See also... Figure 2 As shown, in the PolarDB database system, read-write nodes and read-only nodes synchronize their in-memory states through physical replication. Because the amount of data being synchronized is relatively small, the probability of replication latency is much lower compared to traditional MySQL replication. The physical replication process will be briefly described below:

[0053] The primary node (read-write node) periodically sends a notification to the replica node (read-only node) informing it of the upper limit of log positions that the replica node can safely read (i.e., the maximum sequence number of logs that can be read). Within this period, the replica node can safely read logs below this position. If it exceeds this position, it may read logs that the primary node is currently writing. The replica node periodically reports the position of the applied logs, indicating the maximum sequence number of logs it has applied. Logs below this position have definitely been applied, while logs above this position may not have been applied or are being applied. The difference between the log position currently written by the primary node and the position applied by the replica node is the replication latency. If the replication latency is large, the replica node will fall behind the primary node.

[0054] Figure 2 The LGWR (Log Writer) in read / write node 11 writes all logs from the log buffer to the corresponding log files on the shared storage disk. This log writing process starts automatically when the instance starts.

[0055] The disk stores a first sequence identifier corresponding to a data page. For example, the first sequence identifier is contained within the data page. When the copied page meets the conditions for being written to disk, the read / write node can update the data page in the disk to the copied page, thus simultaneously completing the identifier update operation and the data information update operation (or data write-to-disk operation). The identifier update operation updates the first sequence identifier corresponding to the data page to a second sequence identifier. This operation allows the read-only node to quickly delete the accumulated logs in memory, enabling it to receive new logs. The data information update operation (or data write-to-disk operation) updates the data information of the data page in the disk to the data information included in the copied page. This data information update operation advances the progress of writing the data page to disk (i.e., writing to disk).

[0056] The functions and interactions of each component unit in this system embodiment, such as read-write node 11, read-only node 12, and shared storage 13, can be found in the descriptions of the following method embodiments.

[0057] Figure 3 A flowchart illustrating a data processing method according to an embodiment of this application is shown. The method provided in this embodiment is applicable to read / write nodes in the aforementioned database system. Specifically, the method includes:

[0058] 101. When a data page does not meet the conditions for writing to disk, a copy page is generated; wherein, the copy page includes: a first sequence identifier of the log corresponding to the data page at a first time, a second sequence identifier of the log corresponding to the data page at a second time, and data information of the data page after multiple modifications during the period from the first time to the second time; the first time is earlier than the second time.

[0059] 102. If the copied page meets the conditions for writing to the disk, the operation of updating the first sequence identifier corresponding to the data page in the disk to the second sequence identifier is performed.

[0060] In the above 101, the data page can be a hot page with a write request frequency greater than a preset frequency.

[0061] Assume that the read / write nodes continuously receive write requests for data page P1. Based on these requests, the nodes modify P1 and generate a corresponding log for each modification, assigning each log a sequence identifier (e.g., an LSN, an auto-incrementing sequence number). For example, Table 1 below lists the data pages and logs generated after each write request for P1:

[0062]

[0063] The identifiers 1, 2, 3, 4, 5, ... after each log entry are the sequence number (LSN) corresponding to that log entry.

[0064] If write requests for data page P1 continue to be received, data page P1 will need to be continuously modified, resulting in the LSN of the latest modified log being constantly updated. Therefore, data page P1 will never meet the conditions for writing to disk (flushing or erasing to disk). To address this, in step 101 of this embodiment, a copy page is generated for the data page. For example, as shown in the table above, a copy page is generated. This copy page contains the first sequence identifier (e.g., LSN=1) corresponding to the log modified by data page P1 at the first time, the second sequence identifier (e.g., LSN=5) corresponding to the log modified by data page P1 at the second time, and the data information of data page P1 after multiple modifications between the first and second times (i.e., the data information of data page P15). This copy page is considered a page that is no longer modified; that is, the second sequence identifier corresponding to the log modified at the second time of the copy page is fixed and no longer changes. At this point, it is determined that the copy page definitely meets the conditions for writing to disk.

[0065] In this embodiment, updating the first sequence identifier corresponding to the data page on the disk to the second sequence identifier can advance the progress of the read-only node in deleting the accumulated logs in its memory, enabling it to receive new logs without affecting its performance. In essence, besides updating the corresponding first sequence identifier on the disk to the second sequence identifier, the data page on the disk can also be updated to the copied page to complete the data information update operation. That is, the method provided in this embodiment may further include the following steps:

[0066] The data page in the disk is updated to the copied page to complete the data information update operation;

[0067] The data information update operation refers to the operation of updating the data information included in the data page in the disk to the data information included in the copied page.

[0068] In practical implementation, if the data page stored on the disk (i.e., the disk in shared storage) contains a sequence identifier (for example, the data page mentioned in this embodiment contains the first sequence identifier), then the data page on the disk containing the first sequence identifier can be directly updated to the copied page. Simply put, the data page on the disk containing the first sequence identifier is directly replaced with the copied page. At this time, the data information and LSN of the data page on the disk become the data information contained in the copied page and the second sequence identifier (i.e., the latest modified LSN of the copied page, newest_modification_lsn), respectively, thereby advancing the progress of the read / write node writing the data page to the disk. That is, the data page stored on the disk is changed to the data information modified at the second time, completing the data page writing to disk. Simultaneously, because the first sequence identifier corresponding to the data page is updated to the second sequence identifier, the read-only node can quickly delete the accumulated logs in its memory, enabling it to receive new logs and maintain good performance.

[0069] As shown in Table 1 above, another copy page can be generated (hereinafter referred to as the copy page to distinguish it from the copy page above). This copy page contains the third sequence identifier (e.g., LSN=6) corresponding to the log of data page P1 modified at the third time, the fourth sequence identifier (e.g., LSN=10) corresponding to the log of data page P1 modified at the fourth time, and the data information of data page P1 after multiple modifications during the period from the third time to the fourth time (i.e., data page P1). 10 (Data information). Then, the operation of updating the copy page containing the second sequence identifier on the disk to the copy page is performed.

[0070] As can be seen, using the method provided in this embodiment, when the read / write node performs checks on the disk (or flush or write to disk) in the background, if the data page does not meet the check conditions, this embodiment provides a page replication mechanism. Because the data page is constantly being modified, it may remain in a state where it cannot be flushed to disk for a long time. However, by creating a copy of the data page, the second sequence identifier of the log corresponding to the latest modification of this copy page is fixed. Therefore, in subsequent checks, this copy page can meet the check conditions and be persisted to disk. This also pushes the read / write node to advance the progress of writing the data page to disk, reducing the problem of log backlog in memory affecting the performance of read-only nodes due to the delayed writing of data pages to disk. With the page replication mechanism, even frequent modifications will not affect the stability of the database system, thus improving the stability of the database system.

[0071] It should be noted that the concept of "data page" is used throughout the description in this application. However, the data storage structure differs depending on the database architecture, and the corresponding data names in this embodiment will also vary. In this embodiment, "data page" can be replaced with terms such as dataset, data item, data group, etc., and this embodiment does not impose any specific limitations on this.

[0072] Furthermore, the method provided in this embodiment may also include the following steps:

[0073] 103. Obtain the sequence identifier of the logs of at least one read-only node application;

[0074] 104. Determine the security sequence identifier based on the sequence identifier of the logs applied by the at least one read-only node;

[0075] 105. When the second sequence identifier is less than or equal to the secure sequence identifier, the copied page meets the conditions for being written to the disk.

[0076] The phrase "determine the security sequence identifier based on the sequence identifier of the log applied by the at least one read-only node" in section 104 above can be specifically defined as follows:

[0077] Determine the minimum sequence identifier based on the sequence identifier of the logs applied by the at least one read-only node;

[0078] The minimum sequence identifier is used as the safe sequence identifier.

[0079] Furthermore, the method provided in this embodiment may also include the following steps:

[0080] 106. Periodically send log read position information to at least one read-only node, wherein the log read position information contains the maximum sequence identifier of the logs that the read-only node can read from the shared storage;

[0081] 107. Periodically receive application log location information fed back by the at least one read-only node, wherein the application log location information contains the sequence identifier of the logs applied by the read-only node.

[0082] Furthermore, the method provided in this embodiment may also include the following steps:

[0083] 108. Receive a write request for the data page;

[0084] 109. Based on the write request, modify the data page in memory and generate the corresponding log;

[0085] 110. Assign a sequence identifier to the log;

[0086] 111. Send the logs to the shared storage disk for storage.

[0087] Figure 4 A flowchart illustrating a data processing method according to another embodiment of this application is shown. The execution entity of the method provided in this embodiment can be a read-only node in the aforementioned database system. Specifically, the method includes:

[0088] 201. Read logs from shared storage;

[0089] 202. Store the read logs in memory;

[0090] 203. After determining that the first sequence identifier corresponding to the data page has been updated to the second sequence identifier included in the copied page, delete the logs corresponding to the multiple modifications of the data page in the memory during the first time period to the second time period.

[0091] The copied page is generated when the data page does not meet the conditions for writing to disk. The copied page includes: the first sequence identifier of the corresponding log of the data page at a first time, the second sequence identifier of the corresponding log of the data page at a second time, and the data information of the data page after multiple modifications during the period from the first time to the second time; the first time is earlier than the second time.

[0092] For details on generating copied pages and how to use copied pages to update the content of data pages on the disk, please refer to the corresponding content in the above embodiments, which will not be described in detail here.

[0093] Furthermore, the method provided in this embodiment may also include the following steps:

[0094] 204. Receive a read request for the data page;

[0095] 205. Modify the data page by using the log corresponding to the third time of the data page stored in the application memory; wherein the third time is later than the second time.

[0096] 206. In response to the read request, provide feedback on the modified data page.

[0097] Figure 5 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application is shown. Figure 5 As shown, the device includes: a generation module 21 and an execution module 22. Wherein:

[0098] The generation module 21 is used to generate a copy page when a data page does not meet the conditions for writing to disk; wherein, the copy page includes: a first sequence identifier of the log corresponding to the data page at a first time, a second sequence identifier of the log corresponding to the data page at a second time, and data information of the data page after multiple modifications during the period from the first time to the second time; the first time is earlier than the second time;

[0099] The execution module 22 is used to perform an operation to update the first sequence identifier corresponding to the data page in the disk to the second sequence identifier when the copied page meets the conditions for writing to the disk.

[0100] Furthermore, the execution module 22 is also used to update the data page in the disk to the copied page, so as to complete the data information update operation;

[0101] The data information update operation refers to the operation of updating the data information included in the data page in the disk to the data information included in the copied page.

[0102] Furthermore, the apparatus described in this embodiment may also include an acquisition module and a determination module. The acquisition module is used to acquire the sequence identifier of the logs of at least one read-only node application. The determination module is used to determine a secure sequence identifier based on the sequence identifier of the logs of the at least one read-only node application; when the second sequence identifier is less than or equal to the secure sequence identifier, the copied page meets the conditions for writing to disk.

[0103] Furthermore, the determining module is specifically used to: determine a minimum sequence identifier based on the sequence identifier of the logs applied by the at least one read-only node; and use the minimum sequence identifier as the safe sequence identifier.

[0104] Furthermore, the apparatus provided in this embodiment may also include a transceiver module. This transceiver module is used to periodically send read log position information to at least one read-only node, the read log position information containing the maximum sequence identifier of logs that the read-only node can read from shared storage; and periodically receive application log position information fed back by the at least one read-only node, the application log position information containing the sequence identifier of logs applied by the read-only node.

[0105] Furthermore, the apparatus provided in this embodiment further includes a processing module. The transceiver module is used to receive write requests for the data page. The processing module is used to modify the data page in memory and generate a corresponding log according to the write request; assign a sequence identifier to the log; and send the log to a shared storage disk for storage.

[0106] Furthermore, the data page is a hot page with a write request frequency greater than a preset frequency.

[0107] It should be noted that the apparatus provided in the above embodiments can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding contents of the above method embodiments, and will not be repeated here.

[0108] Figure 6 A schematic diagram of a data processing apparatus according to another embodiment of this application is shown. As shown, the apparatus includes: a reading module 31, a storage module 32, and a deletion module 33.

[0109] Read module 31 is used to read logs from shared storage;

[0110] Storage module 32 is used to store the read logs into memory;

[0111] The deletion module 33 is used to delete the logs corresponding to the multiple modifications of the data page in the memory during the period from the first time to the second time after determining that the first sequence identifier corresponding to the data page has been updated to the second sequence identifier included in the copied page;

[0112] The copied page is generated when the data page does not meet the conditions for writing to disk. The copied page includes: the first sequence identifier of the corresponding log of the data page at a first time, the second sequence identifier of the corresponding log of the data page at a second time, and the data information of the data page after multiple modifications during the period from the first time to the second time; the first time is earlier than the second time.

[0113] Furthermore, the apparatus provided in this embodiment may further include: a receiving module, a modification module, and a feedback module. The receiving module is used to receive a read request for the data page. The modification module is used to modify the data page by applying a log corresponding to a third time modification of the data page stored in memory; wherein the third time is later than the second time. The feedback module is used to respond to the read request by providing feedback on the modified data page.

[0114] It should be noted that the apparatus provided in the above embodiments can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding contents of the above method embodiments, and will not be repeated here.

[0115] This application also provides an electronic device, such as Figure 7 As shown, the electronic device includes a processor 42 and a memory 41. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor for:

[0116] When a data page does not meet the conditions for writing to disk, a copy page is generated; wherein, the copy page includes: a first sequence identifier of the log corresponding to the data page at a first time, a second sequence identifier of the log corresponding to the data page at a second time, and data information of the data page after multiple modifications during the period from the first time to the second time; the first time is earlier than the second time;

[0117] The copied page meets the conditions for being written to the disk, and the operation of updating the first sequence identifier corresponding to the data page contained on the disk to the second sequence identifier is performed.

[0118] The aforementioned memory 41 can be configured to store various other data to support operation on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device. Memory 41 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0119] When the processor 42 executes the program in the memory 41, in addition to the functions described above, it can also perform other functions, as detailed in the descriptions of the preceding embodiments.

[0120] Furthermore, such as Figure 7As shown, the electronic device also includes other components such as a communication component 43, a display 44, a power supply component 45, and an audio component 46. Figure 7 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 6 The components are shown. In specific implementations, the electronic device provided in this embodiment can be a read / write node in a database system.

[0121] Another embodiment of this application provides an electronic device, the structure of which is the same as... Figure 7 Specifically, the electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor for:

[0122] Read logs from shared storage;

[0123] Store the read logs in memory;

[0124] After determining that the first sequence identifier corresponding to the data page has been updated to the second sequence identifier included in the copied page, the logs corresponding to the multiple modifications of the data page in the memory during the first time period to the second time period are deleted.

[0125] The copied page is generated when the data page does not meet the conditions for writing to disk. The copied page includes: the first sequence identifier of the corresponding log of the data page at a first time, the second sequence identifier of the corresponding log of the data page at a second time, and the data information of the data page after multiple modifications during the period from the first time to the second time; the first time is earlier than the second time.

[0126] In addition to the functions described above, the processor described above can also perform other functions when executing programs stored in memory, as detailed in the descriptions of the preceding embodiments.

[0127] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the data processing methods provided in the above embodiments.

[0128] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the data processing methods provided in the above embodiments.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method applicable to read-write nodes, wherein the read-write nodes share the same underlying data storage with multiple read-only nodes, the method comprising: When a data page does not meet the conditions for writing to disk, a copy page is generated; wherein, the copy page is a page that is no longer modified, and the copy page includes: a first sequence identifier of the log corresponding to the data page at a first time modification, a second sequence identifier of the log corresponding to the data page at a second time modification, and data information of the data page after multiple modifications during the period from the first time to the second time; the first time is earlier than the second time. The copied page meets the conditions for writing to the disk, and the operation of updating the first sequence identifier corresponding to the data page in the disk to the second sequence identifier is performed, so that after the first sequence identifier is updated to the second sequence identifier, the read-only node deletes the logs corresponding to the multiple modifications of the data page during the first time period to the second time period; The conditions for writing to the disk include: the second sequence identifier is less than or equal to the security sequence identifier, wherein the security sequence identifier is the smallest sequence identifier of the logs applied by at least one read-only node.

2. The method according to claim 1, further comprising: The data page in the disk is updated to the copied page to complete the data information update operation; The data information update operation refers to the operation of updating the data information included in the data page in the disk to the data information included in the copied page.

3. The method according to claim 1 or 2, further comprising: Obtain the sequence identifier of the logs for at least one read-only node application; Determine the security sequence identifier based on the sequence identifier of the logs applied by the at least one read-only node; When the second sequence identifier is less than or equal to the secure sequence identifier, the copied page meets the conditions for being written to disk.

4. The method according to claim 3, wherein determining a security sequence identifier based on the sequence identifier of the log applied by the at least one read-only node includes: Determine the minimum sequence identifier based on the sequence identifier of the logs applied by the at least one read-only node; The minimum sequence identifier is used as the safe sequence identifier.

5. The method according to claim 3, further comprising: Periodically send read log position information to at least one read-only node, the read log position information containing the maximum sequence identifier of the logs that the read-only node can read from the shared storage; The system periodically receives application log location information fed back by the at least one read-only node, the application log location information containing the sequence identifier of the logs applied by the read-only node.

6. The method according to claim 1 or 2, further comprising: Receive a write request for the data page; Based on the write request, the data page in memory is modified and a corresponding log is generated; Assign a sequence identifier to the log; The logs are sent to a shared storage disk for storage.

7. The method according to claim 6, wherein the data page is a hot page with a write request frequency greater than a preset frequency.

8. A data processing method applicable to read-only nodes, wherein the read-only nodes and read-write nodes share the same underlying data storage, comprising: Read logs from shared storage; Store the read logs in memory; After determining that the first sequence identifier corresponding to the data page has been updated to the second sequence identifier included in the copied page, the logs corresponding to the multiple modifications of the data page in the memory during the first time period to the second time period are deleted. The copied page is a page that is no longer modified. The copied page is generated when the data page does not meet the conditions for writing to disk. The copied page includes: the first sequence identifier of the log corresponding to the data page at a first time, the second sequence identifier of the log corresponding to the data page at a second time, and the data information of the data page after multiple modifications during the period from the first time to the second time; the first time is earlier than the second time. Among them, the conditions for not being able to write to disk include: the second sequence identifier is greater than the security sequence identifier, wherein the security sequence identifier is the smallest sequence identifier of the logs applied by at least one read-only node.

9. The method according to claim 8, further comprising: Receive a read request for the data page; The application uses a log corresponding to a third time-based modification of the data page stored in memory to modify the data page; wherein the third time-based modification is later than the second time-based modification. In response to the read request, the modified data page is returned.

10. A database system, comprising: A read-write node is used to generate a copy page when a data page does not meet the conditions for writing to disk. The copy page is a page that is no longer modified and includes: a first sequence identifier of the log corresponding to the data page's first modification, a second sequence identifier of the log corresponding to the data page's second modification, and data information of the data page after multiple modifications between the first and second times. The first time is earlier than the second time. When the copy page meets the conditions for writing to disk, the operation of updating the first sequence identifier corresponding to the data page on the disk to the second sequence identifier is performed. Shared storage, including the aforementioned disk, is used to store data pages and logs; A read-only node is used to read logs from shared storage; store the read logs in memory; and after determining that the first sequence identifier corresponding to the data page has been updated to the second sequence identifier included in the replicated page, delete the logs corresponding to the multiple modifications of the data page in memory during the first time period to the second time period. The conditions for writing to the disk include: the second sequence identifier is less than or equal to the security sequence identifier, wherein the security sequence identifier is the smallest sequence identifier of the logs applied by at least one read-only node.

11. An electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, at least one program, code set, or instruction set being loaded and executed by the processor to implement any one of claims 1 to 6, or the data processing method of claim 7 or 8.

12. A computer program product comprising a computer program or instructions that, when executed by a processor, cause the processor to perform the steps of the method of any one of claims 1 to 6, or the steps of the method of claim 7 or 8.

13. A computer-readable storage medium storing a computer program, which, when executed by a computer, enables the implementation of the steps in the method of any one of claims 1 to 6, or the steps in the method of claim 7 or 8.

Images