A data page processing method, device and system
By receiving the application log sequence number of the slave node and determining the minimum application log sequence number to identify the data page persistence, the problem of data page persistence caused by the synchronization delay of the master and slave node is solved, ensuring that the checkpoint event of the master node is normal and the slave node memory is achieved. Reasonable use.
Patent Information
- Application Number
- CN202010956140.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-11
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-09-11
AI Technical Summary
In the database, synchronization delay between master and slave nodes causes frequently modified data pages to not be persisted in time, resulting in abnormal progress of checkpoint events of master nodes, and the slave node's forced cache queue continues to increase, which seriously consumes memory.
By receiving the application log sequence number sent by the slave node, determine whether the minimum application log sequence number is greater than or equal to the log sequence number of the last data page persistence operation. If so, mark the data page as persistence and notify the slave after the master node is persisted. The node eliminates the corresponding data page, thereby decoupling the persistence operation of the master node and the change operation of the slave node.
It realizes that the master node can timely persist data pages when hotspot data is updated, avoid abnormal progress of checkpoint events, reduce memory consumption of slave nodes, and ensure stable database operation.
Smart Images

Figure CN112100141B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of database technology, and in particular to a data page processing method, device and system. Background Art
[0002] In database applications, a computing and storage separation architecture is adopted, and multiple computing nodes share storage layer data. In order to cope with the huge amount of data access, computing nodes usually adopt a primary-replica architecture. The primary node is responsible for writing data, and the slave node synchronizes the data update of the primary node by obtaining the redo log RedoLog from the storage layer. Currently, the primary node can only perform persistence operations on the updated data page Page after all slave nodes have completed the update of the data page.
[0003] However, when frequent modifications are made to the same data page, the corresponding redo logs are modified multiple times between the group and the slave nodes, and the slave nodes continuously apply the redo logs to synchronize the data page changes. Due to the synchronization delay between the master and slave nodes, when the modifications are frequent, the persistence conditions cannot be met, and the master node cannot persist the updated data page. In this way, the CheckPoint event of the master node is abnormally advanced. In addition, due to the continuous application of redo logs to synchronize data page updates by the slave nodes, the Force_Cache_List chain list of the forced cache queue continues to increase, the page elimination mechanism fails, and a large amount of memory of the slave nodes is seriously consumed. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, embodiments of the present application provide a data page processing method, device and system.
[0005] According to one aspect of an embodiment of the present application, a data page persistence method is provided, including:
[0006] Receive a first application log sequence number sent by a slave node, where the first application log sequence number is obtained after the slave node applies a first redo log corresponding to a first data page;
[0007] Determine a minimum application log sequence number from the first application log sequence numbers sent by all the slave nodes;
[0008] When it is determined that the minimum application log sequence number is greater than or equal to the first persistence log sequence number corresponding to the last data page persistence operation, the second data page after the current change operation is marked as being persistable.
[0009] Optionally, the method further includes:
[0010] After performing a first change operation on a data page, a first data page is obtained, and a first redo log corresponding to the first change operation is written into a storage layer;
[0011] Generate a first notification message, the notification message including a first offset of the first redo log in the storage layer;
[0012] The first notification message is sent to the slave node, and the slave node reads the first redo log from the storage layer according to the first offset, obtains the first data page after applying the first redo log, generates a first application log sequence number, and adds the first data page to the forced cache queue of the slave node.
[0013] Optionally, the method further includes:
[0014] After the persistence operation on the second data page is completed, a data page elimination instruction including a data page identifier corresponding to the second data page and a second persistence log sequence number is generated;
[0015] The data page elimination instruction is sent to the slave node.
[0016] Optionally, marking the second data page after the current change operation as persistent includes:
[0017] Setting the persistence flag corresponding to the second data page to valid;
[0018] The method further comprises:
[0019] After the persistence operation on the second data page is completed, the persistence flag is cleared.
[0020] According to another aspect of an embodiment of the present application, a data page processing method is provided, including:
[0021] Receive a data page elimination instruction sent by the master node after the persistence operation on the second data page, wherein the data page elimination instruction includes a data page identifier corresponding to the second data page and a second persistence log sequence number;
[0022] Querying a third data page corresponding to the data page identifier;
[0023] When the third application log sequence number corresponding to the third data page is greater than or equal to the second persistent log sequence number, the second data page is marked as being eligible for elimination from the mandatory cache queue.
[0024] Optionally, the method further includes:
[0025] Reading a third redo log from the storage layer according to a third notification message sent by the master node, wherein the third notification message includes a third offset of the third redo log in the storage layer;
[0026] Applying the third redo log to implement a third change operation corresponding to the third redo log to obtain a third data page;
[0027] Adding the third data page to the mandatory cache queue;
[0028] Send the third application log sequence number corresponding to applying the third redo log to the master node.
[0029] According to another aspect of an embodiment of the present application, a data page processing device is provided, including:
[0030] A receiving module, configured to receive a first application log sequence number sent by a slave node, wherein the first application log sequence number is obtained after the slave node applies a first redo log corresponding to a first data page;
[0031] A determination module, used to determine a minimum application log sequence number from the first application log sequence numbers sent by all the slave nodes;
[0032] The identification module is used to identify the first data page after the current change operation as being persistable when it is determined that the minimum application log sequence number is greater than or equal to the first persistence log sequence number corresponding to the last data page persistence operation.
[0033] According to another aspect of an embodiment of the present application, a data page processing device is provided, including:
[0034] A receiving module, used to receive a data page elimination instruction sent by the master node after the persistence operation on the second data page, wherein the data page elimination instruction includes a data page identifier corresponding to the second data page and a second persistence log sequence number;
[0035] A query module, used for querying a third data page corresponding to the data page identifier;
[0036] The identification module is used to identify the second data page as being eligible for elimination from the mandatory cache queue when the third application log sequence number corresponding to the third data page is greater than or equal to the second persistent log sequence number.
[0037] According to another aspect of an embodiment of the present application, there is provided a data page processing system, comprising: a master node and at least one slave node connected to each other, wherein the master node and the slave node are respectively connected to a storage layer;
[0038] The slave node is used to obtain a first application log sequence number after applying a first redo log corresponding to a first data page, and send the first application log sequence number to the master node;
[0039] The master node is used to determine the minimum application log sequence number from the first application log sequence numbers sent by all the slave nodes; when it is determined that the minimum application log sequence number is greater than or equal to the first persistence log sequence number corresponding to the last data page persistence operation, the second data page after the current change operation is marked as persistent.
[0040] Optionally, the master node is further configured to generate a data page elimination instruction including a data page identifier corresponding to the second data page and a second persistent log sequence number after the persistence operation on the second data page is completed; and send the data page elimination instruction to the slave node;
[0041] The slave node is also used to receive the data page elimination instruction; query the third data page corresponding to the data page identifier; when the third application log sequence number corresponding to the third data page is greater than or equal to the second persistent log sequence number, mark the second data page as being eligible for elimination from the forced cache queue.
[0042] Optionally, the master node is further configured to, after performing a first change operation on the data page, obtain a first data page, write a first redo log corresponding to the first change operation into a storage layer; generate a first notification message, the notification message including a first offset of the first redo log in the storage layer; and send the first notification message to the slave node;
[0043] The slave node is also used to read the first redo log from the storage layer according to the first offset, obtain the first data page after applying the first redo log, generate a first application log sequence number, and add the first data page to the forced cache queue of the slave node.
[0044] Optionally, the master node is further used to set the persistence flag corresponding to the second data page to be valid; when the persistence operation on the second data page is completed, the persistence flag is cleared.
[0045] According to another aspect of the embodiment of the present application, there is provided an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0046] The memory is used to store computer programs;
[0047] The processor is used to implement the above method steps when executing a computer program.
[0048] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method steps are implemented.
[0049] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:
[0050] When hot data appears and the master node frequently updates the data page, it does not have to wait for all slave nodes to complete the current changes to the data page. As long as all slave nodes have cached at least the last persisted data page, the master node can persist the current changed data page. In this way, the persistence of the current changed data page by the master node is decoupled from the completion of the current changes to the data page by all slave nodes, so that even if hot data updates occur, the checkpoint event of the master node can continue to advance and perform data page persistence operations. Subsequent updates to the data page can also be performed normally, avoiding database exceptions. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0053] Figure 1 A flowchart of a data page processing method provided in an embodiment of the present application;
[0054] Figure 2 A flowchart of a data page processing method provided by another embodiment of the present application;
[0055] Figure 3 A flowchart of a data page processing method provided by another embodiment of the present application;
[0056] Figure 4 A flowchart of the interaction between the master node and the slave node provided in the embodiment of the present application;
[0057] Figure 5 A block diagram of a data page processing device provided in an embodiment of the present application;
[0058] Figure 6 A block diagram of a data page processing device provided by another embodiment of the present application;
[0059] Figure 7A block diagram of a data page processing device provided by another embodiment of the present application;
[0060] Figure 8 A block diagram of a data page processing system provided by another embodiment of the present application;
[0061] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0063] The following first describes the technical terms involved in the embodiments of the present application.
[0064] The log sequence number (LSN) is mainly used to recover data when a downtime occurs. LSN is an integer that increases continuously, indicating the total number of bytes written to the log by the transaction. LSN not only exists in the redo log RedoLog, but also has a corresponding LSN in the header of each data page (Page). A data page may be modified multiple times at different times. The oldest (that is, the first) modification LSN, namely oldest_modification, and the latest (that is, the last) modification LSN, namely newest_modification, are recorded on the data page. After each change to the data page is persisted, that is, stored in the storage layer, its newest_modification is updated.
[0065] In the embodiment of the present application, a resolution is formed between the master node and the slave node:
[0066] After the master node executes the data page change, as long as the application log sequence number corresponding to the current application redo log of the slave node exceeds the persistence log sequence number of the last master node persistent data page (that is, newest_modification), the master node can persist the data page without waiting for all slave nodes to complete the change to the data page.
[0067] After the master node persists the data page, it can notify the slave node to eliminate the data page after applying the redo log corresponding to the data page.
[0068] In this way, the data page persistence of the master node and the elimination of the data page of the slave node can be promoted normally.
[0069] In this embodiment, the entire process of data page synchronization includes the data page persistence process of the master node and the data page elimination process of the slave node. The data page persistence and elimination processes are described in detail below.
[0070] First, a data page processing method for performing data page persistence on a master node provided by an embodiment of the present invention is introduced.
[0071] Figure 1 Flow chart of a data page processing method provided in an embodiment of the present application. Figure 1 As shown, the method is applied to the master node and includes the following steps:
[0072] Step S11, receiving a first application log sequence number sent by a slave node, where the first application log sequence number is obtained after the slave node applies a first redo log corresponding to a first data page.
[0073] Step S12: determining a minimum application log sequence number from the first application log sequence numbers sent by all slave nodes.
[0074] There are multiple slave nodes in the computing cluster, and the processing progress of each slave node may be different. Therefore, each slave node currently applies different redo logs, and the application log sequence numbers returned to the master node are different. Determining the minimum application log sequence number means determining that all slave nodes have currently completed the cached data pages.
[0075] Step S13: When it is determined that the minimum application log sequence number is greater than or equal to the first persistence log sequence number corresponding to the last data page persistence operation, the second data page after the current change operation is marked as being persistable.
[0076] The change operation corresponding to the second data page occurs after the change operation corresponding to the first data page. If the master node determines that the data page currently cached by all slave nodes has exceeded the last persisted first data page, the second data page after the current change operation can be marked as persistable.
[0077] Optionally, a persistence flag may be added to each data page to identify whether it can be persisted. When the minimum application log sequence number is greater than or equal to the first persistence log sequence number, the persistence flag is set to valid, such as setting the persistence flag to 1. When the master node completes the persistence operation on the second data page, the persistence flag is cleared, such as setting the persistence flag to 0.
[0078] In this embodiment, when hot data appears and the master node frequently updates the data page, it is not necessary to wait for all slave nodes to complete the current changes to the data page. As long as all slave nodes have cached at least the last persisted data page, the master node can persist the currently changed data page. In this way, the persistence of the currently changed data page by the master node is decoupled from the completion of the current changes to the data page by all slave nodes, so that even if hot data updates occur, the checkpoint event of the master node can continue to advance and execute the data page persistence operation, and subsequent updates to the data page can also be executed normally, avoiding database abnormalities.
[0079] Figure 2 FIG. 1 is a flow chart of a data page processing method provided by another embodiment of the present application. Figure 2 As shown, the method further comprises the following steps:
[0080] Step S21, after performing a first change operation on a data page, a first data page is obtained, and a first redo log corresponding to the first change operation is written into a storage layer.
[0081] Among them, the master node performs data page change operations in the database buffer pool BufferPool.
[0082] Step S22: Generate a first notification message, where the notification message includes a first offset of the first redo log in the storage layer.
[0083] The offset is also called the redo log address (redo byte address, RBA). RBA is a string of address numbers, which consists of three parts: the log file sequence number, the log file block number, and the starting offset byte number of the redo log record in the log block.
[0084] Step S23, sending the first notification message to the slave node, so that the slave node reads the first redo log from the storage layer according to the first offset, obtains the first data page after applying the first redo log, generates a first application log sequence number, and adds the first data page to the forced cache queue of the slave node.
[0085] The master node synchronizes data page change operations to the slave node by synchronizing the redo log. After applying the redo log, the slave node adds the corresponding data page to the forced cache queue and returns the application log sequence number corresponding to the application redo log to the master node.
[0086] In fact, after each data page change operation, the master node will notify the slave node to download the redo log corresponding to the change operation from the storage layer. The slave node applies the redo log in sequence according to the log sequence number. For the second data page, after the change, the master node notifies the slave node of the offset of the second redo log. The slave node reads the second redo log from the storage layer according to the offset. After applying the second redo log, the second data page is added to the forced cache queue and the second application log sequence number is returned to the master node.
[0087] In this embodiment, after the master node completes the persistence operation on the second data page, a data page elimination instruction including a data page identifier corresponding to the second data page and a second persistent log sequence number (that is, the updated newest_modification) is generated; the data page elimination instruction is sent to the slave node. Through the elimination instruction, the slave node is notified that the data page corresponding to the second persistent log sequence number can be eliminated after applying the redo log corresponding to the second persistent log sequence number.
[0088] In this way, after processing the data page that has been persisted by the master node, the slave node can eliminate the data page from the forced cache queue, thereby avoiding the forced cache queue linked list from continuously increasing the consumption of slave node memory.
[0089] This embodiment also provides a data page processing method for eliminating data pages from a node, which is described in detail below.
[0090] Figure 3 FIG. 1 is a flow chart of a data page processing method provided by another embodiment of the present application. Figure 3 As shown, the method is applied to the slave node and includes the following steps:
[0091] Step S31, receiving a data page elimination instruction sent by the master node after the persistence operation on the second data page, the data page elimination instruction including a data page identifier corresponding to the second data page and a second persistence log sequence number;
[0092] Step S32, querying the third data page corresponding to the data page identifier;
[0093] Step S33: When the third application log sequence number corresponding to the third data page is greater than or equal to the second persistent log sequence number, the second data page is marked as being eligible for elimination from the mandatory cache queue.
[0094] The slave node may add a elimination flag for each data page to identify whether it can be eliminated. When the third application log sequence number is greater than or equal to the second persistent log sequence number, the elimination flag is set to be valid, such as setting the elimination flag to 1. After the slave node eliminates the second data page, the elimination flag is cleared.
[0095] In fact, there are other factors that affect whether the slave node determines whether to eliminate the data page. Therefore, in the above step S33, the data page is only marked as eligible for elimination. The actual timing of eliminating the data page from the mandatory cache queue needs to be determined in combination with other factors.
[0096] In this embodiment, the method further includes:
[0097] Step P1, reading a third redo log from the storage layer according to a third notification message sent by the master node, where the third notification message includes a third offset of the third redo log in the storage layer;
[0098] Step P2, applying the third redo log to implement the third change operation corresponding to the third redo log, and obtaining the third data page;
[0099] Step P3, adding the third data page to the mandatory cache queue;
[0100] Step P4, sending the third application log sequence number corresponding to the application of the third redo log to the primary node.
[0101] In this embodiment, after the slave node processes the data page that has been persisted by the master node, it can eliminate the data page from the forced cache queue, so that the data page elimination can proceed normally, avoiding the failure of the data page elimination mechanism due to frequent updates of hot data, resulting in the forced cache queue linked list continuously increasing the consumption of slave node memory.
[0102] The following is a detailed description of the data page synchronization process including the data page persistence performed by the master node and the data page elimination performed by the slave node using a specific example.
[0103] Figure 4 The flow chart of the interaction between the master node and the slave node provided in the embodiment of the present application. Among them, the data page identifiers corresponding to data pages 1, 3, 5, 7, and 9 are the same, which is data page A. The data page identifiers corresponding to data pages 2, 4, 6, and 8 are the same, which is data page B. Figure 4 As shown, the interaction process between the master node and the slave node is as follows:
[0104] a1. The master node performs a data page A change operation to obtain data page 1, and synchronizes the first redo log corresponding to the change operation to the slave node;
[0105] a2. The master node performs a data page B change operation to obtain data page 2, and synchronizes the second redo log corresponding to the change operation to the slave node;
[0106] a3. The master node performs a change operation on data page A again to obtain data page 3, and synchronizes the third redo log corresponding to the change operation to the slave node;
[0107] a4. Apply the first redo log from the slave node, add data page 1 to the mandatory cache queue, and return the first application log sequence number to the master node;
[0108] a5. Data page A0 (data page A initial data page) has been persisted, the first persistence log sequence number, the master node determines that the first application log sequence number is greater than the first persistence log sequence number, then marks data page 3 as persistent, but data page 3 is not necessarily persisted at this time.
[0109] Every time the master node receives the application log sequence number returned by the slave node, it will determine whether the data page can be persisted;
[0110] a6. The master node performs a change operation on data page B again to obtain data page 4, and synchronizes the fourth redo log corresponding to the change operation to the slave node;
[0111] a7. The master node persists data page 3, including the data page A corresponding to the third data page and the persistent log sequence number of the third persistent log sequence number of the data page elimination instruction is sent to the slave node;
[0112] The master node notifies the slave node that data page 3 can be eliminated after applying the third redo log corresponding to data page 3;
[0113] a8. Apply the second redo log from the slave node, add data page 2 to the mandatory cache queue in sequence, and return only the second application log sequence number to the master node;
[0114] a9. The slave node queries data page A corresponding to data page 1 according to the data page elimination instruction, and determines that the second application log sequence number is less than the third persistent log sequence number, and does not meet the conditions for data page elimination, and does not perform the elimination identification operation;
[0115] a10. The master node determines that the second application log sequence number is less than the third persistence log sequence number, and does not mark the current data page 4 as persistent;
[0116] a11. The master node performs a change operation on data page A again to obtain data page 5, and synchronizes the fifth redo log corresponding to the change operation to the slave node;
[0117] a12. The slave node applies the third and fourth redo logs in sequence, adds data page 3 and data page 4 to the mandatory cache queue in sequence, and returns only the fourth application log sequence number to the master node;
[0118] a13. Query the data pages 1 and 3 corresponding to the data page A from the node according to the data page elimination instruction, and determine that the fourth application log sequence number is greater than the third persistent log sequence number, then the data pages 1 and 3 can be marked as being eliminated;
[0119] a14. The master node performs a change operation on data page B again to obtain data page 6, and synchronizes the sixth redo log corresponding to the change operation to the slave node;
[0120] a15. Apply the fifth redo log from the slave node, add data page 5 to the mandatory cache queue, and return the fifth application log sequence number to the master node;
[0121] a16. The master node determines that the fifth application log sequence number is greater than the third persistence log sequence number, and then marks data page 6 as persistent.
[0122] a17. The master node performs a change operation on data page A again to obtain data page 7, and synchronizes the seventh redo log corresponding to the change operation to the slave node;
[0123] a18. The master node performs a change operation on data page B again to obtain data page 8, and synchronizes the eighth redo log corresponding to the change operation to the slave node;
[0124] a19. Apply the sixth redo log from the slave node, add data page 6 to the mandatory cache queue, and return the sixth application log sequence number to the master node;
[0125] a20. The master node persists data page 7, and sends the data page identifier corresponding to data page 7 and the persistent log sequence number 7 to the slave node.
[0126] The above embodiment takes the interaction between the master node and a slave node as an example to illustrate the above method embodiment. In practice, there are multiple slave nodes in the computing cluster. When the master node interacts with multiple slave nodes, when performing the persistence judgment operation (in the above steps a5, a10, a16, and a20), it is necessary to screen the minimum application log sequence number from the application log sequence numbers returned by all the current slave nodes to determine whether the minimum application log sequence number is greater than or equal to the persistence log sequence number corresponding to the last data page persistence operation. If so, the master node can mark the current changed data page as persistent.
[0127] It can be seen from the above embodiments that when hot data is frequently updated, the master node can normally promote data page persistence, and at the same time, the slave node can also eliminate non-hot data pages in the mandatory cache queue. Data page updates can be performed normally to ensure the stable operation of the database.
[0128] The following are device embodiments of the present application, which can be used to execute method embodiments of the present application.
[0129] Figure 5 This is a block diagram of a data page processing device provided in an embodiment of the present application. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 5As shown, the data page processing device includes:
[0130] A receiving module 51 is used to receive a first application log sequence number sent by a slave node, where the first application log sequence number is obtained after the slave node applies a first redo log corresponding to a first data page;
[0131] A determination module 52, configured to determine a minimum application log sequence number from the first application log sequence numbers sent by all slave nodes;
[0132] The identification module 53 is used to identify the first data page after the current change operation as being persistable when it is determined that the minimum application log sequence number is greater than or equal to the first persistence log sequence number corresponding to the last data page persistence operation.
[0133] Optionally, the identification module 53 is used to set the persistence identification corresponding to the second data page to be valid; and is also used to clear the persistence identification after the persistence operation on the second data page is completed.
[0134] Figure 6 A block diagram of a data page processing device provided in another embodiment of the present application is shown in FIG. Figure 6 As shown, optionally, the data page processing device also includes: a generating module 54, which is used to generate a data page elimination instruction including a data page identifier corresponding to the second data page and a second persistence log sequence number after the persistence operation on the second data page is completed; and a sending module 55, which is used to send the data page elimination instruction to the slave node.
[0135] Figure 7 This is a block diagram of a data page processing device provided by another embodiment of the present application. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 7 As shown, the data page processing device comprises:
[0136] A receiving module 61 is used to receive a data page elimination instruction sent by the master node after the persistence operation on the second data page, where the data page elimination instruction includes a data page identifier corresponding to the second data page and a second persistence log sequence number;
[0137] A query module 62, configured to query a third data page corresponding to the data page identifier;
[0138] The identification module 63 is configured to identify the second data page as being eligible for elimination from the mandatory cache queue when the third application log sequence number corresponding to the third data page is greater than or equal to the second persistent log sequence number.
[0139] The embodiment of the present application also provides a data page processing system. Figure 8 A block diagram of a data page processing system provided in an embodiment of the present application, such as Figure 8As shown, the system includes: a master node 71 and at least one slave node 72 connected to each other. The master node 71 and the slave node 72 are respectively connected to a storage layer 73 to share data of the storage layer 73.
[0140] The slave node 72 is used to obtain a first application log sequence number after applying the first redo log corresponding to the first data page, and send the first application log sequence number to the master node 71;
[0141] The master node 71 is used to determine the minimum application log sequence number from the first application log sequence numbers sent by all the slave nodes 72; when it is determined that the minimum application log sequence number is greater than or equal to the first persistence log sequence number corresponding to the last data page persistence operation, the second data page after the current change operation is marked as persistent.
[0142] Optionally, the master node 71 is further used to generate a data page elimination instruction including a data page identifier corresponding to the second data page and a second persistent log sequence number after the persistence operation on the second data page is completed; and send the data page elimination instruction to the slave node 72. The slave node 72 is also used to receive the data page elimination instruction; query the third data page corresponding to the data page identifier; and when the third application log sequence number corresponding to the third data page is greater than or equal to the second persistent log sequence number, mark the second data page as being eligible for elimination from the mandatory cache queue.
[0143] Optionally, the master node 71 is also used to obtain a first data page after performing a first change operation on the data page, and write a first redo log corresponding to the first change operation to the storage layer; generate a first notification message, the notification message including a first offset of the first redo log in the storage layer; send the first notification message to the slave node 72; the slave node 72 is also used to read the first redo log from the storage layer according to the first offset, obtain the first data page after applying the first redo log, generate a first application log sequence number, and add the first data page to the forced cache queue of the slave node 72.
[0144] Optionally, the master node 71 is further used to set the persistence flag corresponding to the second data page to be valid; when the persistence operation on the second data page is completed, the persistence flag is cleared.
[0145] The present application also provides an electronic device, such as Fig. 9 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .
[0146] Memory 1503, used for storing computer programs;
[0147] The processor 1501 is used to implement the steps of the above method embodiment when executing the computer program stored in the memory 1503.
[0148] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0149] The communication interface is used for communication between the above electronic device and other devices.
[0150] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0151] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0152] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method embodiment described below are implemented.
[0153] It should be noted that, for the above-mentioned device, electronic device and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0154] It should be further noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0155] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A data page processing method, It is characterized in that include: Receive a first application log sequence number sent by a slave node, where the first application log sequence number is obtained after the slave node applies a first redo log corresponding to a first data page; Determine a minimum application log sequence number from the first application log sequence numbers sent by all the slave nodes; When it is determined that the minimum application log sequence number is greater than or equal to the first persistence log sequence number corresponding to the last data page persistence operation, the second data page after the current change operation is marked as persistent; wherein the change operation corresponding to the second data page occurs after the change operation corresponding to the first data page.
2. The method according to claim 1, It is characterized in that The method further comprises: After performing a first change operation on a data page, a first data page is obtained, and a first redo log corresponding to the first change operation is written into a storage layer; Generate a first notification message, the notification message including a first offset of the first redo log in the storage layer; The first notification message is sent to the slave node, so that the slave node reads the first redo log from the storage layer according to the first offset, obtains the first data page after applying the first redo log, generates a first application log sequence number, and adds the first data page to the forced cache queue of the slave node.
3. The method according to claim 1, It is characterized in that The method further comprises: After the persistence operation on the second data page is completed, a data page elimination instruction including a data page identifier corresponding to the second data page and a second persistence log sequence number is generated; The data page elimination instruction is sent to the slave node.
4. The method according to claim 1, It is characterized in that The step of marking the second data page after the current change operation as being persistent includes: Setting the persistence flag corresponding to the second data page to valid; The method further comprises: After the persistence operation on the second data page is completed, the persistence flag is cleared.
5. A data page processing method, It is characterized in that include: Receive a data page elimination instruction sent by the master node after the persistence operation on the second data page, the data page elimination instruction includes a data page identifier and a second persistence log sequence number corresponding to the second data page; wherein the master node determines the minimum application log sequence number from the first application log sequence numbers sent by all slave nodes; the first application log sequence number is obtained after the slave node applies the first redo log corresponding to the first data page; when it is determined that the minimum application log sequence number is greater than or equal to the first persistence log sequence number corresponding to the last data page persistence operation, mark the second data page after the current change operation as persistable; wherein the change operation corresponding to the second data page occurs after the change operation corresponding to the first data page; Querying a third data page corresponding to the data page identifier; When the third application log sequence number corresponding to the third data page is greater than or equal to the second persistent log sequence number, the second data page is marked as being eligible for elimination from the mandatory cache queue.
6. The method according to claim 5, It is characterized in that The method further comprises: Reading a third redo log from the storage layer according to a third notification message sent by the master node, wherein the third notification message includes a third offset of the third redo log in the storage layer; Applying the third redo log to implement a third change operation corresponding to the third redo log to obtain a third data page; Adding the third data page to the mandatory cache queue; Send the third application log sequence number corresponding to applying the third redo log to the master node.
7. A data page processing device, It is characterized in that include: A receiving module, configured to receive a first application log sequence number sent by a slave node, wherein the first application log sequence number is obtained after the slave node applies a first redo log corresponding to a first data page; A determination module, used to determine a minimum application log sequence number from the first application log sequence numbers sent by all the slave nodes; An identification module is used to identify the first data page after the current change operation as persistent when it is determined that the minimum application log sequence number is greater than or equal to the first persistence log sequence number corresponding to the last data page persistence operation; wherein the change operation corresponding to the second data page occurs after the change operation corresponding to the first data page.
8. A data page processing device, It is characterized in that include: A receiving module, for receiving a data page elimination instruction sent by the master node after a persistence operation on the second data page, wherein the data page elimination instruction includes a data page identifier and a second persistence log sequence number corresponding to the second data page; wherein the master node determines a minimum application log sequence number from the first application log sequence numbers sent by all slave nodes; the first application log sequence number is obtained after the slave node applies the first redo log corresponding to the first data page; when it is determined that the minimum application log sequence number is greater than or equal to the first persistence log sequence number corresponding to the last data page persistence operation, the second data page after the current change operation is marked as persistable; wherein the change operation corresponding to the second data page occurs after the change operation corresponding to the first data page; A query module, used for querying a third data page corresponding to the data page identifier; The identification module is used to identify the second data page as being eligible for elimination from the mandatory cache queue when the third application log sequence number corresponding to the third data page is greater than or equal to the second persistent log sequence number.
9. A data page processing system, It is characterized in that include: A master node and at least one slave node connected to each other, wherein the master node and the slave node are respectively connected to the storage layer; The slave node is used to obtain a first application log sequence number after applying a first redo log corresponding to a first data page, and send the first application log sequence number to the master node; The master node is used to determine the minimum application log sequence number from the first application log sequence numbers sent by all the slave nodes; When it is determined that the minimum application log sequence number is greater than or equal to the first persistence log sequence number corresponding to the last data page persistence operation, the second data page after the current change operation is marked as persistent; wherein the change operation corresponding to the second data page occurs after the change operation corresponding to the first data page.
10. The system according to claim 9, It is characterized in that The master node is further configured to generate a data page elimination instruction including a data page identifier corresponding to the second data page and a second persistent log sequence number after the persistence operation on the second data page is completed; and send the data page elimination instruction to the slave node; The slave node is also used to receive the data page elimination instruction; query the third data page corresponding to the data page identifier; when the third application log sequence number corresponding to the third data page is greater than or equal to the second persistent log sequence number, mark the second data page as being eligible for elimination from the forced cache queue.
11. The system according to claim 9, It is characterized in that The master node is further configured to obtain a first data page after performing a first change operation on the data page, and write a first redo log corresponding to the first change operation into the storage layer; Generate a first notification message, the notification message including a first offset of the first redo log in the storage layer; Sending the first notification message to the slave node; The slave node is also used to read the first redo log from the storage layer according to the first offset, obtain the first data page after applying the first redo log, generate a first application log sequence number, and add the first data page to the forced cache queue of the slave node.
12. The system according to claim 9, It is characterized in that The master node is further used to set the persistence flag corresponding to the second data page to be valid; when the persistence operation on the second data page is completed, the persistence flag is cleared.
13. An electronic device, It is characterized in that include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is used to implement the method steps described in any one of claims 1-6 when executing the computer program.
14. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.
