Data page processing method and device, electronic equipment and storage medium

Abstract

The application provides a data page processing method and device, electronic equipment and a storage medium, wherein the method comprises: parsing redo logs of a first batch to obtain a plurality of application tasks, a target copy page containing a target original page in a target memory space, the target original page being a data page associated with a target operation parsed from the redo logs of the first batch, the target operation being a split or merge operation of the data page; distributing the plurality of application tasks to a plurality of worker threads, each worker thread in the plurality of worker threads being used for processing part of the plurality of application tasks; executing the plurality of application tasks through the plurality of worker threads, an application task corresponding to the target original page in the plurality of application tasks being applied to the target copy page; and in the case that the plurality of application tasks are executed, updating data of the target copy page to the target original page by adding an exclusive lock to a target index, the target index being an index in which the target original page is located.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a data page processing method and device, an electronic device, and a storage medium. BACKGROUND

[0002] At present, an InnoDB storage engine (the default storage engine of MySQL) organizes the storage of table data in the form of a B+ tree index structure. Each node in the tree structure represents a data page, and the node types in the tree are divided into leaf nodes and non-leaf nodes. When data is inserted into or deleted from a table, the splitting or merging of related nodes in the tree may be caused. SMO (Split and Merge Operations) of an InnoDB data page refers to the splitting and merging operations of the data pages in which the InnoDB internally stores data. When SMO occurs, the number of corresponding changed pages is greater than or equal to 2.

[0003] Physical replication of InnoDB is performed through redo logs (i.e., redo logs) rather than binlog logs, and data synchronization is performed by reading the changes of specific data pages in the master library through redo logs and synchronizing the changes of the data pages in the master library to the corresponding pages in the slave library.

[0004] The slave library reads the redo logs sent by the master library in batches through a coordinator thread and parses the MRT (Mini-Transaction, the finest granularity atomic operation) according to the redo logs, distributes the parsed atomic tasks to multiple worker threads, and applies the tasks in a Pingpang hash parallel manner to synchronize the changes to the corresponding data pages, so that the master and slave data are consistent.

[0005] When the coordinator thread parses SMO, because SMO involves changes to multiple pages, in order to ensure the atomicity of SMO operations, the related art generally applies SMO in a serial manner for SMO physical replication: from the start of parsing SMO by the coordinator thread, the coordinator thread is blocked (in order to wait for the worker to apply the content before SMO), then the worker thread is blocked (at this time, the coordinator thread no longer parses the log to start applying SMO), after the SMO application is completed, the coordinator thread starts to parse the log, and the worker thread starts to work.

[0006] In addition, in order to ensure atomicity, the coordinator thread applies the SMO process to add an X lock (exclusive lock) to the index, blocks external requests for the index, and reduces the concurrency of the entire system.

[0007] Therefore, the serial application of SMO in the related art has the problems of low efficiency of physical replication and low concurrency. SUMMARY

[0008] The present application provides a data page processing method and device, an electronic device and a storage medium, to at least solve the problems of low efficiency of physical replication and low concurrency in the serial application of SMO in the related art.

[0009] According to an aspect of an embodiment of the present application, a data page processing method is provided, including: parsing a first batch of redo logs to obtain a plurality of application tasks, wherein a target copy page containing a target original page is in a target memory space, the target original page is a data page associated with a target operation parsed from the first batch of redo logs, and the target operation is a split or merge operation of a data page; distributing the plurality of application tasks to a plurality of worker threads, wherein each worker thread in the plurality of worker threads is used to process part of the plurality of application tasks; executing the plurality of application tasks through the plurality of worker threads, wherein an application task corresponding to the target original page in the plurality of application tasks is applied to the target copy page; and updating data of the target copy page to the target original page by adding an exclusive lock to a target index in a case where the plurality of application tasks are executed, wherein the target index is an index where the target original page is located.

[0010] According to another aspect of an embodiment of the present application, a data page processing device is also provided, including: a parsing unit configured to parse a first batch of redo logs to obtain a plurality of application tasks, wherein a target copy page containing a target original page is in a target memory space, the target original page is a data page associated with a target operation parsed from the first batch of redo logs, and the target operation is a split or merge operation of a data page; a distribution unit configured to distribute the plurality of application tasks to a plurality of worker threads, wherein each worker thread in the plurality of worker threads is used to process part of the plurality of application tasks; a first execution unit configured to execute the plurality of application tasks through the plurality of worker threads, wherein an application task corresponding to the target original page in the plurality of application tasks is applied to the target copy page; and a first update unit configured to update data of the target copy page to the target original page by adding an exclusive lock to a target index in a case where the plurality of application tasks are executed, wherein the target index is an index where the target original page is located.

[0011] According to a further aspect of the embodiments of the present application, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; wherein the memory is configured to store a computer program; and the processor is configured to execute the method steps in any of the above embodiments by running the computer program stored in the memory.

[0012] According to a further aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, wherein the computer program is configured to execute the method steps in any of the above embodiments when running.

[0013] In the embodiments of the present application, the copy page of the original page associated with the target operation (i.e., SMO) is stored in a special memory space, the first batch of redo logs is parsed to obtain a plurality of application tasks, wherein the target copy page containing the target original page in the target memory space is obtained, the target original page is a data page associated with the target operation (i.e., SMO) parsed from the first batch of redo logs, and the target operation is a split or merge operation of the data page; the plurality of application tasks are distributed to a plurality of worker threads, wherein each worker thread in the plurality of worker threads is configured to process part of the plurality of application tasks; the plurality of application tasks are executed by the plurality of worker threads, wherein the application task corresponding to the target original page in the plurality of application tasks is applied to the target copy page; in the case that the plurality of application tasks are executed, the data of the target copy page is updated to the target original page by adding an exclusive lock to the target index, wherein the target index is an index where the target original page is located. Since the SMO related original page is copied to a block of memory space when the SMO is parsed, and then the application task is distributed to the worker, the worker applies the application task of the SMO related original page to the copy of the original page when applying, and the content of the copy page is updated to the original page by adding an X lock to the index after the application of a batch of logs is completed, the application of the SMO can be performed in parallel by the worker, which does not block the work of the coordinator and the worker thread. In addition, the X lock is added to the index only when the copy page related to the SMO is copied to the original page content after the log of a batch is completed, instead of adding the index X lock in the whole process of applying the SMO, which can achieve the purpose of reducing the locking time, and achieves the technical effects of guaranteeing the atomicity of the SMO application, improving the efficiency of the physical replication and the concurrency of the system, thereby solving the problems of low efficiency of physical replication and low concurrency in the related art caused by the serial application of the SMO. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings required to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the field, under the premise of no creative labor, other drawings can also be obtained according to these drawings.

[0016] Figure 1 is a schematic diagram of a hardware environment of an optional data page processing method according to an embodiment of the present application;

[0017] Figure 2 is a flowchart of an optional data page processing method according to an embodiment of the present application;

[0018] Figure 3 is a schematic diagram of an optional data page processing method according to an embodiment of the present application;

[0019] Figure 4 is a schematic diagram of an optional SMO application atomicity according to an embodiment of the present application;

[0020] Figure 5 is a schematic diagram of another optional data page processing method according to an embodiment of the present application;

[0021] Figure 6 is a schematic diagram of an optional coordination thread processing flow according to an embodiment of the present application;

[0022] Figure 7 is a flowchart of another optional data page processing method according to an embodiment of the present application;

[0023] Figure 8 is a schematic diagram of an optional work thread processing flow according to an embodiment of the present application;

[0024] Figure 9 is a flowchart of yet another optional data page processing method according to an embodiment of the present application;

[0025] Figure 10 is a structural block diagram of an optional data page processing device according to an embodiment of the present application;

[0026] Figure 11 is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the personnel in the technical field better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described in the following combined with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should be within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] According to an aspect of the embodiments of the present application, a data page processing method is provided. Optionally, in the present embodiment, the above-mentioned data page processing method can be applied in a hardware environment composed of a terminal 102 and a server 104 as shown in the figure. Figure 1 As shown in the figure, the server 104 is connected with the terminal 102 through a network, which can be used to provide services (such as game services, application services, etc.) for the terminal or the client installed on the terminal, and a database can be set on the server or independently of the server, which is used to provide data storage services for the server 104. Figure 1 The above-mentioned network can include but is not limited to at least one of the following: wired network, wireless network. The above-mentioned wired network can include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above-mentioned wireless network can include but is not limited to at least one of the following: WIFI (Wireless Fidelity, Wireless Fidelity), Bluetooth. The terminal 102 can not be limited to PC, mobile phone, tablet computer, etc.

[0030] The data page processing method of the present application can be executed by the server 104, or by the terminal 102, or by the server 104 and the terminal 102 together. Wherein, the terminal 102 executing the data page processing method of the present application can also be executed by the client installed thereon.

[0031]

[0032] ​Taking an example of performing the data page processing method in the embodiment by the server 104, Figure 2 is a flowchart of an optional data page processing method according to an embodiment of the present application, as shown in the method, the flowchart can include the following steps: Figure 2

[0033] In step S202, the first batch of redo logs is parsed to obtain a plurality of application tasks, wherein the target copy page in the target memory space contains the target original page, and the target original page is a data page associated with the target operation parsed from the first batch of redo logs, and the target operation is a split or merge operation of the data page.

[0034] The data page processing method in the embodiment can be applied to the scenario of master-slave replication through redo logs, that is, the changes of specific data pages in the master database are read through the redo logs, and the changes of the master database pages are synchronized to the corresponding pages in the slave database to perform data synchronization. The data page processing method in the embodiment can be executed by a database server (i.e., a slave) or a database storage engine. Here, the database can be MySQL, and the database storage engine can be InnoDB. Correspondingly, the master-slave replication through redo logs is physical replication of InnoDB.

[0035] Optionally, in the embodiment, the above-mentioned data page processing method is taken as an example to be executed by a database server (which can be a slave) or InnoDB. In the case of no contradiction, other devices capable of performing master-slave replication through redo logs can also be used to execute the data page processing method in the embodiment.

[0036] The target slave can read the redo logs sent by the target master in batches through a target coordination thread, for example, the first batch of redo logs. The target coordination thread can parse the redo logs, for example, according to MTR, distribute the parsed atomic tasks to a plurality of worker threads, and apply the tasks in a Pingpanghash parallel manner by the worker threads, so as to synchronize the changes to the corresponding data pages and make the master-slave data consistent.

[0037] For example, as shown in Figure 3 ​As shown, for InnoDB, the Apply_coordinator_thread of the slave can read a batch of redo logs (e.g., 100 redo logs) from the Ib_logfile group (i.e., the log file group), and distribute the parsed atomic tasks to multiple Log worker threads, where each Log worker thread can correspond to two hash buckets, and each hash bucket can be considered as a group of hash tables, including at least one system hash table and one user hash table. For a Log worker thread, the application tasks (i.e., the above-mentioned atomic tasks) parsed from each batch of redo logs and distributed to the Log worker thread are distributed to one hash bucket of the Log worker thread.

[0038] The worker thread applies the tasks in a Pingpang hash parallel manner, that is, a batch of application tasks are distributed to one hash bucket of the worker thread, and the coordinator thread can continue to parse the next batch of redo logs, distribute the parsed application tasks to another hash bucket of the worker thread, and continuously perform the above-mentioned parsing and distribution steps.

[0039] In the process of parsing a batch of redo logs by the coordinator thread, if the coordinator thread parses an SMO (i.e., a target operation), since the SMO involves changes to multiple data pages, in order to ensure the atomicity of the SMO operation (i.e., an operation or multiple operations are either all executed and the execution process is not interrupted by any factor, or none of them are executed), the related art usually adopts a serial physical replication manner of SMO to solve the application of SMO, and the main operation process is as follows:

[0040] Step 1, the coordinator thread parses the SMO, and no longer distributes tasks to the worker thread, but blocks and waits until the worker thread group applies all the redo logs before the current SMO to ensure that the data pages related to the SMO are in the state before the SMO occurs;

[0041] Step 2, the coordinator thread adds an X lock (exclusive lock) to the entire index, blocks external read requests from accessing the data pages related to the SMO, and then parses and applies the SMO by the coordinator thread, and releases the X lock of the index after the application is completed (in this process, the worker thread is blocked and waits);

[0042] Step 3, after the coordinator thread applies the SMO, it continues to parse the redo logs and distributes application tasks to the worker thread, and the worker thread starts to execute the application tasks.

[0043] For the above-mentioned serial application SMO mode, the efficiency and concurrency of physical replication are greatly reduced, and the index X lock is added during the entire SMO application process, which blocks external requests for the index and reduces the concurrency of the entire system.

[0044] Optionally, in the embodiment, a piece of memory, that is, a target memory space, is added. The target coordination thread can parse the first batch of redo logs to obtain a plurality of application tasks. In the process of parsing, if a target operation is parsed, the target operation is a split or merge operation of a data page (that is, SMO), and the data page associated with the target operation can be a target original page. The target coordination thread can ensure that the target copy page containing the target original page is included in the target memory space.

[0045] For example, if a copy page of the target original page (that is, a target copy page) already exists in the target memory space, the copy page can be created when parsing the batch of redo logs before, or can be created when parsing the redo logs before the same batch, the target coordination thread can no longer create a new copy page. If there is no copy page of the target original page in the target memory space, the target coordination thread can copy (that is, copy) a copy to the target memory space to obtain the target copy page.

[0046] Exemplarily, a piece of memory SMO apply buffer (SMO application buffer, an example of the target memory space) can be added, and when the coordination thread parses the SMO, the data page related to the SMO is copied to the SMO apply buffer to obtain a copy page (which can be called a shadow page).

[0047] In step S204, the plurality of application tasks are distributed to a plurality of worker threads, wherein each worker thread in the plurality of worker threads is used to process part of the plurality of application tasks.

[0048] For the plurality of application tasks parsed, the target coordination thread can distribute the plurality of application tasks to a plurality of worker threads. The plurality of worker threads can be worker threads in the target slave library for processing application tasks parsed from the redo logs, for example, Log worker thread.

[0049] Optionally, the parsing of the redo log and the distribution of the application task can be serially executed, that is, the distribution of the application task is performed after the parsing of a batch of redo logs, or can be executed in parallel, that is, the distribution of the application task is performed after the parsing of a redo log. The embodiment does not limit this.

[0050] In the application task distribution, the application tasks corresponding to the same data page can be distributed to the same worker thread. In order to ensure the efficiency and rationality of the task distribution, the number of worker threads included in the plurality of worker threads and the identification information of the data page corresponding to the application task can be hashed, and the application tasks can be distributed according to the hash values.

[0051] Optionally, the identification information of the data page can include but is not limited to space_id (table space ID) and page_no (page number in the table space). For other identification information capable of uniquely identifying a data page, the above-mentioned application task distribution manner is also applicable.

[0052] In step S206, the plurality of application tasks are executed by the plurality of worker threads, wherein the application task corresponding to the target original page is applied to the target copy page.

[0053] For each worker thread in the plurality of worker threads, the worker thread can execute the application task distributed to the worker thread. If the data page corresponding to an application task does not belong to the target original page, the application task can be executed on the data page corresponding to the application task. If the data page corresponding to an application task belongs to the target original page, the application task can be executed on the target copy page of the target original page.

[0054] The determination of whether the data page corresponding to an application task belongs to the target original page (that is, whether the data page has a copy page) can be performed in various ways. The mapping information can be used to record whether each data page has a copy page and the location of the copy page in the target memory space. Alternatively, the information in the block (that is, the data block, the storage unit of the data page) where each data page is located, that is, the block information, can be used to record whether each data page has a copy page and the location of the copy page in the target memory space. In this embodiment, the way of determining whether each data page has a copy page is not limited.

[0055] In step S208, when the plurality of application tasks are executed, the data of the target copy page is updated to the target original page by adding an exclusive lock to the target index, wherein the target index is the index where the target original page is located.

[0056] After the plurality of application tasks are executed, the data of the target copy page can be updated to the target original page by the target coordination thread, any one of the plurality of worker threads, or other threads. When the data of the target copy page is updated to the target original page, the X lock (that is, Index_X_lock) of the target index where the target original page is located can be added first, and the X lock of the target index is released after the target original page is updated.

[0057] In the embodiment, the atomicity of the SMO application is ensured by means of the copy page, the application work of the SMO is executed in parallel by multiple worker threads, and the data of the original page is updated after the application is completed, so that the correctness of the SMO replication is ensured.

[0058] In addition, the copy of the content of the copy page to the original page involved in the SMO is performed on the index X lock only after the completion of one batch of logs, and the index X lock is released after the completion of the replication of all the copy pages. Compared with the index X lock during the entire SMO application process, the locking time can be greatly reduced, and the concurrency of the system is improved.

[0059] By the steps S202 to S208, the redo logs of the first batch are parsed to obtain a plurality of application tasks, wherein the target copy page containing the target original page in the target memory space, the target original page is a data page associated with the target operation parsed from the redo logs of the first batch, and the target operation is a splitting or merging operation of the data page; the plurality of application tasks are distributed to a plurality of worker threads, wherein each worker thread in the plurality of worker threads is used to process part of the application tasks in the plurality of application tasks; the plurality of application tasks are executed by the plurality of worker threads, wherein the application task corresponding to the target original page in the plurality of application tasks is applied to the target copy page; in the case that the plurality of application tasks are executed, the data of the target copy page is updated to the target original page by adding an exclusive lock to the target index, wherein the target index is the index where the target original page is located, thereby solving the problems of low efficiency and low concurrency of physical replication in the related art serial SMO application mode, ensuring the atomicity of the SMO application, and improving the efficiency of the physical replication and the concurrency of the system.

[0060] As an optional embodiment, before the data of the target copy page is updated to the target original page by adding an exclusive lock to the target index, the method further includes:

[0061] S11, receiving a target request message sent by a target client, wherein the target request message is used to request reading of target data in the target original page;

[0062] S12, in response to the target request message, sending the target data in the target original page to the target client.

[0063] When the log content of the SMO operation is applied, the application tasks of different pages involved in the SMO may be distributed to different worker threads, and the atomicity of the SMO cannot be ensured. As shown in FIG. 1, during the process of applying the SMO log content, the user simultaneously issues a read request for the related page, and if the data is searched in the copy page, an error of not finding the data may be caused due to the incomplete application of a certain page. Figure 4

[0064] ​Optionally, in this embodiment, while the worker threads are applying SMO in parallel on the replica pages, the original pages in the index can be used to respond to external requests, without blocking the work of the coordinating thread and the worker threads, or blocking external requests.

[0065] Before acquiring an exclusive lock on the target index, if the target slave receives an external read request, such as a target request message sent by the target client requesting to read target data from the target original page, the target slave can directly send the target data from the target original page to the target client in response to the target request message.

[0066] This embodiment improves the timeliness of request response by allowing data to be searched directly on the original page without blocking external requests, while avoiding a decrease in user experience due to the inability to find data.

[0067] As an optional embodiment, the above method further includes the following steps during the parsing of the first batch of redo logs:

[0068] S21, if the target redo log in the first batch of redo logs is parsed, the application task corresponding to the target redo log is pre-executed by the target coordination thread. The target redo log is the redo log associated with the creation of the data page.

[0069] Since the target original page can be an old original page related to a split or merge, or a newly split data page, if the newly split data page belongs to a newly requested extent (i.e., zone), the corresponding extent needs to be created based on the redo log before it can be obtained through the data page's identification information.

[0070] For example, the data pages associated with the MTR read during SMO parsing are divided into two categories: new data pages split from the old original B+Tree pages related to B+Tree splits or merges. If both types of data pages belong to the same initialized extent, they are directly obtained from the Buffer Pool or disk using space_id and page_no. If the newly split data page belongs to a newly allocated extent, the corresponding extent needs to be created based on the redo log, and then obtained using space_id and page_no.

[0071] To ensure the timely retrieval of target raw pages, during the parsing of the first batch of redo logs, if a redo log related to data page creation is encountered, the target coordination thread can execute the application task corresponding to the target redo log to complete the data page creation. The application task for the target redo log is completed before other application tasks; that is, it is pre-completed.

[0072] From the above processing logic of the library coordination thread, the related type of redo log can be created for the data page in the process of parsing the redo log, and the application is completed in advance, so that no matter which type of data page copy, when the SMO is parsed, the space_id and page_no can be directly used to realize the acquisition.

[0073] Through the embodiment, by the library coordination thread creating the related type of redo log for the data page in advance, the timeliness of the original page acquisition related to the SMO can be ensured.

[0074] As an optional embodiment, before the plurality of application tasks are distributed to the plurality of worker threads, the above method further comprises:

[0075] S31, a target available memory of a target size is applied in a target memory space, wherein the target size is the size of the target original page;

[0076] S32, the data in the target original page is copied into the target available memory to obtain a target copy page.

[0077] Before the plurality of application tasks are distributed to the plurality of worker threads, or at least before the application task corresponding to the target operation is distributed to a certain worker thread in the plurality of worker threads, the creation of the target copy page can be completed first. The creation of the target copy page can be performed by the target coordination thread when parsing the first batch of redo logs, or can be performed by the target coordination thread when parsing a batch of redo logs before the first batch.

[0078] The target coordination thread can first apply the memory space of the target copy page, that is, apply a target available memory of a target size in a target memory space, and here, the target size is the size of the target original page. Alternatively, if the target original page is a plurality of original pages, the timing of applying the memory space of the copy page of each original page can be different, and the size of the memory space of the copy page of different original pages can be the same or different.

[0079] For example, the copy page memory address can be applied, that is, an available memory of a data page size is obtained through the buf_written of the SMO apply buffer, and the buf_written is updated to the next address offset of the currently allocated memory address. Here, the buf_written can be considered as the first address pointing to the available memory space in the SMO apply buffer.

[0080] After the target available memory is applied, the target coordination thread can copy the data in the target original page into the target available memory to obtain the target copy page.

[0081] For example, the data of the frame page in the block can be copied to the address space of the applied SMO apply buffer by memcpy (i.e., a memory copy function) with a size of 16k.

[0082] According to the embodiment, the application of the copy page according to the size of the original page and the data copying can improve the rationality of the utilization of the memory space for storing the copy page.

[0083] As an optional embodiment, in combination with the Pingpang hash parallel replication strategy, the task can be applied by the worker thread in the Pingpang hash parallel mode, that is, at least two groups of hash tables corresponding to each worker thread are used to store the application tasks executed by each worker thread and parsed from different batches of redo logs in turn. Each group of hash tables can include a system hash table and a user hash table.

[0084] Optionally, in the embodiment, the distribution of the multiple application tasks to the multiple worker threads includes:

[0085] S41, distribute the multiple application tasks to the target group hash table corresponding to each worker thread, wherein the application tasks corresponding to the same data page in the multiple application tasks are distributed to the same group hash table, and the application tasks in the same group hash table are executed in the order of the time identifier.

[0086] When the multiple application tasks are distributed, the target coordination thread can distribute the multiple application tasks to the target group hash table corresponding to each worker thread. The target group hash table can refer to a group of hash tables in which each worker thread stores the application tasks parsed from the first batch of redo logs, and the target group hash tables corresponding to different worker threads are different. The application tasks corresponding to the same data page in the multiple application tasks are distributed to the same group hash table, that is, the same target group hash table. Each target group hash table can store the application tasks corresponding to one or more data pages.

[0087] For the application tasks in the same group hash table, they can be executed by the corresponding worker thread in the order of the time identifier. Here, the time identifier can be the time identifier of the redo log, which can be used to represent the generation time of the corresponding redo log, etc. The embodiment does not limit this.

[0088] According to the embodiment, by combining the PingPang hash parallel replication strategy, the application tasks corresponding to the same data page are distributed to the same group hash table, and the application tasks in the same group hash table are executed in the order of the time identifier, which can improve the orderliness of the execution of the application thread and avoid data errors caused by the execution order error.

[0089] As an optional implementation, the number of hash tables corresponding to each worker thread is two. Correspondingly, before multiple application tasks are distributed to multiple worker threads, the above method further includes:

[0090] S51, update the hash table identifier of the target original page to the target table identifier of the target group hash table.

[0091] Optionally, for each data page, the target database can configure identification information for its corresponding hash tables. This identification information can be used to indicate whether the application task stored in the corresponding hash table is executed on the original data page or on a copy of the data page. For example, each worker thread can correspond to three hash tables, and three flag bits can be set in the data block corresponding to each data page: 0 indicates that the application task is executed on the original data page, and 1 indicates that the application task is executed on the copy page. After execution reaches the corresponding hash table, the flag bit of that hash table is set to 0.

[0092] Optionally, if each worker thread corresponds to two sets of hash tables, a corresponding hash table identifier can be set for each data page. The hash table identifier of a data page corresponds to the application task parsed from a batch of redo logs, indicating the application task within that hash table set. This application task needs to be executed on its replica page. For the target original page, the target coordinating thread can update the hash table identifier of the target original page to the target table identifier (or target group identifier) ​​of the target group hash table. Optionally, the hash table identifier of each data page can be included in the block information of the data block containing that data page; that is, in the block information of that data page.

[0093] Optionally, after updating the target replica page's data to the target original page by acquiring an exclusive lock on the target index, the above method further includes:

[0094] S52, Check the hash table identifier of the target original page;

[0095] S53, if the hash table identifier of the target original page is inconsistent with the target table identifier, retain the target copy page;

[0096] S54. If the hash table identifier of the target original page matches the target table identifier, release the target copy page.

[0097] If the data page associated with the parsed SMO contains the target original page when parsing the next batch of redo logs after the first batch of redo logs, the hash table identifier of the target original page is updated to the hash table identifier of another set of hash tables. If the data page associated with the parsed SMO does not contain the target original page when parsing the next batch of redo logs after the first batch of redo logs, the hash table identifier of the target original page remains the hash table identifier of the target set of hash tables, i.e., the target table identifier.

[0098] It should be noted that the hash tables corresponding to the respective worker threads are switched at the same time, that is, the number of hash table identifiers can be the same as the number of sets of hash tables corresponding to one worker thread.

[0099] After updating the data of the target copy page into the target original page, the hash table identifier of the target original page can be checked. If the hash table identifier of the target original page is consistent with the target table identifier, it indicates that the target original page is not the data page associated with the target operation parsed from the next batch of redo logs. In order to improve the utilization of the target memory space, the target copy page can be released so as to reuse the memory space occupied by the target copy page.

[0100] If the hash table identifier of the target original page is inconsistent with the target table identifier, it indicates that the target original page is the data page associated with the target operation parsed from the next batch of redo logs. In order to reduce the operations (such as releasing the copy page, re-applying the memory space, and copying the data page) on the target memory space, the target copy page can be retained, and the application task executed on the target original page and parsed from the next batch of redo logs continues to be executed on the target copy page.

[0101] Through the embodiment, the hash table identifier of the data page is used to control whether to continue to retain the copy page or release the copy page, and the rationality of the utilization of the memory space can be improved.

[0102] As an optional embodiment, executing the plurality of application tasks through the plurality of worker threads comprises:

[0103] S61, obtaining a target application task to be executed through a first worker thread, wherein the first worker thread is one of the plurality of worker threads, and the target application task is one of the plurality of application tasks;

[0104] S62, checking copy page information of a target data page corresponding to the target application task, wherein the copy page information of the target data page is used to indicate whether the target data page has a copy page;

[0105] S63, in a case where it is determined according to the copy page information of the target data page that the target data page has a copy page, performing the target application task on the copy page of the target data page, wherein the target original page contains the target data page, and the target copy page contains the copy page of the target data page.

[0106] In the target library, whether a data page has a copy page can be indicated by copy page information of the data page. The copy page information can be a copy page identifier used to identify whether the data page has a copy page. Alternatively, the copy page information can also be location indication information of the copy page, which can indicate the location of the copy page in the target memory space.

[0107] For any one of the plurality of work threads, i.e., a first work thread, the first work thread can obtain a current application task to be executed, i.e., a target application task, to be applied to a data page, i.e., a target data page.

[0108] The first work thread can check the copy page information of the target data page, and determine whether the target data page has a copy page according to the copy page information of the target data page. If the target data page has a copy page (the target data page belongs to the target original page described above), the first work thread can execute the target application task on the copy page of the target data page (the copy page of the target data page belongs to the target copy page described above).

[0109] Through the embodiment, by identifying whether a data page has a copy page through copy page information and determining whether an application task of the data page is executed on the data page or the copy page according to the copy page information of the data page, the execution efficiency and accuracy of the application task can be improved.

[0110] As an optional embodiment, checking the copy page information of the target data page corresponding to the target application task comprises:

[0111] S71, checking target field information of a target field of the target data page, wherein the target field of a data page is used to store an address pointer of a first address of a copy page of the data page, and the copy page information of the target data page is the target field information.

[0112] The copy page information of a data page can be stored in a target field of the data page. In this scenario, the copy page information of the data page is field information of the target field of the data page, wherein the target field of the data page refers to a target field (e.g., a frame_shadow field) in a data block where the data page is located. The target field of a data page can be used to store an address pointer of a first address of a copy page of the data page, and whether the data page has a copy page can be indicated by whether the target field is empty.

[0113] The first worker thread can check the target field information of the target field of the target data page. If the target field information is empty, it indicates that the target data page has no copy page. If the target field information is not empty, it indicates that the target data page has a copy page. At the same time, the target field information is an address pointer of the starting address of the copy page of the target data page, which can indicate the location of the copy page of the target data page.

[0114] Correspondingly, the target application task is executed on the copy page of the target data page, including:

[0115] S72, the target application task is executed on the copy page pointed to by the target field information.

[0116] Through the target field information, the first worker thread can locate the copy page of the target data page (i.e. the copy page pointed to by the target field information), and execute the target application task on the copy page of the target data page.

[0117] It should be noted that if the copy page information of the data page is only used to identify whether the data page has a copy page, the copy page of the data page can be located by, for example, traversing the target memory space. Compared with the above-mentioned manner, in the embodiment, the address pointer of the starting address of the copy page of the data page is stored in the target field of the data page, which can improve the positioning efficiency of the copy page while making less changes to the existing processing logic.

[0118] As an optional embodiment, after the data of the target copy page is updated into the target original page by taking an exclusive lock on the target index, the above-mentioned method further includes:

[0119] S81, in the case that the target data page is not the data page associated with the target operation parsed from the redo log of the second batch, the copy page information of the target data page is set as the target copy page information, wherein the target copy page information is used to indicate that the target data page has no copy page, and wherein the second batch is the next batch of the first batch.

[0120] After the redo log of the first batch is parsed, the target coordination thread can continue to parse the redo log of the next batch, i.e. the redo log of the second batch. If the target data page is not the data page associated with the target operation parsed from the redo log of the second batch, the application task corresponding to the target original page parsed from the redo log of the second batch is executed on the target original page.

[0121] Optionally, determining whether the target data page is the data page associated with the target operation parsed from the redo log of the second batch can be determined by the identification information (e.g. hash table identification) of the aforementioned hash table, or can be recorded by other information, which is not limited in the embodiment.

[0122] In this case, the target coordination thread can set the copy page information of the target data page as the target copy page information, and release the copy page of the target data page. For example, the copy page identifier of the target data page is set as the target identifier (e.g., 0), and for example, the target field of the target data page is set as empty, and the like.

[0123] If the target data page is a data page associated with the target operation parsed from the second batch of redo logs, the target coordination thread can keep the copy page information of the target data page, and also keep the copy page of the target data page.

[0124] Through the embodiment, by updating the copy page information of the data page in time, the accuracy of the application task execution can be improved.

[0125] As an optional embodiment, the number of target original pages is usually multiple, that is, the target original page can contain multiple original pages. Correspondingly, the target copy page can also contain multiple copy pages, and the multiple original pages and the multiple copy pages are one-to-one corresponding. The multiple original pages can be associated with the same index, or belong to different indexes. The target coordination thread can record the mapping relationship between the multiple original pages and the corresponding indexes, for example, record the indexes containing the original pages and the original pages contained by the indexes.

[0126] Optionally, the target index contains multiple indexes, and the target coordination thread can record the target mapping information, which can be used to indicate the mapping relationship between each index in the multiple indexes and the original pages belonging to each index in the multiple original pages.

[0127] Correspondingly, the data of the target copy page is updated into the target original page by taking an exclusive lock on the target index, including:

[0128] S91, obtaining, by the second worker thread, a current index to be processed in the multiple indexes from the target mapping information, where the second worker thread is the last worker thread completing the application in the multiple worker threads;

[0129] S92, taking, by the second worker thread, an exclusive lock on the current index;

[0130] S93, updating, by the second worker thread, the data of the copy page of the current original page into the current original page, where the current original page is an original page belonging to the current index in the multiple original pages;

[0131] S94, releasing, by the second worker thread, the exclusive lock on the current index.

[0132] The operation of updating the data of the target copy page to the target original page can be performed by a last work thread of the plurality of work threads, i.e., a second work thread. The second work thread can update the data of the original page in units of indexes. The second work thread can traverse the target mapping information to obtain a current index to be processed in the plurality of indexes. The original pages belonging to the current index in the plurality of original pages are current original pages, and the number of the current original pages can be one or more.

[0133] After the current index is determined, the second work thread can first add an X lock to the current index; then, update the data of the copy page of the current original page to the current original page, and if the number of the current original pages is more than one, the second work thread can sequentially update the data of each of the current original pages; and finally, release the X lock of the current index after the data of all the current original pages is updated.

[0134] Optionally, the target coordination thread can also record only the plurality of original pages, and the second work thread updates the data of the original pages in units of original pages, in which case the same index can be locked multiple times. According to the embodiment, the mapping information records the data pages associated with the SMO operation in each index, and the data of the data pages is updated in units of indexes, which can simplify the data update process and improve the efficiency of data update.

[0135] As an optional embodiment, the second work thread traverses the target mapping information to obtain a current index to be processed in the plurality of indexes includes:

[0136] S1001, the second work thread traverses the target tuple set to obtain a current index to be processed in the plurality of indexes, wherein the target mapping information is the target tuple set, and a target tuple in the target tuple set is a tuple with an index identifier of an index in the plurality of indexes as a key and a target set container as a value, and the target set container contains original page identifiers of original pages belonging to the index in the plurality of original pages.

[0137] The target mapping information can be a target tuple set, and the target tuple set can contain one or more target tuples, each of which corresponds to an index. An index and its contained original pages can be recorded by a target tuple in the target tuple set, which can be a tuple with an index identifier (i.e., index id) of the index as a key and a target set container as a value, and the target set container contains original page identifiers of original pages belonging to the index in the plurality of original pages.

[0138] The second worker thread can traverse the target tuple set, determine a target tuple currently to be processed in the target tuple set, and the target tuple and the index have a one-to-one correspondence relationship, so that the target tuple currently to be processed is corresponding to the current index, thereby obtaining a current index to be processed in the plurality of indexes.

[0139] Through the embodiment, by saving the mapping relationship between the index and the data page associated with the SMO in the index, the accuracy and convenience of information saving can be improved.

[0140] The processing method of the data page in the embodiment of the application will be explained and described below in combination with optional examples. In the example, the database storage engine is InnoDB, the target operation is SMO, the worker thread is a Log worker thread, the number of hash tables corresponding to the worker thread is two groups, that is, hash_table_0 and hash_table_1, and the target memory space is SMO apply buffer.

[0141] The example provides an SMO parallel replication optimization scheme based on InnoDB physical replication, increases a piece of memory SMO apply buffer, copies a copy of the data page related to the SMO to the SMO apply buffer when the SMO is parsed, and then distributes the application task to the worker thread. When the application is performed, if the worker thread finds that the data page is related to the SMO, the copy of the data page in the SMO apply buffer is applied, and after the log application of a batch is completed, the copy content of the data page in the SMO apply buffer is updated to the original page in the form of index X lock mutual exclusion.

[0142] In combination with Figure 5 , Figure 6 and Figure 7 , the flow of the processing method of the data page in the present optional example can include the following steps for the coordination thread:

[0143] Step S702, start to parse a batch of redo logs.

[0144] The coordination thread can parse a batch of redo logs, and distribute the parsed application task to the hash_table_0 of each worker thread.

[0145] Step S704, parse the SMO, copy the data page, and save the related information of the copy page.

[0146] The coordination thread can resolve the SMO, acquire the SMO related data page (i.e., original page), for example, can determine the data page from the Buffer Pool according to the space id and page no, and add a S lock (Page_S_lock) to the data page.

[0147] Then, the coordination thread can apply for memory space in the SMO apply buffer, copy the data page content to the SMO apply buffer to obtain a copy page of the data page.

[0148] And the coordination thread can record the address of the copy page in the SMO apply buffer and the serial number of the hash_table to the block corresponding to the data page, and then release the S lock of the data page.

[0149] For example, the coordination thread can update the frame_shadow pointer of the block corresponding to the data page to point to the starting address of the copy page in the SMO apply buffer, update the hash_no to the hash table serial number corresponding to the current coordination thread, wherein 0 represents hash_table_0 and 1 represents hash_table_1, and release the S lock of the data page after the update is completed.

[0150] Step S706, inserting the data page information of the SMO into the SMO_map.

[0151] According to the index id (i.e., index identifier) parsed from the redo log, the data page information of the SMO is recorded in the SMO_map (an example of a target tuple set). The reading and writing of the SMO_map can be protected by the smo_map_mutex mutex.

[0152] If there is already a tuple with the index id as the key in the SMO_map, then the space id and page no of the data page are inserted as elements into the value set (an example of a target collection container) corresponding to the key.

[0153] If there is no tuple with the index id as the key in the SMO map, a set is created, and the space id and page no of the data page are inserted as elements into the set, and then <index id, set> is inserted into the SMO_map.

[0154] Step S708, distribute the application task to different worker threads to complete the resolution.

[0155] The coordination thread can distribute the resolved application tasks to different worker threads, for example, according to spaceid and page no for hashing, and distribute the application tasks to the corresponding worker threads according to the hash values obtained by the hash_table_0.

[0156] The coordination thread can distribute the application tasks once per part of the redo log (for example, 1 redo log, and for example, other redo logs except SMO), or distribute the application tasks together after resolving all redo logs in the current batch.

[0157] After resolving the redo logs in the current batch, the coordination thread can switch to hash_table_1 and continue to resolve the redo logs in the next batch (if any). The resolution process is similar to the foregoing and will not be repeated here.

[0158] In combination with Figure 5 , Figure 8 and Figure 9 , for a worker thread, the flow of the processing method of the data page in the present alternative example can include the following steps:

[0159] Step S902, the worker thread starts to apply the application tasks in hash_table_1.

[0160] When the coordination thread resolves the redo logs and distributes them to hash_table_0, the worker thread is executing the application tasks in hash_table_1, that is, the worker thread starts to apply the logs in hash_table_1.

[0161] Step S904, traverse hash_table_1 and execute the application tasks on the data pages.

[0162] When traversing hash_table_1, the worker thread can add an X lock (i.e., Page_X_Lock) to the current data page to be processed to obtain the data page block, and check whether the field frame_shadow of the data page block is empty (NULL). If frame_shadow is not empty, the application tasks of the data page are executed on the copy page of the data page, and if frame_shadow is empty, the application tasks of the data page are executed on the original page (the original data page).

[0163] When executing the application tasks on the original page, the application can be executed on the data page in the Buffer Pool through the frame pointer in the block structure, and after the application is completed, the X lock of the data page block is released.

[0164] In addition, based on the Pingpang hash design, even if disk IO is caused by the data page not being in the buffer pool, it can not affect the replication efficiency.

[0165] When performing the application task on the replica page, the application on the replica page in the SMO apply buffer can be performed through the frame_shadow pointer in the block structure, and the X lock of the data page block is released after the application is completed.

[0166] In step S906, the last worker thread copies the replica page.

[0167] The copying of the original page using the replica page can be performed by the last worker thread completing the application in the worker thread group. The last worker thread can traverse the SMO_map, and process each tuple composed of <index id, set<space id, page no>> in the SMO_map. When accessing each tuple in the SMO_map, an X lock (i.e., index_X_lock) of the index corresponding to the key (index id) in the tuple can be acquired, and the index lock (i.e., unlock index_X_lock) is released after processing each data page in the corresponding value set.

[0168] When copying the replica page, for each data page, the original page in the buffer pool can be acquired, and an X lock (i.e., Page_X_lock) of the original page is acquired, and the original page is copied through the replica page. At the same time, the hash_no can be checked to determine whether the replica page needs to be released and whether the block information needs to be reset. If reset is needed, the element is deleted from the set. Finally, the X lock of the data page is released.

[0169] If the set container is empty after traversal, the tuple is deleted from the SMO_map, and then the set container of the next tuple in the SMO_map is traversed until all tuples in the SMO_map are traversed.

[0170] Through the parallel replication strategy of the PingPang hash in the example, by increasing the multi-version page of the memory copy of the data page, the worker can apply the data page in parallel under the premise of ensuring the atomicity of the SMO (that is, parallelize the application replication of the SMO), can replicate the SMO in parallel by the worker under the premise of ensuring the atomicity, shorten the blocking time of the external request as much as possible, that is, reduce the time of adding the X lock to the index as much as possible, can improve the efficiency of the physical replication, improve the concurrency of the system, and thus solve the problems of the serialization of the SMO physical replication process and the high mutual exclusion overhead of the read request.

[0171] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0172] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk, or an optical disk), and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device) to execute the method described in each embodiment of the present application.

[0173] According to another aspect of the embodiments of the present application, a data page processing apparatus for implementing the data page processing method is also provided. Figure 10 is a structural block diagram of an optional data page processing apparatus according to an embodiment of the present application, as shown in Figure 10 The apparatus can include:

[0174] The analysis unit 1002 is configured to analyze the first batch of redo logs to obtain a plurality of application tasks, wherein the target copy page containing the target original page in the target memory space is obtained, the target original page is a data page associated with the target operation parsed from the first batch of redo logs, and the target operation is a splitting or merging operation of the data page.

[0175] The distribution unit 1004 is connected with the analysis unit 1002, and is configured to distribute the plurality of application tasks to the plurality of worker threads, wherein each of the plurality of worker threads is configured to process part of the plurality of application tasks.

[0176] The first execution unit 1006 is connected with the distribution unit 1004, and is configured to execute the plurality of application tasks by the plurality of worker threads, wherein the application task corresponding to the target original page in the plurality of application tasks is applied to the target copy page.

[0177] The first update unit 1008 is connected with the first execution unit 1006, and is configured to, in a case where the plurality of application tasks are executed, update the data of the target copy page into the target original page by adding an exclusive lock to the target index, wherein the target index is an index where the target original page is located.

[0178] It should be noted that the analysis unit 1002 in this embodiment can be configured to perform the step S202, the distribution unit 1004 in this embodiment can be configured to perform the step S204, the first execution unit 1006 in this embodiment can be configured to perform the step S206, and the first update unit 1008 in this embodiment can be configured to perform the step S208.

[0179] By the above modules, the first batch of redo logs are parsed to obtain the plurality of application tasks, wherein the target copy page containing the target original page is in the target memory space, the target original page is a data page associated with the target operation parsed from the first batch of redo logs, and the target operation is a split or merge operation of the data page; the plurality of application tasks are distributed to the plurality of worker threads, wherein each of the plurality of worker threads is configured to process part of the plurality of application tasks; the plurality of application tasks are executed by the plurality of worker threads, wherein the application task corresponding to the target original page in the plurality of application tasks is applied to the target copy page; in a case where the plurality of application tasks are executed, the data of the target copy page is updated into the target original page by adding an exclusive lock to the target index, wherein the target index is an index where the target original page is located, thereby solving the problems of low efficiency of physical replication and low concurrency in the related art serial SMO application mode, ensuring the atomicity of SMO application, improving the efficiency of physical replication and the concurrency of the system.

[0180] As an optional embodiment, the apparatus further comprises:

[0181] The second execution unit is configured to, in the process of parsing the first batch of redo logs, in a case where a target redo log in the first batch of redo logs is parsed, execute, by the target coordination thread, an application task corresponding to the target redo log in advance, wherein the target redo log is a redo log associated with data page creation.

[0182] As an optional embodiment, the apparatus further comprises:

[0183] The application unit is configured to apply for a target available memory of a target size in a target memory space before the plurality of application tasks are dispatched to the plurality of worker threads, wherein the target size is a size of a target original page.

[0184] The copying unit is configured to copy data in the target original page into the target available memory to obtain a target copy page.

[0185] As an optional embodiment, the at least two groups of hash tables corresponding to each worker thread are used to alternately store application tasks executed by each worker thread and parsed from different batches of redo logs. Optionally, the dispatch unit 1004 comprises:

[0186] The dispatch module is configured to dispatch the plurality of application tasks to the target group of hash tables corresponding to each worker thread, wherein application tasks corresponding to a same data page in the plurality of application tasks are dispatched to a same group of hash tables, and the application tasks in the same group of hash tables are executed in a chronological order of time identifiers.

[0187] As an optional embodiment, the number of hash tables corresponding to each worker thread is two groups. Optionally, the apparatus further comprises:

[0188] The second updating unit is configured to update a hash table identifier of the target original page to a target table identifier of the target group of hash tables before the plurality of application tasks are dispatched to the plurality of worker threads.

[0189] The checking unit is configured to check the hash table identifier of the target original page after data of the target copy page is updated into the target original page by taking an exclusive lock on the target index.

[0190] The retaining unit is configured to retain the target copy page in a case where the hash table identifier of the target original page is inconsistent with the target table identifier.

[0191] The releasing unit is configured to release the target copy page in a case where the hash table identifier of the target original page is consistent with the target table identifier.

[0192] As an optional embodiment, the first execution unit 1006 comprises:

[0193] The obtaining module is configured to obtain, by the first worker thread, a target application task to be executed, wherein the first worker thread is one of the plurality of worker threads, and the target application task is one of the plurality of application tasks.

[0194] The checking module is configured to check copy page information of a target data page corresponding to a target application task, wherein the copy page information of the target data page is used to indicate whether the target data page has a copy page.

[0195] The execution module is configured to execute the target application task on the copy page of the target data page in a case where it is determined according to the copy page information of the target data page that the target data page has a copy page, wherein the target original page contains the target data page, and the target copy page contains the copy page of the target data page.

[0196] As an optional embodiment, the checking module comprises a checking submodule, and the execution module comprises an execution submodule, wherein,

[0197] The checking submodule is configured to check target field information of a target field of the target data page, wherein the target field of a data page is used to store an address pointer of a head address of a copy page of the data page, and the copy page information of the target data page is the target field information.

[0198] The execution submodule is configured to execute the target application task on the copy page pointed to by the target field information.

[0199] As an optional embodiment, the apparatus further comprises:

[0200] The setting module is configured to, after updating data of the target copy page into the target original page by exclusively locking the target index, set the copy page information of the target data page to the target copy page information in a case where the target data page is not a data page associated with a target operation parsed from a second batch of redo logs, wherein the target copy page information is used to indicate that the target data page has no copy page, and the second batch is a next batch of the first batch.

[0201] As an optional embodiment, the target original page contains a plurality of original pages, and the target index contains a plurality of indexes. Optionally, the first updating unit 1008 comprises:

[0202] The traversal module is configured to traverse the target mapping information by the second working thread to obtain a current index to be processed in the plurality of indexes, wherein the second working thread is a working thread that completes application last in the plurality of working threads, and the target mapping information is used to indicate a mapping relationship between each index in the plurality of indexes and an original page belonging to each index in the plurality of original pages.

[0203] The locking module is configured to exclusively lock the current index by the second working thread.

[0204] The updating module is configured to update data of a copy page of a current original page into the current original page by the second working thread, wherein the current original page is an original page belonging to the current index in the plurality of original pages.

[0205] a releasing module, configured to release the exclusive lock of the current index through the second worker thread.

[0206] As an optional embodiment, the traversing module comprises:

[0207] a traversing submodule, configured to traverse the target tuple set through the second worker thread to obtain the current index to be processed in the plurality of indexes, wherein the target mapping information is the target tuple set, one target tuple in the target tuple set is a tuple with an index identifier of one index in the plurality of indexes as a key and a target set container as a value, and the target set container contains original page identifiers of original pages belonging to the one index in the plurality of original pages.

[0208] As an optional embodiment, the apparatus further comprises:

[0209] a receiving unit, configured to receive a target request message sent by a target client before updating data of the target replica page into the target original page by adding an exclusive lock to the target index, wherein the target request message is used to request reading of target data in the target original page;

[0210] a sending unit, configured to send the target data in the target original page to the target client in response to the target request message.

[0211] It should be noted that the above modules and the examples and application scenarios realized by the corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules as part of the apparatus can run in the hardware environment as shown in Figure 1 , and can be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0212] According to another aspect of the embodiments of the present application, an electronic device for implementing the above-mentioned data page processing method is also provided, which can be a server, a terminal, or a combination thereof.

[0213] Figure 11 is a structural block diagram of an optional electronic device according to an embodiment of the present application, as shown in Figure 11 , comprising a processor 1102, a communication interface 1104, a memory 1106 and a communication bus 1108, wherein the processor 1102, the communication interface 1104 and the memory 1106 complete communication with each other through the communication bus 1108, wherein

[0214] the memory 1106 is configured to store a computer program;

[0215] the processor 1102 is configured to execute the computer program stored in the memory 1106 to implement the following steps:

[0216] The redo log of the first batch is parsed to obtain a plurality of application tasks, wherein the target copy page containing the target original page in the target memory space is obtained, the target original page is a data page associated with the target operation parsed from the redo log of the first batch, and the target operation is a split or merge operation of the data page;

[0217] The plurality of application tasks are distributed to a plurality of worker threads, wherein each worker thread in the plurality of worker threads is used to process part of the application tasks in the plurality of application tasks;

[0218] The plurality of application tasks are executed by the plurality of worker threads, wherein the application task corresponding to the target original page in the plurality of application tasks is applied to the target copy page;

[0219] In the case where the plurality of application tasks are executed, the data of the target copy page is updated into the target original page by taking an exclusive lock on the target index, wherein the target index is the index where the target original page is located.

[0220] Optionally, in the embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For the convenience of representation, Figure 11 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or only one type of bus. The communication interface is used for communication between the electronic device and other devices.

[0221] The memory can include a RAM and can also include a non-volatile memory, for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0222] As an example, the memory 1106 can include, but is not limited to, the parsing unit 1002, the distribution unit 1004, the first execution unit 1006, and the first update unit 1008 in the processing device of the data page. In addition, other module units in the processing device of the data page can also be included, but not limited to, which will not be described in detail in this example.

[0223] The processor can be a general processor, which can include but is not limited to a CPU (Central Processing Unit), a NP (Network Processor), etc. The processor can also be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0224] Optionally, the specific examples in the embodiment can refer to the examples described in the above embodiments, and the embodiment will not be described here.

[0225] Those skilled in the art can understand that the above-mentioned embodiments can be implemented by hardware, software or a combination of hardware and software. Figure 11 The structure shown is only a schematic, and the device implementing the processing method of the above data page can be a terminal device, which can be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, etc. Figure 11 This does not limit the structure of the above electronic device. For example, the electronic device can further include more or less components (such as a network interface, a display device, etc.) than those shown in the above embodiment, or have a different configuration from that shown in the above embodiment. Figure 11 Figure 11 The structure shown is only a schematic, and the device implementing the processing method of the above data page can be a terminal device, which can be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, etc.

[0226] Those skilled in the art can understand that all or part of the steps of the various methods of the above embodiments can be completed by programs instructing the related hardware of the terminal device, and the programs can be stored in a computer readable storage medium, which can include a flash disk, a ROM, a RAM, a magnetic disk or an optical disk, etc.

[0227] According to another aspect of the embodiment of the present application, a storage medium is also provided. Optionally, in the embodiment, the storage medium can be used to execute the program code of the processing method of any one of the above data pages in the embodiment of the present application.

[0228] Optionally, in the embodiment, the storage medium can be located on at least one of the network devices in the network shown in the above embodiment.

[0229] Optionally, in the embodiment, the storage medium is configured to store program code for executing the following steps:

[0230] ​The redo log of the first batch is parsed to obtain a plurality of application tasks, wherein the target copy page containing the target original page in the target memory space is obtained, the target original page is a data page associated with the target operation parsed from the redo log of the first batch, and the target operation is a split or merge operation of the data page;

[0231] The plurality of application tasks are distributed to a plurality of worker threads, wherein each worker thread in the plurality of worker threads is used to process part of the plurality of application tasks;

[0232] The plurality of application tasks are executed by the plurality of worker threads, wherein the application task corresponding to the target original page in the plurality of application tasks is applied to the target copy page;

[0233] In a case where the plurality of application tasks are executed, data of the target copy page is updated into the target original page by taking an exclusive lock on the target index, wherein the target index is an index where the target original page is located.

[0234] Optionally, specific examples in the embodiment can refer to the examples described in the above-described embodiments, and the embodiment will not be described here.

[0235] Optionally, in the embodiment, the storage medium described above can include, but is not limited to, a U disk, a ROM, a RAM, a mobile hard disk, a magnetic disk or an optical disk and various storage program codes.

[0236] The serial numbers of the embodiments of the present application described above are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0237] The integrated units in the above-described embodiments, if realized in the form of software function units and sold or used as independent products, can be stored in the above-described computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application.

[0238] In the above-described embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0239] In several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other manners. Of course, the described apparatus embodiments are merely schematic, and the division of units is merely logical function division, and there can be other division manners in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, units or modules, and can be in electrical, mechanical or other forms.

[0240] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the scheme provided in the embodiments.

[0241] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0242] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method for processing data pages, characterized in that, include: The first batch of redo logs is parsed to obtain multiple application tasks, among which, the target memory space contains a target copy page of the target original page. The target original page is a data page associated with the target operation parsed from the first batch of redo logs. The target operation is a data page splitting or merging operation. The plurality of application tasks are distributed to a plurality of worker threads, wherein each of the plurality of worker threads is used to process a portion of the plurality of application tasks. The multiple application tasks are executed by the multiple worker threads, wherein the application task corresponding to the target original page is applied to the target copy page. Upon completion of the multiple application tasks, the data of the target copy page is updated to the target original page by acquiring an exclusive lock on the target index, wherein the target index is the index where the target original page is located.

2. The method according to claim 1, characterized in that, The method further includes the following steps during the parsing of the first batch of redo logs: If the target redo log is parsed in the first batch of redo logs, the application task corresponding to the target redo log is pre-executed by the target coordination thread, wherein the target redo log is the redo log associated with the creation of the data page.

3. The method according to claim 1, characterized in that, Before distributing the plurality of application tasks to the plurality of worker threads, the method further includes: Allocate a target available memory of a target size in the target memory space, wherein the target size is the size of the target original page; The data in the original target page is copied into the available memory of the target to obtain the target copy page.

4. The method according to claim 1, characterized in that, At least two sets of hash tables corresponding to each worker thread take turns storing the application tasks parsed from different batches of redo logs and executed by each worker thread; Distributing the multiple application tasks to the multiple worker threads includes: The multiple application tasks are distributed to target group hash tables corresponding to each worker thread, wherein application tasks corresponding to the same data page are distributed to the same group hash table, and application tasks within the same group hash table are executed in the order of their time identifiers.

5. The method according to claim 4, characterized in that, The number of hash tables corresponding to each worker thread is two; Before the multiple application tasks are distributed to the multiple worker threads, the method further includes: updating the hash table identifier of the target original page to the target table identifier of the target group hash table; After updating the data of the target replica page to the target original page by acquiring an exclusive lock on the target index, the method further includes: checking the hash table identifier of the target original page; retaining the target replica page if the hash table identifier of the target original page is inconsistent with the target table identifier; and releasing the target replica page if the hash table identifier of the target original page is consistent with the target table identifier.

6. The method according to claim 1, characterized in that, Executing the multiple application tasks through the multiple worker threads includes: The target application task to be executed is obtained through a first worker thread, wherein the first worker thread is one of the plurality of worker threads, and the target application task is one of the plurality of application tasks; Verify the copy page information of the target data page corresponding to the target application task, wherein the copy page information of the target data page is used to indicate whether the target data page has a copy page; If it is determined that the target data page has a copy page based on the copy page information of the target data page, the target application task is executed on the copy page of the target data page, wherein the target original page contains the target data page, and the target copy page contains a copy page of the target data page.

7. The method according to claim 6, characterized in that, Verifying the copy page information of the target data page corresponding to the target application task includes: verifying the target field information of the target field of the target data page, wherein the target field of a data page is used to store the address pointer of the starting address of the copy page of this data page, and the copy page information of the target data page is the target field information; Executing the target application task on a copy page of the target data page includes: executing the target application task on the copy page pointed to by the target field information.

8. The method according to claim 6, characterized in that, After updating the data of the target replica page to the target original page by acquiring an exclusive lock on the target index, the method further includes: If the target data page is not associated with the target operation parsed from the redo logs of the second batch, the replica page information of the target data page is set as the target replica page information, wherein the target replica page information is used to indicate that the target data page has no replica page, and wherein the second batch is the batch following the first batch.

9. The method according to claim 1, characterized in that, The target original page contains multiple original pages, and the target index contains multiple indexes; updating the data of the target replica page to the target original page by adding an exclusive lock to the target index includes: The target mapping information is traversed by the second working thread to obtain the current index to be processed among the multiple indexes. The second working thread is the last working thread to complete the application among the multiple working threads. The target mapping information is used to indicate the mapping relationship between each index among the multiple indexes and the original page belonging to each index among the multiple original pages. The second worker thread acquires an exclusive lock on the current index; The second worker thread updates the data of the copy page of the current original page to the current original page, wherein the current original page is the original page belonging to the current index among the plurality of original pages; Release the exclusive lock on the current index through the second working line.

10. The method according to claim 9, characterized in that, By traversing the target mapping information through the second worker thread, the current index to be processed among the multiple indices is obtained, including: The second worker thread traverses the target tuple set to obtain the current index to be processed among the multiple indexes. The target mapping information is the target tuple set. A target tuple in the target tuple set is a tuple with the index identifier of one of the multiple indexes as the key and the target set container as the value. The target set container contains the original page identifier of the original page belonging to the one of the multiple original pages.

11. The method according to any one of claims 1 to 10, characterized in that, Before updating the data of the target replica page to the target original page by acquiring an exclusive lock on the target index, the method further includes: Receive a target request message sent by the target client, wherein the target request message is used to request to read target data in the target original page; In response to the target request message, the target data in the target original page is sent to the target client.

12. A data page processing apparatus, characterized in that, include: The parsing unit is used to parse the first batch of redo logs to obtain multiple application tasks, wherein the target memory space contains a target copy page of the target original page, the target original page is a data page associated with the target operation parsed from the first batch of redo logs, and the target operation is a data page splitting or merging operation; A distribution unit is used to distribute the plurality of application tasks to a plurality of worker threads, wherein each of the plurality of worker threads is used to process a portion of the plurality of application tasks. The first execution unit is configured to execute the plurality of application tasks through the plurality of worker threads, wherein the application task corresponding to the target original page among the plurality of application tasks is applied to the target copy page; The first update unit is configured to update the data of the target copy page to the target original page by acquiring an exclusive lock on the target index when the multiple application tasks have been completed, wherein the target index is the index where the target original page is located.

13. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, the communication interface, and the memory communicate with each other via the communication bus, characterized in that... The memory is used to store computer programs; The processor is configured to perform the method of any one of claims 1 to 11 by running the computer program stored in the memory.

14. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method of any one of claims 1 to 11 when it is run.

Images