Data processing method and device

By directly copying the memory page to the destination using postcopy method in the in-memory database, the problem of slow speed and memory overhead in the full data synchronization process of the in-memory database is solved, efficient data synchronization is achieved, and serialization and deserialization are reduced. Improves synchronization speed and efficiency.

CN113297265BActive Publication Date: 2024-12-03ALIBABA CLOUD COMPUTING CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110197071.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-02-22
Publication Date
2024-12-03
Estimated Expiration
2041-02-22

AI Technical Summary

Technical Problem

In the prior art, there are problems of slow speed and memory overhead in the process of full data synchronization of memory databases. This is mainly because serialization and deserialization take up a lot of time, resulting in slow data synchronization speed, high memory overhead, and high performance networks. It is difficult to effectively solve the problem.

Method used

By directly forwarding the request to the second database when the first database receives the data processing request and processing the request from the beginning in the second database, the request log accumulation and memory overhead during the synchronization process is reduced, and serialization and deserialization are avoided. , directly copy memory pages to the destination using postcopy method, and accelerate data transmission using network transmission technologies such as RDMA.

Benefits of technology

It speeds up the speed of full data synchronization, reduces the request log accumulation and memory overhead on the first database during synchronization, supports multi-threaded concurrent replication, avoids serialization and deserialization, and improves the efficiency of data synchronization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113297265B_ABST
    Figure CN113297265B_ABST
Patent Text Reader

Abstract

The embodiments of the present specification provide a data processing method and device, wherein the data processing method includes receiving a data processing request forwarded by a first database, and parsing the data processing request; when it is determined based on the parsing result that there is no corresponding memory page in the second database, sending a memory page write request to the first database; receiving the memory page obtained from a data snapshot by the first database in response to the memory page write request, wherein the data snapshot is generated by the first database based on the data processing request; writing the memory page to the second database, and performing data processing on the memory page of the second database based on the data processing request.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and more particularly to a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, and a computer-readable storage medium. Background Art

[0002] At present, in order to ensure the integrity of the data in the memory database (taking Redis as an example, where the full name of Redis is RemoteDictionary Server and the full name of Redis in Chinese is Remote Dictionary Service) and prevent data loss due to illegal power outages, virus infections, etc., the data in the memory database will be backed up and stored. In the prior art, the memory database generally implements full backup of the data in the memory database by serialization, but in this way, serialization and deserialization will consume a lot of time in the process of data synchronization, and because the source end needs to keep recording the log of new requests during the serialization process, the memory overhead is very large. On large Redis instances, the full data synchronization often fails because the size of the new request log exceeds the limit.

[0003] Therefore, there is an urgent need to provide a data processing method that can accelerate the data synchronization process and reduce the memory overhead during the data synchronization process. Summary of the invention

[0004] In view of this, the present specification provides a data processing method. One or more embodiments of the present specification also relate to a data processing device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a data processing method is provided, including:

[0006] receiving a data processing request forwarded by the first database, and parsing the data processing request;

[0007] If it is determined based on the analysis result that the corresponding memory page does not exist in the second database, sending a memory page write request to the first database;

[0008] receiving the memory page obtained by the first database from the data snapshot of the first database in response to the memory page write request;

[0009] The memory page is written into the second database, and data processing is performed on the memory page of the second database based on the data processing request.

[0010] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including:

[0011] a parsing module, configured to receive a data processing request forwarded by the first database, and parse the data processing request;

[0012] a page missing trigger module, configured to send a memory page write request to the first database when it is determined based on the parsing result that the corresponding memory page does not exist in the second database;

[0013] a missing page request module, configured to receive the memory page obtained by the first database from the data snapshot of the first database in response to the memory page write request;

[0014] The missing page writing module is configured to write the memory page into the second database and perform data processing on the memory page of the second database based on the data processing request.

[0015] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0016] Memory and processor;

[0017] The memory is used to store computer executable instructions, which can implement the steps of the data processing method when executed by the processor.

[0018] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the data processing method are implemented.

[0019] An embodiment of the present specification implements a data processing method and device, wherein the data processing method includes receiving a data processing request forwarded by a first database, and parsing the data processing request; sending a memory page write request to the first database when it is determined based on the parsing result that the corresponding memory page does not exist in the second database; receiving the memory page obtained from the data snapshot in response to the memory page write request by the first database, wherein the data snapshot is generated by the first database based on the data processing request; writing the memory page to the second database, and performing data processing on the memory page of the second database based on the data processing request. Specifically, when the first database receives the data processing request, the method directly forwards the data processing request to the second database, so that the second database can process the data processing request from the beginning, reducing the request log accumulation and memory overhead on the first database during the synchronization process, and directly copying the memory page to the destination through the first database avoids serialization and deserialization, thereby speeding up the full data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a data processing method provided by an embodiment of this specification;

[0021] Figure 2 It is a schematic diagram of the structure of a first database and a second database in a data processing method provided by an embodiment of this specification;

[0022] Figure 3 is a processing flow chart of a data processing method provided by an embodiment of this specification;

[0023] Figure 4 is a structural schematic diagram of a data processing device provided by an embodiment of this specification;

[0024] Figure 5 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0025] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0026] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] First, the terms involved in one or more embodiments of this specification are explained.

[0029] Postcopy: refers to migrating the process first, and copying the relevant data of the process after migration. Postcopy is based on the userfault mechanism, and can obtain page fault information through file descriptors in user space. Among them, page fault: refers to when it cannot be found in the page table, the system will generate a page fault exception, enter the kernel space to allocate physical memory, update the process page table, and finally return to the user space to resume the operation of the process. This is a secondary page fault exception.

[0030] In-memory database: A database that stores data in memory and operates directly on it.

[0031] Memory page: The operating system manages memory in units of memory pages. The size of the memory page affects system performance.

[0032] RDB: It is to save data on disk in the form of snapshots; the RDB file is a snapshot of the data stored in Redis memory at a certain moment.

[0033] SAP HANA: is an in-memory computing platform that supports both on-premise and cloud deployment modes. It provides high-performance data query capabilities, allowing users to directly query and analyze large amounts of real-time project data without the need to model or aggregate the project data.

[0034] SQL: Structured Query Language, abbreviated as SQL, is a special-purpose programming language. It is a database query and programming language used to access data and query, update and manage relational database systems.

[0035] At present, the full data synchronization solution of the in-memory database (taking Redis as an example) can be basically divided into four steps: 1. The source serializes the full data and records logs for new requests; 2. Sends the serialized data; 3. The destination deserializes and loads it into the database; 4. Replays the logs accumulated by the source. The advantage of this is that the size of the transmitted data can be reduced through compression during the serialization process, and cross-version compatibility can be provided.

[0036] However, in recent years, network speeds have developed rapidly, and 25Gbps network bandwidths are already common in data centers, and 100Gbps networks are also gradually being implemented. In the above full data synchronization solution for in-memory databases, serialization and deserialization have become the main time overhead in the data synchronization process, which has caused the full data synchronization mechanism of in-memory databases in the existing technology to have problems such as slow speed and large memory overhead during synchronization.

[0037] Based on the above technical problems, the following two solutions can be provided:

[0038] Taking the full data synchronization solution of the community version of Redis's in-memory database as an example, the entire process is divided into four steps: 1. The source side forks and serializes the full data into RDB in the child process, and accumulates new requests in the output buffer; 2. Send RDB; 3. The destination side deserializes RDB and loads it into the database; 4. Replays the logs accumulated by the source side. Steps 1 to 3 can be performed at the same time. However, in a high-performance network, serialization and deserialization take up a lot of time. During steps 1 to 3, the requests need to be accumulated in the output buffer all the time, which has a large memory overhead; and they must be executed after steps 1 to 3 are completed, which slows down the data synchronization speed.

[0039] In addition, in the full data synchronization solution for multi-threaded databases such as SAP HANA, the serialization and deserialization processes support multi-threading, which can improve the speed of serialization and deserialization. At the same time, the request log is stored on the disk, which has less overhead than memory. However, the serialization / deserialization and loading of multi-threads are complex to implement. In particular, the development work of changing a single-threaded program like Redis to a multi-threaded version is very large. The performance of request log writing is affected by disk performance.

[0040] In order to better solve the above technical problems, a data processing method is provided in this specification. This specification also involves a data processing device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0041] See also Figure 1 , Figure 1 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0042] Step 102: Receive a data processing request forwarded by the first database, and parse the data processing request.

[0043] In practical applications, the first database can be understood as the memory database of the source end in the above embodiment, and the second database can be understood as the memory database of the destination end in the above embodiment. The data processing method takes the destination end as the execution subject, and specifically implements the full data synchronization of the source end and the destination end based on the memory page postcopy method, and reduces the memory overhead and increases the data synchronization speed during the full data synchronization process.

[0044] Among them, the data processing request includes but is not limited to a data processing request of a structured query language, that is, a data processing request of an SQL statement; in specific implementation, the data processing request may also be other query logic, as long as the memory database can perform data read and write operations on the data area of ​​the memory database when processing the data processing request.

[0045] In specific implementation, receiving the data processing request forwarded by the first database can be understood as the memory database at the source end receiving the data processing request sent by the user, and forwarding the data processing request to the destination end while processing the data processing request; wherein the data processing request includes but is not limited to a data write request, a data read request, or a data update request, etc., which can be set according to the actual application and is not limited here.

[0046] Specifically, the receiving the data processing request forwarded by the first database and parsing the data processing request includes:

[0047] A data processing request in a structured query language forwarded by a first database is received, and the data processing request is parsed to determine the data to be processed in the data processing request and index information corresponding to the data to be processed.

[0048] The data processing request in the structured query language can be understood as an SQL statement. For example, the data processing request in the structured query language is: update scores set score=100where id=1.

[0049] In actual applications, after receiving the data processing request in the structured query language forwarded by the first database, the destination end parses the data processing request to determine the data to be processed in the data processing request and the index information corresponding to the data to be processed.

[0050] Still taking the data processing request of structured query language: update scores set score=100whereid=1 as an example, the data processing request is parsed. By parsing the data processing request, it can be determined that the processing of the data processing request is to modify the score value of the record with id=1 in the score table to 100. Then it can be determined that the data to be processed in the data processing request is the score value of id=1 in the score table, and the index information corresponding to the data to be processed is the index of the id in the score table.

[0051] In the embodiment of the present specification, after receiving the data processing request, the source end directly forwards the data processing request to the destination end while processing the data processing request. Through this postcopy method, the destination end can process the data processing request from the very beginning, reducing the request log accumulation and memory overhead on the source end during the data synchronization process; and by parsing the data processing request, the subsequent destination end access to the memory page is achieved based on the parsing result.

[0052] Step 104: When it is determined based on the analysis result that the corresponding memory page does not exist in the second database, a memory page write request is sent to the first database.

[0053] In a specific implementation, when it is determined based on the analysis result that the corresponding memory page does not exist in the second database, sending a memory page write request to the first database includes:

[0054] determining a memory address from a first data area of ​​the second database based on the index information;

[0055] When it is determined based on the memory address that there is no corresponding memory page in the second data area of ​​the second database, a memory page write request is sent to the first database.

[0056] In actual applications, in order to facilitate process migration, the data areas of the source end's memory database (i.e., the first database) and the destination end's memory database (i.e., the second database) are separately segmented and processed, that is, the memory data area is segmented into other memory areas and data areas. All data is stored in the data area, and other memory areas are used to store some parameters, such as the IP address of the current host, the port of the current host, account number, etc.; and only the external area is allowed to reference the data area, and the data area is not allowed to reference the external area, where the external area can be understood as other memory areas.

[0057] See also Figure 2 , Figure 2 A schematic diagram of the structure of a first database and a second database in a data processing method provided according to an embodiment of the present specification is shown.

[0058] Figure 2 The structure of the memory database in the memory database can represent the structure of the first database at the source end and the structure of the second database at the destination end. Specifically, the memory database includes other memory areas and data areas, wherein the other memory areas of the memory database represent the first data areas of the first database and the second database, and the data areas of the memory database represent the second data areas of the first database and the second database.

[0059] The first data area of ​​the second database can be understood as other memory areas, and the second data area can be understood as a data area.

[0060] Specifically, after parsing the data processing request to obtain index information corresponding to the data to be processed, a memory address is determined from other memory areas of the second database based on the index information. When it is determined based on the memory address that there is no corresponding memory page in the data area of ​​the second database, a page fault interrupt is triggered and the execution of the second database is suspended, and a memory page write request is sent to the first database.

[0061] In actual applications, the second database can be regarded as a data storage component of the destination end, and the destination end also includes a CPU, an operating system, and a page fault interrupt processing component. Specifically, after determining the memory address from the second data area of ​​the second database of the destination end based on the index information, the second database of the destination end accesses the memory address. When the CPU of the destination end finds that there is no corresponding memory page for the memory address, a page fault interrupt will be triggered. The operating system of the destination end will capture the page fault interrupt in real time, and then suspend the execution process of the second database, and at the same time notify the page fault interrupt processing component through userfaultd. The notification carries the identifier of the memory page corresponding to the memory address corresponding to the page fault interrupt. After the page fault interrupt processing component of the destination end receives the page fault interrupt notification carrying the identifier, the page fault interrupt processing component of the destination end will send a memory page write request to the first database requesting the memory page corresponding to the memory address.

[0062] In an embodiment of the present specification, when the second database accesses the memory address of a page in the data area based on the data processing request forwarded from the first database, and the CPU finds that there is no corresponding memory page for the memory address, a page fault interrupt will be triggered to obtain the missing page from the first database through the operating system and the page fault interrupt processing component to ensure the correct execution of the data processing request and avoid data asynchrony between the first database and the second database.

[0063] In addition, when the second database accesses the memory address of a page in the data area based on the data processing request forwarded by the first database, if there is a memory page corresponding to the memory address, data processing can be directly performed on the corresponding memory page, thereby realizing synchronous data processing of the first database and the second database. The specific implementation method is as follows:

[0064] After receiving the data processing request forwarded by the first database and parsing the data processing request, the method further includes:

[0065] When it is determined based on the analysis result that the corresponding memory page of the second database exists, data processing is performed on the memory page of the second database based on the data processing request.

[0066] Specifically, when it is determined based on the parsing result that the corresponding memory page of the second database exists, data processing is performed on the memory page of the second database based on the data processing request, including:

[0067] determining a memory address from a first data area of ​​the second database based on the index information;

[0068] When it is determined based on the memory address that a corresponding memory page exists in the second data area of ​​the second database, data processing is performed on the memory page of the second database based on the data processing request.

[0069] Using the above example, when it is determined based on the memory address that a corresponding memory page exists in the data area of ​​the second database, the corresponding memory page is found, and the score value of the record with id=1 in the scores table is modified to 100.

[0070] Step 106: Receive the memory page obtained by the first database from the data snapshot of the first database in response to the memory page write request.

[0071] The data snapshot of the first database is generated when full data synchronization of the first database and the second database is started.

[0072] Specifically, after the page fault interrupt processing component of the destination end sends a memory page write request to the source end, the first database of the source end responds to the memory page write request, obtains the memory page corresponding to the missing page requested in the memory page write request from the data snapshot, and then sends the memory page to the destination end; wherein the missing page requested in the memory page write request is the above-mentioned memory page corresponding to the memory address.

[0073] In actual applications, after receiving a data processing request, the first database at the source end generates a data snapshot based on the second data area (i.e., the data area) of the first database. When receiving a memory page write request sent by the destination end based on the data processing request, the corresponding memory page is obtained from the data snapshot and returned to the destination end.

[0074] Specifically, the receiving the memory page obtained from the data snapshot by the first database in response to the memory page write request includes:

[0075] The memory page is received by the first database from a data snapshot of a second data area of ​​the first database based on a memory page identifier carried in the memory page write request.

[0076] In actual applications, in order to facilitate the first database to accurately obtain the memory page of the second data area of ​​the second database, when the destination end sends a memory page write request to the first database of the source end, the memory page write request carries a memory page identifier, so that the first database can quickly and accurately obtain the corresponding memory page from the data snapshot based on the memory page identifier and return it to the destination end.

[0077] In addition, in order to speed up the transmission speed of memory pages and reduce CPU overhead, the source end can compress the memory pages before transmission, and use network transmission technologies such as RDMA to achieve the above technical effects. The specific implementation methods are as follows:

[0078] The sending a memory page write request to the first database includes:

[0079] A memory page write request is sent to the first database based on a preset network transmission method.

[0080] Correspondingly, the receiving the memory page obtained by the first database from the data snapshot in response to the memory page write request includes:

[0081] The first database responds to the memory page write request based on the preset network transmission mode, and obtains and compresses the memory page from the data snapshot.

[0082] The preset network transmission mode includes but is not limited to the RDMA network transmission mode.

[0083] In the embodiments of the present specification, after receiving a memory page write request sent by the destination, the source end obtains the corresponding memory page from the data snapshot, compresses the memory page and then transmits it to the destination end, thereby exchanging computing power for a reduction in the transmission volume and improving the transmission effect. When data is transmitted between the source and destination ends, network transmission technologies such as RDMA can be used to accelerate data transmission, further improve the transmission rate, and reduce CPU overhead.

[0084] In another embodiment of the present specification, before receiving the data processing request forwarded by the first database and parsing the data processing request, the method further includes:

[0085] An initial memory page that meets a preset condition and is sent by the first database is received, and the initial memory page is written into a second data area of ​​the second database.

[0086] Among them, the preset conditions can be set according to the actual application, and this application does not make any limitation on this. Based on different actual applications, the key memory pages are also different.

[0087] Specifically, before receiving the data processing request forwarded by the first database, it can be understood that before the first database and the second database realize data synchronization for the first time, the second database at the destination end receives the initial memory page (for example, a critical memory page) sent by the first data at the source end and determined according to actual needs, and writes the critical memory page into the second data area of ​​the second database for backup.

[0088] In actual applications, if data synchronization between the first database and the second database is implemented for the first time, then before starting postcopy, a round of precopy can be performed to copy some key memory pages of the second data area of ​​the first database on the source side to the second data area of ​​the second database on the destination side in advance, so as to avoid a large number of page missing in the second data area of ​​the second database when the destination side is just started.

[0089] Step 108: Write the memory page into the second database, and perform data processing on the memory page of the second database based on the data processing request.

[0090] In a specific implementation, writing the memory page into the second database, and performing data processing on the memory page of the second database based on the data processing request includes:

[0091] The memory page is written into a second data area of ​​the second database, and based on the data processing request, the to-be-processed data is processed in the memory page of the second database.

[0092] In actual application, the page fault interrupt processing component of the destination end receives the memory page sent by the first database of the source end, and then writes the memory page into the second data area of ​​the second database. Then the page fault interrupt processing component of the destination end notifies the operating system through userfaultd to resume the execution of the second database, and processes the data to be processed in the memory page of the second database based on the data processing request.

[0093] Using the above example, after the memory page is written into the second data area of ​​the second database, the score value of the record with id=1 in the scores table is modified to 100 in the memory page.

[0094] In the embodiments of the present specification, serialization and deserialization are avoided by directly copying memory pages from the first database on the source end to the second database on the destination end, thereby accelerating the speed of full data synchronization between the first database and the second database, and through postcopy, the second database on the destination end can process requests from the very beginning, thereby reducing the request log accumulation and memory overhead on the first database on the source end during the data synchronization process; in addition, data replication is performed based on memory pages, avoiding multi-threaded transformation of the entire memory database, and supporting multi-threaded concurrent replication, thereby further accelerating the speed of full data synchronization between the first database and the second database.

[0095] In another embodiment of this specification, the method further includes:

[0096] Acquire a memory page in the second database, match the memory page with a memory page in a preset memory page table, and determine an unmatched memory page in the preset memory page table as a target memory page;

[0097] In a case where the target memory page is not the memory page corresponding to the memory page write request, sending a memory page acquisition request to the first database, wherein the memory page acquisition request carries a page identifier of the target memory page;

[0098] A corresponding target memory page is acquired from the first database based on the page identifier, and the target memory page is written into the second database.

[0099] Among them, the memory pages in the preset memory table are memory pages of the second data area of ​​the first database, that is, it is necessary to determine whether there are uncopied pages in the second data area of ​​the second database through the memory pages in the preset memory table to ensure full data synchronization between the first database and the second database.

[0100] In actual application, the background migration component of the destination end will obtain the memory pages in the second data area of ​​the second database in real time, and then match the obtained memory pages with the memory pages in the preset memory table, and determine the memory pages that are not matched in the preset memory table as the target memory pages, that is, the target memory pages can be understood as memory pages that do not exist in the second data area of ​​the second database.

[0101] Then, when the target memory page is not the memory page corresponding to the memory page write request, the background migration component of the destination end sends a memory page acquisition request to the first database, receives the target memory page obtained from the data snapshot by the first database based on the page identifier carried in the memory page acquisition request, and writes the received target memory page into the second data area of ​​the second database.

[0102] In actual applications, the background migration component of the destination end will write the unreplicated memory page in the second data area of ​​the second database in real time. If the unreplicated memory page is the memory page corresponding to the memory page write request, it can be determined that the page fault interrupt processing component will also request the memory page from the first database. In order to avoid repeated writing of the memory page and causing confusion, when it is determined that the unreplicated memory page is the memory page corresponding to the memory page write request, the acquisition of the memory page will be abandoned to avoid the above-mentioned problem.

[0103] In addition, the background migration component of the destination side can use multiple threads to concurrently obtain the memory pages in the first database that have not been copied to the second database, and the page migration component of the source side can also use multiple threads to respond to the memory page acquisition request sent by the background migration component of the destination side. Therefore, there can be multiple target memory pages. In the case where there are multiple target memory pages, the specific implementation method is as follows:

[0104] The target memory page is at least one;

[0105] Accordingly, when the target memory page is not the memory page corresponding to the memory page write request, sending a memory page acquisition request to the first database includes:

[0106] determining whether each target memory page in the at least one target memory page is a memory page corresponding to the memory page write request,

[0107] If yes, then end.

[0108] If not, at least one memory page acquisition request is sent to the first database.

[0109] Specifically, if there is one target memory page, it can be directly determined whether the target memory page is the memory page corresponding to the memory page write request. If so, the result page is obtained. If not, a memory page acquisition request is sent to the page migration component of the source end to obtain the memory page from the data snapshot of the second data area of ​​the first database.

[0110] If there are multiple target memory pages, the above judgment needs to be performed on each target memory page to ensure the accuracy of memory page acquisition. In addition, when there are multiple target memory pages and none of them correspond to the memory page write request, the background migration component of the destination end can use multiple threads to concurrently acquire the target memory pages in the first database that have not been copied to the second database, and the page migration component of the source end can also use multiple threads to respond to the memory page acquisition request sent by the background migration component of the destination end, so as to improve the full data synchronization speed between the first database and the second database.

[0111] In a specific implementation, obtaining a corresponding target memory page from the first database based on the page identifier and writing the target memory page into the second database includes:

[0112] At least one corresponding target memory page is acquired from the first database based on the at least one page identifier, and the at least one target memory page is written into the second database.

[0113] Then, after the background migration component of the destination end sends at least one memory page acquisition request through the page migration component of the source end, the background migration component of the destination end can receive each memory page corresponding to each page identifier obtained from the data snapshot of the second data area of ​​the first database by the page migration component of the source end; and then write all the received memory pages into the second data area of ​​the second database to achieve synchronization of the full data of the first database and the second database.

[0114] The data processing method provided in the embodiment of this specification directly forwards the data processing request to the second database when the first database receives the data processing request, so that the second database can process the data processing request from the beginning, reducing the request log accumulation and memory overhead on the first database during the synchronization process, and directly copying the memory page to the destination end through the first database, avoiding serialization and deserialization, and speeding up the full data synchronization.

[0115] The following combination Figure 3 , taking the application of the data processing method provided in this specification in full synchronization of memory data as an example, the data processing method is further described. Figure 3 A processing flow chart of a data processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0116] Step 302: The memory database at the source end receives a data write request and forwards the data write request to the memory database at the destination end.

[0117] Specifically, the memory database at the source end can be understood as the first database in the above embodiment, and the memory database at the destination end can be understood as the second database in the above embodiment.

[0118] In specific implementation, when the full data synchronization task of the source end's memory database and the destination end's memory database is executed, the source end starts the source end's memory database, page migration component, etc., and the destination end starts the destination end's memory database, page fault interrupt processing component, and background migration component, etc. Among them, the page fault interrupt component can capture its own page fault interrupt through userfaultfd and other technologies when used specifically, or other technologies that can capture page fault interrupts can be used instead of userfaultfd. This manual does not impose any restrictions on this.

[0119] Step 304: The destination end parses the data write request, and based on the parsing result, determines the memory page address corresponding to the data write request from other memory areas of the destination end's memory database. The destination end's memory database accesses the corresponding memory page from the data area of ​​the destination end's memory database based on the memory page address.

[0120] Step 306: The CPU of the destination end determines, based on the memory page address, that a corresponding memory page does not exist in the data area of ​​the memory database of the destination end, and triggers a page fault interrupt.

[0121] Step 308: The operating system of the destination end captures the page fault interrupt, suspends the execution of the memory database of the destination end, and notifies the page fault interrupt processing component of the destination end through the userfaultd technology, wherein the notification carries the identifier of the missing memory page corresponding to the page fault interrupt.

[0122] Step 310: After receiving the notification, the page fault interrupt processing component at the destination end sends a memory page write request for the missing memory page to the page migration component at the source end, requesting the missing memory page.

[0123] Step 312: The page migration component at the source end obtains the memory page from the data area copy of the data area of ​​the memory database at the source end through the identifier of the missing memory page carried in the memory page write request.

[0124] Specifically, a data area copy of the data area of ​​the source end's memory database is generated when full data synchronization between the source end and the destination end is started.

[0125] Step 314: the page migration component at the source end sends the acquired memory page to the page fault interrupt processing component at the destination end.

[0126] Step 316: The page fault interrupt processing component of the destination end writes the received memory page into the data area of ​​the memory database of the destination end.

[0127] Step 318: After the page fault interrupt processing component of the destination end writes the received memory page into the data area of ​​the memory database of the destination end, it notifies the operating system of the destination end to resume the execution of the memory database of the destination end.

[0128] Specifically, the above steps 304 to 308 are repeatedly executed until the synchronization of the full data of the memory database at the source end and the memory database at the destination end is completed.

[0129] In actual applications, the target end's memory database can also be regarded as a data processing component of the target end. When the target end starts the memory database, the target end's page fault interrupt processing component will also be started. Specifically, when the memory page corresponding to a certain memory address read and written by the target end's memory database is not copied to the data area of ​​the target end's memory database, the target end's CPU will trigger a page fault interrupt; after the target end's operating system captures the page fault interrupt, it will suspend the process of the target end's memory database, and at the same time notify the target end's page fault interrupt processing component through the userfaultd technology, wherein the notification will also send the identifier of the specific missing memory page to the target end's page fault interrupt processing component; when the target end's page fault interrupt processing component receives the notification, it sends a request to obtain the missing memory page to the source end's page migration component, and after the source end's page migration component sends the missing memory page to the target end's page fault interrupt processing component, the target end's page fault interrupt processing component writes the missing memory page to the data area of ​​the target end's memory database; at the same time, the userfaultd technology is used to notify the operating system to resume the process execution of the target end's memory database from the place where it was suspended.

[0130] Step 320: The background migration component of the destination side searches for unreplicated memory pages in the data area of ​​the destination side's memory database.

[0131] Step 322: When the background migration component of the target end finds that there are uncopied memory pages in the data area of ​​the memory database of the target end, and the memory page is not the memory page obtained by the main thread of the target end, a memory page acquisition request is sent to the page migration component of the source end.

[0132] Step 324: After the page migration component at the source end receives the memory page acquisition request sent by the background migration component at the destination end, the page migration component at the source end acquires the uncopied memory page from the data area copy of the data area of ​​the source end's memory database based on the memory page identifier in the memory page acquisition request.

[0133] Step 326: The page migration component at the source end sends the acquired un-copied memory page to the background migration component at the destination end.

[0134] Step 328: The background migration component of the destination end writes the received uncopied memory page into the data area of ​​the destination end's memory database, and then continues to execute step 320 until the full data synchronization between the source end's memory database and the destination end's memory database is completed.

[0135] In specific implementation, when the source enters the full synchronization mode, it first performs a fork to create a memory copy of the local data area (i.e., the above data snapshot). Then the page migration component is started to respond to the memory page read request of the destination. While processing the new request (data processing request), the source also sends the request to the destination at the same time. When the source receives the page request from the destination, it sends the corresponding page in the memory copy of the local data area to the remote end.

[0136] In the embodiments of this specification, the data processing method can directly synchronize data based on memory pages, avoiding serialization and deserialization, and at the same time facilitate multi-threaded replication and speed up synchronization. In addition, the postcopy replication method is adopted, so that the destination end can start working in advance, reducing the accumulation of request logs on the source end, and further speeding up the data synchronization.

[0137] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 4 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 4 As shown, the device comprises:

[0138] The parsing module 402 is configured to receive a data processing request forwarded by the first database and parse the data processing request;

[0139] A page missing trigger module 404 is configured to send a memory page write request to the first database when it is determined based on the parsing result that the corresponding memory page does not exist in the second database;

[0140] A missing page request module 406 is configured to receive the memory page obtained by the first database from the data snapshot of the first database in response to the memory page write request;

[0141] The missing page writing module 408 is configured to write the memory page into the second database, and perform data processing on the memory page of the second database based on the data processing request.

[0142] Optionally, the device further comprises:

[0143] The background migration module is configured as follows:

[0144] Acquire a memory page in the second database, match the memory page with a memory page in a preset memory page table, and determine an unmatched memory page in the preset memory page table as a target memory page;

[0145] In a case where the target memory page is not the memory page corresponding to the memory page write request, sending a memory page acquisition request to the first database, wherein the memory page acquisition request carries a page identifier of the target memory page;

[0146] A corresponding target memory page is acquired from the first database based on the page identifier, and the target memory page is written into the second database.

[0147] Optionally, the parsing module 402 is further configured to:

[0148] A data processing request in a structured query language forwarded by a first database is received, and the data processing request is parsed to determine the data to be processed in the data processing request and index information corresponding to the data to be processed.

[0149] Optionally, the page fault triggering module 404 is further configured to:

[0150] determining a memory address from a first data area of ​​the second database based on the index information;

[0151] When it is determined based on the memory address that there is no corresponding memory page in the second data area of ​​the second database, a memory page write request is sent to the first database.

[0152] Optionally, the missing page writing module 408 is further configured to:

[0153] Based on the data processing request, the data to be processed is processed in the memory page of the second database.

[0154] Optionally, the page missing request module 406 is further configured to:

[0155] The memory page is received by the first database from a data snapshot of a second data area of ​​the first database based on a memory page identifier carried in the memory page write request.

[0156] Optionally, the target memory page is at least one;

[0157] Accordingly, the background migration module is further configured as follows:

[0158] determining whether each target memory page in the at least one target memory page is a memory page corresponding to the memory page write request,

[0159] If yes, then end.

[0160] If not, at least one memory page acquisition request is sent to the first database.

[0161] Optionally, the background migration module is further configured to:

[0162] At least one corresponding target memory page is acquired from the first database based on the at least one page identifier, and the at least one target memory page is written into the second database.

[0163] Optionally, the page fault triggering module 404 is further configured to:

[0164] A memory page write request is sent to the first database based on a preset network transmission method.

[0165] Accordingly, the page fault request module 406 is further configured to:

[0166] The first database responds to the memory page write request based on the preset network transmission mode, and obtains and compresses the memory page from the data snapshot.

[0167] Optionally, the device further comprises:

[0168] The preset module is configured to receive an initial memory page that meets a preset condition and is sent by the first database, and write the initial memory page into a second data area of ​​the second database.

[0169] Optionally, the device further comprises:

[0170] The data processing module is configured to perform data processing on the memory page of the second database based on the data processing request when it is determined based on the analysis result that the corresponding memory page of the second database exists.

[0171] The data processing device provided in the embodiment of this specification directly forwards the data processing request to the second database when the first database receives the data processing request, so that the second database can process the data processing request from the beginning, reducing the request log accumulation and memory overhead on the first database during the synchronization process, and directly copying the memory page to the destination end through the first database, avoiding serialization and deserialization, and speeding up the full data synchronization.

[0172] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0173] Figure 5 The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0174] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0175] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0176] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 500 may also be a mobile or stationary server.

[0177] The processor 520 is used to execute the following computer executable instructions, which implement the steps of the data processing method when executed by the processor.

[0178] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0179] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the data processing method when executed by a processor.

[0180] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0181] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0182] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0183] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0184] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0185] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, applied to a second database, include: receiving a data processing request forwarded by the first database, and parsing the data processing request, wherein the data processing request is forwarded to the second database while processing the data processing request when the first database receives the data processing request sent by the user; If it is determined based on the analysis result that the corresponding memory page does not exist in the second database, sending a memory page write request to the first database; receiving the memory page obtained by the first database from the data snapshot of the first database in response to the memory page write request; The memory page is written into the second database, and data processing is performed on the memory page of the second database based on the data processing request.

2. The data processing method according to claim 1, further comprising: include: Acquire a memory page in the second database, match the memory page with a memory page in a preset memory page table, and determine an unmatched memory page in the preset memory page table as a target memory page; In a case where the target memory page is not the memory page corresponding to the memory page write request, sending a memory page acquisition request to the first database, wherein the memory page acquisition request carries a page identifier of the target memory page; A corresponding target memory page is acquired from the first database based on the page identifier, and the target memory page is written into the second database.

3. The data processing method according to claim 1 or 2, wherein the receiving the data processing request forwarded by the first database and parsing the data processing request, include: A data processing request in a structured query language forwarded by a first database is received, and the data processing request is parsed to determine the data to be processed in the data processing request and index information corresponding to the data to be processed.

4. The data processing method according to claim 3, wherein when it is determined based on the analysis result that the corresponding memory page does not exist in the second database, a memory page write request is sent to the first database, include: determining a memory address from a first data area of ​​the second database based on the index information; When it is determined based on the memory address that there is no corresponding memory page in the second data area of ​​the second database, a memory page write request is sent to the first database.

5. The data processing method according to claim 3, wherein the data processing is performed on the memory page of the second database based on the data processing request. include: Based on the data processing request, the data to be processed is processed in the memory page of the second database.

6. The data processing method according to claim 3, wherein the receiving the memory page obtained from the data snapshot by the first database in response to the memory page write request include: The memory page is received by the first database from a data snapshot of a second data area of ​​the first database based on a memory page identifier carried in the memory page write request.

7. The data processing method according to claim 2, wherein the target memory page is at least one; Correspondingly, when the target memory page is not the memory page corresponding to the memory page write request, a memory page acquisition request is sent to the first database, include: determining whether each target memory page in the at least one target memory page is a memory page corresponding to the memory page write request, If yes, then end. If not, at least one memory page acquisition request is sent to the first database.

8. The data processing method according to claim 7, wherein the corresponding target memory page is obtained from the first database based on the page identifier, and the target memory page is written into the second database, include: At least one corresponding target memory page is acquired from the first database based on the at least one page identifier, and the at least one target memory page is written into the second database.

9. The data processing method according to claim 1 or 2, wherein the sending of a memory page write request to the first database comprises: include: Sending a memory page write request to the first database based on a preset network transmission method; Correspondingly, the receiving the memory page obtained by the first database from the data snapshot in response to the memory page write request includes: The first database responds to the memory page write request based on the preset network transmission mode, and obtains and compresses the memory page from the data snapshot.

10. The data processing method according to claim 4, wherein before receiving the data processing request forwarded by the first database and parsing the data processing request, include: An initial memory page that meets a preset condition and is sent by the first database is received, and the initial memory page is written into a second data area of ​​the second database.

11. The data processing method according to claim 1 or 2, wherein after receiving the data processing request forwarded by the first database and parsing the data processing request, include: When it is determined based on the analysis result that the corresponding memory page of the second database exists, data processing is performed on the memory page of the second database based on the data processing request.

12. A data processing device, applied to a second database, include: a parsing module, configured to receive a data processing request forwarded by the first database, and parse the data processing request, wherein the data processing request is forwarded to the second database while processing the data processing request when the first database receives the data processing request sent by the user; a page missing trigger module, configured to send a memory page write request to the first database when it is determined based on the parsing result that the corresponding memory page does not exist in the second database; a missing page request module, configured to receive the memory page obtained by the first database from the data snapshot of the first database in response to the memory page write request; The missing page writing module is configured to write the memory page into the second database and perform data processing on the memory page of the second database based on the data processing request.

13. A computing device, include: Memory and processor; The memory is used to store computer executable instructions, which, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 11.