Data persistence synchronization method, device and equipment based on cloud environment
By building a three-tier shared architecture for a distributed database in a cloud computing environment and utilizing distributed vector timestamp technology, the reliability and performance issues of data storage in the cloud computing environment are solved, enabling fast data persistence and transaction execution, and improving the reliability and scalability of the system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-21
AI Technical Summary
In existing cloud computing environments, data storage suffers from problems such as long transaction log and data persistence times, and weak system reliability. In particular, the slow transmission speed from the shared cache layer to the shared storage layer limits the performance of the shared cache, resulting in slow data read and write speeds and long persistent write times for transaction logs.
A three-tier shared architecture for a distributed database is constructed, comprising a shared cache layer, a shared persistent memory layer, and a shared storage layer. Distributed vector timestamp technology is used to assign timestamps to transactions and maintain visibility intervals. The shared cache layer determines the logical page, quickly reads the page requested by the transaction, and executes the transaction locally. Redo logs are written to the log area in persistent memory, and a background thread replays and generates the page data in the shared persistent memory layer.
It improves system caching performance, solves the scalability bottleneck of transaction timestamps in multi-node environments, shortens data persistence time, enhances system reliability, reduces the latency of a single write operation within a transaction, and improves system performance.
Smart Images
Figure CN121907856A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of database technology, and in particular to a data persistence and synchronization method, apparatus and device based on a cloud environment. Background Technology
[0002] With the rapid development of cloud computing technology, data storage has gradually shifted from within enterprises to the cloud. While cloud computing environments offer unprecedented data storage, processing, and sharing capabilities, promoting the widespread application of cloud databases, they also present significant challenges in maintaining data accuracy, integrity, and consistency. As a key technology in current system design, distributed systems present unprecedented challenges and opportunities for data management.
[0003] In existing technologies, shared caching in cloud scenarios is limited. Slow data transfer speeds from the shared cache layer to the shared storage layer lead to page volatileness in the shared cache layer, hindering the rapid evicting of cold pages. This limits the performance of the shared cache, resulting in slow data read / write speeds and long persistent write times for transaction logs. Furthermore, the shared cache layer in distributed databases uses global timestamps, which has thread-level scalability limitations. The shared data cache cannot be scaled to multiple nodes, leading to inefficient evicting of cold pages to the storage layer and long data persistence times.
[0004] Regarding the aforementioned technologies, the inventors discovered that existing cloud data storage methods suffer from problems such as long transaction log and data persistence times, and relatively weak system reliability. Summary of the Invention
[0005] To shorten the persistence time of transaction logs and data and enhance system reliability, this application provides a data persistence and synchronization method, apparatus and device based on a cloud environment.
[0006] Firstly, this application provides a data persistence and synchronization method based on a cloud environment.
[0007] This application is achieved through the following technical solution: A data persistence and synchronization method based on a cloud environment includes the following steps: Build a distributed database and configure a three-tier shared architecture, including a shared cache layer for storing accessed pages, a shared persistent memory layer for storing persistent logs, log replay generated pages, and cache directories, and a shared storage layer for storing all data pages in shared cache and transaction persistent memory storage. The corresponding transactions are assigned to each node, and after the node updates the data, it is synchronously written to the shared cache layer. A timestamp is assigned to a transaction, and the timestamp maintains a visibility interval. All timestamps generated by the previous computing server within the visibility interval are recorded as a timestamp range, and the timestamp range satisfies monotonically increasing property. Determine whether a logical page exists in the shared cache layer; If a logical page exists, obtain the current timestamp of the transaction as the read timestamp. For each page requested by the transaction, read it into the cache of the current node, execute the transaction locally, and save the private data generated during the transaction execution locally. The redo logs generated during the transaction execution are written to the log area in persistent memory, and the background thread replays and generates the page data in the shared persistent memory layer.
[0008] In a preferred embodiment, this application can be further configured to include the following steps: If the logical page does not exist, the shared storage layer is accessed to obtain the page requested for the transaction, and a copy of the page is added to the shared cache layer and the shared persistent memory layer.
[0009] In a preferred embodiment, this application can be further configured to include the following steps: If a logical page exists and a cache miss occurs, the physical page corresponding to the logical page, which exists in the shared persistent memory layer, is stored in the shared cache layer via a background thread.
[0010] In a preferred embodiment, this application can be further configured to include the following steps: Eject cold pages that have completed playback to the shared storage layer.
[0011] In a preferred embodiment, this application can be further configured such that the step of the background thread replaying the page data in the shared persistent memory layer includes: Write to each node's private log file via a background thread; Each node's private log files are sent to a preset node for merging before being sent back to each node; Each node modifies the page data in persistent memory based on the redo log, and a background thread replays and generates the page data in the shared persistent memory layer.
[0012] In a preferred embodiment, this application can be further configured such that the step of obtaining the current timestamp of the transaction as the read timestamp includes: Access the computing server in a preset order to obtain the current timestamp of the transaction; The partial order relationship generated by the timestamp is consistent with the partial order relationship generated by the timestamp read by the previously accessed computing server.
[0013] Secondly, this application provides a data persistence and synchronization device based on a cloud environment.
[0014] This application is achieved through the following technical solution: A data persistence and synchronization device based on a cloud environment includes, The module is used to build a distributed database and configure a three-tier shared architecture, including a shared cache layer for storing accessed pages, a shared persistent memory layer for storing persistent logs, log replay generated pages, and cache directories, and a shared storage layer for storing all data pages in shared cache and transaction persistent memory storage. The node data module is used to allocate corresponding transactions to various nodes. After a node updates its data, it synchronously writes the updated data to the shared cache layer. The timestamp module is used to assign a timestamp to a transaction. The timestamp maintains a visibility interval and records all timestamps generated by the previous computing server within the visibility interval as a timestamp range. The timestamp range satisfies monotonically increasing property. The detection module is used to determine whether a logical page exists in the shared cache layer; The transaction module is used to obtain the current timestamp of the transaction as the read timestamp when a logical page exists. For each page requested by the transaction, it is read into the cache of the current node, the transaction is executed locally, and the private data generated during the transaction execution is saved locally. The persistence module is used to write the redo logs generated during the execution of transactions into the log area in persistent memory, and the background thread replays and generates the page data in the shared persistent memory layer.
[0015] Thirdly, this application provides a computer device.
[0016] This application is achieved through the following technical solution: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described data persistence and synchronization methods in a cloud environment.
[0017] Fourthly, this application provides a computer-readable storage medium.
[0018] This application is achieved through the following technical solution: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described data persistence and synchronization methods in a cloud environment.
[0019] Fifthly, this application provides a computer program product.
[0020] This application is achieved through the following technical solution: A computer program product includes a computer program that, when executed by a processor, implements the steps of any of the above-described data persistence and synchronization methods in a cloud environment.
[0021] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following: A distributed database is built, configured with a three-tier shared architecture, including a shared cache layer for storing accessed pages, a shared persistent memory layer for storing persistent logs, log replay generated pages, and a cache directory, and a shared storage layer for storing all data pages in shared cache and transaction persistent memory. This ensures shared caching among nodes and improves system caching performance. Transactions are distributed across nodes; after a node updates data, it synchronously writes the updated data to the shared cache layer. A timestamp is assigned to each transaction, and each timestamp maintains a visibility interval. All timestamps generated by the previous computing server within this visibility interval are recorded as a timestamp range, which satisfies monotonically increasing properties. Distributed vector timestamp technology is used to maintain timestamps distributed across nodes, providing a consistent timestamp snapshot. This data sharing cache can expand... Deploying to multiple nodes solves the bottleneck problem of transaction timestamp scalability in a multi-node environment under a shared cache architecture, improving the scalability of transaction timestamps. It checks the existence of logical pages in the shared cache layer; if a logical page exists, it obtains the current timestamp of the transaction as the read timestamp. For each page requested by the transaction, it reads it into the cache of the current node, executes the transaction locally, and saves the private data generated during the transaction execution locally. Data does not need to be transferred from the shared cache layer to the shared storage layer, and pages in the shared cache layer are less prone to missing, thus facilitating the rapid eviction of cold pages and shortening data persistence time. Redo logs generated during transaction execution are written to the log area in persistent memory, and a background thread replays and generates page data in the shared persistent memory layer, reducing the latency of individual write operations within a transaction, improving system limits, shortening transaction log persistence time, and thus enhancing system reliability. Attached Figure Description
[0022] Figure 1 This is a schematic diagram of the main process of a data persistence and synchronization method based on a cloud environment, provided as an exemplary embodiment of this application. Detailed Implementation
[0023] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application. To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0024] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.
[0025] To ensure the consistency of data read and write caching among multiple nodes and maintain the reliability and accuracy of the system, this application proposes a data persistence and synchronization method based on the actual scenario of a cloud computing environment. When facing problems such as network latency and node failure, it can ensure that the shared cache among nodes can be achieved, thereby realizing data consistency. By regenerating pages through the redo log replay function, the latency of a single write operation within a transaction is reduced, thereby shortening the persistence time of transaction logs and data and improving system reliability.
[0026] This application provides a data persistence and synchronization method based on a cloud environment. The main steps of the method are described below.
[0027] Build a distributed database and configure a three-tier shared architecture, including a shared cache layer for storing accessed pages, a shared persistent memory layer for storing persistent logs, pages generated by log replay, and cache directories, and a shared storage layer for storing all data pages in shared cache and transaction persistent memory storage. The corresponding transactions are assigned to each node, and after the node updates the data, it is synchronously written to the shared cache layer. A timestamp is assigned to a transaction, and the timestamp maintains a visibility interval. All timestamps generated by the previous computing server within the visibility interval are recorded as a timestamp range, and the timestamp range satisfies monotonically increasing property. Determine whether a logical page exists in the shared cache layer; If a logical page exists, obtain the current timestamp of the transaction as the read timestamp. For each page requested by the transaction, read it into the cache of the current node, execute the transaction locally, and save the private data generated during the transaction execution locally. If no logical page exists, the page requested by the transaction is obtained by accessing the shared storage layer, and a copy of the page is added to the shared cache layer and the shared persistent memory layer. The redo logs generated during the transaction execution are written to the log area in persistent memory, and the background thread replays and generates the page data in the shared persistent memory layer.
[0028] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.
[0029] Reference Figure 1 First, set up a distributed database, ensuring the available space of the server's file system meets the requirements for the amount of data being transferred, and that the total hard drive capacity of the server is no less than that of an enterprise-grade SSD storage disk. Configure read / write node partitions, ensuring that the available capacity of each node is no less than 50% of the total storage capacity after partitioning, to provide sufficient storage space.
[0030] Next, configure the database shared architecture, including a shared cache layer, a shared persistent memory layer, and a shared storage layer. The shared cache layer is located at the top of the distributed database's storage architecture and is used to store frequently accessed pages. The shared persistent memory layer is located in the middle of the distributed database's storage architecture, connecting the shared cache layer and the shared storage layer. The shared persistent memory layer is used to store persistent logs, pages generated by log replay, and the cache directory. The shared storage layer is located at the bottom of the distributed database's storage architecture and is used to store all data pages in shared cache and transaction persistent memory storage.
[0031] Secondly, node data synchronization. The corresponding transactions are distributed to each node, and after the node updates the data, it is synchronously written to the shared cache layer.
[0032] Next, the database system synchronizes data, first checking if the logical page exists in the shared cache layer. For pages that need to be accessed during transaction execution, it is first necessary to determine whether the logical page exists in the shared cache layer.
[0033] A logical page refers to a virtual page that is partitioned into fixed sizes and numbered by the upper-layer application or database, such as a 4 KB page. The logical page is decoupled from the underlying actual storage location by the cache manager through block mapping, mapping the logical page number to an object key. If an object key is detected, it is determined that a logical page exists in the shared cache layer. If no object key is detected, it is determined that a logical page does not exist in the shared cache layer.
[0034] Updating the same logical page will generate a new version of the object and a new object key. The mapping relationship will then be updated in the block mapping, which adapts to the characteristics of object storage that it cannot be partially modified and that is eventually consistent.
[0035] If a logical page exists, a transaction is executed. Before the transaction begins, a timestamp is assigned to it. Utilizing the atomic update capability of RDMA (remote direct memory access), the timestamp is distributed and maintained across various nodes. For each page requested by the transaction, it is read into the local node's cache by the page reader, and the transaction is executed locally. The transaction manager is notified to save the private data generated during the transaction execution locally. The redo logs for committed transactions are temporarily stored in a thread-local log buffer.
[0036] If the logical page is in the shared cache layer, but the corresponding physical page is not in the shared cache layer, a cache miss will be triggered. The background thread will swap the logical page, which exists in the shared persistent memory layer, into the shared cache layer so that the transaction can be read again.
[0037] A timestamp pointer to a vector timestamp (VTS) provides a logical clock for each compute server or thread. At the start of a transaction, the latest version of the VTS needs to be obtained as the read timestamp, the logical clock of the corresponding compute server is incremented by 1, and the memory server is synchronized.
[0038] During transaction execution, distributed vector timestamps continuously generate vector timestamps that satisfy an increasing relationship and are allocated when the transaction manager first accesses the transaction.
[0039] Maintain one for each timestamp of the timestamp server ( , Visibility interval, and These represent the minimum and maximum versions of the timestamp when the previous timestamp server accessed the current timestamp server, respectively; they record the visibility range of timestamps generated by accessing the previous timestamp server within the visibility interval, and the visibility intervals of timestamps satisfy a monotonically increasing relationship; each server traverses backward from the latest timestamp until it finds a timestamp that fits the visibility interval range.
[0040] If the logical page does not exist, the shared storage layer is accessed to retrieve it, execute the transaction, and trigger a cache miss. The transaction accesses the shared storage layer to retrieve the required page, and at the same time adds a copy of the page to the shared cache layer and the shared persistent memory layer.
[0041] Redo logs generated during transaction execution are persisted to the log area of the shared persistent memory layer (PM) and replayed by a background thread. Redo logs generated during transaction execution are written to the log buffer via a log writer for access during transaction execution; and a dedicated background thread writes data logs and metadata logs to the PM's log area. Page replay also occurs in the background, and changes to the persistent logs in the PM are synchronized to the pages in the shared persistent memory. Writing the redo logs of committed transactions from the log buffer to the persistent memory of this node enables system recovery to a consistent state after a crash. Background threads write to each node's private log file, and the background thread replays and generates pages in the shared persistent memory layer, reducing the latency of individual write operations within a transaction.
[0042] Cold pages that have completed replay are evicted to the shared storage layer. Once the replay is complete, the cold pages will be evicted to the shared storage layer to complete the next data synchronization. This reduces the space pressure on the shared persistent memory, shortens the persistence time of transaction logs and data, and improves system reliability.
[0043] In one embodiment, when replaying persistent logs in the system and applying them to pages in shared persistent memory, changes to a single page are scattered across multiple local logs because compute nodes may access any page. The step of the background thread replaying to generate page data in the shared persistent memory layer includes... Write to each node's private log file via a background thread; Each node's private log files are sent to a preset node for merging before being sent back to each node; Each node modifies the page data in persistent memory based on the redo log, and a background thread replays and generates the page data in the shared persistent memory layer.
[0044] By sending the local logs of each node to a designated node for merging during the replay process, and then sending them back to each node, each node modifies the page data in persistent memory based on the redo logs. The page data in the shared persistent memory layer is then replayed in the background, making the replayed page data more accurate.
[0045] In one embodiment, since the timestamps between transactions are independent, the VTS read by the compute node on a single timestamp node satisfies a monotonically increasing relationship. However, reading the VTS across multiple servers will cause it to no longer satisfy this characteristic. To ensure that the timestamps of transactions satisfy the increasing relationship, rule constraints need to be set to make the timestamps read by different compute nodes satisfy the increasing relationship again. The step of obtaining the current timestamp of the transaction as the read timestamp includes: Access the computing servers in a preset order to obtain the current timestamp of the transaction; Among them, the partial order relationship generated by the timestamp is consistent with the partial order relationship generated by the timestamp read by the previous accessed computing server.
[0046] Rule 1: When all computing nodes obtain timestamps, they must access the timestamp server in a fixed order. For example, in the order of timestamp server 1, timestamp server 2, and timestamp server 3; When all computing nodes access each timestamp server in the set order, it can ensure that the access order of the VTS read between multiple servers satisfies monotonic increase. However, since the access delay of each timestamp server is uncontrollable, Rule 2 must be added to make the VTS read between multiple servers maintain monotonic increase according to the initial access order. Specifically as follows: The partial order relationship generated by the VTS read by each computing node at the current timestamp server is consistent with the partial order relationship generated by the VTS read by the previous accessed timestamp server. That is, if for timestamp server n there is ≤ , then there is ≤ , ≤n.
[0047] The partial order relationship of timestamps means that after using timestamps to assign monotonically increasing numbers to events, it can describe the causal order between events: if event a causally precedes event b, then the timestamps satisfy C(a) < C(b). Every time an event occurs within a process, the local counter is incremented by one. The sent message carries the current timestamp; when receiving a message, the local counter is updated to max(local message timestamp) + 1. Concurrent events in different processes have the same timestamp or incomparable timestamps.
[0048] By using the distributed vector timestamp technology, the timestamps are maintained distributively on each node and a consistent timestamp snapshot is provided. The timestamps of transactions satisfy an increasing relationship, enabling the data sharing cache to be extended to multiple nodes, and solving the bottleneck problem of the scalability of transaction timestamps in a multi-node environment under a shared cache architecture.
[0049] In summary, a cloud-based data persistence and synchronization method utilizes a shared caching mechanism to build a distributed database. It partitions read and write nodes according to the architecture and configures a three-layer shared architecture, including a shared cache layer, a shared persistent memory layer, and a shared storage layer. When synchronizing data, the database system first distributes the corresponding transactions to each node. After a node updates its data, it synchronously writes it to the shared cache layer. The transaction execution process is decoupled. During transaction execution, reading occurs first in the shared cache layer, while the shared storage layer synchronously retrieves cold pages and adds page copies to both the shared persistent memory layer and the shared cache layer. Redo logs generated during transaction execution are persisted to the log area in the PM layer. A background thread replays and generates pages in the shared persistent memory layer, evicting replayed cold pages from the shared storage layer. This process repeats for the next data synchronization cycle. Data does not need to be transferred from the shared cache layer to the shared storage layer, and pages in the shared cache layer are less prone to loss, thus facilitating the rapid eviction of cold pages and shortening data persistence time. It also reduces the latency of individual write operations within a transaction, improves system limits, shortens transaction log persistence time, enhances system reliability, and further improves the performance of database transaction execution.
[0050] A cloud-based data persistence and synchronization method redirects access to log data to addresses in a shared cache during transaction execution. This decouples the shared cache layer and the shared persistence layer, satisfying the low latency requirement for page access during transaction execution while improving the efficiency of dirty page eviction in the shared cache layer.
[0051] Since page directories are typically stored on different storage nodes, a data persistence and synchronization method based on a cloud environment proposes a cache consistency protocol. This protocol uses a shared cache and a cache invalidation protocol to ensure page consistency. Organizing data at the page level can reduce the latency of accessing the storage layer.
[0052] A data persistence and synchronization method based on a cloud environment optimizes the transaction execution process. By utilizing persistent memory technology and introducing a shared persistent memory layer, and by regenerating pages with the help of redo log replay function, the latency of a single write operation within a transaction is reduced. Ultimately, this reduces transaction processing latency and improves system limits, achieving fast persistence.
[0053] A cloud-based data persistence and synchronization method improves the scalability of transaction timestamps. Utilizing distributed vector timestamp technology, timestamps are maintained distributed across various nodes, providing a consistent timestamp snapshot. This optimizes the execution flow of database transactions, enabling elastic expansion of the database's computation and storage layers, enhancing timestamp scalability, eliminating page eviction processes in shared caches ahead of time, and achieving rapid persistence. It solves the bottleneck problem of transaction timestamp scalability in multi-node environments under shared cache architectures, providing flexible, reliable, easy-to-use, and scalable database services.
[0054] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0055] This application also provides a data persistence and synchronization device based on a cloud environment, which corresponds one-to-one with the data persistence and synchronization method based on a cloud environment described in the above embodiments. The data persistence and synchronization device based on a cloud environment includes... The module is used to build a distributed database and configure a three-tier shared database architecture, including a shared cache layer for storing accessed pages, a shared persistent memory layer for storing persistent logs, pages generated by log replay, and cache directories, and a shared storage layer for storing all data pages. The node data module is used to allocate corresponding transactions to various nodes. After a node updates its data, it synchronously writes the updated data to the shared cache layer. The timestamp module is used to assign a timestamp to a transaction. The timestamp maintains a visibility interval and records all timestamps generated by the previous computing server within the visibility interval as a timestamp range. The timestamp range satisfies monotonically increasing property. The detection module is used to determine whether a logical page exists in the shared cache layer; The transaction module is used to obtain the current timestamp of the transaction as the read timestamp when a logical page exists. For each page requested by the transaction, it is read into the cache of the current node, the transaction is executed locally, and the private data generated during the transaction execution is saved locally. The persistence module is used to write the redo logs generated during the execution of transactions into the log area in persistent memory, and the background thread replays and generates the page data in the shared persistent memory layer.
[0056] For specific limitations regarding a data persistence and synchronization device based on a cloud environment, please refer to the limitations of a data persistence and synchronization method based on a cloud environment mentioned above, which will not be repeated here.
[0057] The modules in the aforementioned cloud-based data persistence and synchronization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.
[0058] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements any of the aforementioned data persistence and synchronization methods in a cloud environment.
[0059] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described data persistence and synchronization methods based on a cloud environment.
[0060] In one embodiment, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements any of the above-described data persistence and synchronization methods in a cloud environment.
[0061] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. When executed, the computer program may include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0062] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.
Claims
1. A data persistence and synchronization method based on a cloud environment, characterized in that, Includes the following steps, Build a distributed database and configure a three-tier shared architecture, including a shared cache layer for storing accessed pages, a shared persistent memory layer for storing persistent logs, log replay generated pages and cache directories, and a shared storage layer for storing all data pages in a shared cache and transaction persistent memory storage. The corresponding transactions are assigned to each node, and after the node updates the data, it is synchronously written to the shared cache layer. A timestamp is assigned to a transaction, and the timestamp maintains a visibility interval. All timestamps generated by the previous computing server within the visibility interval are recorded as a timestamp range, and the timestamp range satisfies monotonically increasing property. Determine whether a logical page exists in the shared cache layer; If a logical page exists, obtain the current timestamp of the transaction as the read timestamp. For each page requested by the transaction, read it into the cache of the current node, execute the transaction locally, and save the private data generated during the transaction execution locally. The redo logs generated during the transaction execution are written to the log area in persistent memory, and the background thread replays and generates the page data in the shared persistent memory layer.
2. The data persistence and synchronization method based on a cloud environment according to claim 1, characterized in that, It also includes the following steps, If the logical page does not exist, the shared storage layer is accessed to obtain the page requested for the transaction, and a copy of the page is added to the shared cache layer and the shared persistent memory layer.
3. The data persistence and synchronization method based on a cloud environment according to claim 1, characterized in that, It also includes the following steps, If a logical page exists and a cache miss occurs, the physical page corresponding to the logical page, which exists in the shared persistent memory layer, is stored in the shared cache layer via a background thread.
4. The data persistence and synchronization method based on a cloud environment according to claim 1, characterized in that, It also includes the following steps, Eject cold pages that have completed playback to the shared storage layer.
5. The data persistence and synchronization method based on a cloud environment according to claim 4, characterized in that, The step of generating page data in the shared persistent memory layer by replaying the background thread includes: Write to each node's private log file via a background thread; Each node's private log files are sent to a preset node for merging before being sent back to each node; Each node modifies the page data in persistent memory based on the redo log, and a background thread replays and generates the page data in the shared persistent memory layer.
6. The data persistence and synchronization method based on a cloud environment according to any one of claims 1-5, characterized in that, The step of obtaining the current timestamp of the transaction as the read timestamp includes: Access the computing server in a preset order to obtain the current timestamp of the transaction; The partial order relationship generated by the timestamp is consistent with the partial order relationship generated by the timestamp read by the previously accessed computing server.
7. A data persistence and synchronization device based on a cloud environment, characterized in that, include, The module is used to build a distributed database and configure a three-tier shared architecture, including a shared cache layer for storing accessed pages, a shared persistent memory layer for storing persistent logs, log replay generated pages, and cache directories, and a shared storage layer for storing all data pages in shared cache and transaction persistent memory storage. The node data module is used to allocate corresponding transactions to various nodes. After a node updates its data, it synchronously writes the updated data to the shared cache layer. The timestamp module is used to assign a timestamp to a transaction. The timestamp maintains a visibility interval and records all timestamps generated by the previous computing server within the visibility interval as a timestamp range. The timestamp range satisfies monotonically increasing property. The detection module is used to determine whether a logical page exists in the shared cache layer; The transaction module is used to obtain the current timestamp of the transaction as the read timestamp when a logical page exists. For each page requested by the transaction, it is read into the cache of the current node, the transaction is executed locally, and the private data generated during the transaction execution is saved locally. The persistence module is used to write the redo logs generated during the execution of transactions into the log area in persistent memory, and the background thread replays and generates the page data in the shared persistent memory layer.
8. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.