Cloud database data processing method and device, electronic equipment and storage medium

Abstract

The application provides a cloud database data processing method and device, electronic equipment and storage medium, and relates to the technical field of databases.The method comprises the following steps: a storage engine acquires an obsolete data page from a first buffer pool in a managed memory, and stores the obsolete data page in a local SSD storage; if an access request of target data is received and the target data is not queried in the first buffer pool, the target data is read from the obsolete data page in the local SSD storage; and if the target data is not read in the local SSD storage, the target data is read from a cloud storage. In the embodiment, the obsolete data page in the first buffer pool in the storage engine is stored in the local SSD storage, and when data cannot be read in the local SSD storage, data is read in the cloud storage, so that when data cannot be queried in the first buffer pool, data is directly read in the cloud, the number of cloud access times is reduced, IO delay is reduced, and the performance of a database system is improved.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a cloud database data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] The storage engine is a core component of a Database Management System (DBMS), responsible for data storage, retrieval, and updating. Different storage engines are optimized for different tasks and data processing needs, such as transaction processing, data warehouse operations, full-text search, or real-time analytics. The internal buffer pool of a storage engine resides in memory. Due to limited host memory and instance size constraints, the buffer pool configuration is not particularly large. When the volume of hot data exceeds the buffer pool size, a large number of input / output (IO) requests are generated and sent to cloud storage.

[0003] However, cloud storage incurs network and storage software stack overhead, resulting in higher I / O latency. In I / O-intensive scenarios, this high latency negatively impacts database performance. Summary of the Invention

[0004] This application provides a cloud database data processing method, apparatus, electronic device, and storage medium to solve the problem of high IO latency in cloud storage affecting database performance.

[0005] In a first aspect, embodiments of this application provide a cloud database data processing method, the method comprising: a cloud database including compute nodes and storage nodes, the storage nodes including cloud storage and local solid-state drive (SSD) storage of the instance where the compute node is located, the compute node including a storage engine, the storage engine managing a first buffer pool in memory; the storage engine obtaining evicted data pages from the first buffer pool in the managed memory and storing the evicted data pages in the local SSD storage; if an access request for target data is received and the target data is not found in the first buffer pool, then the target data is read from the evicted data pages in the local SSD storage; if the target data is not read in the local SSD storage, then the target data is read from the cloud storage.

[0006] Secondly, embodiments of this application provide a cloud database data processing apparatus, comprising: a cloud database including compute nodes and storage nodes, the storage nodes including cloud storage and local solid-state drive (SSD) storage of the instance where the compute nodes reside, the compute nodes including a storage engine, the storage engine managing a first buffer pool in memory; a data storage module, configured to retrieve evicted data pages from the first buffer pool in the managed memory and store the evicted data pages in the local SSD storage; a first reading module, configured to read the target data from the evicted data pages in the local SSD storage area if an access request for target data is received and the target data is not found in the first buffer pool; and a second reading module, configured to read the target data from the cloud storage if the target data is not found in the local SSD storage area.

[0007] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods described above when executing the computer program.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods described above.

[0009] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the methods described above.

[0010] Compared with the prior art, this application has the following advantages:

[0011] This application provides a cloud database data processing method, apparatus, electronic device, and storage medium. The cloud database includes compute nodes and storage nodes. The storage nodes include cloud storage and local solid-state drive (SSD) storage of the instance where the compute node resides. The compute node includes a storage engine that manages a first buffer pool in memory. The storage engine retrieves evicted data pages from the first buffer pool in memory and stores them in the local SSD storage. If an access request for target data is received, and the target data is not found in the first buffer pool, the target data is read from the evicted data pages in the local SSD storage. If the target data is not found in the local SSD storage, the target data is read from the cloud storage. In this embodiment, storing evicted data pages from the first buffer pool in the storage engine in the local SSD storage, and then reading data from the cloud storage when data cannot be read from the local SSD storage, avoids directly reading data from the cloud when data is not found in the first buffer pool, reduces the number of cloud accesses, lowers IO latency, and improves the performance of the database system.

[0012] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0013] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0014] Figure 1 This is a schematic diagram illustrating an application scenario of the cloud database data processing method provided in this application.

[0015] Figure 2 This is a schematic diagram illustrating the data block state update according to an embodiment of this application.

[0016] Figure 3 This is a flowchart of a cloud database data processing method according to an embodiment of this application.

[0017] Figure 4 This is a flowchart of a cloud database data processing method according to an embodiment of this application.

[0018] Figure 5 This is a structural block diagram of a cloud database data processing device according to an embodiment of this application.

[0019] Figure 6 This is a block diagram of an electronic device used to implement embodiments of this application. Detailed Implementation

[0020] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0021] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0022] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0023] Figure 1 A flowchart illustrating an application scenario of the cloud database data processing method provided in this application. InnoDB is a storage engine for the relational database MySQL. Figure 1 The InnoDB Buffer Pool shown is the internal buffer pool of the storage engine (i.e., the first buffer pool). There can be one or more InnoDB Buffer Pools, all located in memory. (Extra buffer pool...) Figure 1 The data structure shown (ebuf pool) is located in memory, i.e., the second buffer pool, which caches the evicted data pages of the InnoDB Buffer Pool (the number of evicted data pages can be one or more) and then flushes them to the local solid state drive (SSD) storage. The local SSD storage can be the disk used to store temporary files in the current physical machine, or it can be other disks, such as Non-Volatile Memory Express Solid State Drives (NVME SSDs).

[0024] Each InnoDB buffer pool corresponds to an extra buffer pool (e.g., ...). Figure 1The InnoDB buffer pool includes the ebuf pool and the data file. The InnoDB buffer pool contains a list of Least Recently Used (LRU) data pages (e.g., ...). Figure 1 The buf pool LRU shown is an LRU list of data pages obtained based on the LRU algorithm. The buf pool LRU includes the page identifiers of the data pages stored in the buf pool. The extrabuffer pool includes the ebuf pool LRU (e.g., ...). Figure 1 The diagram shows data structures including the ebuf LRU list, ebuf flushlist, free list, mem pool, and page hash table. The ebuf LRU contains a mapping between the page identifier of the evicted data page and the block identifiers corresponding to the data blocks into which the evicted data page is divided. The ebuf flushlist contains the page identifiers of the evicted data pages to be written to the SSD. The mempool includes a set of page frames used to cache evicted data pages. When there are no free page frames in the mem pool, the evicted data pages need to be flushed to the SSD to free up space. The page hash table contains the hash value of the page identifier of the evicted data page.

[0025] When the amount of data in the data file stored in the InnoDB buffer pool exceeds a preset data volume threshold, the data page is evicted from the InnoDB buffer pool. Evicted data pages are considered clean data pages, meaning their content in memory is identical to their content on disk; that is, the data page has not been modified since being loaded into memory. Evicted data pages are divided into 16KB data blocks. A mapping relationship is established between the page identifier of the evicted data page and the data block identifier of the data block; for example, one page identifier corresponds to one data block identifier. The mapping relationship (space, page no) → m_page_no is stored in the data structure ebuf_block_t, where (space, page no) represents the page identifier of the evicted data page, where space represents the tablespace identifier of the evicted data page, page no represents the data page sequence number, and m_page_no represents the data block identifier, i.e., the page identifier of the data page corresponding to the data block.

[0026] like Figure 1As shown, data pages at the tail end (data pages whose access time is more than a preset time threshold or whose access count is less than a preset number of accesses) are evicted. The mapping relationship corresponding to the evicted data page is inserted into the 3 / 8 position of the ebuf LRU list (any position can be inserted, 3 / 8 is an empirical value). The ebuf_page_hash list includes the hash value of the page identifier of the evicted data page stored in each ebuf_block_t. The ebuf flushlist list records the page identifiers of the evicted data pages that need to be flushed. A background thread flushes the evicted data pages (corresponding to the data blocks identified by m_page_no) to the SSD, i.e., as shown... Figure 1 The example shown is an ebuf pool data file on a local SSD. The background threads include a coordinating thread and worker threads. The coordinating thread retrieves evicted data pages and flushes them to the SSD; alternatively, the coordinating thread retrieves evicted data pages, and the worker threads flush the corresponding data blocks to the SSD.

[0027] In real-world scenarios, the amount of data evicted from a data page may exceed the size of the extra buffer pool. To improve the hit rate of the extra buffer pool, an LRU algorithm is used to manage the ebuf pool LRU list. When a data page is evicted from the buffer pool to the extra buffer pool, if the extra buffer pool is already full, the least accessed data block is evicted from the end of the ebuf pool LRU list, and the data block corresponding to the evicted data page is inserted into the ebuf pool LRU list (e.g., ...). Figure 2 The header and tail of the ebuf_pool->LRU list shown are inserted at the 3 / 8 position by default. Figure 2 The position below 20 is shown in the diagram. Frequently accessed data blocks will be reinserted to the head of the ebuf pool LRU list. If the position of a stored data block in the ebuf pool LRU list exceeds the error zone after insertion 3 / 8 positions (e.g., ...), it will be removed from the list. Figure 2When the EBUF_LRU_OLD_TOLERANCE error zone (as shown) reaches its lower boundary, the state of the stored data blocks is modified. Data blocks within the stored data block region ebuf_pool->LRU_old, from the boundary to the lower boundary of the error zone, and those exceeding the lower boundary of the error zone, are marked as old data blocks. Specifically, the "old" flag of the data blocks in that region is updated to true or false. Old data blocks are deleted if the amount of data stored on the SSD exceeds a preset data volume threshold. The key difference is that the data block state is not modified every time a data block is added or removed, but only when it exceeds the lower boundary of the error zone. This is to avoid performance degradation caused by frequent modifications to the data block state.

[0028] In addition, during the storage of evicted data pages, the process first checks if the hash value of the page identifier of the evicted data page exists in `ebuf_page_hash`. If it does not exist, a free `ebuf block_t` is obtained from the free list to store the mapping relationship between the page identifier of the currently evicted data page and the data block identifier of the data block. A page frame is then allocated from the memory pool to cache the evicted data page. If the hash value of the evicted data page already exists in `ebuf_page_hash`, and the log sequence number (LSN) corresponding to the evicted data page and the existing data page are the same, it means that it has already been stored, and the data page is skipped and not stored again. If the hash value of the evicted data page already exists in `ebuf_page_hash`, but the log sequence number corresponding to the evicted data page and the existing data page are different, the evicted data page is cached in the extra buffer pool and then flushed to the SSD.

[0029] Upon receiving an access request for target data, the request is parsed to obtain the target page identifier. If the hash value corresponding to the target page identifier is found in ebuf_page_hash, the data block's m_page_no is obtained according to the mapping relationship, and the target data is read based on m_page_no. If the hash value corresponding to the target page identifier is not found in ebuf_page_hash, the target data is read from the tablespace file in cloud storage.

[0030] Among them, evicted data pages cached in the extra buffer pool that have not yet been flushed to the SSD can be read. Evicted data pages cached in the extra buffer pool that have not yet been flushed to the SSD may be continuously modified by the upper buffer pool, and only the latest evicted data page is flushed.

[0031] Additionally, when writing or reading data blocks to local SSD storage, a token bucket algorithm is used to limit the number of input / output operations per second (IOPS). Reads and writes are rate-limited independently, and the token count is refreshed when tokens are exhausted. Enabling and disabling the extra buffer pool can be done asynchronously to prevent excessive input / output pressure during data file creation, and disabling it requires waiting for any initiating but incomplete IO requests to finish.

[0032] In this embodiment, a new cache layer, the extra buffer pool, is constructed based on local persistent storage. This local persistent storage can utilize high-performance NVMe SSDs with storage capacities reaching several terabytes, far exceeding the performance of native memory-based buffer pools. This allows for caching more data pages, reducing the number of accesses to cloud storage and lowering IO latency. The read latency of the local NVMe SSD is 80-90µs, and the single-path IO latency is lower than the hundreds ofµs latency of cloud storage. The latency reduction effect of continuous IO access to the extra buffer pool is even more significant.

[0033] This application provides a cloud database data processing method. This method can be applied to computing devices, which may include servers, user terminals, etc. The cloud database includes computing nodes and storage nodes. The storage nodes include cloud storage and local solid-state drive (SSD) storage within the instance where the computing node resides. The computing nodes include a storage engine that manages a first buffer pool in memory. Figure 3 The diagram shown is a flowchart of a cloud database data processing method according to an embodiment of this application, including:

[0034] In step S301, the storage engine retrieves evicted data pages from the first buffer pool in the managed memory and stores the evicted data pages in the local SSD storage.

[0035] In this context, evicted data pages are those removed from the first buffer pool of the storage engine using an eviction algorithm. An eviction algorithm is an algorithm used to manage data items in the cache or memory. When cache or memory space is insufficient to accommodate new data, the eviction algorithm determines which older data should be removed to make room for new data. For example, the Least Recently Used (LRU) algorithm evicts data that has not been accessed for the longest time. This algorithm assumes that data not recently accessed is unlikely to be accessed in the future, and therefore prioritizes its eviction. The number of evicted data pages can be one or more.

[0036] The first buffer pool is located in the memory of the storage engine. Local SSD storage can be a disk used to store temporary files on the current physical machine, or it can be other disks, such as Non-Volatile Memory Express (NVME SSD).

[0037] Step S302: If an access request for the target data is received and the target data is not found in the first buffer pool, then the target data is read from the evicted data page in the local SSD storage.

[0038] If an access request for the target data is received, the system first queries the first buffer pool. If the target data is not found, it queries the local SSD storage. If the target data is found, the data is read.

[0039] Step S303: If the target data is not read from the local SSD storage, then the target data is read from the cloud storage.

[0040] If the target data is not found in local SSD storage, it will be queried and retrieved from cloud storage. Cloud storage is a data storage model that allows users to store data on servers in remote data centers, which users can access via the network.

[0041] The cloud database data processing method provided in this application embodiment includes a cloud database comprising compute nodes and storage nodes. The storage nodes include cloud storage and local solid-state drives (SSDs) of the instance where the compute node resides. Each compute node includes a storage engine that manages a first buffer pool in memory. The storage engine retrieves evicted data pages from the first buffer pool and stores them in the local SSD storage. If an access request for target data is received, and the target data is not found in the first buffer pool, the target data is read from the evicted data pages in the local SSD storage. If the target data is not found in the local SSD storage, the target data is read from the cloud storage. In this embodiment, storing evicted data pages from the first buffer pool in the storage engine in the local SSD storage, and then reading data from the cloud storage only when the data is not found in the local SSD storage, avoids directly reading data from the cloud when the data is not found in the first buffer pool, reducing the number of cloud accesses, lowering IO latency, and improving the performance of the database system. The specific implementation methods of the above steps are described below through various implementation methods:

[0042] In one implementation, before storing the evicted data pages to local SSD storage, the data processing method further includes: creating a second buffer pool and using the second buffer pool to cache the evicted data pages; storing the evicted data pages to local SSD storage includes: using a background thread to persistently store the evicted data pages to local SSD storage.

[0043] For example, the second buffer pool includes data structures such as an LRU list, a flush list, a free list, a memory pool (mem pool), and a data page hash table. These data structures are located in memory, and the data files are located in local SSD storage. The second buffer pool first caches evicted data pages, and then uses a background thread to persist the evicted data pages cached in the second buffer pool to local SSD storage.

[0044] In practical applications, the data structures of the second buffer pool can occupy less space than the first buffer pool, thus taking up less memory.

[0045] In this embodiment, a background thread is used to refresh the evicted data page to the local SSD storage, which does not affect the caching and reading of the evicted data page in the foreground, thus improving data processing efficiency.

[0046] In one implementation, the background thread includes a coordinating thread and a worker thread; the background thread is used to persistently store the evicted data pages in the local SSD storage, including: using the coordinating thread to determine the evicted data pages to be stored in the evicted data pages, and using the worker thread to persistently store the evicted data pages to be stored in the local SSD storage.

[0047] In practical applications, multiple background threads can be created to handle the process separately. These background threads include a coordination thread and multiple worker threads. The coordination thread can be used to determine which cached data pages need to be persisted and flush the evicted data pages to be stored to the local SSD storage. Alternatively, the coordination thread can be used to determine which cached data pages need to be persisted and multiple worker threads can be used to flush the evicted data pages to be stored to the local SSD storage.

[0048] In this embodiment, parallel processing using a coordination thread and worker threads can further improve the efficiency of data processing.

[0049] In one implementation, storing obsolete data pages in local SSD storage includes: dividing the obsolete data page into at least one data block, storing at least one data block in local SSD storage, and establishing a mapping relationship between the data block identifier corresponding to the at least one data block and the page identifier of the obsolete data page.

[0050] Among them, the evicted data page is the clean data page. A clean data page means that the content of the data page in memory is consistent with the content on the disk, that is, the data page has not been modified since it was loaded into memory.

[0051] The evicted data pages are divided into data blocks. The size of the data block can be the same as the size of the evicted data page, for example, 16KB. A mapping relationship is established between the page identifier of the evicted data page and the data block identifier of the data block; for example, one page identifier corresponds to one data block identifier. The mapping relationship (space, page no) → m_page_no is stored in the data structure ebuf_block_t. (space, page no) represents the page identifier of the evicted data page, where space represents the tablespace identifier of the evicted data page, page no represents the data page sequence number, and m_page_no represents the data block identifier, that is, the page identifier of the data page corresponding to the data block.

[0052] In one implementation, reading target data from a vacated data page in local SSD storage includes: determining the target page identifier corresponding to the target data based on the access request; if the target page identifier matches the page identifier of the vacated data page, determining the target data block corresponding to the target page identifier based on the page identifier and mapping relationship of the vacated data page, and reading the target data from the target data block.

[0053] In practical applications, parsing the access request yields the page identifier of the target data carried in the request, i.e., the target page identifier. The cache pool is then searched based on the target page identifier. If not found, the local SSD storage is searched. If found, meaning the target page identifier matches the page identifier of an evicted data page already stored in the local SSD storage, the target data block is determined. This involves obtaining the data block identifier of the target data block based on the mapping relationship, and then searching for the location of the target data block in the local SSD storage based on the data block identifier, from which the target data is read.

[0054] When performing data queries, a hash function can be used to check if the hash value of the target page identifier exists in the list of hash values ​​of page identifiers of evicted data pages in the local SSD storage. Using hash functions for data storage and retrieval can improve the efficiency of data retrieval.

[0055] In this embodiment, evicted data pages in the buffer pool of the storage engine are stored in local SSD storage. When data needs to be read, the data is read from the local SSD storage first. If the data cannot be read from the local SSD storage, the data is then read from the cloud storage. This avoids directly reading data from the cloud when the data cannot be found in the first buffer pool, reduces the number of cloud accesses, reduces IO latency, and improves the performance of the database system.

[0056] In one implementation, if the target data is not found in the local SSD storage, the target data is read from the cloud storage, including: determining the target page identifier corresponding to the target data based on the access request; if the target page identifier does not match the page identifier of the evicted data page, the target data is read from the tablespace file in the cloud storage.

[0057] In practical applications, if the target data is not found in the local SSD storage, that is, the target page identifier does not match the page identifier of the evicted data page stored in the local SSD storage, the target data is read from the tablespace file in the cloud storage.

[0058] In a database management system, a tablespace is a storage space concept used to manage the physical storage of data in the database. A tablespace consists of one or more files that actually exist in the file system and can be directly managed by the database system.

[0059] In MySQL, especially when using the InnoDB storage engine, a tablespace can be a single file (usually a .ibd file) or a shared tablespace, where InnoDB table data and indexes are stored in a shared tablespace file.

[0060] In one implementation, the data processing method further includes: adding the data block identifier corresponding to the data block stored in the local SSD storage to the LRU data page list; if the position of the data block identifier corresponding to the data block stored in the local SSD storage in the LRU data page list exceeds a preset error range, then marking the data blocks within the preset error range and those exceeding the preset error range as old data blocks, and deleting the old data blocks if the amount of data in the local SSD storage exceeds a preset data amount threshold.

[0061] The size of the preset error zone can be set according to specific needs. After adding the data block identifier corresponding to the data block stored in the local SSD to the LRU data page list, if the position of the data block identifier corresponding to the data block stored in the local SSD in the LRU data page list exceeds the preset error zone, then the data blocks within the preset error zone and those exceeding the preset error zone are marked as old data blocks. That is, the status of the old flag of the data blocks within the preset error zone and those exceeding the preset error zone is updated to true or false. If the amount of data stored in the local SSD exceeds the preset data amount threshold, the old data blocks are deleted.

[0062] In this embodiment, the state of the data block is not modified every time a data block is added or removed. Instead, the state of the data block is modified only when the lower boundary of the error region is exceeded. This avoids the performance loss caused by frequently modifying the state of the data block.

[0063] In one implementation, the storage engine retrieves evicted data pages from a first buffer pool in the managed memory, including: if the amount of data in the first buffer pool exceeds a preset data amount threshold, then the data pages in the first buffer pool that meet preset conditions are determined as evicted data pages; wherein, the preset conditions include at least one of the following: the data page access time is more than a preset time threshold from the current time or the number of data page accesses is less than a preset number threshold.

[0064] In this embodiment, data pages whose access time is more than a preset time threshold from the current time can be discarded, i.e., data pages that have not been accessed recently. Alternatively, data pages whose access count is less than a preset number of accesses can be discarded, i.e., rarely accessed data pages. Or, the least recently accessed data page can be discarded.

[0065] For example, the LRU algorithm can be used to determine which data pages should be evicted. The LRU algorithm evicts data that has not been accessed for the longest time; this algorithm assumes that data that has not been accessed recently is unlikely to be accessed in the future, and therefore evicts it first. The number of data pages to be evicted can be one or more.

[0066] In one implementation, before storing the evicted data page in the local SSD storage, the data processing method further includes: determining, based on the page identifier of the evicted data page, that the evicted data page does not exist in the local SSD storage; or determining, based on the page identifier of the evicted data page, that the evicted data page exists in the local SSD storage, and that the log sequence numbers corresponding to the evicted data page and the existing data page do not match.

[0067] In practical applications, before storing evicted data pages to local SSD storage, to avoid duplicate storage, it's necessary to determine whether the evicted data page has already been stored or modified. If it's confirmed that it hasn't been stored, the current data page is stored. Alternatively, if it has been stored but modified, the modified new data page is stored. Whether the data page has been modified can be determined by whether the LSN is the same.

[0068] In a database management system, the LSN is a unique identifier used to identify database modification operations. The LSN is an incrementing value, typically a long integer. It is incremented whenever data in the database changes (such as during insert, update, or delete operations). Thus, each data page has an associated LSN, representing the version of the record on that page.

[0069] In one implementation, the data processing method further includes: using a token bucket algorithm to limit the rate of storing evicted data pages in local SSD storage or reading target data.

[0070] In practical applications, when writing or reading data blocks in local SSD storage, the token bucket algorithm is used to limit the number of read / write IOPS. Reads and writes are rate-limited independently, and the token count is refreshed when all tokens are exhausted.

[0071] In a storage system, the specific process of IOPS rate limiting using the token bucket algorithm is as follows: Setting rate limiting parameters: Determine the maximum allowed IOPS per second and set the token bucket capacity. The capacity can be set to be equal to or greater than the number of tokens generated per second, as needed. Initializing the token bucket: Initially, the bucket can be empty or pre-filled with some tokens. Generating tokens: Generate tokens at a constant rate and add them to the bucket until the bucket capacity is reached. Handling I / O requests: When a read / write request arrives, check if there are enough tokens in the bucket. If so, the request is allowed to proceed, and the corresponding number of tokens are removed from the bucket. If not, the request waits. Adjusting the strategy: Adjust the token generation rate or the token bucket capacity based on actual I / O performance and demand. In this way, the token bucket algorithm can smoothly handle read / write requests, avoid performance issues caused by sudden I / O operations, and ensure that the performance of the storage system meets expectations.

[0072] This application provides a cloud database data processing method. This method can be applied to computing devices, which may include servers, user terminals, etc. The cloud database includes computing nodes and storage nodes. The storage nodes include cloud storage and local solid-state drive (SSD) storage within the instance where the computing node resides. The computing nodes include a storage engine that manages a first buffer pool in memory. Figure 4 The diagram shown is a flowchart of a cloud database data processing method according to an embodiment of this application, including:

[0073] In step S401, the storage engine retrieves evicted data pages from the first buffer pool in the managed memory.

[0074] In this context, evicted data pages are those removed from the first buffer pool of the storage engine using an eviction algorithm. An eviction algorithm is an algorithm used to manage data items in the cache or memory. When cache or memory space is insufficient to accommodate new data, the eviction algorithm determines which older data should be removed to make room for new data. For example, the Least Recently Used (LRU) algorithm evicts data that has not been accessed for the longest time. This algorithm assumes that data not recently accessed is unlikely to be accessed in the future, and therefore prioritizes its eviction. The number of evicted data pages can be one or more.

[0075] Step S402: Create a second buffer pool and use the second buffer pool to cache evicted data pages.

[0076] The second buffer pool first caches evicted data pages, and then uses a background thread to persistently store the evicted data pages cached in the second buffer pool to local SSD storage. In practical applications, the size of the second buffer pool can be smaller than the size of the first buffer pool, occupying less space in memory.

[0077] Step S403: Create background threads, which include a coordinating thread and worker threads. The coordinating thread determines the data pages to be evicted from the evicted data pages, and the worker threads persist the evicted data pages to local SSD storage.

[0078] In practical applications, multiple background threads can be created to handle the data processing. These background threads include a coordination thread and multiple worker threads. The coordination thread determines which cached data pages need to be persisted, while the worker threads flush the evicted data pages to the local SSD storage. In this embodiment, parallel processing using the coordination thread and worker threads further improves data processing efficiency.

[0079] Step S404: If an access request for target data is received, and the target data is not found in the first buffer pool and the second buffer pool, then the target data is read from the evicted data page in the local SSD storage.

[0080] If an access request for the target data is received, the system first queries the first and second buffer pools. If the target data is not found, it queries the local SSD storage. If the target data is found, the data is read.

[0081] Step S405: If the target data is not read from the local SSD storage, then read the target data from the cloud storage.

[0082] If the target data is not found in local SSD storage, it will be queried and retrieved from cloud storage. Cloud storage is a data storage model that allows users to store data on servers in remote data centers, which users can access via the network.

[0083] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a cloud database data processing device. The cloud database includes compute nodes and storage nodes. The storage nodes include cloud storage and local solid-state drive (SSD) storage of the instance where the compute node resides. The compute node includes a storage engine, which manages a first buffer pool in memory; such as Figure 5 The diagram shown is a structural block diagram of a cloud database data processing apparatus according to an embodiment of this application. The apparatus includes:

[0084] The data storage module 501 is used by the storage engine to retrieve evicted data pages from the first buffer pool in the managed memory and store the evicted data pages to the local SSD storage.

[0085] The first reading module 502 is used to read the target data from the evicted data page in the local SSD storage if it receives an access request for the target data and the target data is not found in the first buffer pool.

[0086] The second reading module 503 is used to read the target data from cloud storage if the target data is not read from the local SSD storage.

[0087] The data processing apparatus provided in this application embodiment includes a cloud database comprising compute nodes and storage nodes. The storage nodes include cloud storage and local solid-state drives (SSDs) of the instance where the compute node resides. Each compute node includes a storage engine that manages a first buffer pool in memory. The storage engine retrieves evicted data pages from the first buffer pool and stores them in the local SSD storage. If an access request for target data is received, and the target data is not found in the first buffer pool, the target data is read from the evicted data pages in the local SSD storage. If the target data is not found in the local SSD storage, it is read from the cloud storage. In this embodiment, storing evicted data pages from the first buffer pool in the storage engine in the local SSD storage, and then reading data from the cloud storage only when data is not found in the local SSD storage, avoids directly reading data from the cloud when data is not found in the first buffer pool, reducing the number of cloud accesses, lowering IO latency, and improving the performance of the database system.

[0088] In one implementation, the apparatus is further configured to: create a second buffer pool before storing the evicted data page in the local SSD storage, and use the second buffer pool to cache the evicted data page; when storing the evicted data page in the local SSD storage, the data storage module 501 is configured to: persistently store the evicted data page in the local SSD storage using a background thread.

[0089] In one implementation, the background thread includes a coordinating thread and a worker thread; when the data storage module 501 uses the background thread to persistently store the evicted data page to the local SSD storage, it is used to: use the coordinating thread to determine the evicted data page to be stored in the evicted data page, and use the worker thread to persistently store the evicted data page to be stored to the local SSD storage.

[0090] In one implementation, when storing obsolete data pages to local SSD storage, the data storage module 501 is used to: divide the obsolete data page into at least one data block, store at least one data block in local SSD storage, and establish a mapping relationship between the data block identifier corresponding to at least one data block and the page identifier of the obsolete data page.

[0091] In one implementation, when the first reading module 502 reads target data from the evicted data page in the local SSD storage, it is used to: determine the target page identifier corresponding to the target data based on the access request; if the target page identifier matches the page identifier of the evicted data page, determine the target data block corresponding to the target page identifier based on the page identifier and mapping relationship of the evicted data page, and read the target data from the target data block.

[0092] In one implementation, the device is further configured to: add the data block identifier corresponding to the data block stored in the local SSD storage to the least recently used data page list; if the position of the data block identifier corresponding to the data block stored in the local SSD storage in the least recently used data page list exceeds a preset error range, then mark the data blocks within the preset error range and those exceeding the preset error range as old data blocks, and delete the old data blocks if the amount of data in the local SSD storage exceeds a preset data amount threshold.

[0093] In one implementation, the second reading module 503 is used to: determine the target page identifier corresponding to the target data based on the access request; if the target page identifier and the page identifier of the evicted data page do not match, then read the target data from the tablespace file in the cloud storage.

[0094] In one implementation, when the storage engine retrieves evicted data pages from the first buffer pool in the managed memory, the data storage module 501 is configured to: if the amount of data in the first buffer pool exceeds a preset data amount threshold, determine the data pages in the first buffer pool that meet the preset conditions as evicted data pages; wherein, the preset conditions include at least one of the following: the data page access time is more than a preset time threshold from the current time or the number of data page accesses is less than a preset number threshold.

[0095] In one implementation, the apparatus is further configured to: determine, based on the page identifier of the evicted data page, that the evicted data page does not exist in the local SSD storage before storing the evicted data page in the local SSD storage; or determine, based on the page identifier of the evicted data page, that the evicted data page exists in the local SSD storage, and the log sequence numbers corresponding to the evicted data page and the existing data page do not match.

[0096] In one implementation, the device is also used to: use a token bucket algorithm to rate limit the storage of evicted data pages in local SSD storage or the reading of target data.

[0097] The functions of each module in the embodiments of this application can be found in the corresponding descriptions in the above methods, and they have corresponding beneficial effects, which will not be repeated here.

[0098] Figure 6 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 6 As shown, the electronic device includes a memory 610 and a processor 620. The memory 610 stores a computer program that can run on the processor 620. When the processor 620 executes the computer program, it implements the methods described in the above embodiments. The number of memories 610 and processors 620 can be one or more.

[0099] The electronic device also includes:

[0100] The communication interface 630 is used to communicate with external devices and perform data exchange and transmission.

[0101] If the memory 610, processor 620, and communication interface 630 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0102] Optionally, in a specific implementation, if the memory 610, processor 620, and communication interface 630 are integrated on a single chip, then the memory 610, processor 620, and communication interface 630 can communicate with each other through an internal interface.

[0103] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0104] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the method provided in this application.

[0105] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0106] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0107] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0108] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0109] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0110] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0111] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0112] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0113] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0114] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0116] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A cloud database data processing method, characterized in that, The method includes: The cloud database includes compute nodes and storage nodes. The storage nodes include cloud storage and local SSD storage of the instance where the compute node is located. The compute node includes a storage engine, which manages a first buffer pool in memory. The storage engine retrieves evicted data pages from the first buffer pool in the managed memory and stores the evicted data pages in the local SSD storage; wherein, storing the evicted data pages in the local SSD storage includes: dividing the evicted data pages into at least one data block and storing the at least one data block in the local SSD storage; The data block identifier corresponding to the data block stored in the local SSD storage is added to the least recently used data page list; if the position of the data block identifier corresponding to the data block stored in the local SSD storage in the least recently used data page list exceeds the lower boundary of a preset error region, the data blocks within the preset error region and exceeding the lower boundary of the preset error region are marked as old data blocks, and the old data blocks are deleted if the amount of data in the local SSD storage exceeds a preset data amount threshold. If an access request for target data is received, and the target data is not found in the first buffer pool, then the target data is read from the evicted data page in the local SSD storage; If the target data is not found in the local SSD storage, then the target data is read from the cloud storage.

2. The method according to claim 1, characterized in that, Before storing the evicted data pages to local SSD storage, the method further includes: Create a second buffer pool and use the second buffer pool to cache the evicted data pages; The step of storing the obsolete data pages to the local SSD storage includes: The evicted data pages are persistently stored in the local SSD storage using a background thread.

3. The method according to claim 2, characterized in that, The background thread includes a coordination thread and a worker thread; the step of using the background thread to persistently store the evicted data pages to the local SSD storage includes: The coordination thread determines the data pages to be stored from the evicted data pages, and the worker thread persists the evicted data pages to the local SSD storage.

4. The method according to claim 1, characterized in that, The method further includes: Establish a mapping relationship between the data block identifier corresponding to the at least one data block and the page identifier of the evicted data page.

5. The method according to claim 4, characterized in that, The step of reading the target data from the evicted data page in the local SSD storage includes: Based on the access request, determine the target page identifier corresponding to the target data; If the target page identifier matches the page identifier of the obsolete data page, then based on the page identifier of the obsolete data page and the mapping relationship, the target data block corresponding to the target page identifier is determined, and the target data is read from the target data block.

6. The method according to claim 4, characterized in that, If the target data is not found in the local SSD storage, then reading the target data from the cloud storage includes: Based on the access request, determine the target page identifier corresponding to the target data; If the target page identifier and the page identifier of the evicted data page do not match, the target data is read from the tablespace file in the cloud storage.

7. The method according to any one of claims 1-4, characterized in that, The storage engine retrieves evicted data pages from the first buffer pool in the managed memory, including: If the amount of data in the first buffer pool exceeds a preset data amount threshold, then the data pages in the first buffer pool that meet the preset conditions are determined as the evicted data pages; The preset conditions include at least one of the following: the data page access time is more than a preset time threshold from the current time or the number of data page accesses is less than a preset number threshold.

8. The method according to any one of claims 1-4, characterized in that, Before storing the evicted data pages to local SSD storage, the method further includes: Based on the page identifier of the evicted data page, it is determined that the evicted data page does not exist in the local SSD storage; or Based on the page identifier of the evicted data page, it is determined that the evicted data page exists in the local SSD storage, and the log sequence numbers corresponding to the evicted data page and the existing data page do not match.

9. The method according to any one of claims 1-4, characterized in that, The method further includes: The token bucket algorithm is used to limit the rate of storing the evicted data pages in the local SSD storage or reading the target data.

10. A cloud database data processing device, characterized in that, The device includes: The cloud database includes compute nodes and storage nodes. The storage nodes include cloud storage and local solid-state drive (SSD) storage of the instance where the compute node is located. The compute node includes a storage engine, which manages a first buffer pool in memory. A data storage module is used by the storage engine to retrieve evicted data pages from the first buffer pool in the managed memory and store the evicted data pages in local SSD storage; wherein, storing the evicted data pages in local SSD storage includes: dividing the evicted data pages into at least one data block, storing the at least one data block in local SSD storage; adding the data block identifier corresponding to the data block stored in local SSD storage to a least recently used data page list; if the position of the data block identifier corresponding to the data block stored in local SSD storage in the least recently used data page list exceeds the lower boundary of a preset error region, then marking the data blocks within the preset error region and exceeding the lower boundary of the preset error region as old data blocks, and deleting the old data blocks if the amount of data in local SSD storage exceeds a preset data amount threshold; The first reading module is configured to read the target data from the evicted data page in the local SSD storage if it receives an access request for the target data and the target data is not found in the first buffer pool. The second reading module is used to read the target data from the cloud storage if the target data is not read from the local SSD storage.

11. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

Images