Data reading and writing method, device and equipment

Abstract

The invention provides a data reading and writing method, device and equipment, and the method comprises the steps: receiving a data reading and writing request which carries a target key; performing read-write processing on the target key in a cache to obtain a cache processing result; searching a target data segment corresponding to the target key in a disk based on the cache processing result; the target data segment comprises a preset number of data pages continuously stored on the disk, each data page comprises data corresponding to keys in different ranges, and the keys in different ranges corresponding to each data page are mutually continuous; determining a target data page corresponding to the target key in a preset number of data pages continuously stored on the disk and included in the target data segment; and performing read-write processing on the to-be-read-write data corresponding to the target key based on the target data page. According to the embodiment of the invention, the data reading and writing efficiency can be improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a data reading and writing method, apparatus, and device. Background Technology

[0002] Existing key-value stores (such as RocksDB and LevelDB) use Log Merge Trees (LSMs) to optimize write performance. LSMs are a data structure used to manage persistent data and are typically suitable for write-intensive applications.

[0003] However, while LSMs make writing efficient, it comes at the cost of reading performance. This is because LSMs distribute data across multiple levels of structures (multiple files in memory and on disk), and reading data requires searching through these structures, resulting in relatively low read performance for LSMs.

[0004] Therefore, it is particularly important to provide a data read / write method that can simultaneously improve data write performance and data read performance. Summary of the Invention

[0005] To address the technical problem that while LSMs design makes writing efficient, it comes at the cost of reading performance, this application provides a data read / write method, apparatus, and device that can simultaneously improve both data write and data read performance.

[0006] On the one hand, this application proposes a data read / write method, the method comprising:

[0007] Receive a data read / write request, wherein the data read / write request carries a target key;

[0008] The target key is read and written in the cache to obtain the cache processing result;

[0009] Based on the cache processing result, the target data segment corresponding to the target key is searched in the disk; the target data segment includes a preset number of data pages stored contiguously on the disk, each data page includes data corresponding to a different range of keys, and the different ranges of keys corresponding to each data page are contiguous to each other;

[0010] Among the predetermined number of data pages continuously stored on the disk included in the target data segment, the target data page corresponding to the target key is determined;

[0011] Based on the target data page, read and write operations are performed on the data to be read and written corresponding to the target key.

[0012] On the other hand, this application proposes a data read / write device, the device comprising:

[0013] A read / write request receiving module is used to receive data read / write requests, wherein the data read / write request carries a target key;

[0014] The first read / write processing module is used to perform read / write processing on the target key in the cache to obtain the cache processing result;

[0015] The data segment lookup module is used to search for the target data segment corresponding to the target key in the disk based on the cache processing result; the target data segment includes a preset number of data pages stored contiguously on the disk, each data page includes data corresponding to a different range of keys, and the different ranges of keys corresponding to each data page are contiguous to each other;

[0016] The target data page determination module is used to determine the target data page corresponding to the target key from a preset number of data pages continuously stored on the disk included in the target data segment;

[0017] The second read / write processing module is used to perform read / write processing on the data to be read / written corresponding to the target key based on the target data page.

[0018] On the other hand, this application proposes an electronic device for reading and writing data, the electronic device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the data reading and writing method as described above.

[0019] On the other hand, this application proposes a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the data read / write method as described above.

[0020] On the other hand, this application proposes a computer program product, including a computer program that, when executed by a processor, implements the data read / write method as described above.

[0021] This application proposes a data read / write method, apparatus, and device. The method involves receiving a data read / write request carrying a target key, performing read / write processing on the target key in a cache to obtain a cached processing result, searching for a target data segment corresponding to the target key on the disk based on the cached processing result, identifying the target data page corresponding to the target key among a predetermined number of contiguous data pages stored on the disk within the target data segment, and performing read / write processing on the data to be read / written corresponding to the target key based on the target data page. This approach achieves two key benefits: firstly, processing the data read / write request in the cache beforehand for fast read and write operations; and secondly, enabling data to be read / written quickly. Data page grouping technology ensures that the target data segment comprises a predetermined number of contiguous data pages stored on the disk. Each data page contains data corresponding to a key within a different range, and the keys within each data page are contiguous. In other words, the key range is divided into several groups by data page grouping technology, and each group is stored contiguously on the disk. This allows related data pages to be stored contiguously on the disk. Consequently, when performing data read and write operations on the disk based on the cache processing results, the entire data segment can be scanned from the disk at once, rather than accessing each data page individually, thus improving data page scanning efficiency and further enhancing data read and write efficiency. Attached Figure Description

[0022] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating the implementation environment of a data read / write method according to an exemplary embodiment.

[0024] Figure 2 This is a flowchart illustrating a data read / write method according to an exemplary embodiment. Figure 1 .

[0025] Figure 3 This is a flowchart illustrating a data read / write method according to an exemplary embodiment. Figure 2 .

[0026] Figure 4 This is a flowchart illustrating a data read / write method according to an exemplary embodiment. Figure 3 .

[0027] Figure 5 This is a block diagram illustrating a data read / write device according to an exemplary embodiment.

[0028] Figure 6 This is a hardware structure block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0029] Existing key-value stores (such as RocksDB and LevelDB) typically use Log Merge Trees (LSMs) for construction. An LSM write operation first writes to the log, then to memory, and finally flushes to a file on disk. These files are periodically compressed and merged in the background, providing excellent write performance. LSMs leverage the fact that sequential disk writes are far more efficient than random writes, ensuring that all disk write operations are sequential.

[0030] However, while LSMs make writing efficient, this comes at the cost of read performance because records may exist in multiple locations on disk. Therefore, when reading data, since LSMs distribute data across multiple levels of structure (multiple files in memory and on disk), data needs to be searched within these structures. This results in relatively low read performance for LSMs, especially in random read scenarios.

[0031] Based on this, embodiments of this application provide a novel key-value pair storage system to offer superior data read and write performance. Specifically, embodiments of this application introduce two core concepts that work together to improve the performance of key-value storage on modern storage devices:

[0032] 1. Caching:

[0033] The cache stores frequently accessed records for fast read and write operations. When a record is modified, it is first updated in the cache, and then the corresponding disk data page is updated when the cache entry is evicted. This mechanism reduces the number of disk writes, thereby lowering the write amplification factor.

[0034] The cache uses a clock algorithm to evict entries, prioritizing the eviction of entries that have not been modified recently, in order to reduce input / output (I / O) overhead.

[0035] 2. Data page grouping:

[0036] The key range is divided into several groups, each group stored contiguously on disk. This organizes data pages on disk into logically adjacent groups, ensuring that these data pages are also physically contiguous. In other words, range scan performance is optimized by storing logically adjacent key range data pages in contiguous locations on disk. This layout allows range scans to be performed through physically contiguous reads, rather than scattered random reads. In-memory indexes only need to store the boundaries, linear model, and disk location of each data segment, thus reducing the memory footprint of the in-memory index.

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0039] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0040] Figure 1 This is a schematic diagram illustrating an implementation environment for a data read / write method according to an exemplary embodiment. For example... Figure 1 As shown, the implementation environment may include at least a client 01 and a server 02. The client 01 and the server 02 may be directly or indirectly connected through wired or wireless communication. This embodiment of the application does not impose any limitations on this.

[0041] Specifically, server 02 can be used to receive data read / write requests, perform read / write processing on the target key in the cache to obtain a cache processing result; based on the cache processing result, search for the target data segment corresponding to the target key in the disk; determine the target data page corresponding to the target key among a preset number of data pages continuously stored on the disk included in the target data segment; and perform read / write processing on the data to be read / written corresponding to the target key based on the target data page. Optionally, server 02 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0042] Specifically, the client 01 can be used to display the object recognition results. The client 01 can be, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc.

[0043] It should be noted that, Figure 1 This is just one example. Other implementation environments may also be included in other scenarios.

[0044] It should be noted that in the specific implementation of this application, user information, such as data to be read and written, is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0045] Figure 2 This is a flowchart illustrating a data read / write method according to an exemplary embodiment. Figure 1 This method can be used for Figure 1 In the implementation environment described in this specification, the method operation steps are as illustrated in the embodiments or flowcharts. However, based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown in the embodiments or drawings... Figure 2 As shown, the method may include:

[0046] S101. Receive a data read / write request, which carries the target key.

[0047] Optionally, when a client account wants to read or write data, it can send a data read / write request to the server, which carries the target key of the data to be read or written. This target key can be used to uniquely identify or represent the data to be read or written; for example, the target key can be the client account's identity information (ID).

[0048] In one implementation, the data read / write request is a data read request, the data to be read / written is the data to be read, and the target key is the target key of the data to be read. In another implementation, the data read / write request is a data write request, the data to be read / written is the data to be written, and the target key is the target key of the data to be written.

[0049] S103. Perform read and write operations on the target key in the cache to obtain the cache processing result.

[0050] Optionally, the cache is an in-memory data structure that employs a Recently and Most Frequently Used (RMFU) strategy. Recently accessed records and / or most frequently accessed records are cached in memory. Recently accessed records refer to cache entries whose access time and current time differ from a preset time threshold. Most frequently accessed records refer to cache entries whose access frequency exceeds a first preset frequency threshold. It should be noted that the preset time threshold and the first preset frequency threshold can be set according to actual business needs and are not specifically limited. Furthermore, when insert, update, or delete operations are performed, the corresponding records are stored in the cache and assigned a priority. The cache assigns a priority to each cache entry, and this priority increases as the record is accessed more frequently. Each access increments the priority by 1.

[0051] After the server receives a data read / write request, it can first perform read / write processing on the target key in the cache to obtain the cache processing result.

[0052] In one implementation, for a data read request, the data to be read / written corresponding to the target key can be searched in the cache to obtain a cache processing result. This cache processing result can be either: the cache does not contain the data to be read / written, or the cache does contain the data to be read / written. If the cache processing result is that the cache does not contain the data to be read / written, the data can be directly returned to the client from the cache. If the cache processing result is that the cache contains the data to be read / written, the location of the data to be read / written can be searched on the disk, i.e., the subsequent steps S105-S109 can be executed.

[0053] In another implementation, for data write requests, to ensure consistency after a crash, a write-ahead log (WAL) is used to record critical operations during the write process. This allows the system to be restored to a consistent state after a crash. When a write operation (insert, update, or delete) occurs, an entry is first created or updated in the cache. If the cache is not full, the data to be read / written corresponding to the target key is directly written to the cache. If the cache is full, a clock algorithm can be used to evict a cache entry, and the data to be read / written corresponding to the target key is written to the cache. To maintain consistency between cached data and disk data, cached entries (records that have been updated but not yet written to disk, i.e., the data to be read / written corresponding to the target key) are periodically written to disk in the background, i.e., the subsequent steps S105-S109 are executed.

[0054] S105. Based on the cache processing result, search the disk for the target data segment corresponding to the target key; the target data segment includes a preset number of data pages stored contiguously on the disk; each data page includes data corresponding to a different range of keys, and the different ranges of keys corresponding to each data page are contiguous.

[0055] S107. Among the preset number of data pages that are continuously stored on the disk and included in the target data segment, determine the target data page that corresponds to the target key.

[0056] Optionally, all data pages on the disk can be pre-grouped using data page grouping technology. The core idea of ​​data page grouping is to divide the key range into several groups, with each group stored contiguously on the disk according to the size of the corresponding key. This organizes the data pages on the disk into logically adjacent groups, ensuring that these data pages are also physically contiguous. In other words, by storing logically adjacent key range data pages in contiguous locations on the disk, range scan performance is optimized. Thus, when performing a range scan operation, this contiguous data page layout can be utilized to improve scan efficiency through sequential rather than random reads. Page grouping can also achieve a balance between small and large pages. In page grouping, small pages are maintained to reduce space usage and I / O operations, while large-scale contiguous data reads are achieved through page grouping. Furthermore, page grouping may perform differently on different datasets and workloads. For example, page grouping can work more effectively on datasets with many "linear regions" because it makes it easier to fit linear models.

[0057] In practical applications, without data page grouping, these keys can be scattered across multiple random locations on the disk, causing range scans to require many random read operations, which can significantly reduce performance.

[0058] For example, the layout before grouping data pages can be as follows:

[0059] Disk location: 0 1 2 3 4 5 6 7;

[0060] Key: {10,20,30,40,50,60,70,80};

[0061] In the example above, each disk block stores data corresponding to a key, and the keys are stored in a distributed manner. If it is necessary to scan the range of keys from 20 to 70, six random read operations would be required to retrieve these keys.

[0062] The layout after grouping data pages can be as follows:

[0063] Disk location: 0 1 2 3 4 5 6 7;

[0064] Keys: {10-20}{21-40}{41-60}{61-80};

[0065] Data page grouping technology was used to divide the key ranges into several groups: {10-20}, {21-40}, {41-60}, and {61-80}. Each group corresponds to a data page, and each data page contains data corresponding to different ranges of keys. The keys within each data page are contiguous, ensuring that each group is stored contiguously on disk in order of the size of its corresponding keys. It should be noted that "the keys within each data page are contiguous" means that the values ​​of the keys within each range are contiguous. For example, if a data page contains keys 20, 30, and 40, these keys can be considered numerically contiguous.

[0066] For the group {10-20}, the corresponding key range is 10-20, and the keys within this range are consecutive within the group; for the group {21-40}, the corresponding key range is 21-40, and the keys within this range are consecutive within the group; for the group {41-60}, the corresponding key range is 41-60, and the keys within this range are consecutive within the group; for the group {61-80}, the corresponding key range is 61-80, and the keys within this range are consecutive within the group. Since the key ranges 10-20, 21-40, 41-60, and 61-80 are consecutive, the groups are stored on the disk in sequential order of their corresponding keys. For example, the groups corresponding to the key range 61-80 can be stored at the beginning of the disk and the groups corresponding to the key range 10-20 can be stored at the end of the disk, with the corresponding keys arranged from largest to smallest.

[0067] Now, if you want to perform the same range scan (from 20 to 70), you only need to read the three groups {21-40}, {41-60}, and {61-80}, obtaining all relevant keys sequentially. This reduces the original six random reads to just three sequential reads, significantly improving scanning efficiency.

[0068] The four data pages obtained in the example above can be:

[0069] Data page 1: Keys 10-20;

[0070] Data page 2: Keys 21-40;

[0071] Data page 3: Keys 41-60;

[0072] Data page 4: Keys 61-80.

[0073] If the system frequently needs to perform range scans, such as querying all data with keys between 30 and 50, randomly accessing each data page will be inefficient. To optimize this range scan, we can first create data segments by co-locating the data pages generated during each reorganization operation. Since each data page reorganization changes the data pages, adjacent data pages can be merged into a data segment based on their adjacency. It's important to note that a data segment represents a region on the disk. For example, after reorganization, we get data page 1, data page 2, data page 3, and data page 4. We create data segment 1 based on the adjacent data pages 1 and 2, establishing a mapping between data pages 1, 2, and segment 1. Similarly, we create data segment 2 based on the adjacent data pages 3 and 4, establishing a mapping between them. By grouping these data pages, related data pages are stored contiguously on the disk. This allows the entire data segment to be read at once during a range scan, instead of accessing each data page individually. After grouping, we get:

[0074] Data segment 1: Keys 10-40 (Data page 1, Data page 2);

[0075] Data segment 2: Keys 41-80 (Data page 3, Data page 4).

[0076] Therefore, each data segment includes a set number of data pages stored contiguously on the disk. Each data page contains data corresponding to keys within a different range, and the keys within each data page belonging to the same data segment are consecutive. For example, data segment 1 includes data page 1 and data page 2. Data page 1 contains data corresponding to keys 10-20, and data page 2 contains data corresponding to keys 21-40. The keys corresponding to data page 1 (i.e., keys 10-20) are consecutive in data page 1, and the keys corresponding to data page 2 (i.e., keys 21-40) are consecutive in data page 2. Because keys 10-20 and keys 21-40 are consecutive, data page 1 and data page 2 can be stored contiguously on the disk according to the size order of their corresponding keys. For example, data segment 2 includes data page 3 and data page 4. Data page 3 includes data corresponding to keys 41-60, and data page 4 includes data corresponding to keys 61-80. The keys corresponding to data page 3 (i.e., keys 41-60) are consecutive, and the keys corresponding to data page 4 (i.e., keys 61-80) are consecutive. Since keys 41-60 and keys 61-80 are consecutive, data page 3 and data page 4 can be stored contiguously on the disk according to the size order of their corresponding keys.

[0077] It should be noted that the above grouping results not only reflect the mapping relationship between data pages and data segments (for example, data page 1 and data page 2 correspond to data segment 1, and data page 3 and data page 4 correspond to data segment 2), but also reflect the mapping relationship between keys and data segments (for example, keys 10-40 correspond to data segment 1, and keys 41-80 correspond to data segment 2).

[0078] Therefore, after obtaining the cache processing result, we can first search for the target data segment corresponding to the target key on the disk based on the mapping relationship between the key and the data segment. Here, the target data segment is a region on the disk. Continuing with the example above, if the target key is 30, the target data segment is data segment 1; if the target key is 50, the target data segment is data segment 2.

[0079] Next, based on the mapping relationship between keys and data pages, the target data page corresponding to the target key is determined from a preset number of contiguous data pages stored on the disk within the target data segment. Continuing with the example above, if the target key is 30, then the target data segment is data segment 1, and the target data page is data page 2. If the target key is 50, then the target data segment is data segment 2, and the target data page is data page 3. Therefore, when performing a scan range, the entire data segment can be read at once, instead of accessing each data page individually, effectively improving data read and write efficiency.

[0080] S109. Perform read and write processing on the data to be read and written corresponding to the target key based on the target data page.

[0081] In this embodiment of the application, after determining that the target data page has been obtained, the data to be read and written corresponding to the target key can be processed on the target data page.

[0082] In one implementation, when a data write request is received, after determining that the target data page has been obtained, the data to be read or written corresponding to the target key can be written to the target data page.

[0083] In another implementation, for a data read request, after determining that the target data page has been obtained, the data to be read and written corresponding to the target key can be found in the target data page, and the found data to be read and written can be sent to the client.

[0084] Therefore, on the one hand, upon receiving a data read / write request, it first processes the data in the cache to enable fast read and write operations. On the other hand, it utilizes data page grouping technology to ensure that the target data segment comprises a predetermined number of contiguous data pages stored on the disk. Each data page contains data corresponding to a different range of keys, and the keys corresponding to different ranges of each data page are contiguous. In other words, the key ranges are divided into several groups by data page grouping technology, and each group is stored contiguously on the disk. This ensures that related data pages are stored contiguously on the disk. Consequently, when performing data read and write operations on the disk based on the cache processing results, the entire data segment can be scanned from the disk at once, instead of accessing each data page individually, improving data page scanning efficiency and further enhancing data read and write efficiency. It is evident that this embodiment, through the combined effect of caching technology and disk page grouping technology, can improve data write performance while retaining its advantages in data read operations.

[0085] The following explanation uses a data read / write request as an example to illustrate the above data read / write method.

[0086] Figure 3 This is a flowchart illustrating a data read / write method according to an exemplary embodiment. Figure 2 ,like Figure 3 As shown, in an optional embodiment, in step S103 above, the reading and writing process of the target key in the cache to obtain the cache processing result may include:

[0087] S1031-1. When the data read / write request is a data write request, determine whether the remaining storage space of the cache is less than a first preset storage threshold.

[0088] S1031-3. When it is determined that the remaining storage space of the cache is less than the first preset storage threshold, the target cache item is determined in the cache based on the clock algorithm, the target cache item is deleted from the cache, and the data to be read and written is written to the cache to obtain the cache processing result.

[0089] S1031-5. If it is determined that the remaining storage space of the cache is greater than or equal to the first preset storage threshold, the data to be read and written is written to the cache to obtain the cache processing result.

[0090] The cache contains cache items whose access time and current time difference is less than a preset time threshold, and / or cache items whose access frequency is greater than a first preset frequency threshold. The target cache item is a cache item whose access frequency is greater than the first preset frequency threshold and less than the second preset frequency threshold.

[0091] In this embodiment, upon receiving a data write request from the client, the server can first determine whether the cache is full. In one approach, it can be determined whether the remaining storage space of the cache is less than a first preset storage threshold. If so, the cache is considered full; otherwise, the cache is not full. This first preset storage threshold can be set according to actual business needs and is not specifically limited. In another approach, it can be determined whether the storage capacity of the cached items already stored in the cache exceeds the maximum storage capacity of the cache. If so, the cache is considered full; otherwise, the cache is not full.

[0092] If the remaining storage space in the cache is determined to be less than or equal to a first preset storage threshold, the data to be read or written is directly written to the cache, and a priority is assigned to this data to obtain the cache processing result. To ensure the consistency between cached data and operational data, data written to the cache can be written to the disk periodically.

[0093] If the remaining storage space in the cache is determined to be less than a first preset storage threshold, a clock algorithm can be used to determine the target cache items to be evicted. The clock algorithm is used for cache eviction, especially when cache space is insufficient to accommodate all items, requiring a decision on which items should be removed to free up space. This algorithm is an implementation of the "Least Recently Used" (LRU) strategy, based on the simple assumption that if data has been recently accessed, it is more likely to be accessed again in the future. In determining the cache items to be evicted, the clock algorithm skips higher-priority cache items, prioritizes evicting lower-priority cache items, writes the data to be read / written into the cache, assigns a priority to the data, and obtains the cache processing result.

[0094] Since the priority of cached items is determined by their access frequency—higher access frequency means higher priority and less likely eviction, and lower access frequency means lower priority and more likely eviction—the target cached item can be a cached item whose access frequency is greater than a first preset frequency threshold but less than a second preset frequency threshold. In other words, the target cached item is the least frequently accessed cached item among the most recently used cached items. It should be noted that the second preset frequency threshold can be set according to the actual needs of the business users and is not specifically limited thereto.

[0095] Therefore, during the data writing process, the data can be processed in the cache first. If the cache is not full, the data to be processed can be written directly to the cache so that it can be written to the disk at regular intervals, thereby improving the data writing efficiency. If the cache is full, a clock algorithm is used to select a cache item with a lower access frequency for eviction and write the data to be read and written to the cache, thereby reducing I / O overhead.

[0096] In an optional embodiment, the cache can be represented based on a circular data structure, where the data structure is a clock structure, and each cache item in the cache is located at a tick mark of the clock structure. Then, in steps S1031-3 above, the determination of the target cache item in the cache based on the clock algorithm includes:

[0097] Get the current tick mark that the pointer in the clock structure is currently pointing to.

[0098] If the cache entry corresponding to the current tick is a clean cache entry, then the cache entry corresponding to the current tick will be determined as the target cache entry.

[0099] If the cache entry corresponding to the current tick is not a clean cache entry, move the pointer clockwise to the next tick and re-establish the next tick as the current tick. Repeat the operation of determining whether the cache entry corresponding to the current tick is a clean cache entry until a clean cache entry is obtained, and then set the obtained clean cache entry as the target cache entry. Here, a clean cache entry is a cache entry that has not been modified, and a not-clean cache entry is a cache entry that has been modified but has not been written back to disk.

[0100] First, let's introduce the working principle of the clock algorithm:

[0101] Clocking algorithms typically use a circular data structure to represent the cache. This structure can be understood as a clock structure, with each cache item located at a tick mark on the clock structure.

[0102] When accessing data in the cache, if the data exists, it is moved to the latest position of the "clock" (i.e., the "current time" of the clock).

[0103] If there isn't enough space in the cache to store new data, the algorithm will find a candidate for eviction. This is typically the position the pointer is currently pointing to in a clock algorithm.

[0104] If the cache entry pointed to by the pointer is "dirty," that is, a non-clean cache entry (i.e., it has been modified but has not yet been written back to the storage device), then the pointer continues to move clockwise to find the next eviction candidate until a "clean" cache entry is found.

[0105] The pointer (or "clock hand") moves in a circular structure within the cache, checking each time whether the item it is currently pointing to can be evicted.

[0106] Based on the above working principle, when it is determined that the remaining storage space of the cache is less than the first preset storage threshold, the server can obtain the current scale that the pointer in the clock structure is pointing to, and further determine whether the cache item at the current scale is a clean cache item, that is, whether it is an unmodified cache item. If so, the cache item at the current scale is directly identified as the target cache item to be evicted. If not, it indicates that the cache item at the current scale is a "dirty" or non-clean cache item. The pointer can then be moved clockwise to the next scale, and the server can determine whether the cache item at the next scale is a clean cache item. If so, the cache item at the next scale is identified as the target cache item to be evicted. If not, the pointer is moved clockwise to the next scale again, and the server continues to determine whether the cache item at the next scale is a clean cache item. If so, the server identifies the cache item at the next scale as the target cache item to be evicted. If not, the pointer is moved clockwise to the next scale again, and the server continues to determine whether the cache item at the next scale is a clean cache item. This process continues until a clean cache item is found.

[0107] Non-clean cache entries are data that has been modified but not yet written to disk—that is, recently accessed data, or data with high access frequency. This type of data is likely to be accessed again in the future, so it should be retained for future user access. Clean cache entries, on the other hand, are data that has not been modified—that is, data that has not been accessed recently, or data with low access frequency. This type of data is less likely to be accessed in the future and can be evicted. Therefore, using a clock algorithm to determine clean cache entries as target cache entries for eviction not only improves the accuracy of determining target cache entries but also ensures that the eviction of target cache entries does not affect the user's data writing experience.

[0108] In an optional embodiment, continue as follows Figure 3 As shown, in step S105 above, the process of searching for the target data segment corresponding to the target key on the disk based on the cache processing result may include:

[0109] S1051-1. If the cache processing result is to write the data to be read or written to the cache, obtain the memory index.

[0110] S1051-3. Locate the target data segment to which the target key belongs in the disk using a memory index.

[0111] The memory index represents the mapping relationship between the key and the data segment, and the data segment corresponds to the storage area in the disk.

[0112] During the data page grouping process, a mapping relationship between keys and data segments needs to be established. This mapping relationship maps keys to specific data segments. Optionally, the mapping relationship between keys and data segments can be represented by a memory index, that is, the key is mapped to a specific data segment through the memory index. The memory index is a mapping structure that maps the smallest key of each data segment to a physical segment identifier on disk. The memory index only needs to store the boundaries, linear model, and disk location of each data segment. This means a smaller memory index can be maintained in memory, rather than maintaining an index entry for each individual data page, thus reducing the memory footprint of the memory index.

[0113] In this context, the boundary of a data segment refers to the starting key of the key corresponding to the data page of that data segment. For example, for the above data segment 1: keys 10-40 (data page 1, data page 2), data page 1 includes keys 10-20, and data page 2 includes keys 21-40. Therefore, the boundary of the data segment can be keys 10, 20, 21, and 40.

[0114] The linear model represents the mapping relationship between keys and data pages. For a data segment comprising multiple data pages (i.e., a multi-page data segment), this embodiment of the application can use this linear model to map keys to data pages. This linear model allows for quick location of the data page containing the data corresponding to the key without reading the entire data segment. For example, if data page 1 includes keys 10-20, data page 2 includes keys 21-40, data page 3 includes keys 41-60, and data page 4 includes keys 61-80, then a first mapping relationship can be established between data page 1 and keys 10-20, a second mapping relationship between data page 2 and keys 21-40, a third mapping relationship between data page 3 and keys 41-60, and a fourth mapping relationship between data page 4 and keys 61-80. The linear model is generated based on these first, second, third, and fourth mapping relationships.

[0115] The disk location refers to the data segment and the data pages included in the data segment.

[0116] In an optional embodiment, in step S107 above, determining the target data page corresponding to the target key from the preset number of data pages continuously stored on the disk included in the target data segment may include:

[0117] Using a linear model in the memory index, the target data page to which the target key belongs is determined from a predetermined number of contiguous data pages stored on the disk, including the target data segment; wherein, the linear model represents the mapping relationship between the key and the data page.

[0118] The linear model in memory indexes refers to the mapping relationship between keys and data pages. For data segments (i.e., multi-page data segments) that consist of a predetermined number of data pages stored contiguously on disk, this linear model can be used to map target keys to target data pages. This allows for quick location of the target data page containing the data corresponding to the target key without reading the entire data segment, thereby improving data write efficiency.

[0119] In an optional embodiment, continue as follows Figure 3 As shown, in step S109 above, the read / write processing of the data to be read / written corresponding to the target key based on the target data page may include:

[0120] S1091-1. If the remaining storage space in the target data page is greater than the second preset storage threshold, write the data to be read / written to the target data page.

[0121] S1091-3. If the storage space in the target data page is less than or equal to the second preset storage threshold, create a new data page and write the data to be read / written to the new data page.

[0122] Optionally, for a data write request, after determining that the target data page has been obtained, it can be further determined whether the target data page has sufficient storage space, that is, further determined whether the remaining storage space in the target data page is greater than the second preset storage threshold. If so, it is considered that the target data page has sufficient storage space and the data to be read and written can be written to the target data page. If not, it means that the target data page has sufficient storage space and a new data page can be created and the data to be read and written can be written to the new data page.

[0123] It should be noted that the second preset storage threshold can be set according to actual business needs, and no specific limitation is made here.

[0124] Therefore, data to be written will only be written if there is sufficient storage space on the target data page; otherwise, a new data page will be created for the data to be written. This not only ensures that the data to be written can be accurately written to the disk, but also ensures that the storage capacity of the data page does not exceed its storage limit, further improving data writing performance.

[0125] In an optional embodiment, the above method may further include:

[0126] If the number of data pages included in the target data segment exceeds a preset threshold, a new data segment is created.

[0127] Write a portion of the data pages from the target data segment into a new data segment.

[0128] Optionally, for a data write request, it can also be determined whether the number of data pages included in all data segments on the disk exceeds a preset threshold. If so, a reorganization process can be triggered. During the reorganization process, a new data segment will be created, and data from some data pages in the original data segment will be written into the new data segment. The target data segment is included in all data segments on the disk.

[0129] It should be noted that this preset quantity threshold can be set according to actual business needs, and no specific limitation is made.

[0130] For example, taking a data segment as the target data segment, since the keys corresponding to the data pages included in the target data segment are consecutive, if the number of data pages included in the target data segment is greater than a preset number threshold, a portion of the data pages can be determined from all the data pages included in the target data segment according to the size of the keys corresponding to the data pages, and these portions of data pages can be written into a new data segment in the order of the size of the keys corresponding to these portions of the data pages.

[0131] For example, if the target data segment includes 15 data pages and the preset quantity threshold is 10, then 5 data pages exceed the storage range of the target data segment. Then, based on the key size of the data included in these 15 data pages, the 5 data pages with the lowest key size can be found, and these 5 data pages can be written into the new data segment in sequence according to the corresponding key size.

[0132] Therefore, when the pages in a certain data segment are too full, a reorganization process can be triggered. This reorganization process writes the extra pages in the data segment, avoiding the poor data writing efficiency caused by the data segment being too full.

[0133] In an optional embodiment, the above method may further include:

[0134] In the case of a data read / write request being a data write request, after the data to be written is written to the target data page, the memory index can also be updated to reflect the newly written data page and any changed data segment information.

[0135] The following explanation uses a data read / write request as an example to illustrate the above data read / write method. This method may include:

[0136] 1. Write a log

[0137] To ensure consistency after a crash, a write-ahead log (WAL) is used to record critical operations during the write process. This way, after a system crash, these logs can be used to restore the system to a consistent state.

[0138] 2. Caching:

[0139] When a write operation (insert, update, or delete) occurs, an entry is first created or updated in the cache. The cache is an in-memory data structure used to store recently modified records.

[0140] 3. Cache eviction:

[0141] When the cache is full, a clock algorithm is used to select a record for eviction. Please refer to steps S1031-1 to S1031-5 above for details, which will not be repeated here.

[0142] 4. Memory index location:

[0143] For the key to be written, a memory index is used to determine where the data should be written to on the disk. The memory index contains information about the data segments, each corresponding to a region on the disk. Please refer to steps S1051-1 to S1051-3 above for details, which will not be repeated here.

[0144] 5. Page positioning and writing:

[0145] The corresponding data page is located using the memory index. For details, please refer to step S107 above, which will not be repeated here.

[0146] If the page has sufficient space, the update will be performed directly on the page. If the page space is insufficient, a new page will be created. For detailed steps, please refer to steps S1091-1 to S1091-3 above; they will not be repeated here.

[0147] 6. Page Reorganization:

[0148] When a data segment becomes too full (for example, a data segment has more than 10 pages), a reorganization process is triggered. During the reorganization process, a new data segment is created, and the records from the old pages are written to the new pages in sequence.

[0149] 7. Memory index update:

[0150] After the write operation is complete, the memory index is updated to reflect the newly written data pages and any changed data segment information.

[0151] The following explanation uses data download requests as an example to illustrate the above data read / write methods.

[0152] Figure 4 This is a flowchart illustrating a data read / write method according to an exemplary embodiment. Figure 3 ,like Figure 4As shown, in an optional embodiment, in step S103 above, the reading and writing process of the target key in the cache to obtain the cache processing result may include:

[0153] S1033-1. When the data read / write request is a data read request, determine whether the cache contains the data to be read / written corresponding to the target key.

[0154] S1033-3. If the cache does not contain the data to be read or written corresponding to the target key, determine that the cache processing result is the result in which the cache does not contain the data to be read or written.

[0155] S1033-5. If the cache contains the data to be read or written corresponding to the target key, determine that the cache processing result is the result in which the cache contains the data to be read or written.

[0156] In this embodiment, for a data read request, after the server receives the data read / write request sent by the client, it can first search the cache for the data to be read / written corresponding to the target key, that is, determine whether the cache includes the data to be read / written corresponding to the target key. If the cache does not contain the data to be read / written corresponding to the target key, the cache processing result is determined to be the result where the cache does not contain the data to be read / written; if the cache includes the data to be read / written corresponding to the target key, the cache processing result is determined to be the result where the cache includes the data to be read / written.

[0157] Therefore, during the data read and write process, the data to be read and written corresponding to the target key can be searched in the cache first, and the cache processing result can be determined based on whether the data to be read and written corresponding to the target key exists in the cache, thereby improving the data reading efficiency.

[0158] In an optional embodiment, the above method may further include:

[0159] If the cache processing result is determined to be a result in which the cache contains the data to be read and written, the data to be read and written included in the cache will be returned to the client.

[0160] In this embodiment, for a data read request, if it is determined that the cache contains the result of the data to be read or written, the data to be read or written can be returned directly from the cache, and the priority of the data to be written is incremented by 1, thereby avoiding disk I / O.

[0161] In an optional embodiment, continue as follows Figure 4 As shown, in step S105 above, the process of searching for the target data segment corresponding to the target key on the disk based on the cache processing result may include:

[0162] S1053-1. If the cache processing result is a result in which the data to be read or written is not included in the cache, obtain the memory index.

[0163] S1053-3. Locate the target data segment containing the target key on the disk using a memory index.

[0164] The memory index represents the mapping relationship between the key and the data segment, and the data segment corresponds to the storage area in the disk.

[0165] During the data page grouping process, it is necessary to establish a mapping relationship between keys and data segments. Through this mapping relationship, keys are mapped to specific data segments.

[0166] Alternatively, if the data to be read or written is not included in the cache, a memory index can be used to determine the possible disk location of the data. Specifically, the target key can be mapped to the target data segment using this memory index. This allows for the maintenance of a smaller memory index in memory, which is used to find the target data segment containing the target key, without needing to maintain an index entry for each individual data page, thus reducing the index's memory footprint.

[0167] In an optional embodiment, in step S107 above, determining the target data page corresponding to the target key from the preset number of data pages continuously stored on the disk included in the target data segment may include:

[0168] Using a linear model in the memory index, the target data page containing the data to be read and written corresponding to the target key is determined from a preset number of data pages that are continuously stored on the disk, including the target data segment.

[0169] The linear model represents the mapping relationship between the key and the data page.

[0170] The linear model in memory indexes refers to the mapping relationship between keys and data pages. For data segments (i.e., multi-page data segments) that consist of a predetermined number of data pages stored contiguously on disk, this linear model can be used to map keys to data pages. This allows for quick location of the data page containing the data to be read or written corresponding to the target key without reading the entire data segment, thereby improving data reading efficiency.

[0171] In an optional embodiment, continue as follows Figure 4 As shown, in step S109 above, the read / write processing of the data to be read / written corresponding to the target key based on the target data page may include:

[0172] S1093-1. Load the target data page from the disk into memory, and search for the data to be read and written corresponding to the target key in the target data page in memory.

[0173] S1093-3. Send the found data to be read / written to the client.

[0174] Optionally, for a data write request, after determining that the target data page has been obtained, the target page can be loaded from the disk into memory, and the data to be read and written corresponding to the target key can be searched within the target page. The found data to be read and written can then be sent to the client.

[0175] Since the data on the target page is ordered, binary search or other search algorithms can be effectively executed to quickly locate records, thereby improving data reading efficiency.

[0176] In some embodiments, for a data write request, if the target data corresponding to the target key is not found in the target data page in memory, a report indicating that the target data corresponding to the target key was not found is sent to the management account.

[0177] In an optional embodiment, the above method may further include:

[0178] If the remaining storage space in the cache is greater than the third preset storage threshold, the target data page and / or the data to be read and written will be written to the cache.

[0179] The target data page and / or data to be read and written are written to the cache to process the next data read request.

[0180] For data write requests, if there is remaining space in the memory cache during the reading process, for example, if the remaining storage space is greater than the third preset storage threshold, the target data page or the data to be read or written will be cached so that subsequent read operations can be completed more efficiently.

[0181] The following explanation uses a data read / write request as an example to illustrate the above data read / write method. This method may include:

[0182] 1. Cache check:

[0183] The read operation first checks the cache, an in-memory cache that stores recently accessed or modified records. If the data to be read / written corresponding to the requested target key is in the cache, the data is returned directly from the cache. Simultaneously, the priority of the data to be read / written is incremented by 1. For details, please refer to steps S1033-1 to S1033-5 above; no specific limitations are provided.

[0184] 2. Memory index lookup:

[0185] If the data to be read or written corresponding to the target key is not in the cache, a memory index will be used to determine the possible disk location of the data. A memory index is a mapping structure that maps the smallest key of each data segment to a physical segment identifier on disk.

[0186] 3. Data segment location:

[0187] The target data segment containing the target key is located using a memory index. A data segment is a series of contiguous pages stored on the disk, which together contain a range of keys. See steps S1053-1 to S1053-3 above for details; no specific limitations are imposed.

[0188] 4. Page access:

[0189] Next, the specific data page within the data segment is accessed based on the information in the memory index. If the data segment is multi-page, a linear model stored in the memory index will be used to determine the data page containing the target record. See step S107 above for details; no specific limitations are specified.

[0190] 5. Page loading and search:

[0191] After finding the correct target data page, load the target data page from disk into memory and search for the target key within the target data page. The data within the target data page is ordered, allowing for efficient execution of binary search or other search algorithms to quickly locate the record. For details, please refer to steps S1093-1 to S1093-3 above; no specific limitations are imposed.

[0192] 6. Returned results:

[0193] If the data to be read / written corresponding to the target key can be found on the target data page, it will be returned to the user. If the data to be read / written corresponding to the target key does not exist on the accessed target data page or overflow data page, a report will be sent to the management terminal stating that the data to be read / written corresponding to the target key was not found.

[0194] 7. Caching strategy:

[0195] During the reading process, if there is remaining space in the memory cache, the target data page or the data to be read and written corresponding to the target key will be cached so that subsequent reading operations can be completed more efficiently.

[0196] This application addresses the write performance challenges of traditional in-place update designs while retaining their advantages in read operations. It achieves performance superiority over traditional LSM-based key-value storage systems on NVMe SSDs through caching and page grouping technologies. Here, SSD refers to solid-state drives, and NVMe is a data storage access and transmission protocol for SSDs. Specifically, the caching in this application differs from the caching in existing LSMs in the following ways:

[0197] Cache granularity:

[0198] The caching in this embodiment is based on individual records, allowing for fast read and update operations on these records without needing to read the entire data page.

[0199] LSMs typically use block caching or data page caching, with the granularity of caching being a block of data or a data page, which may contain multiple key-value pairs.

[0200] Cache update:

[0201] In the cache of this application embodiment, update operations can be performed directly in memory, and then the updated records are flushed back to disk as needed. This approach reduces the write amplification factor because only the changed records need to be written back to disk.

[0202] In LSMs, update operations typically begin in an in-memory structure (such as a memtable). When the memtable is full, it is written to disk, triggering a merge operation. This can lead to write amplification because entire blocks or pages of data need to be copied.

[0203] Cache eviction:

[0204] This application uses a clock algorithm to evict entries in the record cache, prioritizing the eviction of entries that have not been modified recently, in order to reduce I / O overhead.

[0205] In LSMs, cache eviction strategies can be more complex, requiring consideration of merge operations and hierarchical structures, as well as how to use the cache most effectively to reduce read operations in the tree.

[0206] It should be noted that the data read / write method provided in this application embodiment can be applied to various types of storage systems, such as cloud storage, instant messaging software storage, etc. Taking cloud storage as an example, when the data read / write request is a data write request, data can be written to the cloud storage in the manner described in this application embodiment; when the data read / write request is a data read request, data can be read from the cloud storage in the manner described in this application embodiment.

[0207] Figure 5 This is a block diagram illustrating a data read / write device according to an exemplary embodiment, such as... Figure 5 The data read / write device includes:

[0208] The read / write request receiving module 201 is used to receive data read / write requests, wherein the data read / write requests carry a target key;

[0209] The first read / write processing module 203 is used to perform read / write processing on the target key in the cache to obtain the cache processing result;

[0210] The data segment lookup module 205 is used to look up the target data segment corresponding to the target key in the disk based on the cache processing result; the target data segment includes a preset number of data pages stored continuously on the disk, each data page includes data corresponding to a key of a different range, and the keys of different ranges corresponding to each data page are consecutive to each other;

[0211] The target data page determination module 207 is used to determine the target data page corresponding to the target key from the preset number of data pages continuously stored on the disk included in the target data segment;

[0212] The second read / write processing module 209 is used to perform read / write processing on the data to be read / written corresponding to the target key based on the target data page.

[0213] In an optional embodiment, the first read / write processing module includes:

[0214] A storage space determination unit is used to determine whether the remaining storage space of the cache is less than a first preset storage threshold when the data read / write request is a data write request.

[0215] A cache item processing unit is configured to, when determining that the remaining storage space of the cache is less than the first preset storage threshold, determine a target cache item in the cache based on a clock algorithm, delete the target cache item from the cache, and write the data to be read / written into the cache to obtain a cache processing result;

[0216] The data writing unit is used to write the data to be read / written into the cache when it is determined that the remaining storage space of the cache is greater than or equal to the first preset storage threshold, so as to obtain the cache processing result;

[0217] The cache stores cache items whose access time and current time difference is less than a preset time threshold, and / or cache items whose access frequency is greater than a first preset frequency threshold. The target cache item is a cache item in the cache whose access frequency is greater than the first preset frequency threshold and less than a second preset frequency threshold.

[0218] In an optional embodiment, the cache item processing unit includes:

[0219] The current scale acquisition subunit is used to acquire the current scale that the pointer in the clock structure is currently pointing to;

[0220] The target cache item determination subunit is used to determine the cache item corresponding to the current scale as the target cache item when the cache item corresponding to the current scale is a clean cache item;

[0221] The repeated execution unit is configured to, when the cache item corresponding to the current scale is a non-clean cache item, move the pointer clockwise to the next scale, and re-determine the next scale as the current scale; repeatedly execute the operation of determining whether the cache item corresponding to the current scale is the clean cache item, until the clean cache item is obtained, and determine the obtained clean cache item as the target cache item;

[0222] The clean cache entry is a cache entry in the cache that has not been modified, and the non-clean cache entry is a cache entry in the cache that has been modified and has not been written back to disk.

[0223] In an optional embodiment, the data segment lookup module includes:

[0224] The first memory index acquisition unit is used to acquire a memory index when the cache processing result is to write the data to be read / written into the cache;

[0225] The first lookup unit is used to locate the target data segment to which the target key belongs in the disk using the memory index;

[0226] The memory index represents the mapping relationship between the key and the data segment, the data segment corresponds to the storage area in the disk, and the data segment includes the target data segment.

[0227] In an optional embodiment, the module for determining the target data page includes:

[0228] The page determination unit is used to determine the target data page to which the target key belongs, based on a linear model in the memory index and among a preset number of data pages continuously stored on the disk, which are included in the target data segment;

[0229] The linear model represents the mapping relationship between the key and the data page.

[0230] In an optional embodiment, the second read / write processing module includes:

[0231] The first data writing unit is used to write the data to be read and written into the target data page when the remaining storage space in the target data page is greater than the second preset storage threshold.

[0232] The second data writing unit is used to create a new data page and write the data to be read / written to the new data page when the storage space in the target data page is less than or equal to the second preset storage threshold.

[0233] In an optional embodiment, the apparatus further includes:

[0234] The data segment creation unit is used to create a new data segment when the number of data pages included in the target data segment exceeds a preset number threshold.

[0235] A partial data writing unit is used to write a portion of the data pages from the target data segment into the new data segment.

[0236] In an optional embodiment, the first read / write processing module includes:

[0237] The read / write data determination unit is used to determine whether the cache contains the data to be read / written corresponding to the target key when the data read / write request is a data read request.

[0238] The first result determination unit is used to determine that the cache processing result is that the cache does not contain the data to be read or written corresponding to the target key when the cache does not contain the data to be read or written.

[0239] The second result determination unit is used to determine, when the cache includes the data to be read and written corresponding to the target key, that the cache processing result is the result in which the cache includes the data to be read and written.

[0240] In an optional embodiment, the apparatus further includes:

[0241] The return module is used to return the data to be read and written included in the cache to the client when it is determined that the cache processing result is a result in which the cache includes the data to be read and written.

[0242] In an optional embodiment, the data segment lookup module includes:

[0243] The second memory index acquisition unit is used to acquire the memory index when the cache processing result is that the cache does not contain the data to be read or written;

[0244] The second search unit is used to search for a target data segment including the target key in the disk using the memory index;

[0245] The memory index represents the mapping relationship between the key and the data segment, and the data segment corresponds to the storage area in the disk.

[0246] In an optional embodiment, the target data page determination module includes:

[0247] The corresponding page determination unit is used to determine, through the linear model in the memory index, a target data page including the data to be read and written corresponding to the target key among a preset number of data pages continuously stored on the disk included in the target data segment;

[0248] The linear model represents the mapping relationship between the key and the data page.

[0249] In an optional embodiment, the second read / write processing module includes:

[0250] The loading and searching unit is used to load the target data page from the disk into memory and search for the data to be read and written corresponding to the target key in the target data page in memory.

[0251] The sending unit is used to send the found data to be read or written to the client.

[0252] In an optional embodiment, the apparatus further includes:

[0253] A cache writing unit is used to write the target data page and / or the data to be read and written into the cache when the remaining storage space of the cache is greater than a third preset storage threshold.

[0254] The target data page and / or the data to be read and written are written to the cache to process the next data read request.

[0255] It should be noted that the device embodiments provided in this application are based on the same inventive concept as the method embodiments described above.

[0256] This application also provides an electronic device for reading and writing data. The electronic device includes a processor and a memory. The memory stores at least one instruction or at least one program. The processor loads and executes the at least one instruction or at least one program to implement the data reading and writing method provided in any of the above embodiments.

[0257] This application also provides an electronic device for reading and writing data. The electronic device includes a processor and a memory. The memory stores at least one instruction or at least one program. The processor loads and executes the at least one instruction or at least one program to implement the data reading and writing method provided in any of the above embodiments.

[0258] Embodiments of this application also provide a computer-readable storage medium that can be disposed in a terminal to store at least one instruction or at least one program for implementing a data read / write method in the method embodiments. The at least one instruction or at least one program is loaded and executed by a processor to implement the data read / write method provided in the above method embodiments.

[0259] Optionally, in the embodiments of this specification, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0260] The memory described in this specification can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for functions, etc.; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0261] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data read / write method provided in the above-described method embodiments.

[0262] The data read / write method embodiments provided in this application can be executed in a terminal, computer terminal, server, or similar computing device. Taking running on a server as an example, Figure 6 This is a hardware structure block diagram of an electronic device according to an exemplary embodiment. For example... Figure 6As shown, the electronic device 300 can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 310 (CPUs 310 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 330 for storing data, and one or more storage media 320 (e.g., one or more mass storage devices) for storing application programs 323 or data 322. The memory 330 and storage media 320 may be temporary or persistent storage. The program stored in the storage media 320 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 310 may be configured to communicate with the storage media 320 and execute the series of instruction operations in the storage media 320 on the server 300. Electronic device 300 may also include one or more power supplies 360, one or more wired or wireless network interfaces 350, one or more input / output interfaces 340, and / or one or more operating systems 321, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0263] The input / output interface 340 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 300. In one example, the input / output interface 340 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 340 may be a radio frequency (RF) module for wireless communication with the Internet.

[0264] Those skilled in the art will understand that Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device 300 may also include... Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown.

[0265] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0266] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and server embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0267] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0268] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data read / write method, characterized in that, The data read / write method includes: Receive a data read / write request, wherein the data read / write request carries a target key; The target key is read and written in the cache to obtain the cache processing result; Based on the cache processing result, the target data segment corresponding to the target key is searched in the disk; the target data segment includes a preset number of data pages stored contiguously on the disk, each data page includes data corresponding to a different range of keys, and the different ranges of keys corresponding to each data page are contiguous to each other; Among the predetermined number of data pages continuously stored on the disk included in the target data segment, the target data page corresponding to the target key is determined; Based on the target data page, read and write operations are performed on the data to be read and written corresponding to the target key.

2. The data read / write method according to claim 1, characterized in that, The step of reading and writing the target key in the cache to obtain the cache processing result includes: If the data read / write request is a data write request, determine whether the remaining storage space of the cache is less than a first preset storage threshold. If it is determined that the remaining storage space of the cache is less than the first preset storage threshold, a target cache item is determined in the cache based on a clock algorithm, the target cache item is deleted from the cache, and the data to be read and written is written into the cache to obtain the cache processing result; If it is determined that the remaining storage space of the cache is greater than or equal to the first preset storage threshold, the data to be read and written is written into the cache to obtain the cache processing result; The cache stores cache items whose access time and current time difference is less than a preset time threshold, and / or cache items whose access frequency is greater than a first preset frequency threshold. The target cache item is a cache item in the cache whose access frequency is greater than the first preset frequency threshold and less than a second preset frequency threshold.

3. The data read / write method according to claim 2, characterized in that, The cache is represented by a circular data structure, which is a clock structure. Each cache item in the cache is located at a tick mark of the clock structure. The clock-based algorithm for determining the target cache item in the cache includes: Obtain the current tick mark that the pointer in the clock structure is currently pointing to; If the cache entry corresponding to the current scale is a clean cache entry, then the cache entry corresponding to the current scale is determined as the target cache entry; If the cache item corresponding to the current tick is not a clean cache item, move the pointer clockwise to the next tick and re-determine the next tick as the current tick; repeat the operation of determining whether the cache item corresponding to the current tick is a clean cache item until the clean cache item is obtained, and determine the obtained clean cache item as the target cache item; The clean cache entry is a cache entry in the cache that has not been modified, and the non-clean cache entry is a cache entry in the cache that has been modified and has not been written back to disk.

4. The data read / write method according to claim 2, characterized in that, The step of searching for the target data segment corresponding to the target key in the disk based on the cache processing result includes: If the cache processing result is to write the data to be read / written into the cache, obtain the memory index; The target data segment to which the target key belongs is located in the disk using the memory index; The memory index represents the mapping relationship between the key and the data segment, the data segment corresponds to the storage area in the disk, and the data segment includes the target data segment.

5. The data read / write method according to claim 4, characterized in that, Determining the target data page corresponding to the target key from the predetermined number of data pages continuously stored on the disk included in the target data segment includes: Using the linear model in the memory index, the target data page to which the target key belongs is determined from a preset number of data pages continuously stored on the disk, including the target data segment; The linear model represents the mapping relationship between the key and the data page.

6. The data read / write method according to any one of claims 2 to 5, characterized in that, The process of reading and writing data corresponding to the target key based on the target data page includes: If the remaining storage space in the target data page is greater than the second preset storage threshold, the data to be read and written will be written to the target data page. If the storage space in the target data page is less than or equal to the second preset storage threshold, a new data page is created, and the data to be read and written is written to the new data page.

7. The data read / write method according to any one of claims 2 to 5, characterized in that, The method further includes: If the number of data pages included in the target data segment exceeds a preset threshold, a new data segment is created. Write a portion of the data pages from the target data segment into the new data segment.

8. The data read / write method according to claim 1, characterized in that, The step of reading and writing the target key in the cache to obtain the cache processing result includes: If the data read / write request is a data read request, determine whether the cache includes the data to be read / written corresponding to the target key; If the cache does not contain the data to be read or written corresponding to the target key, the cache processing result is determined to be the result where the cache does not contain the data to be read or written. If the cache includes the data to be read / written corresponding to the target key, the cache processing result is determined to be the result in which the cache includes the data to be read / written.

9. The data read / write method according to claim 8, characterized in that, The method further includes: If the cache processing result is determined to be a result in which the cache includes the data to be read and written, the data to be read and written included in the cache is returned to the client.

10. The data read / write method according to claim 8, characterized in that, The step of searching the disk for the target data segment corresponding to the processed target key based on the cache processing result includes: If the cache processing result is that the cache does not contain the data to be read or written, obtain the memory index; The target data segment containing the target key is located in the disk using the memory index; The memory index represents the mapping relationship between the key and the data segment, the data segment corresponds to the storage area in the disk, and the data segment includes the target data segment.

11. The data read / write method according to claim 10, characterized in that, Determining the target data page corresponding to the target key from the predetermined number of data pages contiguously stored on the disk, including: Using the linear model in the memory index, a target data page containing the data to be read and written corresponding to the target key is determined from a preset number of data pages continuously stored on the disk, including the target data segment. The linear model represents the mapping relationship between the key and the data page.

12. The data read / write method according to claim 8, characterized in that, The process of reading and writing data corresponding to the target key based on the target data page includes: Load the target data page from the disk into memory, and search for the data to be read and written corresponding to the target key in the target data page in memory; The found data to be read or written is sent to the client.

13. The data read / write method according to any one of claims 8 to 12, characterized in that, The method further includes: If the remaining storage space in the cache is greater than a third preset storage threshold, the target data page and / or the data to be read and written shall be written into the cache. The target data page and / or the data to be read and written are written to the cache to process the next data read request.

14. A data read / write device, characterized in that, The data read / write device includes: A read / write request receiving module is used to receive data read / write requests, wherein the data read / write request carries a target key; The first read / write processing module is used to perform read / write processing on the target key in the cache to obtain the cache processing result; The data segment lookup module is used to search for the target data segment corresponding to the target key in the disk based on the cache processing result; the target data segment includes a preset number of data pages stored contiguously on the disk, each data page includes data corresponding to a different range of keys, and the different ranges of keys corresponding to each data page are contiguous to each other; The target data page determination module is used to determine the target data page corresponding to the target key from a preset number of data pages continuously stored on the disk included in the target data segment; The second read / write processing module is used to perform read / write processing on the data to be read / written corresponding to the target key based on the target data page.

15. An electronic device for reading and writing data, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data read / write method as described in any one of claims 1-13.

Images