Data storage method and device and data reading method and device
By introducing a cache mechanism in the LSM-Tree storage system, flexible hierarchical storage of data is achieved, and the problem of difficulty in balancing cost and performance of data storage in the prior art is solved, and efficient and flexible data storage and reading processes are achieved.
Patent Information
- Application Number
- CN202411874795.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-04-11
AI Technical Summary
When the LSM-Tree storage system processes large amounts of data, it is difficult for the prior art to realize flexible layered storage of data, resulting in difficult to balance cost and performance.
By introducing a caching mechanism in the LSM-Tree storage system, data is stored in object storage and flexibly cached on local disks according to access frequency. The specific steps include writing the data in the write request to the write-pre-log and memory table, converting the memory table into read-only mode after reaching the threshold and organizing it into a sorted string table, storing it to the disk cache and uploading it to the object store synchronously.
It realizes flexible layered data storage, which not only ensures the random expansion of storage capacity, but also achieves the effect of performance close to local disks, achieving a balance between cost and performance.
Smart Images
Figure CN119806410A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to data storage, data reading methods and devices. Background Art
[0002] LSM-Tree (Log-Structured merge tree) is a data structure based on log append writing, with a certain structure and merged tree. It is widely used in various NoSQL (Not Only Structured Query Language) key-value pair storage systems, including but not limited to: open source systems LevelDB, RocksDB, Rockset, etc.
[0003] The LSM-Tree storage system usually consists of three parts in its implementation: WAL (Write Ahead Log) on disk, memtable (memory table) in memory, and sorted string table sstable (Sorted StringTable) on disk.
[0004] The LSM-Tree storage system mainly uses two storage media when implementing data storage: one is to store data in the local disk and access data to the local disk for reading and writing, such as LevelDB and RocskDB; the other is to store data in object storage and access data to the object storage for reading and writing, such as Rockset. Summary of the invention
[0005] The embodiments of the present disclosure provide data storage and data reading methods and devices.
[0006] In a first aspect, an embodiment of the present disclosure provides a data storage method, the method comprising: writing data in a write request into a write-ahead log on a disk, and synchronously writing the data into a memory table on a memory; in response to the memory table reaching a preset storage threshold, setting the memory table to read-only mode, and generating a new memory table; organizing the memory table in read-only mode into a sorted string table, storing the table in a cache on the disk, and synchronously uploading the table to an object storage; and updating the information of the sorted string table to the cache on the disk.
[0007] In a second aspect, an embodiment of the present disclosure provides a data reading method, including: searching a memory table in a memory based on a read request; in response to not finding the data corresponding to the read request in the memory table in the memory, searching a sorted string table in a cache on a disk; in response to not finding the data corresponding to the read request in the sorted string table in the cache, searching a sorted string table in a storage object; in response to finding the data corresponding to the read request in the sorted string table in the storage object, returning the data corresponding to the read request that was found.
[0008] In a third aspect, an embodiment of the present disclosure provides a data storage device, which includes: a write module, configured to write data in a write request into a pre-write log on the disk, and synchronously write it into a memory table on the memory; a setting and generation module, configured to set the memory table to read-only mode in response to the memory table reaching a preset storage threshold, and generate a new memory table; a storage and upload module, configured to organize the memory table in read-only mode into a sorted string table, store it in a cache on the disk, and synchronously upload it to the object storage; an update module, configured to update the information of the sorted string table to the cache on the disk.
[0009] In a fourth aspect, an embodiment of the present disclosure provides a data reading device, comprising: a first search module, configured to search a memory table in a memory based on a read request; a second search module, configured to search a sorted string table in a cache on a disk in response to not finding the data corresponding to the read request in the memory table in the memory; a third search module, configured to search a sorted string table in a storage object in response to not finding the data corresponding to the read request in the sorted string table in the cache; and a return module, configured to return the data corresponding to the read request in response to finding the data corresponding to the read request in the sorted string table in the storage object.
[0010] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect or the second aspect.
[0011] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect or the second aspect.
[0012] In a seventh aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which, when executed by a processor, implements the method described in any implementation manner of the first aspect or the second aspect.
[0013] The method provided by the above embodiment of the present disclosure adopts a method of combining local disk and object storage for LSM-Tree storage system, and realizes flexible hierarchical storage of data by introducing cache. By maintaining a cache for storing sorted string tables in the LSM-Tree storage system, data can be stored in object storage and flexibly cached on the local disk according to the access frequency, achieving a balance between cost and performance.
[0014] The method provided by the above-mentioned embodiment of the present disclosure optimizes the reading and writing process by introducing a cache mechanism into the reading and writing process, thereby realizing both arbitrary expansion and contraction of storage capacity and performance close to that of a local disk. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0016] Figure 1 is a flow chart of an embodiment of a data storage method according to the present disclosure;
[0017] Figure 2 Is to write a flow chart;
[0018] Figure 3 It is a schematic diagram of cache implementation;
[0019] Figure 4 is a flow chart of an embodiment of a data reading method according to the present disclosure;
[0020] Figure 5 It is a reading flow chart;
[0021] Figure 6 is a schematic structural diagram of an embodiment of a data storage device according to the present disclosure;
[0022] Figure 7 is a structural schematic diagram of an embodiment of a data reading device according to the present disclosure;
[0023] Figure 8 It is a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION
[0024] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0025] It should be noted that in the technical solution of the present invention, the collection / collection, update, analysis, use, transmission, storage and other aspects of user personal information involved are in compliance with the provisions of relevant laws and regulations, are used for legal and reasonable purposes, are not shared, leaked or sold outside of these legal uses, and are subject to supervision and management by national regulatory authorities. Necessary measures should be taken for user personal information to selectively block the use or access to personal information data to prevent illegal access to such personal information data, ensure that persons who have access to personal information data comply with the provisions of relevant laws and regulations, and ensure the security of user personal information. In addition, once these user personal information data are no longer needed, the risks should be minimized by limiting or even prohibiting data collection and / or deleting data.
[0026] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0027] Figure 1 A process 100 of an embodiment of a data storage method according to the present disclosure is shown. The data storage method comprises the following steps:
[0028] Step 101, write the data in the write request into the write-ahead log on the disk, and simultaneously write it into the memory table on the memory.
[0029] In this embodiment, when the LSM-Tree storage system receives a write request, the data in the write request can be written to the WAL on the disk and synchronously written to the memtable in the memory. After the data is written to the memtable in the memory, a write success can be returned.
[0030] WAL is a common method in database systems to ensure the atomicity and persistence of data operations. In a system using WAL, all modifications must be written to the log file before being submitted.
[0031] Memtable is an in-memory data structure used in databases and storage systems. Memtable can temporarily store recently written data for fast access and efficient writing, and then periodically flush it to disk. Data synchronization written to memtable in memory can be implemented using the skiplist data structure.
[0032] Step 102: In response to the memory table reaching a preset storage threshold, the memory table is set to a read-only mode, and a new memory table is generated.
[0033] In this embodiment, when the memtable reaches a preset storage threshold, the memtable can be set to immutable (read-only) mode and a new memtable can be generated. The immutable mode is a mode that cannot be written but can only be read.
[0034] Step 103: organize the read-only memory table into a sorted string table, store it in the cache on the disk, and upload it to the object storage synchronously.
[0035] In this embodiment, for a memtable in immutable mode, it can be organized into a sorted string table sstable, stored in a cache on the disk, and synchronously uploaded to the object storage.
[0036] sstable can be used to store internal data files. It is an efficient key-value pair file storage format. sstable is a temporary file on disk. sstable on disk can be logically stored in layers. Specifically, the ssttable file newly formed by the memtable in immutable mode will be stored in layer 0. When the total size of the sstable files in layer 0 exceeds a certain threshold, the sstable in layer 0 can be merged and compressed and sunk to the next layer, and so on.
[0037] Object storage is a technology that stores and manages data in unstructured formats (called objects). Modern organizations need to create and analyze large amounts of unstructured data, including but not limited to photos, videos, emails, web pages, sensor data, and audio files. Object storage systems distribute this data across multiple physical devices, allowing users to efficiently access content from a single virtual repository. Object storage solutions are ideal for building cloud-native applications that require expansion and flexibility, and these solutions can be used to import existing data stores for analysis, backup, or archiving.
[0038] Step 104: Update the information of the sorted string table to the cache on the disk.
[0039] In this embodiment, after the sstable is stored in the cache on the disk, the information of the sorted sstable can be updated to the cache on the disk.
[0040] The cache on the disk can be an LRU (Least Recently Used) cache, or other caches with the same effect, which is not limited here.
[0041] In some embodiments, the cache may include a doubly linked list and a hash table. Each node of the doubly linked list may store a key-value pair <filename, fd>. Here, filename is the file name of the sstable, and fd is the file handle for opening the sstable in the cache. When accessing the sstable on the disk, reading is performed through fd. The content stored in the hash table may also be a key-value pair <filename, pointer>. The pointer may be a pointer to a node of the doubly linked list.
[0042] In the write process, information about the newly generated sstable can be added to the cache. The cache is updated by adding the filename of an sstable and the corresponding fd for opening the sstable. Meanwhile, the cache can also be set with a capacity size as needed, and this capacity size can be adjusted according to different usage scenarios. When the capacity of the cache has not reached the preset capacity threshold, the information of the sstable can be added to the head of the doubly linked list. When the capacity of the cache reaches the preset capacity threshold, the information of the sstable can be added to the head of the doubly linked list, and the element at the tail of the doubly linked list is deleted.
[0043] The method provided in the above embodiments of the present disclosure adopts a method of combining a local disk and an object storage for the LSM-Tree storage system, and realizes flexible hierarchical storage of data by introducing a cache. By maintaining a cache for storing sorted string tables in the LSM-Tree storage system, data is stored in the object storage and can be flexibly cached on the local disk according to the access frequency, achieving a balance between cost and performance.
[0044] Figure 2 A schematic diagram of the write process is shown. The steps of the write process are as follows:
[0045] Step 1, after detecting a write operation, write the record to the WAL log.
[0046] Step 2, after writing to the WAL log, synchronously write it to the memtable in the memory.
[0047] Step 3, after writing to the memtable in the memory, return write success.
[0048] Step 4, when the memtable in the memory reaches a certain size, a new memtable is generated, and at the same time, the original memtable becomes an immutable mode.
[0049] Step 5: Organize the data in the immutable - mode memtable into an sstable in a certain format and store it on disk. At this time, the sstable is a temporary file on disk.
[0050] Step 6: Synchronously upload the sstable to the object storage.
[0051] Step 7: After uploading to the object storage, synchronously update and store the information of the sstable in an LRUCache on disk.
[0052] Figure 3 Shows a schematic diagram of the cache implementation. As Figure 3 shown, the cache can include a doubly - linked list and a hash table. Among them, the content stored in each node of the doubly - linked list can be a key - value pair <filename, fd>. filename is the file name of the sstable. fd is the file handle for opening the sstable in the cache, and when accessing the sstable on disk, it needs to be read through fd. The content stored in the hash table can also be a key - value pair <filename, pointer>. The pointer can be a pointer to the node of the doubly - linked list.
[0053] Figure 4 Shows a flow 300 of an embodiment of the data reading method according to the present disclosure. The data reading method includes the following steps:
[0054] Step 401: Based on the read request, search for the memtable in memory.
[0055] In this embodiment, when the LSM - Tree storage system receives a read request, it can search for the memtable in memory based on the read request. If the data corresponding to the read request is found in the memtable in memory, the data corresponding to the found read request can be returned; if the data corresponding to the read request is not found in the memtable in memory, step 402 can be executed.
[0056] The memtable is a memory data structure used in databases and storage systems. The memtable can temporarily store recently written data for quick access and efficient writing, and then periodically flush it to disk.
[0057] Step 402: In response to not finding the data corresponding to the read request in the memtable in memory, search for the sorted string table in the cache on disk.
[0058] In this embodiment, if the data corresponding to the read request cannot be found in the memtable in the memory, the sstable in the cache on the disk can be searched. If the data corresponding to the read request is found in the sstable in the cache, the data corresponding to the read request that is found can be returned. At the same time, the information of the sstable where the data corresponding to the read request is located can also be moved to the head of the doubly linked list. If the data corresponding to the read request cannot be found in the sstable in the cache, step 403 can be executed.
[0059] Among them, the sstable can be used for the storage of internal data files and is an efficient key-value pair type file storage format. The cache can include a doubly linked list and a hash table. Among them, the content stored in each node of the doubly linked list can be a key-value pair <filename, fd>. filename is the file name of the sstable. fd is the file handle for opening the sstable in the cache, and reading the sstable on the disk needs to be performed through fd. The content stored in the hash table can also be a key-value pair <filename, pointer>. The pointer can be a pointer pointing to the node of the doubly linked list.
[0060] Step 403: In response to the data corresponding to the read request not being found in the sorted string table in the cache, search for the sorted string table in the storage object.
[0061] In this embodiment, if the data corresponding to the read request cannot be found in the sstable in the cache, the sstable in the storage object can be searched. If the data corresponding to the read request is found in the sstable in the storage object, step 404 is executed; if the data corresponding to the read request cannot be found in the sstable in the storage object, the result of not finding the data is returned.
[0062] Step 404: In response to the data corresponding to the read request being found in the sorted string table in the storage object, return the data corresponding to the read request that is found.
[0063] In this embodiment, if the data corresponding to the read request is found in the sstable in the storage object, the data corresponding to the read request that is found can be returned.
[0064] In some embodiments, if the data corresponding to the read request is found in the sstable in the storage object, the sstable where the data corresponding to the read request is located can also be downloaded to the cache, and the information of the sstable where the data corresponding to the read request is located can be updated to the cache. Specifically, the sstable downloaded from the object storage can complete the cache update by adding an sstable filename and the corresponding fd to open the sstable in the cache. At the same time, the cache can also set the capacity size as needed, and the capacity size can be adjusted in combination with different usage scenarios. When the capacity of the cache does not reach the preset capacity threshold, the sstable information can be added to the head of the doubly linked list. When the capacity of the cache reaches the preset capacity threshold, the sstable information can be added to the head of the doubly linked list, and the elements at the tail of the doubly linked list are deleted.
[0065] For the traversal reading scenario, due to the capacity limit of the cache, each downloaded sstable will be quickly replaced by the newly downloaded sstable after it is updated to the cache, causing batch cache invalidation and reducing the cache hit rate. For this scenario, certain optimization processing can be added. For example, a new set of caches is added, and the first set of caches stores the sstables added for the first time. If the sstable is reaccessed within a preset period (such as 10 seconds) after the data is written to the cache, the sstable can be migrated to the second set of caches, thereby ensuring that in the case of batch traversal access, only the hit rate of the first set of caches will be reduced, without affecting the hit rate of the second set of caches, thereby improving the overall cache hit rate.
[0066] The method provided by the above-mentioned embodiment of the present disclosure optimizes the reading and writing process by introducing a cache mechanism into the reading and writing process, thereby realizing both arbitrary expansion and contraction of storage capacity and performance close to that of a local disk.
[0067] Figure 5 The reading process diagram is shown in FIG. The steps of the reading process are as follows:
[0068] Step 1: After the read operation is detected, first check whether there is a corresponding record in the memtable. If it exists, return, corresponding to step 1'.
[0069] Step 2: If the corresponding record is not found in step 1, search for the corresponding record in the memtable in immutable mode. If it exists, return, corresponding to step 2'.
[0070] Step 3: If the corresponding record is not found in step 2, the corresponding record is searched in the disk cache first. If found, the result is returned, corresponding to step 3′.
[0071] Step 4: If the corresponding record is not found in step 3, search in the object storage. Whether the corresponding record is found or not, the corresponding result is returned, corresponding to step 4'.
[0072] Step 5: If the corresponding record is found in step 4, the sstable corresponding to the record is downloaded to the cache, and the sstable information is updated to the cache.
[0073] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data storage device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic equipment.
[0074] like Figure 6 As shown, the data storage device 600 provided in this embodiment includes a writing module 601, a setting and generating module 602, a storage and uploading module 603 and an updating module 604. Among them, the writing module 601 is configured to write the data in the write request into the pre-write log on the disk, and synchronously write it into the memory table on the memory; the setting and generating module 602 is configured to set the memory table to read-only mode in response to the memory table reaching a preset storage threshold, and generate a new memory table; the storage and uploading module 603 is configured to organize the memory table in the read-only mode into a sorted string table, store it in the cache on the disk, and synchronously upload it to the object storage; the updating module 604 is configured to update the information of the sorted string table to the cache on the disk.
[0075] In this embodiment, in the data storage device 600, the specific processing of the writing module 601, the setting and generating module 602, the storage and uploading module 603 and the updating module 604 and the technical effects thereof can be referred to in Figure 1 The relevant descriptions of steps 101 - 104 in the corresponding embodiment are not repeated here.
[0076] In some optional implementations of this embodiment, the cache includes a doubly linked list and a hash table, the doubly linked list is used to store the file name of the sorted string table in the cache and the file handle for opening the sorted string table in the cache, and the hash table is used to store the file name of the sorted string table in the cache and the pointers to the nodes of the doubly linked list.
[0077] In some optional implementations of this embodiment, the update module 604 is further configured to: in response to the capacity of the cache not reaching a preset capacity threshold, add the information of the sorted string table to the head of the doubly linked list; in response to the capacity of the cache reaching a preset capacity threshold, add the information of the sorted string table to the head of the doubly linked list and delete the elements at the tail of the doubly linked list.
[0078] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data reading device, which is similar to Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic equipment.
[0079] like Figure 7 As shown, the data reading device 700 provided in this embodiment includes a first search module 701, a second search module 702, a third search module 703 and a first return module 704. The first search module 701 is configured to search the memory table in the memory based on the read request; the second search module 702 is configured to search the sorting string table in the cache on the disk in response to not finding the data corresponding to the read request in the memory table in the memory; the third search module 703 is configured to search the sorting string table in the storage object in response to not finding the data corresponding to the read request in the sorting string table in the cache; the first return module 704 is configured to return the data corresponding to the read request in response to finding the data corresponding to the read request in the sorting string table in the storage object.
[0080] In this embodiment, in the data reading device 700, the specific processing of the first search module 701, the second search module 702, the third search module 703 and the first return module 704 and the technical effects thereof can be referred to in Figure 4 The relevant descriptions of steps 401-404 in the corresponding embodiment are not repeated here.
[0081] In some optional implementations of this embodiment, the data reading device 700 further includes: a second returning module configured to return the data corresponding to the read request in response to finding the data corresponding to the read request in the memory table in the memory.
[0082] In some optional implementations of this embodiment, the data reading device 700 further includes: a third returning module configured to return the data corresponding to the read request in response to finding the data corresponding to the read request in the sorted string table in the cache.
[0083] In some optional implementations of this embodiment, the cache includes a doubly linked list and a hash table, the doubly linked list is used to store the file name of the sorted string table in the cache and the file handle for opening the sorted string table in the cache, and the hash table is used to store the file name of the sorted string table in the cache and the pointers to the nodes of the doubly linked list.
[0084] In some optional implementations of this embodiment, the data reading device 700 also includes: a moving module, which is configured to move information of the sorted string table where the data corresponding to the read request is located to the head of the bidirectional linked list in response to finding the data corresponding to the read request in the sorted string table in the cache.
[0085] In some optional implementations of this embodiment, the data reading device 700 also includes: a download and update module, which is configured to, in response to finding the data corresponding to the read request in the sorted string table in the storage object, download the sorted string table where the data corresponding to the read request is located to the cache, and update the information of the sorted string table where the data corresponding to the read request is located to the cache.
[0086] In some optional implementations of this embodiment, the download and update module is further configured to: in response to the capacity of the cache not reaching a preset capacity threshold, add the information of the sorted string table where the data corresponding to the read request is located to the head of the doubly linked list; in response to the capacity of the cache reaching a preset capacity threshold, add the information of the sorted string table where the data corresponding to the read request is located to the head of the doubly linked list, and delete the elements at the tail of the doubly linked list.
[0087] In some optional implementations of this embodiment, the data reading device 700 also includes: a migration module, which is configured to migrate the sorted string table containing the data corresponding to the read request to a new cache in response to finding the data corresponding to the read request in the sorted string table in the cache, and the reception time of the read request is within a preset period after the data corresponding to the read request is written to the cache.
[0088] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device (e.g., a server) 800 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0089] like Figure 8As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0090] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 8 Each block shown in the figure may represent one device, or may represent multiple devices as required.
[0091] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0092] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0093] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: writes the data in the write request to the pre-write log on the disk, and synchronously writes it to the memory table on the memory; in response to the memory table reaching a preset storage threshold, sets the memory table to read-only mode and generates a new memory table; organizes the memory table in read-only mode into a sorted string table, stores it in the cache on the disk, and synchronously uploads it to the object storage; updates the information of the sorted string table to the cache on the disk.
[0094] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0095] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0096] The modules involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The modules described may also be set in a processor. For example, they may be described as: a processor includes a write module, a setting and generation module, a storage and upload module, and an update module. The names of these modules do not constitute a limitation on the modules themselves in some cases. For example, the write module may also be described as "a module that writes the data in the write request to the pre-write log on the disk and synchronously writes it to the memory table on the memory."
[0097] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.
Claims
1. A data storage method, comprising: Write the data in the write request to the write-ahead log on disk and simultaneously write it to the memtable in memory; In response to the memory table reaching a preset storage threshold, setting the memory table to a read-only mode and generating a new memory table; Organizing the read-only mode memory table into a sorted string table, storing it in the cache on the disk, and synchronously uploading it to the object storage; The information of the sorted string table is updated into the cache on the disk.
2. The method according to claim 1, wherein: The cache includes a doubly linked list and a hash table, the doubly linked list is used to store the file name of the sorted string table in the cache and the file handle for opening the sorted string table in the cache, and the hash table is used to store the file name of the sorted string table in the cache and the pointer of the node of the doubly linked list.
3. The method according to claim 2, wherein: The updating of the information of the sorted string table to the cache on the disk comprises: In response to the capacity of the cache not reaching a preset capacity threshold, adding information of the sorted string table to a head of the bidirectional linked list; In response to the capacity of the cache reaching the preset capacity threshold, the information of the sorted string table is added to the head of the bidirectional linked list, and the elements at the tail of the bidirectional linked list are deleted.
4. A data reading method, comprising: Look up the memtable in memory based on the read request; In response to not finding the data corresponding to the read request in the memory table in the memory, searching the sorted string table in the cache on the disk; In response to not finding the data corresponding to the read request in the sorted string table in the cache, searching the sorted string table in the storage object; In response to finding the data corresponding to the read request in the sorted string table in the storage object, the found data corresponding to the read request is returned.
5. The method according to claim 4, wherein: The method further comprises: In response to finding the data corresponding to the read request in the memory table in the memory, the found data corresponding to the read request is returned.
6. The method according to claim 5, wherein: The method further comprises: In response to finding the data corresponding to the read request in the sorted string table in the cache, the found data corresponding to the read request is returned.
7. The method according to claim 6, wherein: The cache includes a doubly linked list and a hash table, the doubly linked list is used to store the file name of the sorted string table in the cache and the file handle for opening the sorted string table in the cache, and the hash table is used to store the file name of the sorted string table in the cache and the pointer of the node of the doubly linked list.
8. The method according to claim 7, wherein: The method further comprises: In response to finding the data corresponding to the read request in the sorted string table in the cache, information of the sorted string table where the data corresponding to the read request is located is moved to the head of the bidirectional linked list.
9. The method according to claim 7, wherein: The method further comprises: In response to finding the data corresponding to the read request in the sorted string table in the storage object, downloading the sorted string table where the data corresponding to the read request is located to the cache, and updating the information of the sorted string table where the data corresponding to the read request is located to the cache.
10. The method according to claim 9, wherein: The updating of information of the sorted string table where the data corresponding to the read request is located into the cache includes: In response to the capacity of the cache not reaching a preset capacity threshold, adding information of the sorted string table where the data corresponding to the read request is located to a head of the bidirectional linked list; In response to the capacity of the cache reaching the preset capacity threshold, information of the sorted string table where the data corresponding to the read request is located is added to the head of the bidirectional linked list, and elements at the tail of the bidirectional linked list are deleted.
11. The method according to claim 6, wherein: The method further comprises: In response to finding the data corresponding to the read request in the sorted string table in the cache, and the read request is received within a preset period after the data corresponding to the read request is written into the cache, the sorted string table where the data corresponding to the read request is located is migrated to a new cache.
12. A data storage device, comprising: The write module is configured to write the data in the write request to the write-ahead log on the disk and synchronously write it to the memory table on the memory; A setting and generating module, configured to set the memory table to a read-only mode and generate a new memory table in response to the memory table reaching a preset storage threshold; A storage and upload module, configured to organize the read-only mode memory table into a sorted string table, store it in the cache on the disk, and synchronously upload it to the object storage; The update module is configured to update the information of the sorted string table to the cache on the disk.
13. A data reading device, comprising: A first search module is configured to search a memory table in the memory based on a read request; a second search module, configured to search a sorted string table in a cache on a disk in response to not finding data corresponding to the read request in the memory table in the memory; a third search module, configured to search the sorted string table in the storage object in response to not finding the data corresponding to the read request in the sorted string table in the cache; The first returning module is configured to return the found data corresponding to the read request in response to finding the data corresponding to the read request in the sorted string table in the storage object.
14. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-3 or 4-11.
15. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method described in any one of claims 1-3 or 4-11 is implemented.
16. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1-3 or 4-11.