Data storage method and device, electronic device, and storage medium
Through data reorganization technology, random small blocks of write data are converted into sequential large block requests, combined with dynamic mapping of cache blocks and segmented data space, the problem of low processing efficiency of random write-back requests in the data storage system is solved, and data is improved Processing efficiency and overall performance of the storage system.
Patent Information
- Application Number
- CN202210493066.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-07
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-05-07
AI Technical Summary
When existing data storage systems write data back from the cache layer to the data layer, they cannot effectively process random writeback requests, resulting in excessive pressure on the data disk, especially when the data disk is composed of HDD or QLC SSD, the processing efficiency is low. .
Using data reorganization technology, random small blocks of write data from the front end are converted into write back requests in sequential large blocks, and the write back process is optimized and the data is reduced through the dynamic mapping relationship between pre-divided cache blocks and segmented data space. Random write pressure of the disk.
Through data reorganization technology, the access speed of hybrid storage media is improved, the rapid data sorting and processing efficiency is improved, and random write-back requests are effectively responded to, reducing the load on the data disk.
Smart Images

Figure CN114879909B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data storage, and in particular to a data storage method and device, electronic equipment, and a storage medium. Background Art
[0002] In the related art, the current data storage system generally includes a variety of storage media (for example, hard disk drives HDD and solid-state drives SATA SSD) or mixed media as a cost-effective solution, which is usually divided into a cache layer (or cache disk) and a data layer (or data layer). The cache layer generally includes high-speed disks (such as NVMe SSD or ordinary SSD), and the data layer includes relatively slow disks (for example, ordinary SSD, HDD or QLC SSD), wherein the capacity of the cache disk is smaller than the capacity of the data disk. In this way, the request from the client is first written to the cache disk, and it can be returned immediately, and then the data is slowly written to the data disk through the writeback mechanism (writeback). Since the writeback process is performed in the background, it will not interfere with the front-end IO.
[0003] In the process of data storage and write-back through the data storage system, the core is the Cache algorithm. The core of the write-back mechanism in the Cache algorithm is the mapping relationship between the cache disk Block and the data disk Block. The current mechanism usually divides the cache disk into several Blocks (the Block size can be 4KB, 8KB, etc.), and then we also divide the data disk into several corresponding Blocks; when the front-end request is to write to the data disk Block A, the Cache algorithm will select a Block A' of the SSD, write the data to Block A' first, and then mark this Block A' as dirty data (Dirty), at this time, the mapping relationship between the cache disk Block A' and the data disk Block A is established.
[0004] The above cache algorithm matches the blocks of the high-speed cache disk to the blocks of the low-speed data disk one by one. However, this process does not re-regulate the requests from the front end, that is, if the front-end IO is random, then the write-back process will also be random. Random write-back puts a lot of pressure on the data disk. If the data disk is composed of HDD or QLC SSD, then the pressure on the data disk will be very high.
[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0006] Embodiments of the present invention provide a data storage method and device, an electronic device, and a storage medium to at least solve the technical problem that when writing data from a cache layer to a data layer, random write-back requests cannot be effectively processed.
[0007] According to one aspect of an embodiment of the present invention, there is provided a data storage method, which is applied to a cache layer in a data storage system, wherein the cache space corresponding to the cache layer includes a plurality of pre-divided cache blocks, and comprises: receiving a data processing request initiated by a client, wherein the data processing request carries at least: business data to be processed; storing the business data in a plurality of cache blocks, and identifying the plurality of cache blocks to obtain a plurality of identified cache blocks; combining the plurality of identified cache blocks to obtain a write-back request when the number of the plurality of identified cache blocks is greater than a preset number threshold; writing the write-back request to each segmented data space in the data layer, and determining a dynamic mapping relationship of data from the identified cache block to the segmented data space.
[0008] Optionally, the step of combining the multiple identification cache blocks to obtain a write-back request includes: obtaining the cache interval to which each identification cache block belongs, and determining the interval start identifier of the cache interval; based on the interval start identifier, merging the cache intervals to obtain the write-back request.
[0009] Optionally, after writing the write-back request to each segmented data space in the data layer, it also includes: obtaining the interval information of the cache interval and the pre-stored business data; pre-processing the business data to obtain business processing data and pre-processing information; determining the business processing data as segmented data, and determining the pre-processing information as segmented header information; combining the segmented header information and the segmented data into segmented record information; and appending the segmented record information to the segmented data space.
[0010] Optionally, the step of determining the data dynamic mapping relationship from the identification cache block to the segmented data space includes: determining the mapping start bit based on the interval start identifier of the cache interval to which the identification cache block belongs and the interval size of the cache interval; obtaining the segment start bit of the segmented data space in the overall data space in the data layer; determining the segment space start bit and the segment space size of the cache interval to which the identification cache block belongs in the segmented data space based on the segment start bit; and establishing the data dynamic mapping relationship from the mapping start bit to the segment space start bit.
[0011] Optionally, after determining the dynamic mapping relationship of data from the identification cache block to the segmented data space, it also includes: when a new write-back request is generated, allocating a valid segmented data space to the new write-back request; and writing the new write-back request to the allocated valid segmented data space.
[0012] Optionally, it also includes: when a new write-back request is generated, determining whether the cache interval used by the new write-back request is reused; when the cache interval used by the new write-back request is reused, reclaiming the segmented data space pointed to by the reused cache interval in the historical process, and marking the recovered segmented data space invalid.
[0013] Optionally, the step of reclaiming the segmented data space pointed to by the reused cache interval in the historical process includes: when there are multiple segmented data spaces to be reclaimed, analyzing the space utilization rate and the internal storage data volume of the segmented data space; and reclaiming the multiple segmented data spaces to be reclaimed respectively according to the sorting order of the space utilization rate and the internal storage data volume.
[0014] According to another aspect of an embodiment of the present invention, a data storage device is also provided, which is applied to a cache layer in a data storage system, wherein the cache space corresponding to the cache layer includes multiple pre-divided cache blocks, including: a receiving unit, used to receive a data processing request initiated by a client, wherein the data processing request carries at least: business data to be processed; a storage unit, used to store the business data in multiple cache blocks, and identify the multiple cache blocks to obtain multiple identified cache blocks; a combining unit, used to combine the multiple identified cache blocks to obtain a write-back request when the number of the multiple identified cache blocks is greater than a preset number threshold; a writing unit, used to write the write-back request to each segmented data space in the data layer, and determine the dynamic mapping relationship of data from the identified cache block to the segmented data space.
[0015] Optionally, the combination unit includes: a first acquisition module, used to obtain the cache interval to which each of the identified cache blocks belongs, and determine the interval start identifier of the cache interval; a first merging module, used to merge the cache intervals based on the interval start identifier to obtain the write-back request.
[0016] Optionally, the data storage device also includes: a second acquisition module, used to obtain the interval information of the cache interval and the pre-stored business data after writing the write-back request to each segmented data space in the data layer; a pre-processing module, used to pre-process the business data to obtain business processing data and pre-processing information; a first determination module, used to determine the business processing data as segmented data, and determine the pre-processing information as segment header information; a second merging module, used to combine the segment header information and the segmented data into segment record information; a first writing module, used to append the segment record information to the segmented data space.
[0017] Optionally, the write unit includes: a first determination module, used to determine the mapping start bit based on the interval start identifier of the cache interval to which the identification cache block belongs and the interval size of the cache interval; a third acquisition module, used to obtain the segmentation start bit of the segmented data space in the overall data space in the data layer; based on the segmentation start bit, determine the segmentation space start bit and segmentation space size of the cache interval to which the identification cache block belongs in the segmented data space; a first establishment module, used to establish a data dynamic mapping relationship from the mapping start bit to the segmentation space start bit.
[0018] Optionally, the data storage device also includes: a first allocation module, which is used to allocate a valid segmented data space to a new write-back request after determining the dynamic mapping relationship of data from the identification cache block to the segmented data space when a new write-back request is generated; and a second writing module, which is used to write the new write-back request to the allocated valid segmented data space.
[0019] Optionally, the data storage device also includes: a first judgment module, used to determine whether the cache interval used by the new write-back request is reused when a new write-back request is generated; a recycling module, used to recycle the segmented data space pointed to by the reused cache interval in the historical process when the cache interval used by the new write-back request is reused, and mark the recovered segmented data space invalid.
[0020] Optionally, the recycling module includes: an analysis submodule, used to analyze the space utilization rate and the internal storage data volume of the segmented data space when there are multiple segmented data spaces to be recycled; and a recycling submodule, used to recycle the multiple segmented data spaces to be recycled respectively according to the sorting order of the space utilization rate and the internal storage data volume.
[0021] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned data storage methods by executing the executable instructions.
[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the data storage methods described above.
[0023] In an embodiment of the present invention, a data processing request initiated by a receiving client is adopted, wherein the data processing request carries at least: business data to be processed, the business data is stored in multiple cache blocks, and the multiple cache blocks are identified to obtain multiple identification cache blocks, and when the number of multiple identification cache blocks is greater than a preset number threshold, the multiple identification cache blocks are combined to obtain a write-back request, the write-back request is written to each segmented data space in the data layer, and the dynamic mapping relationship of data from the identification cache block to the segmented data space is determined. In this embodiment, the random small block write data from the front end can be converted into a sequential large block write-back request through data re-regularization technology, so that random write-backs in different situations can be handled, the access speed of the hybrid storage medium can be improved, data can be quickly organized, and data processing efficiency can be improved, thereby solving the technical problem that random write-back requests cannot be effectively processed when writing data from the cache layer to the data layer. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0025] Figure 1 is a flow chart of an optional data storage method according to an embodiment of the present invention;
[0026] Figure 2 is a schematic diagram of an optional data storage device according to an embodiment of the present invention;
[0027] Figure 3 It is a hardware structure block diagram of an electronic device (or mobile device) according to a data storage method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0030] To facilitate those skilled in the art to understand the present invention, some terms or nouns involved in the embodiments of the present invention are explained below:
[0031] Mechanical disk, Hard Disk Drive, referred to as HDD;
[0032] Solid State Disk, referred to as SSD, is the flash memory particles of the storage medium;
[0033] NVMe SSD, SSD with NVMe interface, a high-speed SSD;
[0034] QLC (quad-level cell) SSD is a high-density SSD that is cheaper than previous SSDs, but has a shorter write hit rate.
[0035] The present invention can be applied to various data storage systems. The data storage systems adopt multiple persistence devices, which may include but are not limited to: a cache layer and a data layer, wherein the cache layer may include a high-speed disk (such as NVMeSSD or ordinary SSD), and the data layer may include a relatively slow disk (for example, ordinary SSD, HDD or QLC SSD), wherein the capacity of the cache layer is smaller than the capacity of the data layer.
[0036] The present invention adopts a hybrid media management mechanism in the data storage system. The mechanism can manage a disk or a logical space composed of a group of disks. That is, several cache disks can be combined into a logical cache space, and relatively low-speed data disks can be combined into a logical low-speed data space. According to the same strategy, a write-back mechanism from the cache space to the low-speed data space can be implemented.
[0037] In the present invention, a logical data space solution composed of multiple groups of disks can solve the problem of low efficiency of data disk in processing random small IO data when the cache space is insufficient and data needs to be written back to the data layer under the premise of random small IO. This application introduces data re-regularization technology to convert random small IO into sequential large IO, fully utilizing the large IO processing capacity of the data disk, that is, the re-regularization technology can be used to conveniently fill data without dividing the data layer into data blocks (blocks) that correspond one to one with the cache layer.
[0038] In the present application, the write-back data can be set to meet the storage conditions of the disk storage medium, so as to improve the processing capacity of the disk (for example, if the resource pool adopts EC as the copy strategy, then it can be aligned with the EC stripe by filling data; for example, the erase block size of the data pool SSD can be aligned to optimize the SSD write behavior).
[0039] The present invention is described below in conjunction with various embodiments.
[0040] Embodiment 1
[0041] According to an embodiment of the present invention, a data storage method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0042] This embodiment provides a data storage method, which is applied to a cache layer in a data storage system, wherein a cache space corresponding to the cache layer includes a plurality of pre-divided cache blocks.
[0043] Figure 1 is a flow chart of an optional data storage method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:
[0044] Step S102, receiving a data processing request initiated by a client, wherein the data processing request carries at least: business data to be processed;
[0045] Step S104, storing the business data in a plurality of cache blocks, and marking the plurality of cache blocks to obtain a plurality of marked cache blocks;
[0046] Step S106, when the number of the plurality of identification cache blocks is greater than a preset number threshold, combining the plurality of identification cache blocks to obtain a write-back request;
[0047] Step S108, writing the write-back request into each segmented data space in the data layer, and determining a data dynamic mapping relationship from the identification cache block to the segmented data space.
[0048] Through the above steps, a data processing request initiated by a client can be received, wherein the data processing request carries at least: business data to be processed, the business data is stored in multiple cache blocks, and multiple cache blocks are identified to obtain multiple identification cache blocks, and when the number of multiple identification cache blocks is greater than a preset number threshold, multiple identification cache blocks are combined to obtain a write-back request, the write-back request is written to each segmented data space in the data layer, and the dynamic mapping relationship of data from the identification cache block to the segmented data space is determined. In this embodiment, the random small block write data from the front end can be converted into a sequential large block write-back request through data re-regularization technology, so that different write-back requests can be responded to, data can be quickly organized, and data processing efficiency can be improved, thereby solving the technical problem that random write-back requests cannot be effectively processed when writing data from the cache layer to the data layer.
[0049] This embodiment introduces data re-regularization technology. For the Cache algorithm used by the data storage system, when establishing the mapping relationship between the blocks of the cache layer and the blocks of the data layer, there is no need to correspond the blocks of the high-speed cache layer and the blocks of the low-speed data layer one by one. By splitting into Log Append writing technology, the random small-block write data from the client (or front end) is converted into sequential large-block write requests. Sequential writing effectively gives play to the sequential large-block write capability of the data layer.
[0050] The embodiments of the present invention are described in detail below in combination with the above-mentioned implementation steps.
[0051] Step S102: receiving a data processing request initiated by a client, wherein the data processing request carries at least business data to be processed.
[0052] In this embodiment, the business write IO from the client / front end will be sent to the cache layer / cache disk / cache space through the data processing request. The data writing of the cache layer is fast and the data cache can be regularized in time. All business write IO from the client / front end can be written to the cache space of the cache layer first.
[0053] Step S104: store the business data in a plurality of cache blocks, and mark the plurality of cache blocks to obtain a plurality of marked cache blocks.
[0054] In this embodiment, the cache space in the cache layer may include at least one high-speed storage disk, and the cache space is pre-divided into multiple cache blocks, which are managed at the cache block block granularity, and the entire cache space can be divided into N blocks. Optionally, the size of the divided cache blocks in this embodiment is divided according to the size of the data written in each request, and the size of the cache blocks is inconsistent, for example, 4K, 8K, 16K, 128K, etc.
[0055] Each time new business data is written, a corresponding number of block spaces are allocated. After the data is written to these blocks, the cache blocks that have been written with data can be marked to obtain identified cache blocks. For example, these blocks can be marked as dirty blocks. After completing the block marking, the write success information can be returned to the client / front end.
[0056] As the client continues to write data, the blocks in the cache layer will gradually be consumed. When the consumption reaches a certain level, the data in the cache disk needs to be written back to the data layer to free up enough space to process new requests. Step S106: When the number of multiple identified cache blocks is greater than a preset number threshold, multiple identified cache blocks are combined to obtain a write-back request.
[0057] That is, as cache blocks are continuously allocated and data is written in the cache space, the number of identified cache blocks increases, and the consumption of cache space also increases. Then, according to the preset write-back policy, the adjustment can be triggered (for example, when the number of cache blocks reaches a certain threshold, the cache duration reaches a certain preset duration threshold, and the usage rate of the cache space reaches a certain usage rate threshold), these identified cache blocks can be pieced together to form a large write-back request, ready to be written to the data space in the data layer. That is, as the blocks are consumed, the number of dirty blocks increases. At this time, we can piece these blocks together to form a large write-back request, ready to be written to the data space.
[0058] In this embodiment, for the data layer in the data storage system, the data space is pre-divided into logical segmented data space - Segment. The size of the Segment is flexibly configured, generally larger than 1M, and each Segment corresponds to a physical space in the data space.
[0059] Optionally, the step of combining multiple identification cache blocks to obtain a write-back request includes: obtaining the cache interval to which each identification cache block belongs, and determining the interval start identifier of the cache interval; based on the interval start identifier, merging the cache intervals to obtain the write-back request.
[0060] In the cache space, since the size of the data written each time is inconsistent and the number of data blocks divided each time is inconsistent, it is necessary to set multiple cache intervals to identify the cache blocks for one or more data processing requests. In this way, the size of the write-back request is also inconsistent, and the requests from the write-back are not necessarily continuous. It contains several cache intervals, marked as A (off, len), B (off, len), C (off, len), etc., and these intervals are merged - Merge. For example, for cache interval A in the SSD, its size is 4K, from 0K-4k, starting from the 1Mth in the cache space, then the interval to which cache interval A belongs is from 1M+0K to 1M+4K, and the interval starting identifier of cache interval A is 1M; for cache interval B in the SSD, its size is 4K, from 8k-12K, starting from the 2Mth in the cache space, then the interval to which cache interval B belongs is from 2M+8k to 2M+12K, and the interval starting identifier of cache interval B is 2M+8k; for cache interval C in the SSD, its size is 16K, from 16k-32K, starting from the 3Mth in the cache space, then the interval to which cache interval C belongs is from 3M+16k to 3M+32K, and the interval starting identifier of cache interval C is 3M+16k.
[0061] In this embodiment, the selection of the cache interval during the write-back process is flexibly configured and can also be selected based on certain characteristics to reduce the cost of subsequent dynamic mapping and persistence of data in the mixed medium. For example, the application can select the corresponding cache space based on the characteristics of persistence, so that subsequent persistence only needs to update a small amount of data.
[0062] Step S108, writing the write-back request into each segmented data space in the data layer, and determining a data dynamic mapping relationship from the identification cache block to the segmented data space.
[0063] Another optional method, after writing the write-back request to each segmented data space in the data layer, also includes: obtaining the interval information of the cache interval and the pre-stored business data; pre-processing the business data to obtain business processing data and pre-processing information; determining the business processing data as segmented data, and determining the pre-processing information as segmented header information; combining the segmented header information and the segmented data into segmented record information; and appending the segmented record information to the segmented data space.
[0064] The data written back at one time can be very large. Each cache interval is in the form of a segment record information - SegmentEntry. The Entry records the information and data of these cache intervals, which is called segment header information - EntryHeader and segment data, and records data information - EntryData. The data can be processed as needed (checksum or compression, etc.). This additional information can be easily added to the EntryHeader and then written to the above segment data space - Segment in an append-write manner.
[0065] Optionally, the step of determining a dynamic mapping relationship of data pointed to by an identification cache block to a segmented data space includes: determining a mapping start bit based on an interval start identifier of a cache interval to which the identification cache block belongs and an interval size of the cache interval; obtaining a segment start bit of the segmented data space in the overall data space in the data layer; determining a segmented space start bit and a segmented space size of the cache interval to which the identification cache block belongs in the segmented data space based on the segment start bit; and establishing a dynamic mapping relationship of data pointed to by a mapping start bit to a segmented space start bit.
[0066] The dynamic mapping of mixed media data is the core idea of the entire algorithm. By continuously updating the mapping, large blocks of sequential data space can be written. This dynamic data mapping relationship can be called metadata mapping, which needs to be persisted. Since each cache space needs to form a mapping, the persistence process of the mapping may bring additional overhead.
[0067] Although the data dynamic mapping established each time is very small, since persistence must at least align the logical blocks of the disk, updating only one mapping each time is very wasteful of the disk. In this embodiment, the overhead of the mapping information part can be reduced by batch updating.
[0068] During the write-back process, it is necessary to record the mapping relationship between the cache space and the segmented data space. For example, the mapping between the cache space A(off,len) and the segment space SA(off,len) is recorded, which is called dynamic data mapping of hybrid media.
[0069] In this embodiment, for a resource pool composed of several disks, the write-back process can fill data so that the written-back data meets a certain condition, under which certain physical characteristics of the data disk can be better utilized (for example, if the data space adopts EC as the copy strategy, it can be aligned with the EC stripe by filling data; for example, the erase block size of the data pool SSD can be aligned to optimize the SSD write behavior).
[0070] As an optional implementation of the present invention, after determining the dynamic mapping relationship of data pointed to the segmented data space by the identification cache block, it also includes: when a new write-back request is generated, allocating a valid segmented data space to the new write-back request; writing the new write-back request to the allocated valid segmented data space.
[0071] Optionally, the present application also includes: when a new write-back request is generated, determining whether the cache interval used by the new write-back request is reused; when the cache interval used by the new write-back request is reused, recycling the segmented data space pointed to by the reused cache interval in the historical process, and marking the recycled segmented data space invalid.
[0072] When an overwrite occurs and a new write-back request is formed, the same cache space A(off, len) will correspond to different Segment spaces SB(off, len). At this time, the old space SA(off, len) will be marked as invalid and require garbage collection. In this application, a bitmap can be used to mark the valid / invalid space of the segmented data space. The bitmap can reduce invalid data movement in the subsequent garbage collection stage.
[0073] An optional step of reclaiming the segmented data space pointed to by the reused cache interval in the historical process includes: when there are multiple segmented data spaces to be reclaimed, analyzing the space utilization rate and internal storage data volume of the segmented data space; and reclaiming the multiple segmented data spaces to be reclaimed respectively according to the sorting order of the space utilization rate and the internal storage data volume.
[0074] In this embodiment, the garbage collection algorithm is started according to the used capacity of the data disk, and when the capacity usage rate is low, the data space with a high amount of garbage can be recycled first.
[0075] Another option is to take more stringent measures during the off-peak period of business, such as performing garbage collection earlier or more vigorously to free up space in advance.
[0076] The space recycling process is the process of recycling invalid segmented data space - Segment space. During the recycling process, the valid data space is read according to the segment bitmap content, and then a new segment is allocated and written to the new data space in an append-write manner. Then, the dynamic data mapping of the mixed medium is modified so that the cache space points to the new data space. After the modification is completed, the old data space is recycled for subsequent data write-back.
[0077] In this embodiment, hot spot data is cached by a cache disk, and colder data is written back to the data disk, thereby giving full play to the advantages of high performance and long life of the cache disk; the write back uses the append write technology to give full play to the advantages of some types of disks with poor random write capabilities but good large-block sequential write capabilities; the append write process can conveniently fill in some data, including reprocessing the data (such as checksum and compression, etc.), and the reprocessing process is of great significance for giving full play to certain characteristics of the data disk.
[0078] The present application is described below in conjunction with another optional embodiment.
[0079] Embodiment 2
[0080] This embodiment provides a data storage device, which is applied to a cache layer in a data storage system. The cache space corresponding to the cache layer includes multiple pre-divided cache blocks. The data storage device provides multiple implementation units, each implementation unit corresponds to each implementation step in the above-mentioned embodiment one.
[0081] Figure 2 is a schematic diagram of an optional data storage device according to an embodiment of the present invention, such as Figure 2 As shown, the device may include: a receiving unit 21, a storage unit 23, a combining unit 25, and a writing unit 27, wherein:
[0082] The receiving unit 21 is used to receive a data processing request initiated by a client, wherein the data processing request carries at least: business data to be processed;
[0083] The storage unit 23 is used to store the business data in a plurality of cache blocks, and mark the plurality of cache blocks to obtain a plurality of marked cache blocks;
[0084] A combining unit 25, configured to combine the multiple identification cache blocks to obtain a write-back request when the number of the multiple identification cache blocks is greater than a preset number threshold;
[0085] The writing unit 27 is used to write the write-back request into each segmented data space in the data layer, and determine the data dynamic mapping relationship from the identification cache block to the segmented data space.
[0086] The above-mentioned data storage device can receive a data processing request initiated by a client through a receiving unit 21, wherein the data processing request carries at least: business data to be processed, the business data is stored in multiple cache blocks through a storage unit 23, and multiple cache blocks are identified to obtain multiple identification cache blocks, and when the number of multiple identification cache blocks is greater than a preset number threshold, the multiple identification cache blocks are combined through a combination unit 25 to obtain a write-back request, and the write-back request is written to each segmented data space in the data layer through a writing unit 27, and a dynamic mapping relationship of data from the identification cache block to the segmented data space is determined. In this embodiment, the random small block write data from the front end can be converted into a sequential large block write-back request through a data re-regularization technology, so that different write-back requests can be responded to, data can be quickly organized, and data processing efficiency can be improved, thereby solving the technical problem that random write-back requests cannot be effectively processed when writing data from the cache layer to the data layer.
[0087] Optionally, the combination unit includes: a first acquisition module, used to obtain the cache interval to which each identified cache block belongs, and determine the interval start identifier of the cache interval; a first merging module, used to merge the cache intervals based on the interval start identifier to obtain a write-back request.
[0088] Optionally, the data storage device also includes: a second acquisition module, used to obtain the interval information of the cache interval and the pre-stored business data after writing the write-back request to each segmented data space in the data layer; a pre-processing module, used to pre-process the business data to obtain business processing data and pre-processing information; a first determination module, used to determine the business processing data as segmented data, and determine the pre-processing information as segmented header information; a second merging module, used to combine the segmented header information and the segmented data into segmented record information; a first writing module, used to append the segmented record information to the segmented data space.
[0089] Optionally, the write unit includes: a first determination module, used to determine the mapping start bit based on the interval start identifier of the cache interval to which the cache block belongs and the interval size of the cache interval; a third acquisition module, used to obtain the segmentation start bit of the segmented data space in the overall data space in the data layer; based on the segmentation start bit, determine the segmentation space start bit and the segmentation space size of the cache interval to which the cache block belongs in the segmented data space; a first establishment module, used to establish a data dynamic mapping relationship from the mapping start bit to the segmentation space start bit.
[0090] Optionally, the data storage device also includes: a first allocation module, which is used to allocate a valid segmented data space to a new write-back request after determining the dynamic mapping relationship of data pointed to the segmented data space by the identification cache block and in the event that a new write-back request is generated; and a second writing module, which is used to write the new write-back request to the allocated valid segmented data space.
[0091] Optionally, the data storage device also includes: a first judgment module, used to determine whether the cache interval used by the new write-back request is reused when a new write-back request is generated; a recycling module, used to recycle the segmented data space pointed to by the reused cache interval in the historical process when the cache interval used by the new write-back request is reused, and mark the recovered segmented data space invalid.
[0092] Optionally, the recycling module includes: an analysis submodule, used to analyze the space utilization rate and internal storage data volume of the segmented data space when there are multiple segmented data spaces to be recycled; and a recycling submodule, used to recycle the multiple segmented data spaces to be recycled separately according to the sorting order of the space utilization rate and the internal storage data volume.
[0093] The above-mentioned data storage device may also include a processor and a memory. The above-mentioned receiving unit 21, storage unit 23, combination unit 25, writing unit 27, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.
[0094] The processor includes a kernel, which retrieves the corresponding program unit from the memory. The kernel can be set to one or more, and the kernel parameters are adjusted to write the write-back request to each segmented data space in the data layer, and determine the dynamic mapping relationship of the data from the identification cache block to the segmented data space.
[0095] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one storage chip.
[0096] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above data storage methods by executing the executable instructions.
[0097] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above data storage methods.
[0098] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: receiving a data processing request initiated by a client, wherein the data processing request carries at least: business data to be processed; storing the business data in multiple cache blocks, and identifying the multiple cache blocks to obtain multiple identification cache blocks; when the number of the multiple identification cache blocks is greater than a preset number threshold, combining the multiple identification cache blocks to obtain a write-back request; writing the write-back request to each segmented data space in the data layer, and determining a dynamic mapping relationship of data from the identification cache block to the segmented data space.
[0099] Figure 3 1 is a hardware structure block diagram of an electronic device (or mobile device) according to a data storage method according to an embodiment of the present invention. Figure 3 As shown, the electronic device may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations are shown.
[0100] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0101] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0104] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
[0106] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A data storage method, It is characterized in that A cache layer applied to a data storage system, wherein the cache space corresponding to the cache layer includes a plurality of pre-divided cache blocks, including: Receiving a data processing request initiated by a client, wherein the data processing request carries at least: business data to be processed; The business data is stored in a plurality of cache blocks, and the plurality of cache blocks are marked to obtain a plurality of marked cache blocks; When the number of the plurality of identification cache blocks is greater than a preset number threshold, combining the plurality of identification cache blocks to obtain a write-back request; Writing the write-back request to each segmented data space in the data layer, and determining a dynamic data mapping relationship from the identification cache block to the segmented data space; wherein, during the write-back process, the mapping relationship from the cache space to the segmented data space is recorded; After writing the write-back request into each segmented data space in the data layer, the method further includes: obtaining interval information of the cache interval and the pre-stored business data; pre-processing the business data to obtain business processing data and pre-processing information; determining the business processing data as segmented data, and determining the pre-processing information as segment header information; combining the segment header information and the segmented data into segment record information; and appending the segment record information into the segmented data space; The step of determining the data dynamic mapping relationship from the identification cache block to the segmented data space includes: determining the mapping start bit based on the interval start identifier of the cache interval to which the identification cache block belongs and the interval size of the cache interval; obtaining the segment start bit of the segmented data space in the overall data space in the data layer; determining the segment space start bit and the segment space size of the cache interval to which the identification cache block belongs in the segmented data space based on the segment start bit; and establishing the data dynamic mapping relationship from the mapping start bit to the segment space start bit.
2. The method according to claim 1, It is characterized in that The step of combining the multiple identification cache blocks to obtain a write-back request includes: Obtaining the cache interval to which each identified cache block belongs, and determining the interval start identifier of the cache interval; Based on the interval start identifier, the cache intervals are merged to obtain the write-back request.
3. The method according to claim 2, It is characterized in that After determining the data dynamic mapping relationship from the identification cache block to the segmented data space, the method further includes: In the case of generating a new write-back request, allocating a valid segmented data space for the new write-back request; The new write-back request is written to the allocated valid segment data space.
4. The method according to claim 3, It is characterized in that Also includes: In the case where a new write-back request is generated, determining whether the cache interval used by the new write-back request is reused; In the case where the cache interval used by the new write-back request is reused, the segmented data space pointed to by the reused cache interval in the historical process is recycled, and the recycled segmented data space is marked invalid.
5. The method according to claim 4, It is characterized in that The step of reclaiming the segmented data space pointed to by the reused cache interval in the historical process comprises: In the case where there are a plurality of segmented data spaces to be reclaimed, analyzing the space usage and the amount of internally stored data of the segmented data spaces; According to the order of the space usage rate and the amount of internally stored data, the plurality of segmented data spaces to be reclaimed are respectively reclaimed.
6. A data storage device, It is characterized in that A cache layer applied to a data storage system, wherein the cache space corresponding to the cache layer includes a plurality of pre-divided cache blocks, including: A receiving unit, configured to receive a data processing request initiated by a client, wherein the data processing request carries at least: business data to be processed; A storage unit, used for storing the business data in a plurality of cache blocks, and marking the plurality of cache blocks to obtain a plurality of marked cache blocks; A combining unit, configured to combine the plurality of identification cache blocks to obtain a write-back request when the number of the plurality of identification cache blocks is greater than a preset number threshold; A writing unit, used for writing the write-back request into each segmented data space in the data layer, and determining a dynamic mapping relationship of data from the identification cache block to the segmented data space; wherein, during the write-back process, a mapping relationship from the cache space to the segmented data space is recorded; The data storage device further includes: a second acquisition module, used to acquire the interval information of the cache interval and the pre-stored business data after writing the write-back request into each segmented data space in the data layer; a pre-processing module, used to perform pre-processing on the business data to obtain business processing data and pre-processing information; a first determination module, used to determine the business processing data as segmented data, and determine the pre-processing information as segment header information; a second merging module, used to combine the segment header information and the segment data into segment record information; a first writing module, used to append the segment record information to the segment data space; The write unit includes: a first determination module, used to determine the mapping start bit based on the interval start identifier of the cache interval to which the identification cache block belongs and the interval size of the cache interval; a third acquisition module, used to obtain the segmentation start bit of the segmented data space in the overall data space in the data layer; based on the segmentation start bit, determine the segmentation space start bit and segmentation space size of the cache interval to which the identification cache block belongs in the segmented data space; a first establishment module, used to establish a data dynamic mapping relationship from the mapping start bit to the segmentation space start bit.
7. An electronic device, It is characterized in that include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to perform the data storage method described in any one of claims 1 to 5 by executing the executable instructions.
8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the data storage method according to any one of claims 1 to 5.
