Data storage method, device, equipment and computer readable storage medium
By dividing the cloud disk into multiple address spaces according to logical addresses and allocating erasure stripes with different compression ratios, and compressing the data before writing it, the problem of high storage costs in distributed storage systems is solved, achieving cost reduction and performance improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
- Filing Date
- 2024-02-27
- Publication Date
- 2026-07-21
AI Technical Summary
Existing block storage underlying distributed storage systems have high storage costs, especially in the three-replica mode, where storage costs are three times higher than before. Furthermore, the existing EC-then-compression method is complex and cumbersome, affecting system performance.
The cloud disk of the multi-replica node is divided into multiple address spaces according to the preset logical address, and multiple erasure stripes are allocated to each address space. The erasure stripes correspond to different compression ratio ranges and do not overlap. After the data to be written is written to the cloud disk, it is compressed to obtain compressed data, and then written into the corresponding erasure stripe.
Without compromising the reliability of the storage system, it significantly reduces storage costs, simplifies the data recycling process, and improves system performance.
Smart Images
Figure CN120560565B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a data storage method, apparatus, device, and computer-readable storage medium. Background Technology
[0002] The underlying distributed storage system of block storage generally ensures data reliability through multiple replicas. That is, the data written by the user corresponds to multiple complete copies of the data (usually three copies) at the underlying distributed system. This allows the user's data to continue to be accessed by the surviving copies when some machines fail and copies are lost. This approach has high reliability but high storage cost.
[0003] In addition, when writing data, the underlying distributed storage system of block storage often uses a redirected write method. That is, when a user repeatedly writes data to the same offset of the cloud disk, the underlying block does not overwrite the original data when writing to improve performance. Instead, it appends the data to the end of the block and marks the original data as garbage. The background garbage collection process moves the valid data in the block to a new block and releases the original block. The new block still uses a multi-replica mechanism.
[0004] In existing technologies, for a 400GB cloud disk, the original three-replica mode requires 400GB*3=1200GB of storage cost for block data at the underlying level, which is high. Therefore, there is an urgent need for a data storage method to reduce storage costs. Summary of the Invention
[0005] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a data storage method, apparatus, device, and computer-readable storage medium to reduce storage costs.
[0006] In a first aspect, embodiments of this disclosure provide a data storage method, including:
[0007] The cloud disk of the multi-replica node is divided into multiple address spaces according to a preset logical address, and multiple erasure stripes are allocated to each address space. The multiple erasure stripes correspond to different compression ratio ranges, and the compression ratio ranges do not overlap.
[0008] The data to be written is written to the cloud disk of the multi-replica node;
[0009] When recycling data in the cloud disk of the multi-replica node, the data to be written is compressed to obtain compressed data and a compression ratio, wherein the compression ratio is the memory ratio of the compressed data to the data to be written.
[0010] The compressed data is written into an erasure stripe corresponding to the compression ratio.
[0011] Secondly, embodiments of this disclosure provide a data storage device, comprising:
[0012] The allocation module is used to divide the cloud disk of the multi-replica node into multiple address spaces according to a preset logical address, and allocate multiple erasure stripes to each address space. The multiple erasure stripes correspond to different compression ratio ranges, and the compression ratio ranges do not overlap.
[0013] The write module is used to write the data to be written to the cloud disk of the multi-replica node;
[0014] A compression module is used to compress the data to be written when recycling the data in the cloud disk of the multi-replica node, to obtain compressed data and a compression ratio, wherein the compression ratio is the memory ratio of the compressed data to the data to be written.
[0015] The storage module is used to write the compressed data into erasure stripes corresponding to the compression ratio.
[0016] Thirdly, embodiments of this disclosure provide an electronic device, including:
[0017] Memory;
[0018] Processor; and
[0019] Computer programs;
[0020] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first aspect.
[0021] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in the first aspect.
[0022] Fifthly, embodiments of this disclosure also provide a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the method described in the first aspect.
[0023] The data storage method, apparatus, device, and computer-readable storage medium provided in this disclosure divide a cloud disk with multiple replica nodes into multiple address spaces according to a preset logical address, and allocates multiple erasure stripes to each address space. These erasure stripes correspond to different compression ratio ranges, and the compression ratio ranges do not overlap. Data to be written is written to the cloud disk of the multiple replica nodes. During the data reclamation process in the cloud disk of the multiple replica nodes, the data to be written is compressed to obtain compressed data and a compression ratio, where the compression ratio is the memory ratio of the compressed data to the data to be written. The compressed data is then written into the erasure stripe corresponding to the compression ratio. Compared to compression before compression, this disclosure provides a strategy of compressing the data to be written before writing it to erasure stripes. The cloud disk with multiple replica nodes is divided into multiple address spaces according to preset logical addresses, and multiple erasure stripes are allocated to each address space. The multiple erasure stripes correspond to different compression ratio ranges, and these compression ratio ranges do not overlap. When recycling data in the cloud disk, the compressed data is written to the erasure stripe corresponding to the compression ratio. Only one write of compressed data is needed during recycling, reducing cloud disk input and output, and greatly reducing storage costs without reducing the reliability of the storage system. Attached Figure Description
[0024] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 A flowchart illustrating the data storage method provided in this embodiment of the disclosure;
[0027] Figure 2 A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;
[0028] Figure 3 A flowchart illustrating a data storage method provided in another embodiment of this disclosure;
[0029] Figure 4 A schematic diagram of the structure of the data storage device provided in the embodiments of this disclosure;
[0030] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0031] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0032] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0033] The underlying distributed storage system of block storage generally ensures data reliability through multiple replicas. That is, the data written by the user corresponds to multiple complete copies of the data (usually three copies) at the underlying distributed system. This allows the user's data to continue to be accessed by the surviving copies when some machines fail and copies are lost. This approach has high reliability but high storage cost.
[0034] In addition, when writing data, the underlying distributed storage system of block storage often uses a redirected write method. That is, when a user repeatedly writes data to the same offset of the cloud disk, the underlying block does not overwrite the original data when writing to improve performance. Instead, it appends the data to the end of the block and marks the original data as garbage. The background garbage collection process moves the valid data in the block to a new block and releases the original block. The new block still uses a multi-replica mechanism.
[0035] For a 400GB cloud disk, the original three-replica mode requires 400GB * 3 = 1200GB of storage cost for block data at the underlying level. After introducing erasure coding (EC), if a 4+2 striping mode is used, the storage cost for block data at the underlying level is 100GB * 6 = 600GB. Compared to the three-replica mode, the storage cost is directly reduced to 50% of the original, which is the biggest benefit of using EC.
[0036] Besides EC (Extreme Collision Controller), distributed systems can also use compression to reduce costs. The core idea is to compress data before writing it to the underlying storage medium. Using a three-replica compression mode, the storage cost is reduced by the amount of data compression. If EC and compression are used simultaneously, that is, data is sequentially appended to multiple replicas in the foreground and compressed while the background garbage collection process moves the data to the EC stripe, the storage cost of the distributed storage system is greatly reduced.
[0037] If data is stored using an EC-first, compression-later approach—that is, all valid data is moved to the EC stripe, and then compression is performed on the complete block data of each column of the stripe, with each column triggering its own compression operation—it's equivalent to the original data needing to be written twice during the EC compression process (the first time is writing to the EC stripe before compression, and the second time is writing to the compressed block). The metadata needs to record the corresponding offset of each page of data in the compressed block. When reading a page, the metadata is looked up, the specified offset of the block is read, and then decompressed before being returned to the client. If a block corresponding to a column in the stripe becomes inaccessible due to server or hard drive failure, all remaining columns of the stripe need to be read, decompressed, and the data recovered. Then, the recovered data needs to be compressed to restore the abnormal stripe to a normal state. This process is quite complex and cumbersome.
[0038] For a 400GB cloud disk, after introducing erasure coding (EC) compression, assuming a 4+2 striping pattern and a data compression rate of 50%, the underlying storage cost of block data is 100GB * 6 * 50% = 300GB. Compared to the three-replica mode, this is reduced to 25% of the original cost. If the compression rate is 75%, the underlying storage cost of block data is 100GB * 6 * 75% = 450GB, reduced to 37.5% of the original cost. Therefore, how to apply erasure coding compression storage in the distributed storage system corresponding to the block storage layer to reduce storage costs has become an urgent problem to be solved. To address this problem, this disclosure provides a data storage method, which will be described below with reference to specific embodiments.
[0039] Figure 1 This is a flowchart illustrating a data storage method provided in an embodiment of this disclosure. The method can be executed by a data storage device, which can be implemented in software and / or hardware. The data storage device can be configured in an electronic device, such as a smartphone, tablet, laptop, in-vehicle navigation device, smart sports equipment, or other portable mobile device; it can also be a personal computer, smart home appliance, server, or other fixed device. The server can be a single server, a server cluster, a distributed cluster, or a centralized cluster. This method can be applied to scenarios involving data storage, as well as scenarios involving data management in a distributed storage system. It is understood that the data storage method provided in this embodiment can also be applied to other scenarios.
[0040] Figure 1 A flowchart of the data storage method provided in the embodiments of this disclosure is shown below. Figure 1As shown, the specific steps included in this method are as follows:
[0041] S101. Divide the cloud disk of the multi-replica node into multiple address spaces according to a preset logical address, and allocate multiple erasure stripes to each address space. The multiple erasure stripes correspond to different compression ratio ranges, and the compression ratio ranges do not overlap.
[0042] Electronic devices divide the cloud disk of multiple replica nodes into multiple address spaces according to preset logical addresses, and allocate multiple erasure stripes to each address space. For example, the cloud disk is divided into multiple address spaces of 32G size, and multiple erasure stripes are allocated to each address space. The multiple erasure stripes correspond to different compression ratio ranges. The erasure stripes and compression ratio ranges are related, and the compression ratio ranges do not overlap. The total range of the compression ratio is [0-1]. Each erasure stripe is responsible for a fixed address space.
[0043] S102. Write the data to be written to the cloud disk of the multi-replica node.
[0044] The electronic device acquires the data to be written, which may be page data, and writes it to the cloud disk with multiple replica nodes. Specifically, when writing data, a redirected write method is often used, that is, the user repeatedly writes data to the same offset of the cloud disk. When the underlying block is written, in order to improve performance, it does not overwrite the original data, but appends the data to the end of the block.
[0045] S103. When recycling the data in the cloud disk of the multi-replica node, the data to be written is compressed to obtain compressed data and a compression ratio, wherein the compression ratio is the memory ratio of the compressed data to the data to be written.
[0046] When electronic devices recycle data in cloud disks with multiple replica nodes, they compress the data to be written to obtain compressed data and a compression ratio. The length of the compressed data is not fixed, and the compression ratio is the memory ratio of the compressed data to the data to be written.
[0047] Optionally, the electronic device compresses the data to be written using a compression algorithm, such as Defate, Bzip, PPMd, RAR, LZMA, LZMA2, etc.
[0048] S104. Write the compressed data into an erasure stripe corresponding to the compression ratio.
[0049] Electronic devices write compressed data into erasure stripes that correspond to the compression ratio of the compressed data.
[0050] This embodiment of the disclosure divides the cloud disk of multiple replica nodes into multiple address spaces according to preset logical addresses, and allocates multiple erasure stripes to each address space. These erasure stripes correspond to different compression ratio ranges, and the compression ratio ranges do not overlap. Data to be written is written to the cloud disk of the multiple replica nodes. When reclaiming data in the cloud disk of the multiple replica nodes, the data to be written is compressed to obtain compressed data and a compression ratio, where the compression ratio is the memory ratio of the compressed data to the data to be written. The compressed data is then written to the erasure stripe corresponding to the compression ratio. Compared to EC followed by compression, this embodiment of the disclosure provides a strategy of compressing the data to be written before writing it to the erasure stripe. The cloud disk of the multiple replica nodes is divided into multiple address spaces according to preset logical addresses, and multiple erasure stripes are allocated to each address space. These multiple erasure stripes correspond to different compression ratio ranges, and these compression ratio ranges do not overlap. When reclaiming data in the cloud disk, the compressed data is written to the erasure stripe corresponding to the compression ratio. During the reclamation process, only one write operation of the compressed data is required, reducing cloud disk input and output and significantly reducing storage costs without compromising the reliability of the storage system.
[0051] Based on the above embodiments, writing the data to be written to the cloud disk of the multi-replica node includes: obtaining the data to be written, determining the writing position of the data to be written in the cloud disk of the multi-replica node, and appending the data to be written to the writing position in the cloud disk of the multi-replica node.
[0052] The electronic device acquires the data to be written. During the writing process, due to the use of redirected writes, the user repeatedly writes data to the same offset on the cloud disk; this same offset becomes the write location of the data to be written. After determining the write location of the data to be written in the cloud disk of the multi-replica node, the electronic device appends the data to that location. To improve performance, the original data is not overwritten; instead, the data to be written is appended to the end.
[0053] This embodiment of the disclosure obtains the data to be written and determines the writing position of the data to be written in the cloud disk of the multi-replica node; it appends the data to be written to the writing position in the cloud disk of the multi-replica node. Compared with the prior art of overwriting the original data, this embodiment of the disclosure appends the data to be written to the end, thereby improving the performance of the distributed storage system.
[0054] In some embodiments, writing the compressed data into an erasure stripe corresponding to the compression ratio includes: determining a target compression ratio range in which the compression ratio is located, and writing the compressed data into an erasure stripe corresponding to the target compression ratio range.
[0055] The electronic device determines the target compression ratio range and writes the compressed data into the erasure stripe corresponding to the target compression ratio range.
[0056] For example, an electronic device pre-allocates multiple erasure stripes to simultaneously support the writing of compressed data with different compression ratios. Each erasure strip only receives requests that the compression ratio of the compressed data meets the compression ratio range. For example, the compression ratio ranges of the four erasure strips are [0-25%], (25%-50%), (50%-75%), and (75%-1%). When the compression ratio of the compressed data is 20%, since 20% is within the compression ratio range [0-25%], that is, the target compression ratio range corresponding to the compressed data is [0-25%], the compressed data is written to the erasure strip corresponding to [0-25%].
[0057] Optionally, after determining the target compression ratio range in which the compression ratio is located and writing the compressed data into the erasure stripe corresponding to the target compression ratio range, the method further includes: aligning the erasure stripe with all-zero data according to the length of the compressed data to obtain the target data in the erasure stripe, wherein the target data includes the compressed data and the all-zero data.
[0058] The length of the compressed data to be written is not fixed. After the electronic device compresses the data to be written to obtain compressed data, it determines the length of the compressed data and aligns the erasure strip with all-zero data to obtain the target data in the erasure strip. The target data includes compressed data and all-zero data.
[0059] Optionally, according to the length of the compressed data, the erasure stripe is aligned with all-zero data to obtain the target data in the erasure stripe, including: determining the length of the compressed data, calculating the remainder between the length and a preset length; and writing all-zero data at the end of the compressed data according to the difference between the preset length and the remainder to obtain the target data in the erasure stripe.
[0060] The electronic device determines the length of the compressed data and calculates the remainder between the length and the preset length. Based on the difference between the preset length and the remainder, it writes all zeros at the end of the compressed data to obtain the target data in the erasure stripe.
[0061] For example, if the preset length is 1KB, and the compressed data obtained after compressing 4KB of data to be written is 1000B, the compression ratio is calculated as: 1000 / (4*1024), and the compression ratio is approximately 24.41%. This compression ratio is within the compression ratio range of [0-25%], that is, the target compression ratio range is [0-25%]. The remainder between the length 1000B and the preset length 1KB is 1000B. The difference between the preset length 1KB and the remainder 1000B is 24B. The calculation process is: 1024-1000=24B. Therefore, 24B of all-zero data is written at the end of the compressed data to obtain the target data in the erasure stripe. This target data includes 1000B of compressed data and 24B of all-zero data. If the compressed data obtained after compressing 4KB of data to be written is 1900B, the compression ratio is calculated as: 1900 / (4*1024), which is approximately 46.38%. This compression ratio falls within the range of (25%-50%), i.e., the target compression ratio range is (25%-50%). The remainder between the length 1900B and the preset length 1KB is 876B. The difference between the preset length 1KB and the remainder 876B is 148B. The calculation process is: 1024-876=148B. Therefore, 148B of all-zero data is written to the end of the compressed data to obtain the target data in the erasure stripe. This target data includes 1900B of compressed data and 148B of all-zero data. The same implementation process and principle apply when the compression ratio is within the range of (50%-75%) and (75%-1%), which will not be elaborated in this embodiment.
[0062] This embodiment of the disclosure determines the target compression ratio range in which the compression ratio is located, writes the compressed data into the erasure stripe corresponding to the target compression ratio range, determines the length of the compressed data, calculates the remainder between the length and the preset length, and writes all-zero data at the end of the compressed data according to the difference between the preset length and the remainder, thereby obtaining the target data in the erasure stripe. The target data includes compressed data and all-zero data, which improves the space utilization of the distributed storage system without reducing reliability.
[0063] Based on the above embodiments, the erasure stripe includes multiple data columns and multiple check columns; each address space includes multiple block address spaces, and the block address spaces correspond one-to-one with the data columns.
[0064] For example, such as Figure 2As shown, the erasure stripe (stripe1) includes 4 data columns and 2 parity columns. Stripe1 is a pre-allocated erasure stripe containing 6 columns: col[1] to col[4] are data columns, and col[5] to col[6] are parity columns. Each data column corresponds to a block address space. Blocks support sequential append writing, and their space is allocated according to the amount of data written. Each address space is 32G, and each address space can be divided into 4 block address spaces, each block address space being 8G.
[0065] Accordingly, the process of writing the target data into the erasure stripe includes: calling the write stripe interface to move a first data from a first address space of a preset data amount from the target data, and writing the first data into the data column corresponding to the first address space; calling the write stripe interface to move a second data from a second address space of a preset data amount from the target data, and writing the second data into the data column corresponding to the second address space; determining whether the number of times the write stripe interface is called meets a preset condition; if the number of times the write stripe interface is called meets the preset condition, then calculating the verification based on the data in multiple data columns, obtaining the verification data corresponding to each of the multiple verification columns, and writing the verification data into the corresponding verification column; writing the multiple data columns and the multiple verification columns into the hard disk of each data column and the server node corresponding to each verification column, respectively, and calling back the write stripe interface.
[0066] like Figure 2 As shown, when recycling data in the cloud disk of the multi-replica node, the WriteStripe interface is called to move the first data from the first address space of a preset data size, and write the first data into the data column corresponding to the first address space. During recycling, the compressed data written in the foreground is moved into the stripe. The WriteStripe process is called during the movement. Each WriteStripe call only writes to a specific data column, and the data size of a single WriteStripe call is limited to a preset data size. If the data size is less than the preset data size, zeros are added to the end.
[0067] The electronic device calls WriteStripe to move a preset amount of second data from the target data to the second address space, and writes the second data into the data column corresponding to the second address space.
[0068] The electronic device determines whether the number of times the WriteStripe interface is called meets a preset condition. Optionally, it can determine whether the number of times the WriteStripe interface is called has reached a limit, i.e., whether the number of calls has reached the number of data columns. If it is determined that the number of times the WriteStripe interface is called meets the preset condition, the electronic device calculates verification based on the data in multiple data columns, obtains the verification data corresponding to each of the multiple verification columns, and writes the verification data into the corresponding verification columns. In other words, when the WriteStripe call limit is reached, verification is calculated for multiple data columns.
[0069] Since multiple data columns and multiple check columns correspond to different server nodes, the electronic device writes the multiple data columns and multiple check columns to the hard disk of each data column and the server node corresponding to each check column, and then calls back the write stripe interface. Only after the callback can the compressed data continue to be written to the erasure stripe.
[0070] Optionally, determining whether the number of times the write stripe interface is called meets the preset conditions includes: determining whether the number of times the write stripe interface is called is equal to the number of data columns; when the number of times the write stripe interface is called is equal to the number of data columns, it is determined that the number of times the write stripe interface is called meets the preset conditions.
[0071] Optionally, if the number of times the write stripe interface is called is less than the number of data columns, it is determined that the number of times the write stripe interface is called does not meet the preset condition.
[0072] In some embodiments, the method further includes: if the number of times the write stripe interface is called does not meet a preset condition, then the write stripe interface is called to move a preset amount of third data from a third address space in the target data, and the third data is written into the data column corresponding to the third address space; the write stripe interface is called to move a preset amount of fourth data from a fourth address space in the target data, and the fourth data is written into the data column corresponding to the fourth address space, until the number of times the write stripe interface is called meets the preset condition.
[0073] If it is determined that the number of times the write stripe interface is called does not meet the preset condition, the electronic device continues to call WriteStripe to move the third data from the third address space of the target data by a preset amount of data, and write the third data into the data column corresponding to the third address space. Then, WriteStripe is called to move the fourth data from the fourth address space of the target data by a preset amount of data, and write the fourth data into the data column corresponding to the fourth address space, until the number of times the write stripe interface is called meets the preset condition.
[0074] In some embodiments, the method further includes: setting a call time threshold for the write stripe interface; for any call to the write stripe interface, if the time of the call to the write stripe interface is greater than the call time threshold, writing the remaining data column of the erasure stripe into all-zero data, and calculating a verification based on the all-zero data and the data in the written data column.
[0075] For example, an electronic device may set a call time threshold for the WriteStripe interface. If the call time threshold is exceeded after a certain WriteStripe call, the number of WriteStripe calls may not have reached the number of data columns. In this case, the remaining data columns will be written with all zeros, and the verification will be calculated using the all-zero data and the existing WriteStripe data.
[0076] In some embodiments, the method further includes: recording the position and offset of the target data; locating the target block address space of the erasure stripe by means of the position in response to the client reading the data to be written; obtaining compressed data contained in the target data according to the target block address space and the offset; decompressing the compressed data to obtain the data to be written; and sending the data to be written to the client.
[0077] Specifically, the electronic device records the position and offset of the target data; when the client reads the data to be written, it locates the target block address space of the erasure stripe by position; according to the target block address space, offset, and the compression ratio range corresponding to the erasure stripe, it reads a fixed length to obtain the target data, thereby obtaining the compressed data contained in the target data; it decompresses the compressed data to obtain the data to be written; and it sends the data to be written to the client.
[0078] For example, if the compression ratio of the data to be written is below 25%, and the erasure stripe has a compression ratio range of [0-25%], then the target block address space of the erasure stripe is located by position, and the target data 1KB long after the offset in the target block address space is read to obtain the compressed data contained in the target data; the compressed data is then decompressed to obtain the data to be written. The same implementation process and principle apply when the compression ratio of the data to be written is other, and will not be described in detail in this embodiment.
[0079] This disclosure describes in detail the process of writing target data into the erasure stripe, thereby greatly reducing storage costs without compromising the reliability of the storage system.
[0080] Figure 3 A flowchart of a data storage method provided in another embodiment of this disclosure, such as Figure 3 As shown, the method includes the following steps:
[0081] S301, Obtain the data to be written.
[0082] S302. Write the data to be written to the cloud disk of the multi-replica node.
[0083] S303. Divide the cloud disk of the multi-replica node into multiple address spaces according to a preset logical address, and allocate multiple erasure stripes to each address space. The multiple erasure stripes correspond to different compression ratio ranges, and the compression ratio ranges do not overlap.
[0084] S304. When recycling the compressed data in the cloud disk of the multi-replica node, the compressed data is written into the erasure stripe corresponding to the address space where the compressed data is stored.
[0085] S305. When a hard drive or server node failure occurs, the erasure stripe affected by the failure is identified as an abnormal erasure stripe, and the data in the abnormal erasure stripe is recovered and decompressed.
[0086] For example, if a file system is used, the file system will report an error; if an NVMe driver is used, the SPDK will also report an error. When a hard drive failure or server node failure occurs, the address space of a block corresponding to a column in the erasure stripe may become inaccessible. In this case, the electronic device performs data recovery and decompression. The recovery and decompression process involves reading the remaining columns in the erasure stripe, then using the EC algorithm to recover the data, and finally decompressing the recovered data using a decompression algorithm.
[0087] S306. Allocate a new erasure stripe. When recycling the compressed data in the address space corresponding to the abnormal erasure stripe, write the compressed data into the new erasure stripe.
[0088] Before data recovery and decompression are complete, the electronic device can allocate a new erasure stripe to support the writing of compressed data for recycling processing in the same mode. By writing the compressed data into the new erasure stripe, abnormal erasure stripes can be sealed, allowing the focus to be on data recovery without blocking the recycling process.
[0089] This disclosure, when encountering a hard drive or server node failure, identifies the affected erasure stripe as an aberrant erasure stripe, recovers and decompresses the data in the aberrant erasure stripe, and then allocates a new erasure stripe. During the recycling process of compressed data in the address space corresponding to the aberrant erasure stripe, the compressed data is written into the new erasure stripe. Compared to existing technologies, this disclosure focuses on data recovery and decompression of aberrant erasure stripes, allocating a new erasure stripe and writing compressed data into it, thus avoiding blocking data recycling and reducing cloud disk I / O overhead and data decompression overhead.
[0090] Figure 4 This is a schematic diagram of the structure of a data storage device provided in an embodiment of this disclosure. The data storage device may be an electronic device as described in the above embodiments, or it may be a component or assembly within that electronic device. The data storage device provided in this embodiment can execute the processing flow provided in the data storage method embodiments, such as... Figure 4 As shown, the data storage device 40 includes: an allocation module 41, a writing module 42, a compression module 43, and a storage module 44; wherein, the allocation module 41 is used to divide the cloud disk of the multi-replica node into multiple address spaces according to a preset logical address, and allocate multiple erasure stripes to each address space, the multiple erasure stripes corresponding to different compression ratio ranges, and the compression ratio ranges do not overlap; the writing module 42 is used to write the data to be written into the cloud disk of the multi-replica node; the compression module 43 is used to compress the data to be written when the data in the cloud disk of the multi-replica node is recycled, to obtain compressed data and a compression ratio, the compression ratio being the memory ratio of the compressed data to the data to be written; the storage module 44 is used to write the compressed data into the erasure stripe corresponding to the compression ratio.
[0091] Optionally, the writing module 42 is further configured to acquire the data to be written, determine the writing position of the data to be written in the cloud disk of the multi-replica node, and append the data to be written to the writing position in the cloud disk of the multi-replica node.
[0092] Optionally, the storage module 44 is further configured to determine the target compression ratio range in which the compression ratio is located, and write the compressed data into the erasure stripe corresponding to the target compression ratio range.
[0093] Optionally, the storage module 44 is further configured to align the erasure stripe with all-zero data according to the length of the compressed data to obtain target data in the erasure stripe, wherein the target data includes the compressed data and the all-zero data.
[0094] Optionally, the storage module 44 is further configured to determine the length of the compressed data, calculate the remainder between the length and a preset length, and write all-zero data at the end of the compressed data according to the difference between the preset length and the remainder, thereby obtaining the target data in the erasure stripe.
[0095] Optionally, the erasure stripe includes multiple data columns and multiple check columns; each address space includes multiple block address spaces, and the block address spaces correspond one-to-one with the data columns; the storage module 44 is further configured to call the write stripe interface to move a preset amount of first data from the target data in the first block address space, and write the first data into the data column corresponding to the first block address space; call the write stripe interface to move a preset amount of second data from the target data in the second block address space, and write the second data into the data column corresponding to the second address space; determine whether the number of times the write stripe interface is called meets a preset condition; if the number of times the write stripe interface is called meets the preset condition, calculate the check based on the data in the multiple data columns, obtain the check data corresponding to each of the multiple check columns, and write the check data into the corresponding check column; write the multiple data columns and the multiple check columns into the hard disk of each data column and the server node corresponding to each check column, and call back the write stripe interface.
[0096] Optionally, the storage module 44 is also used to determine whether the number of times the write stripe interface is called is equal to the number of data columns; when the number of times the write stripe interface is called is equal to the number of data columns, it is determined that the number of times the write stripe interface is called meets the preset condition.
[0097] Optionally, the storage module 44 is further configured to, if the number of times the write stripe interface is called does not meet a preset condition, call the write stripe interface to move a preset amount of third data from the target data in a third address space, and write the third data into the data column corresponding to the third address space; call the write stripe interface to move a preset amount of fourth data from the target data in a fourth address space, and write the fourth data into the data column corresponding to the fourth address space, until the number of times the write stripe interface is called meets the preset condition.
[0098] Optionally, the storage module 44 is further configured to set a call time threshold for the write stripe interface; for any call to the write stripe interface, if the time of the call to the write stripe interface is greater than the call time threshold, the remaining data column of the erasure stripe is written into all-zero data, and a verification is calculated based on the all-zero data and the data in the written data column.
[0099] Optionally, the data storage device further includes: a recording module, a positioning module, an acquisition module, a decompression module, and a sending module; wherein, the recording module is used to record the position and offset of the target data; the positioning module is used to locate the target block address space of the erasure stripe by means of the position in response to the client reading the data to be written; the acquisition module is used to acquire the compressed data contained in the target data according to the target block address space and the offset; the decompression module is used to decompress the compressed data to obtain the data to be written; and the sending module is used to send the data to be written to the client.
[0100] Optionally, the data storage device further includes a recovery module and a processing module; wherein, the recovery module is used to identify the erasure stripe affected by the failure as an abnormal erasure stripe when a hard disk or server node failure is encountered, and to recover and decompress the data in the abnormal erasure stripe; the processing module is used to allocate a new erasure stripe, and when recycling the compressed data in the address space corresponding to the abnormal erasure stripe, write the compressed data into the new erasure stripe.
[0101] Figure 4 The data storage device shown in the embodiment can be used to execute the technical solution of the above-described data storage method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.
[0102] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. The electronic device can be a terminal as described in the above embodiments. The electronic device provided in this embodiment of the present disclosure can execute the processing flow provided in the data storage method embodiments, such as… Figure 5 As shown, the electronic device 50 includes: a memory 51, a processor 52, a computer program, and a communication interface 53; wherein the computer program is stored in the memory 51 and configured to be executed by the processor 52 using the data storage method described above.
[0103] In addition, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the data storage method described in the above embodiments.
[0104] Furthermore, this disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the data storage method described above.
[0105] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0106] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.
[0107] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0108] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:
[0109] The cloud disk of the multi-replica node is divided into multiple address spaces according to a preset logical address, and multiple erasure stripes are allocated to each address space. The multiple erasure stripes correspond to different compression ratio ranges, and the compression ratio ranges do not overlap.
[0110] The data to be written is written to the cloud disk of the multi-replica node;
[0111] When recycling data in the cloud disk of the multi-replica node, the data to be written is compressed to obtain compressed data and a compression ratio, wherein the compression ratio is the memory ratio of the compressed data to the data to be written.
[0112] The compressed data is written into an erasure stripe corresponding to the compression ratio.
[0113] In addition, the electronic device can also perform other steps in the data storage method described above.
[0114] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0116] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.
[0117] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0118] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0119] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0120] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data storage method, characterized in that, The method includes: The cloud disk of the multi-replica node is divided into multiple address spaces according to a preset logical address, and multiple erasure stripes are allocated to each address space. The multiple erasure stripes correspond to different compression ratio ranges, and the compression ratio ranges do not overlap. The data to be written is written to the cloud disk of the multi-replica node; When recycling data in the cloud disk of the multi-replica node, the data to be written is compressed to obtain compressed data and a compression ratio, wherein the compression ratio is the memory ratio of the compressed data to the data to be written. The compressed data is written into an erasure stripe corresponding to the compression ratio.
2. The method according to claim 1, characterized in that, Write the data to be written to the cloud disk on the multiple replica nodes, including: Obtain the data to be written and determine the write location of the data to be written in the cloud disk of the multi-replica node; The data to be written is appended to the write location in the cloud disk of the multi-replica node.
3. The method according to claim 1, characterized in that, The step of writing the compressed data into an erasure stripe corresponding to the compression ratio includes: Determine the target compression ratio range in which the compression ratio falls, and write the compressed data into the erasure stripe corresponding to the target compression ratio range.
4. The method according to claim 3, characterized in that, After determining the target compression ratio range in which the compression ratio falls, and writing the compressed data into the erasure stripe corresponding to the target compression ratio range, the method further includes: Based on the length of the compressed data, the erasure stripe is aligned using all-zero data to obtain the target data in the erasure stripe, wherein the target data includes the compressed data and the all-zero data.
5. The method according to claim 4, characterized in that, Based on the length of the compressed data, the erasure stripe is aligned using all-zero data to obtain the target data in the erasure stripe, including: Determine the length of the compressed data and calculate the remainder between the length and the preset length; Based on the difference between the preset length and the remainder, all-zero data is written to the end of the compressed data to obtain the target data in the erasure stripe.
6. The method according to claim 5, characterized in that, The erasure stripe includes multiple data columns and multiple check columns; each address space includes multiple block address spaces, and the block address spaces correspond one-to-one with the data columns. The process of writing the target data into the erasure stripe includes: Call the write stripe interface to move the first data from the first address space of the first block of address space with a preset amount of data from the target data, and write the first data into the data column corresponding to the first address space; The write stripe interface is invoked to move a preset amount of second data from the target data to the second address space, and the second data is written into the data column corresponding to the second address space; Determine whether the number of times the write stripe interface is called meets the preset conditions; If the number of times the write stripe interface is called meets the preset conditions, then the verification is calculated based on the data in multiple data columns, the verification data corresponding to each of the multiple verification columns is obtained, and the verification data is written into the corresponding verification column. Write the multiple data columns and the multiple check columns to the hard disk of each data column and the server node corresponding to each check column, and then call back the write stripe interface.
7. The method according to claim 6, characterized in that, The determination of whether the number of times the write stripe interface is called meets the preset conditions includes: Determine if the number of times the write stripe interface is called is equal to the number of data columns; When the number of times the write stripe interface is called is equal to the number of data columns, it is determined that the number of times the write stripe interface is called meets the preset condition.
8. The method according to claim 6, characterized in that, The method further includes: If the number of times the write stripe interface is called does not meet the preset condition, the write stripe interface is called to move the third data from the third address space of the target data by a preset amount of data, and the third data is written into the data column corresponding to the third address space. The write stripe interface is called to move a preset amount of fourth data from the target data to the fourth address space, and the fourth data is written into the data column corresponding to the fourth address space until the number of times the write stripe interface is called meets the preset condition.
9. The method according to claim 6, characterized in that, The method further includes: Set the call time threshold for the write stripe interface; For any call to the write stripe interface, if the time of the call to the write stripe interface is greater than the call time threshold of the write stripe interface, the remaining data column of the erasure stripe is written to all zero data, and a verification is calculated based on the all zero data and the data in the already written data column.
10. The method according to claim 6, characterized in that, The method further includes: Record the position and offset of the target data; In response to the client reading the data to be written, the target block address space of the erasure stripe is located through the position; The compressed data contained in the target data is obtained based on the target block address space and the offset; Decompress the compressed data to obtain the data to be written; The data to be written is sent to the client.
11. The method according to claim 1, characterized in that, The method further includes: When a hard drive or server node failure occurs, the erasure stripe affected by the failure is identified as an abnormal erasure stripe, and the data in the abnormal erasure stripe is recovered and decompressed. A new erasure stripe is allocated, and when the compressed data in the address space corresponding to the abnormal erasure stripe is reclaimed, the compressed data is written into the new erasure stripe.
12. A data storage device, characterized in that, The device includes: The allocation module is used to divide the cloud disk of the multi-replica node into multiple address spaces according to a preset logical address, and allocate multiple erasure stripes to each address space. The multiple erasure stripes correspond to different compression ratio ranges, and the compression ratio ranges do not overlap. The write module is used to write the data to be written to the cloud disk of the multi-replica node; A compression module is used to compress the data to be written when recycling the data in the cloud disk of the multi-replica node, to obtain compressed data and a compression ratio, wherein the compression ratio is the memory ratio of the compressed data to the data to be written. The storage module is used to write the compressed data into erasure stripes corresponding to the compression ratio.
13. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-11.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-11.