Data writing method and device, electronic equipment and computer readable storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XINHUASAN INFORMATION TECH CO LTD
- Filing Date
- 2024-06-25
- Publication Date
- 2026-07-21
Smart Images
Figure CN118838541B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of storage technology, and in particular to data writing methods, apparatus, electronic devices and computer-readable storage media. Background Technology
[0002] With the continuous development of storage technology, compared with traditional disks, flash-based solid-state drives (referred to as flash drives) are widely used in storage systems due to their advantages in performance, reliability, and space utilization. Here, a storage system that uses flash drives as storage media can be called an all-flash system.
[0003] In all-flash systems, garbage collection of junk data is necessary to improve system space utilization. However, since garbage collection often involves the migration of valid data, write amplification occurs. Write amplification refers to a situation where the actual amount of data written to the flash drive by the system is much greater than the amount of data written to the system by other electronic devices. Write amplification increases the number of write operations to the flash drive, leading to problems such as decreased flash speed, shortened lifespan, and reduced stability. Therefore, reducing write amplification is a pressing technical problem that needs to be solved. Summary of the Invention
[0004] In view of this, this application provides a data writing method, apparatus, electronic device, and computer-readable storage medium to reduce write amplification.
[0005] This application provides a data writing method applied to a storage device; the storage device provides X flash drives as memory for an external designated device connected to the storage device, where X is greater than 1; each flash drive includes Y storage blocks, where Y is greater than 1; the storage blocks of the X flash drives are grouped into Z storage segments, where Z is greater than 1; in the initialization state, the storage device is configured with a shared pool that records information of the Z storage segments; the method includes:
[0006] When a data write request sent by an externally designated device is detected, the data write request at least indicates the data to be written and the first storage location identifier corresponding to the storage location where the data is to be written; a first data heat level matching the first storage location identifier is determined; the first data heat level is used to indicate the data update frequency at the storage location corresponding to the first storage location identifier;
[0007] If there is no storage segment in the currently used storage segment that matches the first data popularity level, then based on the information of the free storage segments recorded in the shared pool, a free storage segment is selected as the first target storage segment that matches the first data popularity level, and the data is written to the first target storage segment; if there is a first target storage segment in the currently used storage segment that matches the first data popularity level and has enough free storage space to accommodate the data, then the data is written to the free storage space in the first target storage segment.
[0008] This application embodiment also provides a data writing device configured in a storage device; the storage device provides X flash drives as memory for an external designated device connected to the storage device, where X is greater than 1; each flash drive includes Y storage blocks, where Y is greater than 1; the storage blocks of the X flash drives are grouped into Z storage segments, where Z is greater than 1; in the initialization state, the storage device is configured with a shared pool that records information of the Z storage segments; the device includes:
[0009] The determination module is used to determine, when a data write request sent by an externally specified device is detected, that the data write request at least indicates the data to be written and a first storage location identifier corresponding to the storage location where the data is to be written; determine a first data heat level matching the first storage location identifier; the first data heat level is used to indicate the data update frequency at the storage location corresponding to the first storage location identifier;
[0010] The write module is configured to, if there is no storage segment in the currently used storage segment that matches the first data popularity level, select an idle storage segment as the first target storage segment that matches the first data popularity level based on the information of the idle storage segments recorded in the shared pool, and write the data into the first target storage segment; if there is a first target storage segment in the currently used storage segment that matches the first data popularity level and has enough idle storage space to accommodate the data, then write the data into the idle storage space of the first target storage segment.
[0011] This application also provides an electronic device, which includes:
[0012] Processor; and
[0013] The memory stores computer program instructions that, when executed by the processor, cause the processor to perform the steps described above.
[0014] This application also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the above method.
[0015] As can be seen from the above technical solutions, in this embodiment, when a data write request sent by an external designated device is detected, the data hotness level of the storage location corresponding to the data to be written is first determined. The data hotness level indicates the data update frequency at the storage location corresponding to the data to be written. Then, the target storage segment to be written is determined according to the determined data hotness level for the data to be written. This can make some or all of the data in the same storage segment become invalid and become garbage data in approximately the same time, thereby effectively reducing the amount of valid data in the storage segment when garbage data is subsequently reclaimed, and realizing the reduction of write amplification. Attached Figure Description
[0016] The accompanying drawings, which are incorporated in and form part of this application, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0017] Figure 1 This is a schematic diagram illustrating the SSD disk garbage collection principle provided in an embodiment of this application.
[0018] Figure 2 This is a flowchart illustrating the method provided in an embodiment of this application.
[0019] Figure 3 This is a schematic diagram illustrating the implementation of hot and cold data discrimination provided in an embodiment of this application.
[0020] Figure 4 This is a schematic diagram of an array provided for an embodiment of this application.
[0021] Figure 5 This is a schematic diagram of array weight configuration provided in an embodiment of this application.
[0022] Figure 6 This is a schematic diagram illustrating the implementation of waste recycling as provided in an embodiment of this application.
[0023] Figure 7 This is a structural diagram of the device provided in the embodiments of this application.
[0024] Figure 8 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation
[0025] To facilitate understanding of this solution, before describing this solution, the technical problems existing in the existing storage systems involved in the embodiments of this application will be described first:
[0026] In all-flash systems, garbage collection is primarily addressed at two levels. The first level involves individual SSDs (i.e., flash drives) as storage media, and the second level is the entire all-flash system. Here, garbage collection refers to a mechanism for reclaiming space occupied by invalid data. In storage systems, dynamic writing and deletion of data generate a large amount of invalid data, and the space occupied by this invalid data cannot be immediately released. The main purpose of garbage collection is to identify and reclaim this invalid data, thereby improving the space utilization of the storage system.
[0027] The following section describes garbage collection at the SSD level:
[0028] The reason why SSDs need garbage collection is related to their internal working principle. SSDs use flash memory to store data, and their internal storage units are usually composed of data pages and storage blocks. For example, the size of a data page (also called a page) is usually 4KB, and a block usually contains a fixed number of pages (usually a power of 2). The size of a block usually ranges from hundreds of KB to thousands of KB.
[0029] The smallest unit of writing on an SSD is a page. However, due to the nature of flash memory chips, data cannot be updated locally (i.e., in-place update) after it has been written. The existing data must be erased before new data can be written. Similarly, due to the nature of flash memory chips, erasing operations must be performed in blocks, and compared to reading and writing, erasing operations are particularly time-consuming (in milliseconds; in contrast, reading and writing a page takes about 10 microseconds). This inconsistency between the granularity of the data writing and erasing units, coupled with the excessively slow erasing operation, prevents data from being updated in place; updated data must be written to a new location.
[0030] See Figure 1 As shown, a storage block 100 includes multiple data pages 101; data A, B, and C are first written to three pages in a certain block, which is called Block X (i.e., storage block X). Then, data D, E, and F are written to Block X; and data A, B, and C are updated to A′, B′, and C′. At this point, due to the characteristic that flash memory cannot be updated locally, A′, B′, and C′ need to be written to new pages in Block X (if there is no free space in Block X). Figure 1In this context, "free" represents an idle page (which can be considered as free space within a block). If data is written to the free space of another block, the original data A, B, and C become garbage data, and the pages they occupy cannot be used. To reuse the pages occupied by this garbage data, a free block Y (i.e., storage block Y) needs to be found. The valid data from block X (including data D, E, F, and data A', B', and C') is copied to block Y, and then block X is completely erased. In this way, the space occupied by the garbage data A, B, and C is reclaimed.
[0031] It's important to note that the garbage collection process on SSDs is transparent to upper-layer applications (i.e., applications on electronic devices with all-flash systems). In other words, from the application's or OS's perspective, the logical location of updated data is the file offset or logical block address (LBA). However, the SSD uses a Flash Translation Layer (FTL) to remap these logical addresses (such as LBAs) to the SSD's physical addresses, which is imperceptible to the upper layers. Here, LBA is a method used to identify the location of data storage on the disk.
[0032] The following describes the all-flash system level:
[0033] From the perspective of an all-flash system, there are two types of write strategies: in-place write and log write. In in-place write, data is written directly to its original location. When existing data needs to be updated or modified, in-place write performs the write operation directly at the original location, overwriting the existing data. From the perspective of an SSD, this means it may receive a large amount of random input / output (IO).
[0034] Log writing involves sequentially writing new data into a log. When existing data needs to be updated, the new data is appended sequentially to the log, while the previously written data remains unchanged. Since data is written sequentially to the log, it can achieve high write performance in scenarios with intensive, small I / O writes. However, the updated data becomes garbage data and requires garbage collection to reuse its space. From the perspective of the SSD, its data is written sequentially.
[0035] For both hard disks (HDDs) and SSDs, sequential write performance is far superior to random write performance. Furthermore, if garbage collection is implemented effectively at the system level, it's theoretically unnecessary to perform garbage collection at the SSD level. This significantly reduces the SSD's over-provisioned space (typically occupying more than 10% of the total SSD capacity, which is not directly usable by the user), thereby improving storage utilization. Therefore, all-flash systems based on log-based writing are widely adopted.
[0036] It's clear that garbage collection is a common requirement, both in a single SSD and in an all-flash system based on log-based writing. This is because garbage collection involves the migration of valid data, inevitably leading to write amplification. From a system perspective, this means the actual amount of data written to the SSD by the all-flash system is far greater than the amount of data written by the host application. Write amplification negatively impacts SSDs because it increases the number of write operations to the flash memory chips, leading to problems such as decreased flash speed, shortened lifespan, and reduced stability. Therefore, reducing write amplification at the system level has become a pressing technical challenge.
[0037] In this embodiment, for ease of management and implementation, the used space of the SSD disk as the storage medium can be further grouped. For example, several consecutive fixed-number blocks can be grouped into a storage segment (a segment can be called a segment, section, or zone, etc., referred to as a segment in this embodiment). In this way, an SSD disk will contain several segments. To fully utilize the performance of the all-flash system, the data writing rules meet the following requirements: the all-flash system is implemented based on log-write; the all-flash system is implemented based on log-write; the all-flash system can concurrently write to multiple segments; for each segment, data is written sequentially, and the remaining free segments are written only after the segment is full; garbage collection is performed on a segment-by-segment basis.
[0038] Under the above requirements, it can be seen that initially, the all-flash system is full of free segments, which gradually decrease as data is written. Simultaneously, data updates will cause the amount of garbage data in the segments to gradually increase. At some point, the all-flash system needs to perform garbage collection to obtain free segments. At this point, in order to obtain minimum write amplification, the following two problems exist:
[0039] 1) How should user-written data be laid out in segments (because data can be written concurrently to multiple data segments);
[0040] 2) What strategies should be adopted when performing garbage collection (e.g., which segments should be garbage collected; how to rearrange the data for valid data in garbage segments, etc.)?
[0041] This embodiment first introduces the concept of data lifecycle, which refers to the time data exists before it is updated. After an update, previously written data becomes garbage data. Therefore, if data is updated multiple times, in an all-flash system, all data except the last one written becomes garbage data. Regarding the first problem mentioned above, from a omniscient perspective, when writing data, data with similar or nearly similar lifecycles should be placed in the same segment. This way, the entire segment's data would become garbage data within almost the same timeframe; at this point, reclaiming the segment wouldn't require any effective data migration, and the entire segment could be directly erased, achieving optimal write amplification. However, in reality, there is no omniscient perspective, so we can only estimate the future lifecycle of data based on its past update frequency. For example, if data in a corresponding storage location is frequently updated, it can be considered that this data still has a high probability of being updated in the future; this data can be called hot data. Conversely, if data is not updated for a long time, it can be called cold data. It's important to note that predicting data lifecycle involves considering not only the average number of data updates over a period but also the recency of the updates (recency refers to the time interval between the last update of data at a storage location and the current time). For example, data in storage locations A and B may have been updated 10 times in the past 3 minutes. However, updates in storage location A all occurred within the first minute, while updates in storage location B occurred within the third minute. In this case, the update frequency is the same for both locations, but storage location B has a higher recency. This means that data in storage location B is more likely to be updated in the near future, indicating that the data in storage location B is more "hot." Therefore, in this embodiment, when writing data, hot and cold data need to be separated and placed in different segments so that data within each segment becomes invalid and becomes garbage data within approximately the same time frame, thereby reducing write amplification. Thus, identifying hot and cold data becomes a problem. The aforementioned storage location can be understood as the storage location (such as a logical address) where the data carried in the data write request is to be written.
[0042] In summary, from the perspective of reducing system write amplification, the garbage collection problem of all-flash systems can be abstracted into the following two problems:
[0043] 1) Hot and cold data identification problem (after identification, hot and cold data are placed in different segments);
[0044] 2) In waste recycling, the selection of recycling sections and how to arrange the effective data in the waste sections are issues.
[0045] To address the aforementioned technical problems, embodiments of this application provide a data writing method. To enable those skilled in the art to better understand the technical solutions provided in these embodiments, and to make the aforementioned objectives, features, and advantages of these embodiments more apparent, the technical solutions in these embodiments will be further described in detail below with reference to the accompanying drawings.
[0046] See Figure 2 , Figure 2 This is a flowchart illustrating a method provided in an embodiment of this application. The method is applied to a storage device; the storage device provides X flash drives as memory for an externally designated device connected to it, where X is greater than 1; each flash drive includes Y storage blocks, where Y is greater than 1; the storage blocks of the X flash drives are grouped into Z storage segments, where Z is greater than 1; in the initialization state, the storage device is configured with a shared pool that records information about the Z storage segments. As an example, the storage device may refer to a database server or similar device, and the externally designated device may refer to the electronic device corresponding to the client; this embodiment is not specifically limited.
[0047] like Figure 2 As shown, the process may include the following steps:
[0048] Step 201: When a data write request sent by an externally designated device is detected, the data write request at least indicates the data to be written, the first storage location identifier corresponding to the storage location where the data is to be written, and determines the first data heat level matching the first storage location identifier.
[0049] In this embodiment, when a data write request sent by an externally designated device is detected, a first data heat level matching the first storage location identifier indicated by the data write request is determined. Here, the first data heat level is used to indicate the data update frequency at the storage location corresponding to the first storage location identifier.
[0050] In this embodiment, the first storage location identifier refers to the location identifier corresponding to the storage location where the data to be written is to be written. For example, the first storage location identifier can be the LBA address where the data to be written is to be written. This embodiment does not specifically limit this.
[0051] As an example, K arrays are pre-configured based on actual needs, each array including multiple element positions. Any used element position in these K arrays is used to record the number of data updates at the storage location corresponding to the storage location identifier that matches the used element position. Based on this, determining the first data heat level matching the first storage location identifier in step 201 can be implemented as follows: First, determine whether the currently used array meets the following conditions: the number of used element positions is less than a set number, and / or, the timer duration corresponding to the currently used array has not reached the set timer duration. If so, find the element position matching the first storage location identifier in the currently used array, increment the value at that element position by a set value (which can be considered as updating the number of data updates recorded at that element position), and determine the first data heat level matching the first storage location identifier based on the value at the element position matching the first storage location identifier in each array; here, the set value can be 1. Otherwise, initialize the next array and use it as the currently used array, then return to the step of determining whether the currently used array meets the conditions.
[0052] In this embodiment, any array needs to be initialized before the step of determining whether the currently used array meets the conditions is executed. Here, initialization may specifically include: initializing the value at each element position in the array to a set element value (such as 0), initializing the corresponding used element position record of the array to 0, and starting the timer corresponding to the array, etc., which are not specifically limited in this embodiment.
[0053] As for how to find the element position that matches the first storage location identifier from the currently used array, and how to determine the first data heat level that matches the first storage location identifier based on the value of the element position that matches the first storage location identifier in each array, examples will be given below, and will not be elaborated here.
[0054] Step 202: If there is no storage segment in the currently used storage segment that matches the first data popularity level, then based on the information of the free storage segments recorded in the shared pool, select a free storage segment as the first target storage segment that matches the first data popularity level, and write the data into the first target storage segment; if there is a first target storage segment in the currently used storage segment that matches the first data popularity level and has enough free storage space to accommodate the data, then write the data into the free storage space of the first target storage segment.
[0055] In this embodiment, the segment description information corresponding to any used storage segment contains at least a data popularity level that matches the storage segment. That is, the storage location identifiers of each data in any used storage segment have the same data popularity level. Based on this, in step 202, it is first checked whether there is at least one storage segment in the currently used storage segment that matches the first data popularity level; if not, based on the information of the free storage segments recorded in the shared pool, a free storage segment is selected as the first target storage segment that matches the first data popularity level, and the data is written to the first target storage segment; if there is a first target storage segment in the currently used storage segment that matches the first data popularity level and has enough free storage space to accommodate the data, the data is written to the free storage space in the first target storage segment.
[0056] As one embodiment, the above-mentioned selection of an idle storage segment as the first target storage segment for matching the first data popularity level based on the information of idle storage segments already recorded in the shared pool can be implemented as follows: based on the information of idle storage segments already recorded in the shared pool, an idle storage segment is randomly selected from the shared pool as the first target storage segment for matching the first data popularity level, and the first data popularity level matching the storage segment is added to the segment description information of the first target storage segment. Based on this, after selecting an idle storage segment as the first target storage segment for matching the first data popularity level based on the information of idle storage segments already recorded in the shared pool, the information of the first target storage segment is deleted from the recorded shared pool information.
[0057] This concludes the process. Figure 2 The process is shown below.
[0058] pass Figure 2 As can be seen from the process shown, this embodiment, when detecting a data write request sent by an external designated device, first determines the data heat level matching the storage location identifier of the data to be written. The data heat level indicates the data update frequency at the storage location corresponding to the data to be written. Then, according to the determined data heat level, the target storage segment to be written is determined for the data to be written. This can make some or all of the data in the same storage segment become invalid and become garbage data in approximately the same time, thereby effectively reducing the amount of valid data in the storage segment when garbage data is subsequently reclaimed, and realizing the reduction of write amplification.
[0059] The following describes how to determine the first data popularity level matching the first storage location identifier based on the value at the position of the element in each array that matches the first storage location identifier:
[0060] As an example, the above method of determining the target data popularity level matching the storage location identifier based on the value of the element position matching the first storage location identifier in each array can be implemented in various ways. For example, firstly, find the element position matching the first storage location identifier from the configured K arrays; then, determine the first data popularity level based on the weight of each array and the value of the element position matching the first storage location identifier in each array.
[0061] In this embodiment, as an example, each of the K configured arrays corresponds to a hash function; here, to avoid hash collisions, the hash functions corresponding to each array are different. Based on this, the above-mentioned finding the element position matching the first storage location identifier from the K configured arrays can be implemented as follows: for each array, substitute the first storage location identifier into the hash function corresponding to that array to obtain the hash result; based on the hash result, find the element position corresponding to the hash result in that array.
[0062] Optionally, the above method of finding the element position corresponding to the hash result in the array based on the hash result can be implemented as follows: using the hash result as the element position index to find the element position corresponding to the hash result in the array.
[0063] Correspondingly, the specific implementation of finding the element position that matches the first storage location identifier from the currently used array in step 101 is similar to the specific implementation of finding the element position that matches the first storage location identifier from the configured K arrays, and will not be elaborated here.
[0064] As one example, multiple data popularity levels are pre-configured, each corresponding to a range of popularity values. Based on this, there are many specific implementations for determining the first data popularity level according to the weights of each array and the values at the positions of elements in each array that match the first storage location identifier. For example, firstly, based on the weights of each array and the values at the positions of elements in each array that match the first storage location identifier, a popularity value matching the first storage location identifier is determined; then, based on the configured range of popularity values corresponding to each data popularity level, a data popularity level matching the popularity value is determined from at least one configured data popularity level to serve as the first data popularity level. Here, the popularity value indicates the probability that the data to be written in the data request at the storage location corresponding to the first storage location identifier will be frequently updated in the future; a higher popularity value indicates a higher probability that the corresponding data will be frequently updated in the future.
[0065] Optionally, the weight values corresponding to each array are dynamically changed, depending on the array currently in use, the order of the arrays, and the pre-configured weight rules. How to determine the weights of each array when using their respective weights to determine the first data popularity level will be described with examples below, and will not be elaborated upon here.
[0066] The following describes the garbage collection of used storage segments:
[0067] As one embodiment, when the number of free storage segments determined based on the recorded shared pool information is less than a set threshold, at least one storage segment requiring garbage data reclamation is identified from the currently used storage segments. While valid data is still stored in this storage segment, the valid data is migrated to other storage segments, and garbage data stored in that storage segment is garbage collected. Afterward, this storage segment is designated as a free storage segment and added to the shared pool, and its information is added to the recorded shared pool information. Here, the set threshold can be set based on actual needs, such as 3, 4, or 5 segments.
[0068] In this embodiment, there are many specific ways to determine at least one storage segment that needs to be garbage collected from the currently used storage segments. For example, as an embodiment, the top P storage segments with the largest amount of garbage data are selected from the currently used storage segments as the storage segments that need to be garbage collected, where P is greater than or equal to 1.
[0069] As another embodiment, for each of the currently used storage segments, the recycling reference value corresponding to the storage segment is first determined based on the data update frequency recorded in the segment description information of the storage segment and the effective data occupancy rate corresponding to the storage segment; wherein, the data update frequency refers to the data update frequency at the storage location where the specified data is written, and the specified data is any data within the storage segment; then, the Q storage segments with the largest recycling reference values are selected from the currently used storage segments as the storage segments that need to be garbage collected, where Q is greater than or equal to 1.
[0070] Optionally, as an embodiment, the above-mentioned determination of the recycling reference value corresponding to the storage segment based on the data update frequency recorded in the segment description information of the storage segment and the effective data occupancy rate corresponding to the storage segment can be implemented as follows: The recycling reference value corresponding to the storage segment can be calculated by the following formula: Recycling reference value = (1-U)*(1 / f), where U represents the effective data occupancy rate, U = the number of effective data in the storage segment / the total number of data in the storage segment; f represents the recorded data update frequency.
[0071] Optionally, for any used array, the array description information of the used array must at least contain the timer duration of the most recent use of the used array. There are many specific ways to determine the recorded data update frequency. For example, first, find the element position in each configured array that matches the storage location identifier corresponding to the specified data; then, determine the recorded data update frequency by the ratio between the sum of the values at each found element position and the sum of the timer durations corresponding to each array.
[0072] As one example, the above-mentioned migration of valid data to other storage segments can be implemented in many ways. For instance, for each piece of valid data, based on the second location identifier corresponding to the storage location where the valid data is written, the element position matching the second location identifier is found from the configured K arrays; any array contains multiple element positions. Here, how to find the element position matching the second location identifier from the configured K arrays is similar to how to find the element position matching the first storage location identifier from the configured K arrays, and will not be repeated here.
[0073] Then, based on the weights of each array and the values at the positions of the elements in each array that match the second storage location identifier, the second data popularity level matching the second storage location identifier is determined. The second data popularity level is used to indicate the data update frequency at the storage location corresponding to the second storage location identifier. Here, how to determine the second data popularity level matching the second storage location identifier is similar to how to determine the first data popularity level based on the weights of each array and the values at the positions of the elements in each array that match the first storage location identifier, and will not be repeated here.
[0074] Finally, if there is no storage segment in the currently used storage segment that matches the second data popularity level, then based on the information of the free storage segments recorded in the shared pool, a free storage segment is selected as the second target storage segment that matches the second data popularity level, and the valid data is written to the second target storage segment; if there is a second target storage segment in the currently used storage segment that matches the second data popularity level and has free storage space to accommodate the valid data, then the valid data is written to the free storage space in the second target storage segment.
[0075] To facilitate understanding of the specific implementation process of the above method, it is described below through specific embodiments.
[0076] This embodiment provides a method for garbage data recycling in an all-flash system. The method mainly involves the following aspects:
[0077] 1) Distinguish between hot and cold data for write I / O with minimal computational and memory overhead;
[0078] 2) Hot and cold data can be classified in multiple levels according to their popularity (used to put data with similar life cycles into the same segment);
[0079] 3) The distinction between hot and cold data can simultaneously capture both frequency and recency information;
[0080] 4) During waste recycling, the effective data in the recycled waste section is re-identified as hot or cold data, and the data layout is made accordingly based on the identification results.
[0081] This embodiment optimizes data writing and garbage collection in an all-flash system, reduces system write amplification, improves SSD lifespan and write performance, and enhances system stability.
[0082] The following is a description of the hot / cold data discrimination method in this embodiment:
[0083] See Figure 3 As shown, the method includes the following steps:
[0084] Step 301: After receiving the write IO from the user, extract the LBA address carried in the write IO;
[0085] Step 302: Based on the LBA address of the write IO, determine the data hotness level of the write IO and obtain the determination result;
[0086] Step 302: Based on the judgment result, write the data to be written corresponding to the write IO to the segment with the corresponding popularity.
[0087] It's important to note that all-flash systems can configure the levels of hot and cold data (referred to as data popularity levels). For example, they can be configured with two levels (cold and hot), three levels (cold, warm, and hot), or any L levels, where L is a positive integer greater than or equal to 2. After the parameter L is pre-configured, the data stored in each segment must have the same popularity level. Data in each segment is written sequentially. At any given time, each segment will be in one of four states: "idle," "in use," "full," or "currently undergoing garbage collection." "Idle" indicates that the segment contains no data; "in use" indicates that the segment is being used; "full" indicates that the segment's storage space is full, with no free space; and "currently undergoing garbage collection" indicates that the segment is undergoing garbage data collection.
[0088] The following describes the determination of data popularity levels (i.e., the hot / cold data distinction mentioned above):
[0089] See Figure 4 As shown, the system allocates K ordered arrays, named C1, C2, C3, ..., C... K The array is labeled. Each array contains M (M is a power of 2) element positions, and each element position contains N bits (N = log₂M) of space. The values of K and M can be flexibly configured according to actual needs. Initially, the value of each element position in each array is initialized to 0. At the same time, a hash function is configured for each array, and the hash function is different for different arrays.
[0090] In this embodiment, when a write IO arrives, it is first recorded in an array, then its heat value is calculated, and the data heat level matching the write IO is determined according to the pre-configured data heat level, so that the data to be written corresponding to the write IO is written into the segment that matches the data heat level.
[0091] In general, the number of write I / O operations is recorded in K arrays according to their relative write times. Each array records no more than D (D < M) write I / O operations, where D is configurable. Simultaneously, the I / O time interval (i.e., timer duration) recorded in each array must not exceed T, where T is configurable. In array C... S After completing the recording, begin using the next array C. S+1 Record the data, proceeding sequentially. When array C... K After recording is complete, C1 is used to start recording again, and so on. To achieve the decay effect and avoid cumulative overlap, it must be ensured that the new array is reinitialized when starting a new array count. Additionally, the K hash functions selected by the system satisfy the condition that the Hash(LBA) value falls exactly within the range of 0-(M-1). It should be noted that the parameters P and T are set here primarily to ensure adaptability to different load conditions. Under intensive I / O loads, writing to a certain array does not need to wait for timer T to expire; it can reach the threshold P first, thus starting the counting with the next array. Conversely, under light I / O loads, an array may not reach the threshold P but will reach the timer timeout first, thus starting the counting with the next array. This ensures an adaptive effect of multi-level classification of data heat under different loads.
[0092] The following is an example of the data writing process:
[0093] Step 1, array initialization:
[0094] 1.1 Initialize the current record array C S =0;
[0095] 1.2 Start the timer (period T);
[0096] 1.3, Initialize the single-array LBA address counter D = 0 (D represents the number of used element positions in the array).
[0097] Step 2, For the write IO with LBA address LBA X.
[0098] 2.1, D = D + 1;
[0099] 2.2, Determine whether the update count corresponding to this LBA X can be recorded in the current C S If not, enter the next array C S+1 and perform the above array initialization process on C S+1 ; Otherwise, execute 2.3;
[0100] 2.3, Calculate Hash(X) through the hash function specified by the current array. Assume its value is V, then increment the element value at the element position V in C S by 1;
[0101] 2.4, Calculate the heat value of the current IO, and determine the matching data heat level based on the heat value of this IO, so as to write the data to be written into the segment matching this data heat level;
[0102] Step 3, Wait for a new write IO and transfer to the above Step 2.
[0103] In this embodiment, the method for determining whether the update count corresponding to an LBA can be recorded in the current C S is as follows:
[0104] If both of the following two conditions are met, it is determined as recordable; otherwise, it is determined as non-recordable:
[0105] Condition 1: D < M (the number of LBAs recorded in a single array does not exceed D to control the number of hash conflicts);
[0106] Condition 2: The timer T does not time out.
[0107] The following describes the calculation of the heat value corresponding to the IO:
[0108] C1, C2, C3,..., C K record the number of LBAs of each write IO in the recent period. The following explains how to calculate the heat value corresponding to a write IO. In this embodiment, when calculating the heat value, both the number of its occurrences and its recency need to be considered. Since the frequency and recency information are both recorded in the array, the final heat value only needs to take the weighted sum of the frequency values of each array.
[0109] The weighting is assigned to each array; the array currently in use has the highest weight, and the closer an array is to the currently used array, the higher its corresponding weight. See also Figure 5 As shown, assuming the currently used array is C3, then C3 has the highest weight, and the weights are ranked according to... Figure 5 The weights decrease sequentially along the direction of the dashed lines, with C4 having the lowest weight. As the currently used array changes, the weight of each array also changes dynamically. There are multiple ways to choose the decreasing weight starting from the currently used array. For example, the weight of the current array can be defined as 2. K The second array is defined as 2. K-1 And so on, with the minimum weight array weight defined as 2. For example, the current array weight could be defined as 1, the minimum weight array weight as 0.1, and the intermediate arrays could decrease with a fixed step size (in this example, the step size is 0.9 / (K-1)), etc. This embodiment does not impose specific limitations.
[0110] For example, suppose at some point, the current record array (i.e., the array currently in use) is C. S Then the frequency value of LBA X is C S The array index is Hash S The value at the element position of (X); the values at the element positions corresponding to LBA X in other arrays are deduced similarly, and will not be elaborated here. The frequency values of LBA X in each array are F1, F2, ..., F... K The corresponding array weights are W1, W2, ..., W K The heat value corresponding to this LBAX is
[0111] It should be noted that the frequency values and array weight values mentioned above are dynamically changing.
[0112] The following describes the classification of data popularity levels:
[0113] After calculating the IO heat level, the IO data needs to be written to the corresponding segment. As described earlier, assuming the system configuration divides the data heat level into L levels, let's denote these levels as H1, H2, ..., H... L The higher the level, the higher the popularity. Each level can have a corresponding threshold set, and the thresholds of adjacent levels change continuously but do not overlap. For example, the range of level H1 can be set to [0, V1], and H... L The threshold for the level is set to [V] L For the intermediate level, the average score is (V). L -V1) is sufficient. V LThe values of V1 can be set based on actual needs. Of course, other methods can also be used, such as classifying levels according to the distribution of popularity values, which will not be detailed here.
[0114] The following describes waste recycling methods:
[0115] See Figure 6 As shown, the method includes the following steps:
[0116] Step 601: When the number of currently free storage segments is detected to be less than a set threshold, garbage collection is initiated;
[0117] Step 602: Select at least one segment that needs to be garbage collected based on the garbage collection segment selection strategy;
[0118] Step 603: For each segment, determine the data popularity level of the valid data contained in the segment to obtain the determination result;
[0119] Step 604: Based on the judgment result, migrate the valid data to a segment that matches the data popularity level corresponding to the valid data.
[0120] The following describes the data layout method for effective data in the waste recycling process:
[0121] As described above, IO data written to the system is written to different segments based on its estimated popularity (i.e., predicted lifespan). Probabilistically, it's reasonable to expect that most, or even all, of the data in the same segment will become garbage data around the same time, and thus be garbage collected by the system at some point. However, also probabilistically, there will certainly be some segments where the actual lifespan of some data does not match the system's prediction. When the system garbage collects this segment, some data may still be valid. In this case, it's necessary to copy this data from the segment and write it to an idle segment. The valid data to be copied can be treated as ordinary write IO, and the aforementioned IO popularity calculation and data layout methods are used to classify and write this valid data into segments. The specific valid data migration process can be found in the above description and will not be repeated here.
[0122] The following describes the selection of waste recycling strategies:
[0123] The garbage collection selection strategy refers to which segments the system chooses for garbage collection. This embodiment provides two strategies: a greedy algorithm and a benefit-cost algorithm.
[0124] The greedy algorithm iterates through all segments in the system and then selects the P segments with the most garbage data (the value of P can be preset in the system) for recycling.
[0125] The cost-benefit algorithm comprehensively considers the amount of garbage in a segment and the lifecycle of that segment. In this embodiment, since the data contained in each segment has a similar lifecycle, we assume its value is T' (T' can be recorded as metadata of the segment when data is written, i.e., T' is 1 / f as mentioned above); we also assume that the effective data occupancy rate of the segment is U (0≤U≤1, a dynamically changing value). The cost-benefit value of each segment is defined as (1-U)*T'. At this point, we can calculate the cost-benefit value of all segments and select the segments with the largest values, Q (Q can be preset in the system), for garbage collection. This is done because hot data segments become garbage data in a shorter time, and compared with cold data segments, collecting cold data segments at the same U value will bring greater benefits (lower write amplification).
[0126] As an example, since a hash function is used to hash the LBA address, hash collisions may occur. Hash collisions may ultimately lead to misjudgments of a certain write I / O activity; however, a small number of misjudgments are not a major problem because the activity value is a prediction and inherently possesses probabilistic properties. This example provides a method to reduce hash collisions as follows:
[0127] The upper limit of the number of positions of elements used in each array is controlled by the value D. For example, D can be set to M / 2, trading space for the probability of collisions; and / or,
[0128] By setting different hash functions for different arrays, the cumulative effect of collisions can be avoided. In other words, even if the hash values of two different I / O operations collide in one array, the probability of these two I / O operations colliding in other arrays is very small.
[0129] This concludes the description of the method provided in the embodiments of this application. The apparatus provided in the embodiments of this application will now be described:
[0130] As one embodiment, this embodiment also provides a data writing device. See [link to documentation]. Figure 7 , Figure 7 This is a schematic diagram of the device structure provided in an embodiment of this application. Figure 7As shown, the device 700 is configured in a storage device; the storage device provides X flash drives as memory for an external designated device to which the storage device is connected, where X is greater than 1; each flash drive includes Y storage blocks, where Y is greater than 1; the storage blocks of the X flash drives are grouped into Z storage segments, where Z is greater than 1; initially, information of the Z storage segments is recorded in a shared pool configured in the storage device; the device 700 includes:
[0131] The determination module 701 is used to determine, when a data write request sent by an external designated device is detected, that the data write request at least indicates the data to be written and a first storage location identifier corresponding to the storage location where the data is to be written; determine a first data heat level matching the first storage location identifier; the first data heat level is used to indicate the data update frequency at the storage location corresponding to the first storage location identifier;
[0132] The write module 702 is configured to, if there is no storage segment in the currently used storage segment that matches the first data popularity level, select an idle storage segment as the first target storage segment that matches the first data popularity level based on the information of the idle storage segments recorded in the shared pool, and write the data into the first target storage segment; if there is a first target storage segment in the currently used storage segment that matches the first data popularity level and has enough idle storage space to accommodate the data, then write the data into the idle storage space of the first target storage segment.
[0133] As one embodiment, the storage device is pre-configured with K arrays, each array containing multiple element positions; each used element position in each array is used to record the number of data updates at the storage position corresponding to the storage position identifier that matches the used element position;
[0134] Module 701 is specifically used for:
[0135] Determine if the array currently in use meets the following conditions: the number of used element positions is less than the set number, and / or the timer duration corresponding to the array currently in use has not reached the set timer duration;
[0136] If so, find the element position that matches the first storage location identifier from the currently used array, increase the value at that element position by the set value, and determine the first data heat level that matches the first storage location identifier based on the value at the element position that matches the first storage location identifier in each array.
[0137] Otherwise, initialize the next array and use it as the currently used array, then return the step that determines whether the currently used array meets the condition.
[0138] As one example, determining the target data popularity level matching the storage location identifier based on the value at the element position that matches the first storage location identifier in each array includes:
[0139] Find the element position that matches the first storage location identifier from the K configured arrays; determine the first data popularity level based on the weight of each array and the value of the element position that matches the first storage location identifier in each array.
[0140] As an example, each of the K arrays corresponds to a hash function;
[0141] Finding the element positions that match the first storage location identifier from the K configured arrays includes:
[0142] For each array, the first storage location identifier is substituted into the hash function corresponding to that array to obtain the hash result; based on the hash result, the element position corresponding to the hash result is found in the array.
[0143] As one embodiment, selecting an idle storage segment as the first target storage segment for matching the first data heat level based on the information of the free storage segments recorded in the shared pool further includes: deleting the information of the first target storage segment from the recorded shared pool information;
[0144] The device further includes:
[0145] The recycling module is used to determine at least one storage segment that needs garbage data recycling from the currently used storage segments when the number of free storage segments is less than a set threshold based on the recorded shared pool information. If there is still valid data in the storage segment, the valid data is migrated to other storage segments and the garbage data stored in the storage segment is garbage recycled. Then, the storage segment is added to the shared pool as a free storage segment and the information of the storage segment is added to the recorded shared pool information.
[0146] As one example, determining at least one storage segment from the currently used storage segments that requires garbage data reclamation includes:
[0147] Select the P storage segments with the largest amount of garbage data from the currently used storage segments as the storage segments to be garbage collected, where P is greater than or equal to 1; or,
[0148] For each of the currently used storage segments, the corresponding recycling reference value is determined based on the data update frequency recorded in the segment description information and the effective data occupancy rate of the storage segment. The data update frequency refers to the data update frequency at the storage location where the specified data is written, and the specified data is any data within the storage segment. The top Q storage segments with the largest recycling reference values are selected from the currently used storage segments as the storage segments that need to be garbage collected, where Q is greater than or equal to 1.
[0149] As one example, performing a valid data migration to another storage segment includes:
[0150] For each valid data, based on the second location identifier corresponding to the storage location where the valid data is written, find the element position that matches the second location identifier from the configured K arrays; each array contains multiple element positions; determine the second data heat level matching the second storage location identifier based on the weight of each array and the value of the element position that matches the second storage location identifier in each array; the second data heat level is used to indicate the data update frequency at the storage location corresponding to the second storage location identifier.
[0151] If there is no storage segment in the currently used storage segment that matches the second data popularity level, then based on the information of the free storage segments recorded in the shared pool, a free storage segment is selected as the second target storage segment that matches the second data popularity level, and the valid data is written to the second target storage segment; if there is a second target storage segment in the currently used storage segment that matches the second data popularity level and has free storage space to accommodate the valid data, then the valid data is written to the free storage space in the second target storage segment.
[0152] This concludes the process. Figure 7 Structural description of the device shown.
[0153] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0154] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0155] Please see Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an exemplary embodiment of this application. The electronic device may include a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, communication interface 802, and memory 803 communicate with each other via the communication bus 804. The memory 803 stores computer program instructions; the processor 801 can execute the steps of the method described in the above embodiments by executing the computer program instructions stored in the memory 803. Depending on the actual function of the electronic device, other hardware may also be included, which will not be elaborated further.
[0156] Correspondingly, embodiments of this application also provide a computer-readable storage medium storing a plurality of computer program instructions, which, when executed by a processor, can implement the methods disclosed in the above examples of this application.
[0157] For example, the aforementioned computer-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For instance, computer-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof. The processor and memory may be supplemented by or incorporated into dedicated logic circuitry.
[0158] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A data writing method, characterized in that, The method is applied to a storage device; the storage device provides X flash drives as memory for an external designated device connected to the storage device, where X is greater than 1; each flash drive includes Y storage blocks, where Y is greater than 1; the storage blocks of the X flash drives are grouped into Z storage segments, where Z is greater than 1; in the initialization state, the storage device is configured with a shared pool that records information of the Z storage segments; the method includes: When a data write request sent by the externally designated device is detected, the data write request at least indicates the data to be written and a first storage location identifier corresponding to the storage location where the data is to be written; a first data popularity level matching the first storage location identifier is determined; the first data popularity level is used to indicate the data update frequency at the storage location corresponding to the first storage location identifier; wherein, the storage device is pre-configured with K arrays, each array containing multiple element positions; each used element position in each array is used to record the number of data updates at the storage location corresponding to the storage location identifier matching the used element position; the number of used element positions in each array is controlled to be less than a set number; each array corresponds to a hash function, and the hash functions corresponding to each array are different, and the element position in each array that matches the first storage location identifier depends on the hash function corresponding to the data; the first data popularity level depends on the weight corresponding to each array and the value at the element position in each array that matches the first storage location identifier; the weights corresponding to each array decrease sequentially according to the order of the arrays, with the currently used array having the highest weight; If there is no storage segment in the currently used storage segment that matches the first data popularity level, then based on the information of the free storage segments recorded in the shared pool, a free storage segment is selected as the first target storage segment that matches the first data popularity level, and the data is written into the first target storage segment; if there is a first target storage segment in the currently used storage segment that matches the first data popularity level and has enough free storage space to accommodate the data, then the data is written into the free storage space in the first target storage segment.
2. The method according to claim 1, characterized in that, The step of determining the first data popularity level that matches the first storage location identifier includes: Determine whether the array currently in use meets the following conditions: the number of used element positions is less than a set number, and / or the timer duration corresponding to the array currently in use has not reached the set timer duration; If so, find the element position that matches the first storage location identifier from the currently used array, increase the value at that element position by a set value, and determine the first data heat level matching the first storage location identifier based on the value at the element position that matches the first storage location identifier in each array. Otherwise, initialize the next array and use it as the currently used array, then return to the step of determining whether the currently used array satisfies the stated condition.
3. The method according to claim 2, characterized in that, The step of determining the first data popularity level matching the storage location identifier based on the value at the position of the element in each array that matches the first storage location identifier includes: Find the element position that matches the first storage location identifier from the K configured arrays; determine the first data popularity level based on the weight of each array and the value of the element position that matches the first storage location identifier in each array.
4. The method according to claim 3, characterized in that, Finding the element position that matches the first storage location identifier from the configured K arrays includes: For each array, the first storage location identifier is substituted into the hash function corresponding to the array to obtain the hash result; based on the hash result, the element position corresponding to the hash result is found in the array.
5. The method according to claim 1, characterized in that, The step of selecting an idle storage segment as the first target storage segment for matching the first data heat level based on the information of the idle storage segments recorded in the shared pool further includes: deleting the information of the first target storage segment from the recorded shared pool information; The method further includes: When the number of free storage segments is determined to be less than a set threshold based on the recorded shared pool information, at least one storage segment that needs to be garbage collected is determined from the currently used storage segments. If valid data is still stored in the storage segment, the valid data is migrated to other storage segments and garbage collection is performed on the garbage data stored in the storage segment. Then, the storage segment is added to the shared pool as a free storage segment, and the information of the storage segment is added to the recorded shared pool information.
6. The method according to claim 5, characterized in that, The step of identifying at least one storage segment from the currently used storage segments that requires garbage collection includes: Select the P storage segments with the largest amount of garbage data from the currently used storage segments as the storage segments to be garbage collected, where P is greater than or equal to 1; or, For each of the currently used storage segments, a recycling reference value is determined based on the data update frequency recorded in the segment description information and the effective data occupancy rate corresponding to the storage segment. The data update frequency refers to the data update frequency at the storage location where specified data is written, and the specified data is any data within the storage segment. The Q storage segments with the largest recycling reference values are selected from the currently used storage segments as the storage segments that need to be garbage collected, where Q is greater than or equal to 1.
7. The method according to claim 5, characterized in that, Performing effective data migration to other storage segments includes: For each valid data, based on the second storage location identifier corresponding to the storage location where the valid data is written, find the element position that matches the second storage location identifier from the configured K arrays; each array contains multiple element positions; based on the weight corresponding to each array and the value of the element position that matches the second storage location identifier in each array, determine the second data heat level that matches the second storage location identifier; the second data heat level is used to indicate the data update frequency at the storage location corresponding to the second storage location identifier. If there is no storage segment in the currently used storage segment that matches the second data popularity level, then based on the information of the free storage segments recorded in the shared pool, a free storage segment is selected as the second target storage segment that matches the second data popularity level, and the valid data is written into the second target storage segment; if there is a second target storage segment in the currently used storage segment that matches the second data popularity level and has free storage space to accommodate the valid data, then the valid data is written into the free storage space in the second target storage segment.
8. A data writing device, characterized in that, The device is configured on a storage device; the storage device provides X flash drives as memory for a designated external device connected to the storage device, where X is greater than 1; each flash drive includes Y storage blocks, where Y is greater than 1; the storage blocks of the X flash drives are grouped into Z storage segments, where Z is greater than 1; in the initial state, the storage device is configured with a shared pool that records information of the Z storage segments; the device includes: A determination module is configured to, when a data write request sent by the externally designated device is detected, at least indicate the data to be written and a first storage location identifier corresponding to the storage location where the data is to be written; determine a first data heat level matching the first storage location identifier; the first data heat level is used to indicate the data update frequency at the storage location corresponding to the first storage location identifier; wherein, the storage device is pre-configured with K arrays, each array containing multiple element positions; each used element position in each array is used to record the number of data updates at the storage location corresponding to the storage location identifier matching the used element position; control the number of used element positions in each array to be less than a set number; each array corresponds to a hash function, and the hash functions corresponding to each array are different, the element position in each array matching the first storage location identifier depends on the hash function corresponding to the data; the first data heat level depends on the weight corresponding to each array and the value at the element position in each array matching the first storage location identifier; the weights corresponding to each array decrease sequentially according to the order of the arrays, with the currently used array having the highest weight; The write module is configured to, if there is no storage segment in the currently used storage segment that matches the first data popularity level, select an idle storage segment as the first target storage segment that matches the first data popularity level based on the information of the idle storage segments recorded in the shared pool, and write the data into the first target storage segment; if there is a first target storage segment in the currently used storage segment that matches the first data popularity level and has enough idle storage space to accommodate the data, then write the data into the idle storage space in the first target storage segment.
9. An electronic device, characterized in that, The electronic device includes: Processor; and A memory storing computer program instructions that, when executed by the processor, cause the processor to perform the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 7.