Intelligent multi-stream writing method and device based on interval write frequency distribution and server

Abstract

The application provides a kind of intelligent multi-flow write method, device and server based on interval write frequency distribution, relating to the technical field of solid state disk, comprising: segment processing and statistical processing are carried out for SSD, and the normal distribution of the large particle interval write frequency of SSD is determined;Through the flow model of the preset multiple flows, the normal distribution is analyzed and calculated, the write flow of different addresses is counted, the average value and variance of the large particle interval write frequency are obtained, and the standard normal distribution is constructed according to the average value and variance;According to the probability density function of standard normal distribution and the cold and hot flow division proportion of the flow model of the preset multiple flows, threshold value calculation is carried out, and the target write frequency threshold value for dividing cold data and hot data is determined, so that the target write frequency threshold value is used to divide the cold and hot flows of the data of SSD, so that the cold data and hot data are written into the corresponding flash memory area when data is written and data is recycled.The application can significantly reduce the write amplification of SSD, thereby prolonging the service life of SSD.

Description

Technical Field

[0001] This invention relates to the technical field of solid-state drives (SSDs), and in particular to an intelligent multi-stream writing method, apparatus, and server based on interval write count distribution. Background Technology

[0002] Currently, because solid-state drives (SSDs) cannot be overwritten, writing data to an SSD that already contains data requires erasing the existing data first. When the write and erase sizes are mismatched, write amplification occurs, affecting the SSD's lifespan. Related technologies propose intelligent multi-stream writing, which identifies hot and cold data streams based on access frequency, sequence, and other characteristics. Specifically, it calculates Stream IDs based on data update frequency, sequence, read / write counts, and length statistics to differentiate between hot and cold streams. However, this solution has complex algorithms and high resource consumption, thus failing to effectively solve the SSD write amplification problem. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide an intelligent multi-stream writing method, device and server based on interval write count distribution, which can significantly reduce SSD write amplification and thus extend the life of SSD.

[0004] In a first aspect, embodiments of the present invention provide an intelligent multi-stream writing method based on interval write count distribution. The method includes: performing segmentation and statistical processing on the SSD to determine the normal distribution of large-granularity interval write counts of the SSD; analyzing and calculating the normal distribution using a preset multi-stream traffic model, statistically analyzing the write traffic at different addresses to obtain the average and variance of the large-granularity interval write counts, and constructing a standard normal distribution based on the average and variance; calculating a threshold based on the probability density function of the standard normal distribution and the cold / hot flow division ratio of the preset multi-stream traffic model to determine the target write count threshold for dividing cold data and hot data, so as to use the target write count threshold to divide the data of the SSD into cold and hot flows, so that cold data and hot data are written to the corresponding flash memory areas respectively during data writing and data recycling.

[0005] In one implementation, the steps of segmenting and statistically processing the SSD to determine the normal distribution of the number of write operations in large granular intervals of the SSD include: segmenting the continuous logical address space of the SSD based on a preset address interval division threshold to obtain a set of large granular intervals of the SSD logical address; statistically processing the number of write operations occurring in each large granular interval of the SSD logical address to obtain a statistical result of the number of write operations, and constructing a normal distribution of the number of write operations in large granular intervals based on the statistical result of the number of write operations.

[0006] In one implementation, before the step of calculating the threshold based on the probability density function of the standard normal distribution and the hot and cold flow division ratio of the pre-defined multi-flow flow model, the method includes: obtaining the lifecycle of each data in the SSD, wherein the lifecycle represents the average interval at which the same logical data is rewritten, and is used as a time scale for the degree of hotness or coldness; and determining the hot and cold flow division ratio of the pre-defined multi-flow flow model based on the number of lifecycles and the length of the lifecycle corresponding to each data.

[0007] In one implementation, the step of determining the cold and hot flow division ratio of a traffic model with multiple preset flows based on the number of lifecycles and the length of the lifecycle corresponding to each data includes: if the number of lifecycles in the SSD data exceeds 2, then the flow with the longest lifecycle is determined as the cold flow, and the remaining lifecycles are determined as the hot flow, so as to determine the cold and hot flow division ratio.

[0008] In one implementation, the step of determining the target write count threshold for dividing cold data and hot data by calculating a threshold based on the probability density function of the standard normal distribution and the cold and hot flow division ratio of a pre-defined multi-flow flow model includes: integrating the probability density function to obtain a cumulative distribution function, dividing the area of ​​the cumulative distribution function according to the cold and hot flow division ratio, and determining the threshold corresponding to the division result as the target write count threshold.

[0009] In one implementation, the method further includes: if the data in the SSD is irregular traffic, based on a preset time threshold, performing long-term write counts on the actual irregular traffic, and using a preset irregular traffic model, determining the center point of the hot and cold flows based on the k-means clustering algorithm to determine the hot and cold flow division ratio.

[0010] In one implementation, the step of determining the centroid of cold and hot flows based on the k-means clustering algorithm includes: sorting the statistical results of write operation counts for each SSD logical address large granular interval, and randomly assigning cluster centroids in the sorted sequence; allocating data according to the distance between each data point and the cluster centroid to divide irregular flows into cold and hot flows, and determining the cold and hot flow division ratio.

[0011] Secondly, embodiments of the present invention also provide an intelligent multi-stream writing device based on interval write count distribution. The device includes: a data statistics module, which performs segmentation and statistical processing on the SSD to determine the normal distribution of large-granularity interval write counts of the SSD; a data processing module, which analyzes and calculates the normal distribution using a preset multi-stream flow model, statistically analyzes the write flow of different addresses, obtains the average and variance of large-granularity interval write counts, and constructs a standard normal distribution based on the average and variance; and a threshold determination module, which calculates the threshold based on the probability density function of the standard normal distribution and the cold / hot flow division ratio of the preset multi-stream flow model to determine the target write count threshold for dividing cold data and hot data, so as to divide the data of the SSD into cold and hot flows using the target write count threshold, so that cold data and hot data are written to the corresponding flash memory areas during data writing and data recycling respectively.

[0012] Thirdly, embodiments of the present invention also provide a server, including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement any of the methods provided in the first aspect.

[0013] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement any of the methods provided in the first aspect.

[0014] The embodiments of the present invention bring the following beneficial effects: This invention provides an intelligent multi-stream writing method, device, and server based on interval write count distribution. The method performs segmentation and statistical processing on the SSD to determine the normal distribution of large-granularity interval write counts. Then, it analyzes and calculates the normal distribution using a preset multi-stream traffic model, statistically analyzing write traffic at different addresses to obtain the average and variance of large-granularity interval write counts. A standard normal distribution is then constructed based on the average and variance. Subsequently, a threshold is calculated based on the probability density function of the standard normal distribution and the hot / cold flow division ratio of the preset multi-stream traffic model to determine the target write count threshold for separating cold and hot data. This target write count threshold is used to divide the SSD data into hot and cold flows, ensuring that cold and hot data are written to their respective flash memory regions during data writing and data recycling. This invention can significantly reduce SSD write amplification, thereby extending the SSD's lifespan.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an intelligent multi-stream writing method based on interval write count distribution provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a multi-stream writing method provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of an intelligent multi-stream writing device based on interval write count distribution provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Currently, solid-state drives (SSDs), as a significant technological innovation in data storage, are sparking a storage revolution globally. This is especially true in emerging technology fields such as cloud computing, artificial intelligence, and autonomous driving, which have an urgent need for high-performance storage, as well as in the consumer electronics sector, which continuously seeks fast and lightweight storage solutions.

[0021] The difference between solid-state drives (SSDs) and traditional hard disk drives (HDDs) is that SSDs cannot overwrite data. Therefore, for an SSD containing existing data, a single data write operation consists of two parts: erasing and writing. When the size of the write operation and the size of the erase operation are mismatched, it leads to write amplification, which means that the actual data written to the SSD is much larger than the data written by the user. Since the lifespan of an SSD depends on the number of erase cycles, write amplification significantly impacts its lifespan.

[0022] Existing methods for addressing write amplification include batch writing, reserving OP space, write compression, and the TRIM command. However, traditional AutoStream (i.e., intelligent multi-stream) calculates Stream IDs based on data update frequency, sequence, read / write counts, and length statistics to distinguish between hot and cold streams. This algorithm is complex and resource-intensive. Therefore, the present invention provides an intelligent multi-stream writing method, device, and server based on interval write count distribution. This method can use intelligent multi-stream technology to divide data streams into hot and cold streams based on data distribution characteristics and write them to mutually isolated blocks. This reduces invalid data movement during GC (i.e., garbage collection), thereby reducing write amplification and extending the lifespan of the SSD.

[0023] See Figure 1 The diagram shows a flowchart of an intelligent multi-stream writing method based on interval write count distribution. The method mainly includes the following steps S102 to S106: Step S102 involves segmenting and statistically processing the SSD to determine the normal distribution of write counts for large granular intervals of the SSD. In one implementation, the continuous logical address space of the SSD can be segmented based on a preset address interval division threshold to obtain a set of large granular intervals of the SSD logical address. Then, the number of write operations occurring in each large granular interval of the SSD logical address is statistically processed to obtain the write operation count statistics. Based on the write operation count statistics, a normal distribution of write counts for large granular intervals is constructed.

[0024] In practical applications, we can start from the data distribution characteristics and divide the address space into large-granularity address ranges based on the number of write operations. For example, we can divide a 200G SSD into 100 parts, each address range being 2G in size, and count the number of accesses for each address range. Assuming that data storage access is random, we can plot a normal distribution of the number of write operations for large-granularity address ranges based on the statistics. In other words, we first coarsely divide the SSD address space into large segments, and then count how many write operations occur in each segment, thereby constructing a normal distribution of the number of write operations for large-granularity address ranges.

[0025] Step S104: By using a pre-defined traffic model for multiple flows, the normal distribution is analyzed and calculated. Write traffic at different addresses is statistically analyzed to obtain the average and variance of write counts for large granular intervals. A standard normal distribution is then constructed based on the average and variance. In one implementation, a write count threshold can be determined based on the statistical average and variance. The probability density function formula for the standard normal distribution is:

[0026] The integral of the standard normal distribution N(1,0) is 1. The probability of a given interval is: I=∫f(x)dx Furthermore, the JESD219 traffic model can be used for pre-defined multi-flow traffic models. The JESD219 model has a clear proportional relationship between the number of writes in different address granular intervals. Accurate distribution characteristics can be obtained through simulation, and precise thresholds for various traffic flows can be derived accordingly. JESD219 explicitly describes that 5% of LBA (i.e., contiguous logical address space) accesses account for 50% of the total number of reads, 15% of LBA accesses account for 30%, and 80% of LBA accesses account for 20%. If the address interval is divided into 100 parts, assuming the smallest unit of access is n times, the 5% interval is accessed 40n times, the 15% interval is accessed 8n times, and the remaining 80% interval is accessed n times. Therefore, the average μ = 4n and σ = 8.63n can be calculated, i.e.: μ = 5 * 40n + 15 * 8n + 80 * n / 100 = 4n σ= (1 / N∑(x-μ)^2)=8.63n Based on μ and σ, a normal distribution curve is plotted. According to the integral formula of the normal distribution, the 80% flow threshold is 0.68σ and the 50% flow threshold is 1.28σ.

[0027] Step S106: Calculate the threshold based on the probability density function of the standard normal distribution and the cold / hot flow division ratio of the preset multi-flow flow model to determine the target write count threshold for dividing cold and hot data. Use the target write count threshold to divide the SSD data into cold and hot flows, so that cold and hot data are written to the corresponding flash memory areas during data writing and data recycling, respectively.

[0028] In one implementation, the probability density function can be integrated to obtain a cumulative distribution function. The area of ​​the cumulative distribution function can then be divided according to the hot / cold flow ratio. The threshold corresponding to the division result is determined as the target write count threshold. That is, if 80% of the capacity area only handles 20% of the access volume, then this 80% area is classified as cold flow, and the remaining 20% ​​is classified as hot flow. When determining the target write count threshold, a vertical line is found in the cumulative distribution function obtained after integrating the probability density function, and the area on the left side of the function is equal to 0.8. The corresponding value at this point is then determined as the target write count threshold.

[0029] Furthermore, after determining the target write count threshold, data with a write count greater than the target write count threshold can be identified as hot flow, and data with a write count less than the target write count threshold can be identified as cold flow. Hot and cold data are written to different blocks respectively, reducing the GC burden caused by mixed writes.

[0030] The intelligent multi-stream writing method based on interval write count distribution provided in this invention can accurately identify hot and cold flows by statistically analyzing the write counts of each address interval in the SSD, using clustering algorithms to determine the hot and cold flow ratio, and using current distribution probability analysis to calculate the hot and cold flow thresholds. This allows for the separate writing of hot and cold data to different regions, significantly reducing write amplification and extending the lifespan of the SSD. Furthermore, because it employs large-granular interval statistics and only counts the write counts, it is simple to implement and consumes very few memory and CPU resources.

[0031] In one embodiment, regarding the cold and hot flow division ratio of the preset multiple flow flow model in step S106 above, the present invention provides two calculation schemes for different cases, as detailed below (1) to (2): (1) Obtain the lifecycle of each data in the SSD. Based on the number of lifecycles and the length of the lifecycle corresponding to each data, determine the cold and hot flow division ratio of the traffic model of multiple preset flows. The lifecycle represents the average interval at which the same logical data is rewritten and is used as a time scale for the degree of coldness or hotness. In one implementation, if the number of lifecycles in the SSD data exceeds 2, the flow with the longest lifecycle is determined as the cold flow and the remaining lifecycles are determined as the hot flow, so as to determine the cold and hot flow division ratio.

[0032] Specifically, since the goal of this solution is to separate hot and cold flows, for flow models with more than two lifecycles, the strategy adopted is to treat the flow with the longest lifecycle as a cold flow and the rest as hot flows. For the JESD219 flow model, there are three lifecycle flows: HOT (50% Access, First 5% LBA), WARM (30% Access, Next 15% LBA), and COLD (20% Access, Last 80% LBA). According to this strategy, 20% Access (Last 80% LBA) is designated as a cold flow, and the rest are all set as hot flows. The long-term WAF is a WAF composed of both HOT and WARM flows, and its estimated value is:

[0033] Where cap is the user capacity, bw is the maximum bandwidth, and t is the time taken to write to the disk once. Given the capacity and bandwidth, the expected WAF (i.e., write amplification factor) can be calculated. Taking a user capacity of 200G and a maximum bandwidth of 20MBps as an example, t is the time taken to write to the disk once, t=cap / bw, then theoretically WAF=1.23.

[0034] (2) If the data in the SSD is irregular traffic, based on the preset time threshold, the number of writes of the actual irregular traffic is statistically analyzed over a long period of time, and a preset irregular traffic model is adopted. Based on the k-means clustering algorithm, the center point of the cold and hot flow is determined to determine the cold and hot flow division ratio. In one implementation, the write operation count results of each SSD logical address large granular interval can be sorted, and the cluster centroid is randomly assigned in the sorted sequence. Then, the data is allocated according to the distance between each data point and the cluster centroid to divide the irregular traffic into cold flow and hot flow and determine the cold and hot flow division ratio.

[0035] Specifically, for irregular flow models, the k-means clustering algorithm is used to determine the centroids of hot and cold data points and to define the hot and cold boundaries. First, the collected interval statistics are sorted by size, and cluster centroids are randomly assigned within the sequence. Then, based on the distance of each data point to a cluster centroid, it is assigned to a cluster. After assigning each point to a cluster, a new cluster centroid is assigned. This process is iterated until a suitable cluster is found. Assuming the number of clusters is known, data points are placed into a cluster as follows: dis(Xi,Cj)= (∑(Xit-Cjt)^2) By sequentially comparing the distance of each object to each cluster center, the object is assigned to the cluster with the nearest cluster center, resulting in k clusters {S1, S2, ..., Sk}. The principle of clustering is defined by the center; the cluster center is the mean of all objects within the cluster across all dimensions, and its formula is: Ct=(∑Xi) / |St| Where Ct represents the t-th cluster center, 1≤t≤K, |St| represents the number of objects in the t-th cluster, and Xi represents the i-th object in the t-th cluster, 1≤i≤|St|.

[0036] In other words, when the data in the SSD is irregular traffic, the data distribution is not normal. In this case, after the data is segmented in step S102, the data in each segment is regarded as data points, and the k-means clustering algorithm is used to divide it into two categories: hot and cold. Then, the center value of these two categories is used to determine the threshold.

[0037] The intelligent multi-stream writing method based on interval write count distribution provided in this invention can greatly simplify and perform equivalent calculations for intelligent multi-stream scenarios with two cold and hot streams, requiring very few resources, with a minimum memory footprint of less than 4KB and approximately 100 CPU cycles. It mainly performs statistical analysis of the number of writes over a long period of time within a large-granularity LBA interval (N), with a recommended value of 128 for N. This solution can calculate the average interval write count μ in real time and estimate the variance σ according to JESD219. It then uses the normal distribution probability to inversely calculate the cold and hot stream thresholds, typically using 20% ​​capacity as hot flow (i.e., 80% of the access bandwidth occupies 20% of the capacity). Furthermore, for multi-stream traffic models, the stream with the longest lifespan can be designated as a cold flow, and all other streams as hot flows. For irregular traffic models, the k-means algorithm can be used to determine the cold and hot center points, define the cold and hot boundaries, and thus determine the cold and hot thresholds.

[0038] Even in unknown usage scenarios and without a defined traffic model, the threshold calculated using the JESD219 model can still achieve the beneficial effect of reducing WAF. The WAF is significantly reduced in random write scenarios of the EXT4 file system because there are also read and write hotspots in the internal management items of the EXT4 file system such as journal, inode, dir, and bitmap.

[0039] See Figure 2The diagram illustrates a multi-stream writing method. This invention also provides an implementation that automatically divides hot and cold streams based solely on the write count distribution, without relying on the host. It employs a large-granularity interval write count distribution for stream identification. Compared to methods based on 4K alignment, 2M interval statistics, data length statistics, and multiple statistical fusions, while stream identification accuracy is slightly lower, the computation is greatly simplified, and it exhibits good fault tolerance and universality in complex scenarios. For extremely large database files, current methods such as F2FS and FStream, which rely on file type-based host tagging, do not provide good identification. This solution can help improve the success rate of stream identification, essentially combining fuzzy and deterministic identification, leveraging their respective strengths. When using SQLite applications in the EXT4 file system, enabling AutoStream further reduces WAF (Web Application Firewall).

[0040] In other words, using a small-granularity write count distribution for stream identification results in significant CPU overhead and computational complexity. Furthermore, it fails when encountering irregular, featureless, or host-untagged transactions. However, using a large-granularity interval write count distribution simplifies computation and ensures that every write operation is followed by a read operation, allowing for overall identification even in ambiguous situations. In the flowchart of the multi-stream write method described above, FStream represents the file stream; Separate journal writes represent separate journal writes; inode / block bitmap represents the inode block bitmap; Inode-stream represents the inode stream; Dir-stream represents the directory stream; Journal-stream represents the journal stream; Extn-stream represents the extension stream; Fname-stream represents the filename stream; Misc-stream represents the miscellaneous stream; AutoStream represents automatic stream identification; Hot-stream represents the hot stream; Cold-stream represents the cold stream; and NAND represents flash memory, i.e., the final write target. Figure 2 The left side shows the traditional solution (FStream), where the host manually marks the stream based on file type / extension; the right side shows the solution of this invention (AutoStream), where the SSD controller automatically identifies hot and cold data and allocates the stream.

[0041] In summary, this invention only requires real-time statistics of the number of writes in N intervals and the total number of writes. N can be set according to resource availability, with a suggested value of 128. When N is 128, a shift calculation can be used to calculate the average value, simplifying the computational resource requirements while meeting the overall needs. μ=total_write_counter>>7.

[0042] In real-world SSD usage scenarios, a small amount of address data is frequently accessed. For the JESD219 traffic model, the last 80% of the interval is classified as cold flow, and the remaining 20% ​​is classified as hot flow. In the JESD219 model, the hot flow in the 20% interval accounts for 80% of the access frequency. Based on the normal distribution, the threshold THR for 80% of the access frequency can be estimated as 0.68σ≈0.68*8.63*μ / 4≈1.5μ. The interval with the number of writes greater than the threshold THR is marked as a hot zone, and the rest are marked as cold zones.

[0043] With a 200GB SSD and 20% OP, using the JESD219 traffic model as the incentive, the WAF is 3.6 without enabling AutoStream. With AutoStream enabled, the WAF is 1.8. It can be seen that the WAF of the JESD219 traffic model with equal probability is significantly reduced, by about 50%.

[0044] Regarding the intelligent multi-stream writing method based on interval write count distribution provided in the foregoing embodiments, this invention provides an intelligent multi-stream writing device based on interval write count distribution, see [link to previous document]. Figure 3 The diagram shows a structural schematic of an intelligent multi-stream writing device based on interval write count distribution. The device includes the following parts: Data statistics module 302 performs segmentation and statistical processing on the SSD to determine the normal distribution of write counts in the large granular intervals of the SSD; The data processing module 304 analyzes and calculates the normal distribution by using a pre-set traffic model for multiple streams, statistically analyzes the write traffic of different addresses, obtains the average and variance of the number of writes in large granular intervals, and constructs a standard normal distribution based on the average and variance. The threshold determination module 306 calculates the threshold based on the probability density function of the standard normal distribution and the cold and hot flow division ratio of the preset multi-flow flow model to determine the target write count threshold for dividing cold data and hot data. The target write count threshold is used to divide the SSD data into cold and hot flows so that cold data and hot data are written to the corresponding flash memory areas during data writing and data recycling, respectively.

[0045] The intelligent multi-stream writing device based on interval write count distribution provided in this application embodiment can significantly reduce SSD write amplification, thereby extending the lifespan of the SSD.

[0046] In one embodiment, when performing segmentation and statistical processing on the SSD to determine the normal distribution of the number of write operations in large granular intervals of the SSD, the data statistics module 302 is further configured to: segment the continuous logical address space of the SSD based on a preset address interval division threshold to obtain a set of large granular intervals of the SSD logical address; perform statistical processing on the number of write operations occurring in each large granular interval of the SSD logical address to obtain a statistical result of the number of write operations, and construct a normal distribution of the number of write operations in large granular intervals based on the statistical result of the number of write operations.

[0047] In one embodiment, before performing the threshold calculation step based on the probability density function of the standard normal distribution and the hot and cold flow division ratio of the preset multi-flow flow model, the threshold determination module 306 is further configured to: obtain the lifecycle of each data in the SSD, wherein the lifecycle represents the average interval at which the same logical data is rewritten, and is used as a time scale for the degree of hot and cold; and determine the hot and cold flow division ratio of the preset multi-flow flow model based on the number of lifecycles and the length of the lifecycle corresponding to each data.

[0048] In one embodiment, when performing the step of determining the cold and hot flow division ratio of a preset multi-flow flow model based on the number of lifecycles and the length of the lifecycle corresponding to each data, the threshold determination module 306 is further configured to: if the number of lifecycles in the SSD data exceeds 2, then determine the flow with the longest lifecycle as the cold flow and the remaining lifecycles as the hot flow, so as to determine the cold and hot flow division ratio.

[0049] In one embodiment, when performing the step of calculating the threshold for dividing cold and hot data based on the probability density function of the standard normal distribution and the cold and hot flow division ratio of the flow model of multiple pre-set flows, and determining the target write count threshold for dividing cold data and hot data, the threshold determination module 306 is further configured to: perform integral processing on the probability density function to obtain the cumulative distribution function, and divide the area of ​​the cumulative distribution function according to the cold and hot flow division ratio, and determine the threshold corresponding to the division result as the target write count threshold.

[0050] In one embodiment, the threshold determination module 306 is further configured to: if the data in the SSD is irregular traffic, perform long-term write count statistics on the actual irregular traffic based on a preset time threshold, and use a preset irregular traffic model to determine the center point of the hot and cold flows based on the k-means clustering algorithm, so as to determine the hot and cold flow division ratio.

[0051] In one implementation, when performing the step of determining the centroid of cold and hot flows based on the k-means clustering algorithm, the threshold determination module 306 is further configured to: sort the statistical results of the number of write operations in the large granular range of each SSD logical address, and randomly assign cluster centroids in the sorted sequence; allocate the data according to the distance between each data point and the cluster centroid, so as to divide the irregular flow into cold flow and hot flow, and determine the cold and hot flow division ratio.

[0052] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0053] This invention provides a server, specifically, the server includes a processor and a storage device; the storage device stores a computer program, which, when run by the processor, executes the method described in any of the above embodiments.

[0054] Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. The server 100 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43. The processor 40, the communication interface 43 and the memory 41 are connected through the bus 42. The processor 40 is used to execute executable modules, such as computer programs, stored in the memory 41.

[0055] The memory 41 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 43 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0056] Bus 42 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0057] The memory 41 is used to store programs. After receiving an execution instruction, the processor 40 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0058] Processor 40 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 40 or by instructions in software form. Processor 40 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 41. The processor 40 reads the information in memory 41 and, in conjunction with its hardware, completes the steps of the above method.

[0059] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0060] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0061] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A smart multi-stream writing method based on interval write frequency distribution, characterized in that, The method includes: Segmentation and statistical processing are performed on the SSD to determine the normal distribution of write counts in large granular intervals of the SSD; By using a pre-defined traffic model for multiple streams, the normal distribution is analyzed and calculated. The write traffic of different addresses is statistically analyzed to obtain the average and variance of the number of writes in the large granular interval. A standard normal distribution is then constructed based on the average and variance. Based on the probability density function of the standard normal distribution and the cold and hot flow division ratio of the preset multi-flow flow model, a threshold calculation is performed to determine the target write count threshold for dividing cold data and hot data. The target write count threshold is then used to divide the SSD data into cold and hot flows, so that cold data and hot data are written to the corresponding flash memory areas during data writing and data recycling, respectively. The step of segmenting and statistically processing the SSD to determine the normal distribution of the number of write operations in large granular intervals of the SSD includes: segmenting the continuous logical address space of the SSD based on a preset address interval division threshold to obtain a set of large granular intervals of the SSD logical address; statistically processing the number of write operations occurring in each large granular interval of the SSD logical address to obtain a statistical result of the number of write operations; and constructing a normal distribution of the number of write operations in large granular intervals based on the statistical result of the number of write operations. Before the step of calculating the threshold based on the probability density function of the standard normal distribution and the hot / cold flow division ratio of the preset multi-flow flow model, the method includes: obtaining the lifecycle of each data in the SSD, wherein the lifecycle represents the average interval at which the same logical data is rewritten, and is used as a time scale for the degree of hot / cold flow; and determining the hot / cold flow division ratio of the preset multi-flow flow model based on the number of lifecycles and the length of the lifecycle corresponding to each data. The step of determining the hot and cold flow division ratio of the traffic model of the preset multiple flows based on the number of lifecycles and the length of the lifecycles corresponding to each data includes: if the number of lifecycles in the SSD data exceeds 2, then the flow with the longest lifecycle is determined as the cold flow, and the remaining lifecycles are determined as the hot flow, so as to determine the hot and cold flow division ratio.

2. The intelligent multi-stream writing method based on interval write count distribution according to claim 1, characterized in that, The step of calculating a threshold based on the probability density function of the standard normal distribution and the cold / hot flow division ratio of the preset multi-flow flow model to determine the target write count threshold for the cold and hot data division includes: The probability density function is integrated to obtain the cumulative distribution function. The area of ​​the cumulative distribution function is divided according to the hot and cold flow division ratio. The threshold corresponding to the division result is determined as the target write count threshold.

3. The intelligent multi-stream writing method based on interval write count distribution according to claim 1, characterized in that, The method further includes: If the data in the SSD is irregular traffic, based on a preset time threshold, the number of writes over a long period of time is counted, and a preset irregular traffic model is used to determine the center point of the hot and cold flows based on the k-means clustering algorithm, so as to determine the ratio of the hot and cold flows.

4. The intelligent multi-stream writing method based on interval write count distribution according to claim 3, characterized in that, The step of determining the center point of the cold and heat flow based on the k-means clustering algorithm includes: The write operation counts for large granular ranges of SSD logical addresses are sorted, and cluster centroids are randomly assigned to the sorted sequence. Based on the distance between each data point and the cluster centroid, the data is allocated to divide the irregular flow into cold flow and hot flow, and the ratio of the cold and hot flow is determined.

5. An intelligent multi-stream writing device based on interval write count distribution, characterized in that, The device includes: The data statistics module performs segmentation and statistical processing on the SSD to determine the normal distribution of write counts in the large granular intervals of the SSD. The data processing module analyzes and calculates the normal distribution by using a preset traffic model for multiple streams, statistically analyzes the write traffic of different addresses, obtains the average and variance of the number of writes in the large granular interval, and constructs a standard normal distribution based on the average and variance. The threshold determination module calculates the threshold based on the probability density function of the standard normal distribution and the cold and hot flow division ratio of the preset multi-flow flow model to determine the target write count threshold for dividing cold data and hot data. The target write count threshold is used to divide the SSD data into cold and hot flows so that cold data and hot data are written to the corresponding flash memory areas during data writing and data recycling, respectively. The data statistics module segments the continuous logical address space of the SSD based on a preset address interval division threshold to obtain a set of large-granularity SSD logical address intervals; it then performs statistical processing on the number of write operations occurring in each of the large-granularity SSD logical address intervals to obtain write operation count statistics results, and constructs a normal distribution of the number of write operations in the large-granularity intervals based on the write operation count statistics results. Before completing the threshold calculation, the threshold determination module obtains the lifecycle of each data in the SSD, where the lifecycle represents the average interval at which the same logical data is rewritten, and is used as a time scale for the hot and cold flow degree; based on the number of lifecycles and the length of the lifecycle corresponding to each data, the hot and cold flow division ratio of the preset multi-flow flow model is determined. In determining the cold and hot flow division ratio, if the number of lifetimes in the SSD data exceeds 2, the threshold determination module will determine the flow with the longest lifetime as the cold flow and the remaining lifetimes as the hot flow, so as to determine the cold and hot flow division ratio.

6. A server, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method described in any one of claims 1 to 4.

Images