Data storage method

Abstract

The application discloses a data storage method, relates to the technical field of data storage, and comprises the following steps: dividing target data into first type data, second type data and third type data according to the access probability of the target data from low to high; the first type data has a low access frequency, and loss of the first type data has a small influence on stability, so that the first type data can be encoded, storage cost is saved, and stability is less reduced; the second type data has a medium access frequency, and the second type data is encoded by using a maximum distance separable code to ensure repair and high stability and to increase storage cost little; and the third type data has a high access frequency, and the third type data is encoded by using a target encoding mode to reduce repair bandwidth, ensure high stability and increase storage cost of unit data volume much. Since the third type data volume is not high, the overall storage cost is increased little, and the problem that a storage management method in the prior art is difficult to balance system stability and storage cost is solved.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to data encoding and storage. Background Technology

[0002] Distributed storage systems (DSS) ensure high data reliability and availability by storing redundant data across multiple storage nodes. With the explosive growth of data volume, storage costs have become another major challenge for data center operations. Encoded storage leads to increased recovery bandwidth and storage costs, while unencoded storage results in data loss and business interruptions, leading to poor storage system stability. Large-scale data storage systems urgently need an adaptive storage management method to achieve a comprehensive optimization of system stability and storage costs. Summary of the Invention

[0003] This application provides a data storage method to at least address the problem in related technologies where storage management methods struggle to balance system stability and storage costs.

[0004] This application provides a data storage method, comprising: determining the access probability of target data; if the access probability of the target data is less than or equal to a first probability threshold, determining the target data as first type data and storing the target data; if the access probability of the target data is greater than the first probability threshold and less than a second probability threshold, determining the target data as second type data, encoding the target data using a maximum distance divisible code encoding method, and then storing the target data; if the access probability of the target data is greater than or equal to the second probability threshold, determining the target data as third type data, encoding the target data using a target encoding method, wherein the target encoding method is the encoding method corresponding to the encoding with a repair bandwidth less than the repair bandwidth of the maximum distance divisible code.

[0005] This application also provides a data storage device, comprising: a first determining module for determining the access probability of target data; a first storage module for determining the target data as first type data and storing the target data when the access probability of the target data is less than or equal to a first probability threshold; a second storage module for determining the target data as second type data when the access probability of the target data is greater than the first probability threshold and less than a second probability threshold, encoding the target data using a maximum distance divisible code encoding method and then storing it; and a third storage module for determining the target data as third type data when the access probability of the target data is greater than or equal to the second probability threshold, encoding the target data using a target encoding method, wherein the target encoding method is the encoding method corresponding to an encoding with a repair bandwidth less than the repair bandwidth of a maximum distance divisible code.

[0006] This application also provides a storage system, comprising: a memory for storing a computer program; and a processor for implementing the steps of any data storage method when executing the computer program.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any data storage method.

[0008] According to this application, the above data storage method categorizes target data into three types based on their access probability, from low to high: Type 1 data, Type 2 data, and Type 3 data. Type 1 data, due to its low access frequency, has a relatively small impact on stability if lost, and therefore does not require encoding, saving storage costs and minimizing stability degradation. Type 2 data has a moderate access frequency, and its loss has a significant impact on stability. Encoding with a maximum distance separable code ensures repairability, providing high repair bandwidth, but infrequent repairs do not consume excessive repair bandwidth, ensuring high stability with minimal increase in storage costs. Type 3 data has a high access frequency, and its loss has a significant impact on stability, requiring relatively frequent repairs. Encoding with a target encoding method reduces repair bandwidth while maintaining high stability, but increases storage costs per unit of data. Since the amount of Type 3 data is low, the overall increase in storage costs is minimal, thus solving the problem in related technologies where storage management methods struggle to balance system stability and storage costs. Attached Figure Description

[0009] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a hardware structure block diagram of a data storage method according to an embodiment of this application;

[0011] Figure 2 This is a flowchart of a data storage method according to an embodiment of this application;

[0012] Figure 3 This is a global data block access count histogram according to an embodiment of this application;

[0013] Figure 4 This is a flowchart of a data reading stage according to an embodiment of this application;

[0014] Figure 5 This is a flowchart illustrating a dynamic adaptive compression method according to an embodiment of this application;

[0015] Figure 6 This is a structural block diagram of a data storage device according to an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0017] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0018] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] The specific application environment architecture or specific hardware architecture on which the execution of the data storage method depends is described here.

[0020] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of the data storage method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), microprocessor (MCU), or programmable logic device (FPGA), etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0021] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data storage method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0022] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0023] The embodiments of this application provide a data storage method, and the method is described in detail below in conjunction with the execution flow of the data storage method.

[0024] The following explains the technical terms used in this application:

[0025] MDSCode: Maximum Distance Separable Code.

[0026] This embodiment provides a data storage method. Figure 2 This is a flowchart of a data storage method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0027] Step S202: Determine the access probability of the target data;

[0028] Specifically, the probability of accessing the target data is predicted based on information such as the file type and owner of the target data, which is used to evaluate the access frequency after the target data is stored.

[0029] Step S204: If the access probability of the target data is less than or equal to the first probability threshold, determine that the target data is the first type of data and store the target data.

[0030] Specifically, if the access probability of the target data is less than or equal to the first probability threshold, it indicates that the access frequency after storage is low and the impact of data loss on stability is not significant. The target data is determined to be the first type of data, i.e., cold data. The target data is stored directly without encoding, and there is no need to store the check bit data, thus saving storage costs.

[0031] Step S206: If the access probability of the target data is greater than the first probability threshold and less than the second probability threshold, the target data is determined to be second type data, and the target data is encoded and stored using the maximum distance divisible code encoding method.

[0032] Specifically, if the access probability of the target data is greater than the first probability threshold and less than the second probability threshold, it indicates that the access frequency after storage is moderate. The target data is determined to be second-type data, i.e. warm data. Data loss has a significant impact on stability. The target data is encoded and stored using a maximum distance divisible code encoding method to ensure that data loss can be repaired, maintain stability, and increase storage costs only slightly.

[0033] Step S208: If the access probability of the target data is greater than or equal to the second probability threshold, the target data is determined to be third type data, and the target data is encoded and stored using the target encoding method. The target encoding method is the encoding method corresponding to the repair bandwidth that is less than the maximum distance divisible code.

[0034] Specifically, if the access probability of the target data is greater than the second probability threshold, it indicates that the access frequency after storage is high, thus identifying the target data as the third type of data, i.e., hot data. Data loss has a significant impact on stability. Encoding the target data before storage can repair data loss and ensure stability, but the relatively high repair frequency leads to high repair bandwidth. Therefore, the maximum distance divisible code encoding method is adopted to reduce the repair bandwidth. Although it increases the encoding complexity and storage cost compared to the maximum distance divisible code encoding method, the amount of hot data is not large, so the overall increase in storage cost is not significant.

[0035] Through the above steps, the target data is categorized into three types based on its access probability, from low to high: Type 1 data, Type 2 data, and Type 3 data. Type 1 data, due to its low access frequency, has a minimal impact on stability if lost, and therefore does not require encoding, saving storage costs and minimizing stability degradation. Type 2 data has a moderate access frequency, and its loss has a significant impact on stability. It is encoded using a maximum distance separable code method to ensure repairability. The repair bandwidth is high, but infrequent repairs do not consume too much repair bandwidth, ensuring high stability with minimal increase in storage costs. Type 3 data has a high access frequency, and its loss has a significant impact on stability. Repairs are relatively frequent, and target encoding is used to reduce repair bandwidth while maintaining high stability. However, this increases the storage cost per unit of data significantly. Since the amount of Type 3 data is not high, the overall increase in storage costs is minimal. This solves the problem in related technologies where storage management methods struggle to balance system stability and storage costs.

[0036] As an optional implementation, step S202 above includes:

[0037] Step S2021: Extract multiple features from the target data, including file type values;

[0038] Step S2022: Use the weights of each feature to perform a weighted summation of multiple features to obtain a comprehensive feature value;

[0039] Step S2023: Substitute the comprehensive feature value into the logistic function to obtain the access probability of the target data.

[0040] In the above embodiments, for a target data A that is about to be written... j The system builds a probability model P(A) for future access based on prior knowledge such as its metadata (e.g., file type, owner) and historical access patterns (e.g., the user / application's historical behavior). j This can be represented by a logistic regression model or a weighted scoring function: , where f i (A j ) is from target data A j The extracted i-th feature (e.g., the feature is the file type value; if the file type is video, the feature is 1, otherwise it is 0), m is the number of features, ω i It is the weight of that feature, and σ is the Sigmoid function, i.e. By using a probability model of future access, the access probability of target data can be predicted, so as to classify data types before storage.

[0041] As an optional implementation, after the target data has been stored, the above method further includes:

[0042] Step S301: Obtain the access count value of each data block at predetermined intervals, where the data block includes the data block storing the target data;

[0043] Step S302: Generate a first access count threshold and a second access count threshold based on the access count value of each data block, such that the proportion of data blocks with access count values ​​greater than or equal to the first access count threshold is less than the first proportion threshold, and the proportion of data blocks with access count values ​​greater than the second access count threshold and less than the first access count threshold is less than the second proportion threshold.

[0044] Step S303: Data blocks with access count values ​​greater than or equal to the first access count threshold are identified as third type data;

[0045] Step S304: Data blocks whose access count value is greater than the second access count threshold and less than the first access count threshold are identified as second type data;

[0046] Step S305: Data blocks whose access count value is less than or equal to the second access count threshold are identified as first type data.

[0047] In the above embodiments, after data enters the storage node, the background continuously monitors the access count of the corresponding data block within a specific time interval (generally considered long enough, such as 3 days). The system periodically collects count values ​​from all storage nodes to construct a global histogram of data block access counts, such as... Figure 3 As shown, this histogram divides the count values ​​into multiple bins and dynamically adjusts the first access threshold Q by counting the number of data blocks falling into each bin. hot Second access threshold Q warm The horizontal axis of the histogram represents the number of times a data block is accessed, and the vertical axis represents the number of data blocks. Note that the number of accesses to a data block is expressed in terms of 2^n. i As a reference benchmark, each rectangle in the histogram represents the number of visits falling within [2]. i ,2 i+1 The number of data blocks in this range. The dynamic adjustment of Q here. hot and Q warm This means that if the system sets the proportion of data blocks of the third type of data to not exceed the first proportion threshold (e.g., 40%), and the proportion of data blocks of the second type of data to not exceed the second proportion threshold (e.g., 20%), then the third type of data is hot data, the second type of data is warm data, and the first type of data is cold data, thereby achieving dynamic data type classification that closely reflects actual access situations.

[0048] As an optional implementation, after determining that each data block is one of the third type of data, the second type of data, and the first type of data, the above method further includes:

[0049] Step S401: When the target data corresponding to the data block is third-type data and the data block is second-type data, control the data block to remain unchanged;

[0050] Step S402: If the target data corresponding to the data block is third-type data or second-type data and the data block is first-type data, delete the check bit of the codeword of the data block;

[0051] Step S403: When the target data corresponding to the data block is of the first type and the data block is of the second type, the data block is encoded using the maximum distance divisible code encoding method.

[0052] Step S404: When the target data corresponding to the data block is of type 1 or type 2 and the data block is of type 3, the data block is encoded using the target encoding method.

[0053] In the above embodiments, the third type of data is hot data, the second type of data is warm data, and the first type of data is cold data. Based on the number of times the data is accessed within a specific time period, the data is dynamically classified and re-encoded. Specifically, when hot data becomes warm data, no encoding method conversion is performed, and its data storage method is retained; when hot or warm data becomes cold data, the check bits of the codeword are directly deleted, corresponding to the recycling process in the operating system; when cold data becomes warm data, the data is encoded using MDS; when cold or warm data becomes hot data, the data is encoded using the target encoding method.

[0054] As an optional implementation, step S301 above includes:

[0055] Step S3011: At predetermined intervals, query the hash table of the storage node where each data block is located based on the unique identifier of each data block to obtain the access count value corresponding to the unique identifier of each data block. The hash table is used to track the access count value of each data block on the storage node.

[0056] In the above embodiments, an in-memory hash table (essentially an efficient "dictionary" or "index" that maps a key to a value) is maintained on each storage node to track the access count C of each data block on that node in real time. i C i Let C be the access count value of the i-th data block, where the key is the unique identifier of the data block, and the value is the access count value C of the data block. iYou can then query the access count value of a data block through the hash table of the storage node.

[0057] As an optional implementation, before querying the hash table of the storage node where each data block resides based on its unique identifier at predetermined intervals to obtain the access count value corresponding to the unique identifier of each data block, the above method further includes:

[0058] Step S501: Generate a unique identifier for the target data block based on the access request for the target data block. The target data block is a data block stored by a storage node.

[0059] Step S502: Query the hash table of the storage node where the target data block is located based on the unique identifier of the target data block;

[0060] Step S503: If the hash table of the storage node contains a unique identifier for the target data block, increment the access count value corresponding to the unique identifier of the target data block by 1.

[0061] Step S504: If the hash table of the storage node does not contain a unique identifier for the target data block, the access count value corresponding to the unique identifier of the target data block is initialized to 1.

[0062] In the above embodiments, the hash table of the storage node tracks the access count value C of each data block on the node in real time. i The steps are as follows: 1) Generate Key: The system generates a unique identifier Key for the data block based on the access request information. 2) Search Hash Table: The system performs a fast search in the hash table in memory using this Key. Case A: Found (KeyExists): This indicates that the data block has been accessed before, and its entry already exists in the hash table. The system retrieves the corresponding Value from the hash table, which is a pointer to the metadata structure. Then, the system accesses the access count value C in that structure. i Perform an increment operation (count++). Case B: KeyNotFound: This indicates that this is the first time this data block has been accessed within the current scheduled period. The system will allocate a new metadata structure in memory for this new data block. Set the access count value C of this new structure... i Initialize to 1. Finally, insert this new "Key-Value" pair (i.e., "data block ID" -> "pointer to the new metadata structure") into the hash table. 3) Complete access: After updating the count value, the system continues to perform the original data read and write operations. Because this lookup and update process utilizes the characteristics of the hash table, the average time complexity is close to O(1), which is very fast and can achieve "real-time tracking" without significantly affecting the performance of the main I / O path.

[0063] As an optional implementation, before querying the hash table of the storage node where each data block resides based on its unique identifier at predetermined intervals to obtain the access count value corresponding to the unique identifier of each data block, the above method further includes:

[0064] Step S601: Halve the access count value of the hash table of the storage node at predetermined intervals.

[0065] In the above embodiments, to make the model better reflect recent access activity, the system periodically "cools down" all count values. For example, every predetermined period Δt, all count values ​​are halved. , The access count value C for the i-th data block in the k-th predetermined period. i , The access count value C for the i-th data block in the (k-1)-th predetermined period. i In other words, at the beginning of each predetermined period, the access count value of each data block in the previous predetermined period is halved and the access count value continues to accumulate, so that the access count value can better reflect the recent access popularity.

[0066] As an optional implementation, after determining that the target data is of the first type and storing the target data, the above method further includes:

[0067] Step S701: If the target data is of the first type, read the data block corresponding to the stored target data;

[0068] Step S702: Compress the data block corresponding to the target data to obtain a compressed data block;

[0069] Step S703: Store the compressed data block to the storage node where the target data is located and delete the data block corresponding to the target data.

[0070] In the above embodiment, a background worker thread asynchronously reads the data blocks marked as "cold" (i.e., the target data is of type 1), compresses them using a fast compression algorithm, and writes the compressed data back. Simultaneously, it updates the metadata, marking it as compressed and recording the new size, further reducing storage costs.

[0071] As an optional implementation, after storing the compressed data block to the storage node where the target data resides and deleting the data block corresponding to the target data, the above method further includes:

[0072] Step S801: If the data block requested for access is a compressed data block, read the compressed data block of the storage node;

[0073] Step S802: Decompress the compressed data block to obtain decompressed data, and return the decompressed data to the requester of the access request, while the compressed data block of the storage node remains unchanged.

[0074] In the above embodiments, when a read request accesses a compressed data block, the storage service intercepts the request on the I / O path, reads the compressed data from the disk, quickly decompresses it in memory, and then returns the decompressed data to the user. The data on the disk remains compressed, without increasing storage space usage.

[0075] Specifically, such as Figure 4 As shown, when a data block read request is received, the metadata of the data block is read. If the data block is in a compressed state, the compressed data is read, quickly decompressed in memory, and then the decompressed data is returned to the user. If the data block is in an uncompressed state, the uncompressed data is read directly and returned to the user, and the access count value of the data block is incremented. The data block access count value is then returned to the background for real-time monitoring.

[0076] As an optional implementation, after storing the compressed data block to the storage node where the target data resides and deleting the data block corresponding to the target data, the above method further includes:

[0077] Step S901: If the access count value of the compressed data block is greater than the second access count threshold, read the compressed data block of the storage node, decompress the compressed data block to obtain decompressed data, return the decompressed data to the storage node for storage, and delete the compressed data block.

[0078] In the above embodiments, accessing compressed data also increases its access count. When a compressed data block is accessed frequently, its access count exceeds the second access count threshold Q. warm It will be reclassified as either Type II or Type III data, i.e., "warm / hot data". Another background worker thread is responsible for reading and decompressing this newly added "warm / hot data" from the disk, writing the raw, uncompressed data back to the disk, and updating the metadata. This process avoids repeated on-the-spot decompression of frequently accessed data, thus optimizing read performance.

[0079] Specifically, such as Figure 5 As shown, the backend stores access count metadata for all data blocks, periodically or on demand reads the access count values ​​of data blocks, periodically "cools" all count values, and then constructs a global histogram of data block access counts. Based on the histogram, it determines whether the access count value of each data block is lower than the second access count threshold Q. warmIf yes, then determine whether the data block has been compressed. If the data block has been compressed, keep the state unchanged. If it has been compressed, the background will compress the data block and update the metadata. If no, then determine whether the data block has been compressed. If the data block has not been compressed, keep the state unchanged. If the data block has been compressed, the background will decompress the data block and update the metadata.

[0080] As an optional implementation, step S302 above includes:

[0081] Step S3021: Construct a global cost function. The function value of the global cost function is the weighted average of storage cost, repair cost and access cost. Storage cost is used to characterize the size of storage resources occupied by each data block, repair cost is used to characterize the size of the average repair bandwidth occupied by each data block, and access cost is used to characterize the size of the sum of decompression time for accessing each data block.

[0082] Step S3022: Adjust the second access count threshold until the global cost function is minimized.

[0083] In the above embodiments, the entire system can be modeled as a multi-objective optimization problem. The objective is to minimize a global cost function, where the function value of the global cost function is the weighted average of storage cost, repair cost, and access cost, in order to comprehensively evaluate the cost of the storage system and minimize the global cost function to find the optimal storage strategy.

[0084] As an optional implementation, step S3021 above includes:

[0085] Step S30211: Construct a storage cost function. The value of the storage cost function is inversely proportional to the proportion of compressed data blocks. The proportion of compressed data blocks is the ratio of the number of compressed data blocks to the total number of data blocks.

[0086] Step S30212: Construct a repair cost function. The function value of the repair cost function is the sum of the repair bandwidth for each failed data block.

[0087] Step S30213: Construct an access cost function. The function value of the access cost function is the sum of the decompression time for accessing each data block.

[0088] Step S30214: Generate a storage cost function based on the storage cost function, repair cost function, and access cost function.

[0089] In the above embodiment, the storage cost function is minC. total =λ store C total +λ repair C repair +λ access C accessAmong them, the storage cost function Size(i) is the size of the i-th data block, and n is the number of data blocks. Storage cost is inversely proportional to the ratio of compressed data. Repair cost function b i The total repair bandwidth corresponding to the i-th data block is related to the encoding scheme used for the data and the node failure rate. For data encoded using the target encoding method, the repair cost is lower. The repair cost is defined as the total repair bandwidth of the failed data block (assuming a data block is encoded using [n,k] MDS codes, then the repair bandwidth of that data block is defined as the sum of the repair bandwidths for failures from storage node 1 to storage node n; if the data block has not undergone any encoding, its repair bandwidth is defined as twice the total repair bandwidth using only the initial MDS code). Access cost function. , Let λ be the decompression delay for the j-th compressed data block, m be the number of compressed data blocks, and λ be the decompression delay for the j-th compressed data block. store , λ repair and λ access This is a weighting factor that can be adjusted by the system administrator based on business needs (e.g., sensitivity to cost, performance, or reliability).

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0091] Embodiments of this application also provide a data storage device. Figure 6 This is a structural block diagram of a data storage device according to an embodiment of this application, such as... Figure 6 As shown, the device includes:

[0092] The first determining module 802 is used to determine the access probability of the target data;

[0093] Specifically, the probability of accessing the target data is predicted based on information such as the file type and owner of the target data, which is used to evaluate the access frequency after the target data is stored.

[0094] The first storage module 804 is used to determine that the target data is first type data and store the target data when the access probability of the target data is less than or equal to a first probability threshold.

[0095] Specifically, if the access probability of the target data is less than or equal to the first probability threshold, it indicates that the access frequency after storage is low and the impact of data loss on stability is not significant. The target data is determined to be the first type of data, i.e., cold data. The target data is stored directly without encoding, and there is no need to store the check bit data, thus saving storage costs.

[0096] The second storage module 806 is used to determine that the target data is second type data when the access probability of the target data is greater than the first probability threshold and less than the second probability threshold, and to encode the target data using the maximum distance divisible code encoding method and then store it.

[0097] Specifically, if the access probability of the target data is greater than the first probability threshold and less than the second probability threshold, it indicates that the access frequency after storage is moderate. The target data is determined to be second-type data, i.e. warm data. Data loss has a significant impact on stability. The target data is encoded and stored using a maximum distance divisible code encoding method to ensure that data loss can be repaired, maintain stability, and increase storage costs only slightly.

[0098] The third storage module 808 is used to determine that the target data is third type data when the access probability of the target data is greater than or equal to the second probability threshold, and to encode the target data using the target encoding method and then store it. The target encoding method is the encoding method corresponding to the repair bandwidth of the repair bandwidth that is less than the maximum distance divisible code.

[0099] Specifically, if the access probability of the target data is greater than the second probability threshold, it indicates that the access frequency after storage is high, thus identifying the target data as the third type of data, i.e., hot data. Data loss has a significant impact on stability. Encoding the target data before storage can repair data loss and ensure stability, but the relatively high repair frequency leads to high repair bandwidth. Therefore, the maximum distance divisible code encoding method is adopted to reduce the repair bandwidth. Although it increases the encoding complexity and storage cost compared to the maximum distance divisible code encoding method, the amount of hot data is not large, so the overall increase in storage cost is not significant.

[0100] Using the above device, target data is categorized into three types based on its access probability, from low to high: Type 1 data, Type 2 data, and Type 3 data. Type 1 data, due to its low access frequency, has a minimal impact on stability if lost, and therefore does not require encoding, saving storage costs and minimizing stability degradation. Type 2 data has a moderate access frequency, and its loss has a significant impact on stability. It is encoded using a maximum distance separable code method to ensure repairability, with high repair bandwidth but infrequent repairs that do not consume excessive repair bandwidth, ensuring high stability with minimal increase in storage costs. Type 3 data has a high access frequency, and its loss has a significant impact on stability, requiring relatively frequent repairs. It is encoded using a target encoding method to reduce repair bandwidth while maintaining high stability, but this increases storage costs per unit of data. However, since the amount of Type 3 data is not high, the overall increase in storage costs is minimal, thus solving the problem in related technologies where storage management methods struggle to balance system stability and storage costs.

[0101] As an optional implementation, the first determining module includes:

[0102] The extraction unit is used to extract multiple features from the target data, including file type values.

[0103] The first calculation unit is used to perform a weighted summation of multiple features using the weights of each feature to obtain a comprehensive feature value;

[0104] The second calculation unit is used to substitute the comprehensive feature values ​​into the logistic function to obtain the access probability of the target data.

[0105] In the above embodiments, for a target data A that is about to be written... j The system builds a probability model P(A) for future access based on prior knowledge such as its metadata (e.g., file type, owner) and historical access patterns (e.g., the user / application's historical behavior). j This can be represented by a logistic regression model or a weighted scoring function: , where f i (A j ) is from target data A j The extracted i-th feature (e.g., the feature is the file type value; if the file type is video, the feature is 1, otherwise it is 0), m is the number of features, ω i It is the weight of that feature, and σ is the Sigmoid function, i.e. By using a probability model of future access, the access probability of target data can be predicted, so as to classify data types before storage.

[0106] As an optional implementation, the above-described apparatus further includes:

[0107] The acquisition module is used to acquire the access count value of each data block at predetermined intervals after the target data has been stored. The data blocks include the data blocks that store the target data.

[0108] The first generation module is used to generate a first access count threshold and a second access count threshold based on the access count value of each data block, such that the proportion of data blocks with access count values ​​greater than or equal to the first access count threshold is less than the first proportion threshold, and the proportion of data blocks with access count values ​​greater than the second access count threshold and less than the first access count threshold is less than the second proportion threshold.

[0109] The second determining module is used to determine data blocks whose access count value is greater than or equal to the first access count threshold as third type data;

[0110] The third determination module is used to determine data blocks whose access count value is greater than the second access count threshold and less than the first access count threshold as second type data;

[0111] The fourth determination module is used to determine data blocks whose access count value is less than or equal to the second access count threshold as first type data.

[0112] In the above embodiments, after data enters the storage node, the background continuously monitors the access count of the corresponding data block within a specific time interval (generally considered long enough, such as 3 days). The system periodically collects count values ​​from all storage nodes to construct a global histogram of data block access counts, such as... Figure 3 As shown, this histogram divides the count values ​​into multiple bins and dynamically adjusts the first access threshold Q by counting the number of data blocks falling into each bin. hot Second access threshold Q warm The horizontal axis of the histogram represents the number of times a data block is accessed, and the vertical axis represents the number of data blocks. Note that the number of accesses to a data block is expressed in terms of 2^n. i As a reference benchmark, each rectangle in the histogram represents the number of visits falling within [2]. i ,2 i+1 The number of data blocks in this range. The dynamic adjustment of Q here. hot and Q warm This means that if the system sets the proportion of data blocks of the third type of data to not exceed the first proportion threshold (e.g., 40%), and the proportion of data blocks of the second type of data to not exceed the second proportion threshold (e.g., 20%), then the third type of data is hot data, the second type of data is warm data, and the first type of data is cold data, thereby achieving dynamic data type classification that closely reflects actual access situations.

[0113] As an optional implementation, the above-described apparatus further includes:

[0114] The first update module is used to control the data block to remain unchanged after determining each data block as one of the third type of data, the second type of data, and the first type of data, when the target data corresponding to the data block is the third type of data and the data block is the second type of data.

[0115] The second update module is used to delete the check bits of the codewords of the data block when the target data corresponding to the data block is third type data or second type data and the data block is first type data.

[0116] The third update module is used to encode the data block using the maximum distance divisible code encoding method when the target data corresponding to the data block is of the first type and the data block is of the second type.

[0117] The fourth update module is used to encode the data block using the target encoding method when the target data corresponding to the data block is of type 1 or type 2 and the data block is of type 3.

[0118] In the above embodiments, the third type of data is hot data, the second type of data is warm data, and the first type of data is cold data. Based on the number of times the data is accessed within a specific time period, the data is dynamically classified and re-encoded. Specifically, when hot data becomes warm data, no encoding method conversion is performed, and its data storage method is retained; when hot or warm data becomes cold data, the check bits of the codeword are directly deleted, corresponding to the recycling process in the operating system; when cold data becomes warm data, the data is encoded using MDS; when cold or warm data becomes hot data, the data is encoded using the target encoding method.

[0119] As an optional implementation, the acquisition module includes:

[0120] The acquisition unit is used to query the hash table of the storage node where each data block is located at predetermined intervals based on the unique identifier of each data block, and obtain the access count value corresponding to the unique identifier of each data block. The hash table is used to track the access count value of each data block on the storage node.

[0121] In the above embodiments, an in-memory hash table (essentially an efficient "dictionary" or "index" that maps a key to a value) is maintained on each storage node to track the access count C of each data block on that node in real time. i C i Let C be the access count value of the i-th data block, where the key is the unique identifier of the data block, and the value is the access count value C of the data block. i You can then query the access count value of a data block through the hash table of the storage node.

[0122] As an optional implementation, the above-described apparatus further includes:

[0123] The second generation module is used to generate a unique identifier for a target data block based on the access request of the target data block before querying the hash table of the storage node where each data block is located according to the unique identifier of each data block at a predetermined interval and obtaining the access count value corresponding to the unique identifier of each data block. The target data block is a data block stored in a storage node.

[0124] The query module is used to query the hash table of the storage node where the target data block is located based on the unique identifier of the data block;

[0125] The first control module is used to increment the access count value corresponding to the unique identifier of the target data block by 1 when the hash table of the storage node contains a unique identifier of the target data block.

[0126] The second control module is used to initialize the access count value corresponding to the unique identifier of the target data block to 1 when the hash table of the storage node does not contain the unique identifier of the target data block.

[0127] In the above embodiments, the hash table of the storage node tracks the access count value C of each data block on the node in real time. i The steps are as follows: 1) Generate Key: The system generates a unique identifier Key for the data block based on the access request information. 2) Search Hash Table: The system performs a fast search in the hash table in memory using this Key. Case A: Found (KeyExists): This indicates that the data block has been accessed before, and its entry already exists in the hash table. The system retrieves the corresponding Value from the hash table, which is a pointer to the metadata structure. Then, the system accesses the access count value C in that structure. i Perform an increment operation (count++). Case B: KeyNotFound: This indicates that this is the first time this data block has been accessed within the current scheduled period. The system will allocate a new metadata structure in memory for this new data block. Set the access count value C of this new structure... i Initialize to 1. Finally, insert this new "Key-Value" pair (i.e., "data block ID" -> "pointer to the new metadata structure") into the hash table. 3) Complete access: After updating the count value, the system continues to perform the original data read and write operations. Because this lookup and update process utilizes the characteristics of the hash table, the average time complexity is close to O(1), which is very fast and can achieve "real-time tracking" without significantly affecting the performance of the main I / O path.

[0128] As an optional implementation, the above-described apparatus further includes:

[0129] The fifth update module is used to halve the access count value of the hash table of the storage node at predetermined intervals before querying the hash table of the storage node where each data block is located based on the unique identifier of each data block to obtain the access count value corresponding to the unique identifier of each data block.

[0130] In the above embodiments, to make the model better reflect recent access activity, the system periodically "cools down" all count values. For example, every predetermined period Δt, all count values ​​are halved. , The access count value C for the i-th data block in the k-th predetermined period. i , The access count value C for the i-th data block in the (k-1)-th predetermined period. i In other words, at the beginning of each predetermined period, the access count value of each data block in the previous predetermined period is halved and the access count value continues to accumulate, so that the access count value can better reflect the recent access popularity.

[0131] As an optional implementation, the above-described apparatus further includes:

[0132] The first reading module is used to read the data block corresponding to the stored target data after determining that the target data is of the first type of data and storing the target data, provided that the target data is of the first type of data.

[0133] The compression module is used to compress the data blocks corresponding to the target data to obtain compressed data blocks;

[0134] The sixth update module is used to store compressed data blocks to the storage node where the target data is located and to delete the data blocks corresponding to the target data.

[0135] In the above embodiment, a background worker thread asynchronously reads the data blocks marked as "cold" (i.e., the target data is of type 1), compresses them using a fast compression algorithm, and writes the compressed data back. Simultaneously, it updates the metadata, marking it as compressed and recording the new size, further reducing storage costs.

[0136] As an optional implementation, the above-described apparatus further includes:

[0137] The second reading module is used to read the compressed data block of the storage node after storing the compressed data block to the storage node where the target data is located and deleting the data block corresponding to the target data, when the data block accessed in the access request is a compressed data block;

[0138] The first decompression module is used to decompress the compressed data blocks, obtain the decompressed data, and return the decompressed data to the requester of the access request, while the compressed data blocks of the storage node remain unchanged.

[0139] In the above embodiments, when a read request accesses a compressed data block, the storage service intercepts the request on the I / O path, reads the compressed data from the disk, quickly decompresses it in memory, and then returns the decompressed data to the user. The data on the disk remains compressed, without increasing storage space usage.

[0140] Specifically, such as Figure 4 As shown, when a data block read request is received, the metadata of the data block is read. If the data block is in a compressed state, the compressed data is read, quickly decompressed in memory, and then the decompressed data is returned to the user. If the data block is in an uncompressed state, the uncompressed data is read directly and returned to the user, and the access count value of the data block is incremented. The data block access count value is then returned to the background for real-time monitoring.

[0141] As an optional implementation, the above-described apparatus further includes:

[0142] The second decompression module is used to, after storing the compressed data block to the storage node where the target data is located and deleting the data block corresponding to the target data, read the compressed data block from the storage node, decompress the compressed data block, obtain decompressed data, and return the decompressed data to the storage node for storage and delete the compressed data block after the access count value of the compressed data block is greater than the second access count threshold.

[0143] In the above embodiments, accessing compressed data also increases its access count. When a compressed data block is accessed frequently, its access count exceeds the second access count threshold Q. warm It will be reclassified as either Type II or Type III data, i.e., "warm / hot data". Another background worker thread is responsible for reading and decompressing this newly added "warm / hot data" from the disk, writing the raw, uncompressed data back to the disk, and updating the metadata. This process avoids repeated on-the-spot decompression of frequently accessed data, thus optimizing read performance.

[0144] Specifically, such as Figure 5 As shown, the backend stores access count metadata for all data blocks, periodically or on demand reads the access count values ​​of data blocks, periodically "cools" all count values, and then constructs a global histogram of data block access counts. Based on the histogram, it determines whether the access count value of each data block is lower than the second access count threshold Q. warmIf yes, then determine whether the data block has been compressed. If the data block has been compressed, keep the state unchanged. If it has been compressed, the background will compress the data block and update the metadata. If no, then determine whether the data block has been compressed. If the data block has not been compressed, keep the state unchanged. If the data block has been compressed, the background will decompress the data block and update the metadata.

[0145] As an optional implementation, the first generation module includes:

[0146] The building unit is used to construct the global cost function. The function value of the global cost function is the weighted average of storage cost, repair cost and access cost. Storage cost is used to characterize the amount of storage resources occupied by each data block, repair cost is used to characterize the average repair bandwidth occupied by each data block, and access cost is used to characterize the sum of the decompression time of each data block.

[0147] An adjustment unit is used to adjust the second access count threshold until the global cost function is minimized.

[0148] In the above embodiments, the entire system can be modeled as a multi-objective optimization problem. The objective is to minimize a global cost function, where the function value of the global cost function is the weighted average of storage cost, repair cost, and access cost, in order to comprehensively evaluate the cost of the storage system and minimize the global cost function to find the optimal storage strategy.

[0149] As an optional implementation, the above-mentioned building unit includes:

[0150] The first construction subunit is used to construct the storage cost function. The value of the storage cost function is inversely proportional to the proportion of compressed data blocks. The proportion of compressed data blocks is the ratio of the number of compressed data blocks to the total number of data blocks.

[0151] The second construction subunit is used to construct the repair cost function, the function value of which is the sum of the repair bandwidth of each data block failure;

[0152] The third construction subunit is used to construct the access cost function, the function value of which is the sum of the decompression time for accessing each data block;

[0153] The generation sub-unit is used to generate the storage cost function based on the storage cost function, the repair cost function, and the access cost function.

[0154] In the above embodiment, the storage cost function is minC. total =λ store C total +λ repair C repair +λ access C accessAmong them, the storage cost function Size(i) is the size of the i-th data block, and n is the number of data blocks. Storage cost is inversely proportional to the ratio of compressed data. Repair cost function b i The total repair bandwidth corresponding to the i-th data block is related to the encoding scheme used for the data and the node failure rate. For data encoded using the target encoding method, the repair cost is lower. The repair cost is defined as the total repair bandwidth of the failed data block (assuming a data block is encoded using [n,k] MDS codes, then the repair bandwidth of that data block is defined as the sum of the repair bandwidths for failures from storage node 1 to storage node n; if the data block has not undergone any encoding, its repair bandwidth is defined as twice the total repair bandwidth using only the initial MDS code). Access cost function. , Let λ be the decompression delay for the j-th compressed data block, m be the number of compressed data blocks, and λ be the decompression delay for the j-th compressed data block. store , λ repair and λ access This is a weighting factor that can be adjusted by the system administrator based on business needs (e.g., sensitivity to cost, performance, or reliability).

[0155] For a description of the features in the embodiment corresponding to the data storage device, please refer to the relevant description in the embodiment corresponding to the data storage method, which will not be repeated here.

[0156] Embodiments of this application also provide a storage system including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data storage method embodiments.

[0157] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data storage method embodiments when it is run.

[0158] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0159] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data storage method embodiments.

[0160] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data storage method embodiments.

[0161] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0162] The data storage method, apparatus, and storage system provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data storage method, characterized in that, include: Determine the access probability of the target data; If the access probability of the target data is less than or equal to a first probability threshold, the target data is determined to be first type of data, and the target data is stored. If the access probability of the target data is greater than the first probability threshold and less than the second probability threshold, the target data is determined to be second type data, and the target data is encoded and stored using a maximum distance separable code encoding method. If the access probability of the target data is greater than or equal to the second probability threshold, the target data is determined to be third type data, and the target data is encoded and stored using a target encoding method. The target encoding method is the encoding method corresponding to the encoding where the repair bandwidth is less than the repair bandwidth of the maximum distance divisible code. After the target data is stored, the method further includes: obtaining the access count value of each data block at predetermined intervals, wherein the data block includes the data block storing the target data; constructing a global cost function, wherein the function value of the global cost function is a weighted average of storage cost, repair cost, and access cost, wherein the storage cost is used to characterize the amount of storage resources occupied by storing each data block, the repair cost is used to characterize the amount of repair bandwidth occupied by repairing each data block on average, and the access cost is used to characterize the amount of decompression time of accessing each data block; adjusting a first access count threshold and a second access count threshold until the global cost function is minimized; identifying the data block whose access count value is greater than or equal to the first access count threshold as the third type of data; identifying the data block whose access count value is greater than the second access count threshold and less than the first access count threshold as the second type of data; and identifying the data block whose access count value is less than or equal to the second access count threshold as the first type of data. The method further includes: when the target data corresponding to the data block is the third type of data and the data block is the second type of data, controlling the data block to remain unchanged; when the target data corresponding to the data block is the third type of data or the second type of data and the data block is the first type of data, deleting the check bit of the codeword of the data block; when the target data corresponding to the data block is the first type of data and the data block is the second type of data, encoding the data block using the maximum distance divisible code encoding method; and when the target data corresponding to the data block is the first type of data or the second type of data and the data block is the third type of data, encoding the data block using the target encoding method.

2. The data storage method according to claim 1, characterized in that, Determine the access probability of the target data, including: Extract multiple features from the target data, including a file type value; The weights of each feature are used to perform a weighted summation of the multiple features to obtain a comprehensive feature value; Substituting the comprehensive feature value into the logistic function yields the access probability of the target data.

3. The data storage method according to claim 1, characterized in that, Obtaining the access count value of each data block at predetermined intervals includes: At predetermined intervals, the hash table of the storage node where each data block is located is queried based on the unique identifier of each data block to obtain the access count value corresponding to the unique identifier of each data block. The hash table is used to track the access count value of each data block on the storage node.

4. The data storage method according to claim 3, characterized in that, Before querying the hash table of the storage node where each data block resides based on its unique identifier at predetermined intervals to obtain the access count value corresponding to the unique identifier of each data block, the method further includes: The access count value of the hash table of the storage node is halved at predetermined intervals.

5. The data storage method according to claim 1, characterized in that, After determining that the target data is of the first type of data and storing the target data, the method further includes: If the target data is the first type of data, read the data block corresponding to the stored target data; The data block corresponding to the target data is compressed to obtain a compressed data block; The compressed data block is stored in the storage node where the target data is located, and the data block corresponding to the target data is deleted.

6. The data storage method according to claim 5, characterized in that, After storing the compressed data block to the storage node where the target data is located and deleting the data block corresponding to the target data, the method further includes: If the data block requested for access is the compressed data block, read the compressed data block from the storage node; The compressed data block is decompressed to obtain decompressed data, and the decompressed data is returned to the requester of the access request, while the compressed data block of the storage node remains unchanged.

7. The data storage method according to claim 5, characterized in that, After storing the compressed data block to the storage node where the target data is located and deleting the data block corresponding to the target data, the method further includes: If the access count of the compressed data block is greater than the second access count threshold, the compressed data block of the storage node is read, the compressed data block is decompressed to obtain decompressed data, the decompressed data is returned to the storage node for storage, and the compressed data block is deleted.

Images