Distributed storage method, device and electronic device based on data modification frequency

Abstract

The present application relates to the field of database technology, and specifically to a distributed storage method, device, and electronic device based on data modification frequency. The method can configure different erasure code encoding strategies for different modification frequency levels, wherein the higher the modification frequency level, the smaller the number of data blocks used for encoding, and the modification frequency level is marked in units of data partitions included in the data file, so that when a data file is distributedly stored, an adaptive erasure code encoding strategy can be adopted for the data partition according to the modification frequency, wherein, if the data modification frequency is high, the number of encoded data blocks adopted in the encoding strategy is small, so as to reduce the computing resource occupation and network overhead when the data blocks are modified; if the data modification frequency is low, the number of encoded data blocks adopted in the encoding strategy is large, so as to improve the storage space utilization, thereby improving the flexibility of the distributed storage system and improving the performance of the distributed storage system.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a distributed storage method, device and electronic device based on data modification frequency. Background Art

[0002] Erasure codes (EC) are a data protection method primarily used in data storage and network transmission to ensure data reliability and fault tolerance. They encode data by splitting it into fragments and creating redundant data blocks. These encoded data fragments are stored in different locations, such as different disks, storage nodes, or other geographical locations. Core technologies used in erasure codes include polynomial interpolation and oversampling, which enable the system to recover the original data even if some data is lost.

[0003] An erasure code system can be represented as "n = k + m," where "k" represents the number of original data blocks, "m" represents the number of added parity blocks (redundant data blocks), and "n" represents the total number of encoded data blocks. For example, in an "EC 10 / 16" configuration, six additional parity blocks (m) are added to ten original data blocks (k), and these 16 encoded data fragments (n) are stored in different locations. Even if up to m data blocks are lost or damaged, the original data can still be recovered from the remaining data blocks.

[0004] In practical applications, erasure codes are widely used in distributed storage systems, object storage, cloud computing, and large-scale data transmission.

[0005] In erasure coding technology, if a file is partially modified, the entire file may need to be re-encoded, which may result in large computational and network overhead. Therefore, erasure codes are generally suitable for scenarios where data modifications are infrequent. This application aims to expand the applicable scenarios of erasure codes and reduce related computational and network overhead. Erasure Codes (EC) is a data protection method that is mainly used in data storage and network transmission to ensure data reliability and fault tolerance. It encodes data by splitting it into fragments and creating redundant data blocks. These encoded data fragments are stored in different locations, such as different disks, storage nodes, or other geographical locations. The core technologies of erasure coding include polynomial interpolation or oversampling, which enable the system to recover the original data when some data is lost.

[0006] An erasure code system can be represented as "n = k + m," where "k" represents the number of original data blocks, "m" represents the number of added parity blocks (redundant data blocks), and "n" represents the total number of encoded data blocks. For example, in an "EC 10 / 16" configuration, six additional parity blocks (m) are added to ten original data blocks (k), and these 16 encoded data fragments (n) are stored in different locations. Even if up to m data blocks are lost or damaged, the original data can still be recovered from the remaining data blocks.

[0007] In practical applications, erasure codes are widely used in distributed storage systems, object storage, cloud computing, and large-scale data transmission.

[0008] In erasure coding, if a file is partially modified, the entire file may need to be re-encoded, which can result in significant computational and network overhead. Therefore, erasure coding is generally suitable for scenarios where data modifications are infrequent. This application aims to expand the application scenarios of erasure coding and reduce the associated computational and network overhead. Summary of the Invention

[0009] In view of this, the present application discloses a distributed storage method based on data modification frequency. The method may include obtaining a data partition contained in a data file to be stored, and a target encoding strategy corresponding to the modification frequency level of the data partition; wherein different data modification frequency levels correspond to different erasure code encoding strategies, and the higher the modification frequency level, the smaller the number of data blocks used for encoding indicated by the encoding strategy; according to the first number included in the target encoding strategy, the multiple data blocks included in the data partition are divided into at least one data stripe, and the data stripe includes the first number of data blocks; for each data stripe, the first number of data blocks contained therein are encoded to obtain the second number of check blocks included in the target encoding strategy, and the first number of data blocks and the second number of check blocks are distributed and stored in different storage nodes.

[0010] In some embodiments, the modification frequency level includes at least two levels; the method for generating a coding strategy includes: obtaining a first coding strategy corresponding to the lowest level, the first coding strategy including the number of data blocks involved in encoding in the erasure code technology and the number of check blocks generated by the encoding; according to the distance between other levels and the lowest level, reducing the number of data blocks involved in encoding in the first coding strategy to obtain a second coding strategy corresponding to the other levels.

[0011] In some embodiments, reducing the number of data blocks involved in encoding in the first encoding strategy based on the distance between other levels and the lowest level includes: proportionally reducing the number of data blocks involved in encoding in the first encoding strategy based on the distance; or arithmetically reducing the number of data blocks involved in encoding in the first encoding strategy based on the distance.

[0012] In some embodiments, the method further includes: updating the modification frequency level of the data partition; wherein, in response to the time length between the generation time of the data in the data partition and the current time being within a preset time length, the modification frequency level of the data partition is updated to a higher level; in response to the time length between the generation time of the data in the data partition and the current time exceeding the preset time length, the modification frequency level of the data partition is updated to a lower level; in response to the frequency of the data in the data partition being operated exceeding the preset frequency, the modification frequency level of the data partition is updated to a higher level; in response to the frequency of the data in the data partition being operated within the preset frequency, the modification frequency level of the data partition is updated to a lower level; in response to the modification frequency level of the data partition being adjusted, at the idle time of the distributed storage system, the data of the stored data partition is obtained and distributed storage is performed according to the encoding strategy corresponding to the adjusted modification frequency level.

[0013] In some embodiments, the data file is a data table, and a fixed length is set for at least one column field of the data table to allocate a fixed storage space according to the fixed length to store the field content of the column field, so that modifying the field content will not change the size of the storage space occupied by the field, and will not change the order of the data blocks; the method also includes: establishing a correspondence between the data block and the field content contained in the data block.

[0014] In some embodiments, the method includes: in response to a modification operation on the target field content, obtaining a target data block corresponding to the target field content according to the correspondence; updating the target data block; and updating the target check block corresponding to the target data block according to the updated target data block.

[0015] In some embodiments, updating the target check block corresponding to the target data block according to the updated target data block includes: obtaining the target check block corresponding to the target data block; obtaining an intermediate value through a matrix inverse operation based on the target check block and the target data block before the update; and obtaining an updated target check block through a matrix operation based on the intermediate value and the updated target data block.

[0016] In some embodiments, the method further includes: for at least one of the column fields, counting the longest field length and the frequency of occurrence of the longest field; when the frequency of occurrence of the longest field reaches a preset number of times, updating the fixed length of the column field to the longest field length; when the longest field does not reach the set initial fixed length and the longest field does not change for a preset time length, updating the fixed length of the column field to the longest field length; after updating the fixed length of the column field to the longest field length, in response to the column field receiving a target field content greater than the longest field length, comparing the length of the target field content with the initial fixed length of the column field; if the length of the target field content is greater than the initial fixed length, discarding the target field content; if the length of the target field content is less than or equal to the initial fixed length, updating the fixed length of the column field to the length of the target field content.

[0017] The present application also proposes a distributed storage device based on data modification frequency, the device comprising: an acquisition module, acquiring a data partition contained in a data file to be stored, and a target encoding strategy corresponding to the modification frequency level of the data partition; wherein different data modification frequency levels correspond to different erasure code encoding strategies, and a higher modification frequency level corresponds to a smaller number of data blocks indicated by the encoding strategy for encoding; a division module, dividing the multiple data blocks included in the data partition into at least one data stripe according to a first number included in the target encoding strategy, the data stripe including the first number of data blocks; a distributed storage module, encoding the first number of data blocks contained in each data stripe to obtain a second number of check blocks included in the target encoding strategy, and distributively storing the first number of data blocks and the second number of check blocks to different storage nodes.

[0018] The present application also proposes an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the distributed storage method based on data modification frequency as described in any of the aforementioned embodiments by running the executable instructions.

[0019] In the scheme recorded in any of the aforementioned embodiments, different erasure code encoding strategies can be configured for different modification frequency levels, wherein the higher the modification frequency level, the smaller the number of data blocks used for encoding indicated by the corresponding encoding strategy, and the modification frequency level is marked in units of data partitions included in the data file, so that when a data file is distributedly stored, an adaptive erasure code encoding strategy can be adopted for the data partition according to the modification frequency, wherein, if the data modification frequency is high, the number of encoded data blocks adopted in the encoding strategy is small to reduce the computing resource occupation and network overhead when the data blocks are modified; if the data modification frequency is low, the number of encoded data blocks adopted in the encoding strategy is large to improve the storage space utilization.

[0020] Therefore, compared with a one-size-fits-all distributed storage solution for a data file, the solution described in this application improves the flexibility of the distributed storage system and improves the performance of the distributed storage system.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] The following is a brief introduction to the drawings required for describing the embodiments or related technologies.

[0024] Figure 1 This is a flow chart of a distributed storage method based on data modification frequency shown in this application.

[0025] Figure 2 This is a flow chart of a method for generating a coding strategy shown in this application.

[0026] Figure 3 This is a flow chart of a method for updating a check block shown in this application.

[0027] Figure 4 This is a flow chart of a method for updating a check block shown in this application.

[0028] Figure 5 This is a structural diagram of a distributed storage device based on data modification frequency shown in an embodiment of the present application.

[0029] Figure 6 A schematic diagram of the hardware structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following exemplary embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0031] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "the" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used herein, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0032] In related technologies, when distributing and storing a data file using erasure coding, the system inputs the number of data blocks (k) used in the encoding and the number of check blocks (m) generated by the encoding. The data file is then divided into multiple data stripes, each consisting of k data blocks. m redundant check blocks are then calculated from these k data blocks, and these k+m data blocks are then stored concurrently on different storage nodes.

[0033] Different values ​​for k and m can be set based on the value and storage cost of each file. The storage space utilization ratio is k / (k+m). Generally speaking, the larger k and the smaller m, the higher the storage space utilization, but the lower the protection level, and the more data needs to be read for reconstruction. Conversely, the smaller k and the larger m, the lower the storage space utilization, but the higher the protection level, and the faster the reconstruction.

[0034] In this solution, data files are distributedly stored using a one-size-fits-all erasure coding scheme. Using a large k, small m strategy results in high storage space utilization. However, in scenarios with frequent data modifications, when a data block is modified, k-1 other data blocks must be retrieved and recalculated to create m encoding blocks. The larger k is, the greater the computing resources and network overhead. If a small k, large m strategy is adopted to reduce computing resources and network overhead, storage space utilization will be reduced. This one-size-fits-all erasure coding scheme limits the flexibility of distributed storage and impacts the performance of distributed storage systems.

[0035] In view of this, the present application proposes a distributed storage method based on data modification frequency. This method can configure different erasure code encoding strategies for different modification frequency levels, wherein the higher the modification frequency level, the smaller the number of data blocks used for encoding. The modification frequency level is marked in units of data partitions included in the data file, so that when a data file is distributedly stored, an adaptive erasure code encoding strategy can be adopted for the data partition according to the modification frequency. If the data modification frequency is high, the number of encoded data blocks adopted in the encoding strategy is small to reduce the computing resource occupation and network overhead when the data blocks are modified. If the data modification frequency is low, the number of encoded data blocks adopted in the encoding strategy is large to improve storage space utilization.

[0036] Therefore, compared with a one-size-fits-all distributed storage solution for a data file, the solution described in this application improves the flexibility of the distributed storage system and improves the performance of the distributed storage system.

[0037] The following describes the embodiments with reference to the accompanying drawings.

[0038] See Figure 1 , Figure 1 This is a flow chart of a distributed storage method based on data modification frequency shown in this application. Figure 1 As shown, the method may include S102-S106.

[0039] S102: Acquire data partitions contained in the data file to be stored, and target encoding strategies corresponding to modification frequency levels of the data partitions.

[0040] The data file may include any type of file, such as audio, image, document, data table, etc. The present application does not limit the type of data file.

[0041] The data partitions are obtained by processing data files according to set partitioning rules, and may include horizontal partitions, vertical partitions, and mixed partitions.

[0042] The horizontal partition refers to a data combination obtained by splitting the data into units of at least one row. For example, if the data file is a data table, Table A has 1 million rows, and each 10,000 rows is a horizontal partition, 100 horizontal partitions can be obtained.

[0043] The vertical partition refers to a data combination obtained by splitting data into at least one column. For example, if data table A has 100 columns and each column is a vertical partition, 100 vertical partitions can be obtained.

[0044] Hybrid partitioning refers to a combination of horizontal and vertical partitioning. For example, data table A can be partitioned horizontally by row, and then vertically partitioned by column based on the horizontal partitioning.

[0045] It should be noted that the data partitioning strategy in this application is set according to business needs and does not limit the specific partitioning strategy. In some scenarios, the data files are originally stored by row partitioning. In this application, the partitioning is performed by column and the encoding strategy is set. In this case, the partitioning method of the data files can be changed from horizontal partitioning to vertical partitioning to complete the data storage.

[0046] It can be understood that, in the present application, storing data files in sections actually means storing the data partitions contained in the data files one by one, and this step illustrates the process of storing a data partition of a data file.

[0047] Different from the related art, in this application, the corresponding data modification frequency level can be marked for each data partition to indicate the modification frequency of the data partition and facilitate the subsequent adaptation of the corresponding erasure code encoding strategy.

[0048] The data modification frequency level includes at least two levels, which can be set according to business needs. For example, it can include high frequency and low frequency, or include high frequency, medium frequency and low frequency. For the convenience of explanation, this application uses two levels including high frequency and low frequency as an example.

[0049] In some ways, the levels of data files can be automatically or manually labeled during the partitioning process.

[0050] Automated labeling can pre-maintain attribute terms corresponding to different levels, and then label data partitions based on their attributes. For example, in a data table, product names and phone numbers can be assigned to low-frequency terms, while product prices and quantities can be assigned to high-frequency terms. By mapping levels to attributes, data partitions can be automatically labeled to modify their frequency levels.

[0051] It should be noted that, in some embodiments, the above modification frequency level marking is a preliminary marking, and in subsequent embodiments, an embodiment of dynamically updating the modification frequency level according to some properties of the data partition will be proposed.

[0052] In this application, different data modification frequency levels correspond to different erasure code encoding strategies. The higher the modification frequency level, the smaller the number of data blocks used for encoding in the corresponding encoding strategy indication.

[0053] The encoding strategy may include the number k of data blocks involved in encoding during an erasure coding process and the number m of parity blocks obtained by encoding. In these erasure coding strategies, the higher the modification frequency level, the smaller k. It should be noted that the number m of parity blocks in different encoding strategies can be the same or different and can be set as needed. In some embodiments, a higher modification frequency level not only reduces k but also reduces the ratio of k to m, thereby improving the efficiency of reconstructing the parity code.

[0054] For example, for the low frequency level, k is 8 and m is 2 in the coding strategy, and for the high frequency level, k is 4 and m is 2 in the coding strategy. For another example, for the low frequency level, k is 8 and m is 2 in the coding strategy, and for the high frequency level, k is 4 and m is 1 in the coding strategy.

[0055] The coding strategy can be manually configured or automatically generated. In the manual configuration mode, an interface can be provided for the user to configure the coding strategy corresponding to different modification frequency levels. The automatic generation scheme will be introduced in the subsequent embodiments.

[0056] In this step, after obtaining the data partition to be stored, a corresponding target encoding strategy can be obtained from a stack of encoding strategies according to the modification frequency level marked on the data partition.

[0057] S104 : Divide the multiple data blocks included in the data partition into at least one data stripe according to the first number included in the target encoding strategy, where the data stripe includes the first number of data blocks.

[0058] The data block refers to a data combination of a preset size. For example, 1M or 1K of data is 1 data block.

[0059] The data stripe can be understood as a combination of a certain number of data blocks.

[0060] In the present application, the number k of data blocks involved in an erasure coding process can be obtained from the target coding strategy. Then, based on k, the data blocks included in the data partition are grouped to obtain multiple data stripes. The data stripes include the first number of data blocks.

[0061] For example, the data partition is 100M, the preset data block size is 1M, and k in the target encoding strategy is 4, then the data partition can be divided into 25 data stripes containing 4 data blocks.

[0062] S106: For each of the data stripes, encode the first number of data blocks contained therein to obtain the second number of check blocks included in the target encoding strategy, and distribute and store the first number of data blocks and the second number of check blocks to different storage nodes.

[0063] In this step, each data stripe may be encoded using methods such as array erasure codes (RAID5, RAID6, etc.), RS (Reed-Solomon) erasure codes, and LDPC (Low Density Parity Check Code) erasure codes.

[0064] Taking RS (Reed-Solomon) erasure codes as an example, for k data blocks D1, D2...Dk, the k data blocks can be combined into a vector D and multiplied by the matrix B to obtain a data vector consisting of k data blocks and m check blocks. If a data block is lost, the lost data block can be restored through a series of calculations. RS(k, m) can tolerate the loss of up to m blocks (including data blocks and check blocks). RS encoding and decoding involves matrix inversion, using Gaussian elimination. The relevant technology will not be described in detail here.

[0065] Take an erasure code with redundancy levels k=8 and m=2 as an example. The eight source data blocks D1-D8 are arranged by column into a vector D, and then an (n+m)n matrix B is constructed, called the distribution matrix. Matrix B requires that any k row vectors are independent of each other, meaning that the kk matrix composed of these k row vectors is reversible. Performing matrix-vector multiplication B*D yields two check blocks C1-Cm. If no more than two data blocks or code blocks are lost, data recovery can be performed using the remaining data blocks or code blocks.

[0066] Therefore, when distributing storage for a data file, an adaptive erasure coding strategy can be adopted for different data partitions included in the data file according to the modification frequency, rather than a one-size-fits-all approach.

[0067] For example, low-frequency data partitions can be encoded with k = 8 and m = 2, while high-frequency data partitions can be encoded with k = 4 and m = 2. This allows high-frequency data to be reconstructed using only four data blocks when a data block is modified. A one-size-fits-all storage approach would require the same eight data blocks for reconstruction as for low-frequency data, reducing computing resource usage and network overhead when modifying data blocks. For low-frequency data, the encoding strategy of eight data blocks can be maintained, improving storage space utilization.

[0068] Therefore, compared with a one-size-fits-all distributed storage solution for a data file, the solution described in this application improves the flexibility of the distributed storage system and improves the performance of the distributed storage system.

[0069] In some embodiments, in order to simplify user work and configure a reasonable encoding strategy, the user only needs to configure the encoding strategy corresponding to the lowest level of modification frequency, and encoding strategies of other levels will be automatically generated.

[0070] See Figure 2 , Figure 2 This is a flow chart of a method for generating a coding strategy shown in this application. Figure 2 The illustrated method is an illustration of a scheme for automatically generating a coding strategy. Figure 2 As shown, the method may include S202 - S204 .

[0071] S202: Obtain a first coding strategy corresponding to the lowest level.

[0072] The first coding strategy includes the number k of data blocks involved in coding and the number m of check blocks generated by coding in the erasure coding technology.

[0073] In this step, an interface may be provided for the user to input k and m corresponding to the lowest modification frequency level (eg, low frequency), for example, 8 and 2.

[0074] S204 : reducing the number of data blocks involved in encoding in the first encoding strategy according to the distance between the other levels and the lowest level, to obtain a second encoding strategy corresponding to the other levels.

[0075] The distance is a quantifiable way to express the distance between other levels and the lowest level in order to facilitate the calculation of the number of data blocks. For example, if there are three levels of modification frequency, the low frequency can be represented as 1, the medium frequency as 2, and the high frequency as 3. Then the distance from the medium frequency to the low frequency is 1, and the distance from the high frequency to the low frequency is 2. If there are two levels of modification frequency, the low frequency can be represented as 1 and the high frequency as 2. Then the distance from the high frequency to the low frequency is 1.

[0076] In this step, the number of data blocks can be reduced in at least two ways, namely, proportional reduction and arithmetic difference reduction.

[0077] Mode 1: According to the distance, the number of data blocks involved in encoding in the first encoding strategy is reduced arithmetic progression.

[0078] The formula is: b = as * t; s is the distance between the other levels and the lowest level; t is the preset unit step length; a is the first number before reduction; and b is the first number after reduction.

[0079] If the modification frequency level is the lowest level, then s is 0, which is equivalent to not reducing the value of the first quantity.

[0080] If the frequency level is modified to be one level higher than the lowest level, then s is 1, which is equivalent to reducing the value of the first quantity.

[0081] Mode 2: According to the distance, the number of data blocks involved in encoding in the first encoding strategy is reduced in equal proportion.

[0082] The formula is: d=c / (p+1); p is the distance between the other levels and the lowest level, and p is greater than or equal to 0. c is the first number before reduction, and d is the first number after reduction.

[0083] If the modification frequency level is the lowest level, then l is 0, which is equivalent to not reducing the value of the first quantity.

[0084] If the frequency level is modified to be one level higher than the lowest level, s is 1, which is equivalent to reducing the value of the first quantity by half.

[0085] With the solution described in S202-S204, the user only needs to enter the encoding strategy corresponding to the lowest level, and other encoding strategies will be automatically generated, reducing the user's workload. Furthermore, the solution described in S202-S204 can keep m the same across different encoding strategies, without changing the protection level for the same data file. Compared to manually generated encoding strategies, the generated encoding strategies are more reasonable.

[0086] In some embodiments, in order to maintain the accuracy and rationality of the modification frequency of the data partition and thereby ensure the adaptation of the correct encoding strategy to complete the distributed storage, the modification frequency level of the data partition can be updated periodically or by setting a probe.

[0087] In some embodiments, the following four situations may be included.

[0088] Scenario 1: In response to the time between the generation time of the data in the data partition and the current time being within a preset time, the modification frequency level of the data partition is updated to a higher level.

[0089] Scenario 2: In response to the time length between the generation time of the data in the data partition and the current time exceeding a preset time length, the modification frequency level of the data partition is updated to a lower level.

[0090] Scenario 3: In response to the frequency of data being operated on in the data partition exceeding a preset frequency, the modification frequency level of the data partition is updated to a higher level.

[0091] Scenario 4: In response to the frequency of data being operated in the data partition being within the preset frequency, the modification frequency level of the data partition is updated to a lower level.

[0092] Situation 1 and Situation 2 are equivalent to periodically determining the time interval between the generation time of the data in the data partition and the current time. If it is within the preset time interval (for example, 1 week or 5 days), it means that the data is relatively new and is likely to be modified. The modification frequency level can be updated to high frequency. If it exceeds the preset time interval (for example, 1 week or 5 days), it means that the data is old and is less likely to be modified. The modification frequency level can be updated to low frequency.

[0093] Scenarios 3 and 4 can count the frequency of data operations within a data partition. These operations may include, but are not limited to, additions, deletions, and modifications. If the frequency of data operations within a data partition exceeds a preset number (e.g., 3 or 5), it indicates that the data partition is frequently modified, and the modification frequency level can be updated to high. If the frequency of data operations within a data partition is less than a preset number (e.g., 3 or 5), it indicates that the data partition is infrequently modified, and the modification frequency level can be updated to low.

[0094] In some embodiments, after the level of the data partition changes, in addition to affecting the encoding strategy (storage strategy) of subsequent incremental data in the data table, it also affects the storage strategy of the data that has been stored.

[0095] Specifically, in response to the modification frequency level of the data partition being adjusted, during the idle time of the distributed storage system, the data of the data partition that has been stored is acquired and distributedly stored according to the encoding strategy corresponding to the adjusted modification frequency level.

[0096] For example, a data partition changes from a high-frequency level to a low-frequency level. In this example, the idle rate of the distributed storage system is periodically counted. When the idle rate reaches a preset threshold (e.g., 70% or 80%), the data of the data partition can be obtained and encoded and distributedly stored according to the encoding strategy corresponding to the low-frequency level.

[0097] This not only allows the data storage strategy to be changed synchronously after the level change, but also does not affect the performance of the distributed storage system.

[0098] In related technologies, especially in data table storage scenarios, modifications to the length of one or more fields are often involved. In this case, if the field content changes, the storage space size will change, which will cause the data block size to change. Therefore, it may be necessary to divide the entire data table into data blocks and re-encode the storage, which will affect storage performance.

[0099] To address the above issues, in some embodiments, a fixed length is set for at least one required column field in the data table. A fixed storage space is allocated based on the fixed length to store the field content of the column field. This ensures that modifying the field content does not change the storage space occupied by the field, nor does it change the order of the data blocks. This allows for a correspondence to be established between the data blocks and the field content they contain. Consequently, subsequent modifications to a field require only updating the corresponding data block and recalculating the check block, eliminating the need to re-store the entire data table. This reduces the amount of data computation and improves storage performance.

[0100] For example, a fixed length of 2 bytes is set for the column field, that is, 2 bytes of storage space is allocated to column field A. Assuming that after the data partition is divided, field A belongs to data block B, a corresponding relationship between field A and data block B can be established. No matter how the content of field A is modified, field A always corresponds to data block B. When field A is subsequently modified, only data block B needs to be modified.

[0101] See Figure 3 , Figure 3 This is a flow chart of a method for updating a check block shown in this application. Figure 3 The steps shown in the figure are how to update the check block when the column field length is fixed. Figure 3 As shown, the method may include S302-S306.

[0102] S302 : In response to a modification operation on the target field content, obtaining a target data block corresponding to the target field content according to the corresponding relationship.

[0103] In this step, the correspondence between the field content and the data block is maintained when the data block is segmented. In this step, the target data block below which the target field to be modified is located can be obtained based on the correspondence.

[0104] S304: Update the target data block.

[0105] Since the field content has changed, the target data block stored in the storage node needs to be updated.

[0106] S306: Update the target check block corresponding to the target data block according to the updated target data block.

[0107] In this step, the data stripe described by the target data block can be obtained, and other data blocks included in the data stripe can be obtained, and then re-encoded based on the corresponding encoding strategy to obtain a redundant check block to complete subsequent distributed storage.

[0108] Therefore, through the solution recorded in S302-S306, when a field is subsequently modified, it is only necessary to update the data block corresponding to the field and recalculate the check block. There is no need to re-store the entire data table, which reduces the amount of data calculation and improves storage performance.

[0109] In some embodiments, the amount of system calculations after the field content is modified can be further reduced, improving storage performance. Figure 4 , Figure 4 This is a flow chart of a method for updating a check block shown in this application. Figure 4 The illustrated steps are for explaining some embodiments of S306. Figure 4 As shown, the method may include S402-S406.

[0110] S402: Obtain a target check block corresponding to the target data block.

[0111] When encoding data blocks, the correspondence between data blocks and check blocks can be maintained. In this step, the target check block corresponding to the target data block can be obtained based on this correspondence. After the field content is modified, the target check block is updated, which is equivalent to completing data redundancy storage.

[0112] S404: Obtain an intermediate value through a matrix inverse operation according to the target check block and the target data block before updating.

[0113] In this step, matrix E can be generated based on the target check block group, and matrix A can be generated based on the target data block before updating. The inverse matrix of matrix A can be obtained through matrix inverse operation, and then the inverse matrix of matrix A is multiplied by matrix E to obtain the intermediate value matrix F.

[0114] S406: Obtain an updated target check block through matrix operation according to the intermediate value and the updated target data block.

[0115] In this step, a matrix C can be generated based on the updated target data block, and the updated target check block can be obtained by multiplying the matrix C with the intermediate value matrix F. After that, the original target check block can be replaced.

[0116] Through the scheme recorded in S402-S406, the updated target check block can be obtained based on matrix inverse operation, only based on the target data block before the update, the target data block after the update, and the target check block before the update. Compared with the traditional scheme of obtaining other data blocks in the same data stripe and recalculating the check block, there is no need to obtain other data blocks, which reduces the amount of network data transmission and network occupancy, and helps to improve the performance of distributed storage systems. Especially in large-scale data storage scenarios, saving network occupancy is of great significance to improving system performance.

[0117] In some embodiments, setting a fixed length for a column field relies on manual experience, which may result in the fixed length being set too large, leading to a waste of storage space. To address this issue, a solution for automatically updating the fixed length can be adopted. Specifically, for at least one of the column fields, the longest field length and the frequency of occurrence of the longest field can be counted.

[0118] In this step, for at least one column field that needs to have a fixed length updated, a special identifier may be assigned to count the longest field length and the frequency of occurrence of the longest field within a period of time.

[0119] When the frequency of occurrence of the longest field reaches a preset number of times, the fixed length of the column field is updated to the longest field length.

[0120] The preset number is set based on experience, for example, 2 or 3. If the frequency of the longest field reaches the preset number, it means that the field is the longest, that is, the longest field, and thus the fixed length of the column field can be updated to the longest field length.

[0121] In the case that the longest field does not reach the set initial fixed length and the longest field does not change for a preset time length, the fixed length of the column field is updated to the longest field length.

[0122] The preset time length is not set according to demand, for example, it can be 10 days or 15 days. If the field reaches the maximum field and the maintenance time exceeds the preset time length, it means that the field may not change, and the fixed length corresponding to the field can be shortened.

[0123] After the fixed length of the column field is updated to the longest field length, in response to the column field receiving target field content greater than the longest field length, the length of the target field content is compared with the initial fixed length of the column field. If the length of the target field content is greater than the initial fixed length, the target field content is discarded. If the length of the target field content is less than or equal to the initial fixed length, the fixed length of the column field is updated to the length of the target field content.

[0124] For example, the initial fixed length of a column field in a data table is 10 bytes, and the field length after adjustment according to the previous rules is 6 bytes. If the column subsequently receives 8-byte field content, it means that the adjusted field length is unreasonable, and the field length needs to be increased and the field content needs to be retained. If the column receives 11-byte field content, there is no need to adjust the field length and discard the unreasonable content. In this way, the field length can be dynamically adjusted to achieve the effect of reducing space occupancy and not losing reasonable data. Because the initial fixed length is generally set by experienced personnel, it is unreasonable to output dirty data if the field content exceeds this length, and it is reasonable to output field content less than this length. In this embodiment, the fixed length corresponding to the column field can be dynamically updated to reasonably optimize the field length of the column field. When new data needs to be stored subsequently, storage space can be saved and system performance can be optimized.

[0125] Corresponding to any of the above embodiments, the present application also proposes a distributed storage device based on data modification frequency. Figure 5 , Figure 5 This is a structural diagram of a distributed storage device based on data modification frequency shown in an embodiment of the present application. Figure 5 As shown, the distributed storage device based on data modification frequency includes:

[0126] An acquisition module 510 acquires data partitions included in a data file to be stored, and target encoding strategies corresponding to modification frequency levels of the data partitions; wherein different data modification frequency levels correspond to different erasure coding strategies, and a higher modification frequency level corresponds to an encoding strategy indicating that a smaller number of data blocks is used for encoding;

[0127] A partitioning module 520 is configured to partition the plurality of data blocks included in the data partition into at least one data stripe according to the first number included in the target encoding strategy, wherein the data stripe includes the first number of data blocks;

[0128] The distributed storage module 530 encodes the first number of data blocks contained in each data stripe to obtain the second number of check blocks included in the target encoding strategy, and distributes the first number of data blocks and the second number of check blocks to different storage nodes.

[0129] In some embodiments, the modification frequency level includes at least two levels; the apparatus 500 further includes a module for generating a coding strategy;

[0130] The generated coding strategy module:

[0131] Obtaining a first coding strategy corresponding to the lowest level, the first coding strategy including the number of data blocks involved in encoding and the number of check blocks generated by encoding in an erasure coding technology;

[0132] According to the distance between other levels and the lowest level, the number of data blocks involved in encoding in the first encoding strategy is reduced to obtain a second encoding strategy corresponding to the other levels.

[0133] In some embodiments, the generating coding strategy module further:

[0134] According to the distance, the number of data blocks involved in encoding in the first encoding strategy is reduced in equal proportion; or,

[0135] The number of data blocks involved in encoding in the first encoding strategy is reduced arithmetic difference according to the distance.

[0136] In some embodiments, the apparatus 500 further includes a module for updating and modifying a frequency level:

[0137] The update modification frequency level module:

[0138] Updating the modification frequency level of the data partition;

[0139] wherein, in response to the time length between the generation time of the data in the data partition and the current time being within a preset time length, updating the modification frequency level of the data partition to a higher level;

[0140] In response to a time duration between the generation time of the data in the data partition and the current time exceeding a preset time duration, updating the modification frequency level of the data partition to a lower level;

[0141] In response to a frequency of data being operated on in the data partition exceeding a preset frequency, updating a modification frequency level of the data partition to a higher level;

[0142] In response to a frequency of operations on data in the data partition being within the preset frequency, updating the modification frequency level of the data partition to a lower level;

[0143] In response to the modification frequency level of the data partition being adjusted, during an idle moment of the distributed storage system, the data of the data partition that has been stored is acquired and distributedly stored according to the encoding strategy corresponding to the adjusted modification frequency level.

[0144] In some embodiments, the data file is a data table, and a fixed length is set for at least one column field of the data table, so that a fixed storage space is allocated according to the fixed length to store the field content of the column field, so that modifying the field content does not change the size of the storage space occupied by the field and does not change the order of the data blocks;

[0145] The apparatus 500 further includes a building module; the building module:

[0146] A correspondence between a data block and the field contents contained in the data block is established.

[0147] In some embodiments, the apparatus 500 includes a module for updating a check block;

[0148] The update check block module:

[0149] In response to a modification operation on the target field content, obtaining a target data block corresponding to the target field content according to the corresponding relationship;

[0150] Updating the target data block;

[0151] The target check block corresponding to the target data block is updated according to the updated target data block.

[0152] In some embodiments, the updating check block module further:

[0153] Obtaining a target check block corresponding to the target data block;

[0154] Obtaining an intermediate value by performing a matrix inverse operation based on the target check block and the target data block before updating;

[0155] An updated target check block is obtained through matrix operation according to the intermediate value and the updated target data block.

[0156] In some embodiments, the apparatus 500 further includes a module for updating field length;

[0157] The update field length module:

[0158] For at least one of the column fields, counting the longest field length and the frequency of occurrence of the longest field;

[0159] When the frequency of occurrence of the longest field reaches a preset number of times, updating the fixed length of the column field to the longest field length;

[0160] If the longest field does not reach the set initial fixed length and the longest field does not change within a preset time, updating the fixed length of the column field to the longest field length;

[0161] After the fixed length of the column field is updated to the longest field length, in response to the column field receiving target field content greater than the longest field length, the length of the target field content is compared with the initial fixed length of the column field. If the length of the target field content is greater than the initial fixed length, the target field content is discarded. If the length of the target field content is less than or equal to the initial fixed length, the fixed length of the column field is updated to the length of the target field content.

[0162] In the scheme recorded in the aforementioned device embodiment, different erasure code encoding strategies can be configured for different modification frequency levels, wherein the higher the modification frequency level, the smaller the number of data blocks used for encoding indicated by the corresponding encoding strategy, and the modification frequency level is marked in units of data partitions included in the data file, so that when a data file is distributedly stored, an adaptive erasure code encoding strategy can be adopted for the data partition according to the modification frequency, wherein, if the data modification frequency is high, the number of encoded data blocks adopted in the encoding strategy is small to reduce the computing resource occupation and network overhead when the data blocks are modified; if the data modification frequency is low, the number of encoded data blocks adopted in the encoding strategy is large to improve the storage space utilization.

[0163] Therefore, compared with a one-size-fits-all distributed storage solution for a data file, the solution described in this application improves the flexibility of the distributed storage system and improves the performance of the distributed storage system.

[0164] The embodiment of the distributed storage device based on data modification frequency shown in the present application can be applied to electronic devices. Accordingly, the present application discloses an electronic device, which may include: a processor.

[0165] Memory used to store processor-executable instructions.

[0166] The processor is configured to call the executable instructions stored in the memory to implement the distributed storage method based on data modification frequency shown in any of the aforementioned embodiments.

[0167] See Figure 6 , Figure 6 A schematic diagram of the hardware structure of an electronic device shown in an embodiment of the present application.

[0168] like Figure 6 As shown, the electronic device may include a processor for executing instructions, a network interface for network connection, a memory for storing operating data for the processor, and a non-volatile memory for storing instructions corresponding to a distributed storage device based on a data modification frequency.

[0169] The embodiments of the device can be implemented by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running it. From the hardware level, in addition to Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device is located in the embodiment may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0170] It is understandable that, in order to improve the processing speed, the corresponding instructions of the distributed storage device based on the data modification frequency can also be directly stored in the memory, which is not limited here.

[0171] The present application proposes a computer-readable storage medium storing a computer program, which can be used to enable a processor to execute the distributed storage method based on data modification frequency shown in any of the aforementioned embodiments.

[0172] It will be understood by those skilled in the art that one or more embodiments of the present application may be provided as a method, system, or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to magnetic disk storage, CDROM, optical storage, etc.) containing computer-usable program code.

[0173] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".

[0174] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the data processing device embodiment is generally similar to the method embodiment, so its description is relatively simple. For relevant portions, refer to the description of the method embodiment.

[0175] The foregoing description describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0176] The subject matter and functional loading embodiments described in this application can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that can include the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-transitory program carrier to be executed by a data processing device or to control the loading of a data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0177] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by loading data based on input and generating output. The processes and logic flows can also be performed by, and the apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0178] A computer suitable for executing a computer program may include, for example, a general-purpose and / or special-purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer may be loadably coupled to such a mass storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0179] Computer-readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media, and storage devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0180] Although the present application includes many specific implementation details, these should not be interpreted as limiting the scope of any disclosure or the scope of protection claimed, but are mainly used to describe the features of the specific disclosed embodiments. Certain features described in multiple embodiments of the present application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented in any suitable sub-combination. In addition, although features can work in some combinations as described and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0181] Similarly, although loads are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these loads be executed in the particular order shown or sequentially, or that all illustrated loads be executed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0182] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0183] The above are merely preferred embodiments of one or more embodiments of the present application and are not intended to limit one or more embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

Claims

1. A distributed storage method based on data modification frequency, characterized in that: The method comprises: Obtaining data partitions included in the data file to be stored and target encoding strategies corresponding to modification frequency levels of the data partitions; wherein different data modification frequency levels correspond to different erasure coding strategies, and a higher modification frequency level corresponds to an encoding strategy indicating that a smaller number of data blocks is used for encoding; Dividing the plurality of data blocks included in the data partition into at least one data stripe according to a first number included in the target encoding strategy, wherein the data stripe includes the first number of data blocks; For each of the data stripes, the first number of data blocks contained therein are encoded to obtain the second number of check blocks included in the target encoding strategy, and the first number of data blocks and the second number of check blocks are distributed and stored in different storage nodes.

2. The distributed storage method based on data modification frequency according to claim 1, characterized in that: The modification frequency level includes at least two levels; and the method for generating the coding strategy includes: Obtaining a first coding strategy corresponding to the lowest level, the first coding strategy including the number of data blocks involved in encoding and the number of check blocks generated by encoding in an erasure coding technology; According to the distance between other levels and the lowest level, the number of data blocks involved in encoding in the first encoding strategy is reduced to obtain a second encoding strategy corresponding to the other levels.

3. The distributed storage method based on data modification frequency according to claim 2, characterized in that: The reducing the number of data blocks involved in encoding in the first encoding strategy according to the distances between the other levels and the lowest level includes: According to the distance, the number of data blocks involved in encoding in the first encoding strategy is reduced in equal proportion; or, The number of data blocks involved in encoding in the first encoding strategy is reduced arithmetic difference according to the distance.

4. The distributed storage method based on data modification frequency according to claim 1, characterized in that: The method further comprises: Updating the modification frequency level of the data partition; wherein, in response to the time length between the generation time of the data in the data partition and the current time being within a preset time length, updating the modification frequency level of the data partition to a higher level; In response to a time duration between the generation time of the data in the data partition and the current time exceeding a preset time duration, updating the modification frequency level of the data partition to a lower level; In response to a frequency of data being operated on in the data partition exceeding a preset frequency, updating a modification frequency level of the data partition to a higher level; In response to a frequency of operations on data in the data partition being within the preset frequency, updating the modification frequency level of the data partition to a lower level; In response to the modification frequency level of the data partition being adjusted, during an idle moment of the distributed storage system, the data of the data partition that has been stored is acquired and distributedly stored according to the encoding strategy corresponding to the adjusted modification frequency level.

5. The distributed storage method based on data modification frequency according to claim 1, characterized in that: The data file is a data table, and a fixed length is set for at least one column field of the data table, so that a fixed storage space is allocated according to the fixed length to store the field content of the column field, so that modifying the field content does not change the size of the storage space occupied by the field and does not change the order of data blocks; The method further comprises: A correspondence between a data block and the field contents contained in the data block is established.

6. The distributed storage method based on data modification frequency according to claim 5, characterized in that: The method comprises: In response to a modification operation on the target field content, obtaining a target data block corresponding to the target field content according to the corresponding relationship; Updating the target data block; The target check block corresponding to the target data block is updated according to the updated target data block.

7. The distributed storage method based on data modification frequency according to claim 6, characterized in that: Updating the target check block corresponding to the target data block according to the updated target data block includes: Obtaining a target check block corresponding to the target data block; Obtaining an intermediate value by performing a matrix inverse operation based on the target check block and the target data block before updating; An updated target check block is obtained through matrix operation according to the intermediate value and the updated target data block.

8. The distributed storage method based on data modification frequency according to claim 5, characterized in that: The method further comprises: For at least one of the column fields, counting the longest field length and the frequency of occurrence of the longest field; When the frequency of occurrence of the longest field reaches a preset number of times, updating the fixed length of the column field to the longest field length; If the longest field does not reach the set initial fixed length and the longest field does not change within a preset time, updating the fixed length of the column field to the longest field length; After the fixed length of the column field is updated to the longest field length, in response to the column field receiving target field content greater than the longest field length, the length of the target field content is compared with the initial fixed length of the column field. If the length of the target field content is greater than the initial fixed length, the target field content is discarded. If the length of the target field content is less than or equal to the initial fixed length, the fixed length of the column field is updated to the length of the target field content.

9. A distributed storage device based on data modification frequency, characterized in that: The device comprises: an acquisition module for acquiring data partitions contained in a data file to be stored and target encoding strategies corresponding to modification frequency levels of the data partitions; wherein different data modification frequency levels correspond to different erasure coding strategies, and a higher modification frequency level corresponds to an encoding strategy indicating that a smaller number of data blocks is used for encoding; a partitioning module, configured to partition the plurality of data blocks included in the data partition into at least one data stripe according to a first number included in the target encoding strategy, wherein the data stripe includes the first number of data blocks; The distributed storage module encodes the first number of data blocks contained in each data stripe to obtain the second number of check blocks included in the target encoding strategy, and distributes the first number of data blocks and the second number of check blocks to different storage nodes.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor executes the executable instructions to implement the distributed storage method based on data modification frequency as described in any one of claims 1 to 8.

Images