Distributed storage method and device based on data modification frequency and electronic equipment

Abstract

The invention relates to the technical field of databases, in particular to a distributed storage method and device based on data modification frequency and electronic equipment. According to the method, different erasure code coding strategies can be configured for different modification frequency levels, the higher the modification frequency level is, the smaller the number of data blocks used for coding is indicated by the corresponding coding strategy, and the modification frequency level is marked by taking a data partition included in a data file as a unit. Therefore, when distributed storage is carried out on one data file, an adaptive erasure code encoding strategy can be adopted for the data partition according to the modification frequency, if the data modification frequency is high, the number of encoding data blocks adopted in the encoding strategy is small so as to reduce the occupation of computing resources and network overhead when the data blocks are modified, and the data storage efficiency is improved. And if the data modification frequency is low, the number of coding data blocks adopted in the coding strategy is large, so that the utilization rate of the storage space is improved, the flexibility of the distributed storage system is improved, and the performance of the distributed storage system is improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a distributed storage method, device and electronic device based on data modification frequency. Background Art

[0002] Erasure Codes (EC) is a data protection method mainly used in data storage and network transmission to ensure data reliability and fault tolerance. It encodes data by splitting it into fragments and creating redundant data blocks. These encoded data fragments are stored in different locations, such as different disks, storage nodes, or other geographical locations. The core technologies of erasure coding include polynomial interpolation or oversampling, which enable the system to recover the original data when some data is lost.

[0003] The erasure code system can be expressed as "n = k + m", where "k" represents the number of original data blocks, "m" represents the number of added check blocks (redundant data blocks), and "n" is the total number of encoded data blocks. For example, in the "EC 10 / 16" configuration, 6 additional check blocks (m) are added to 10 original data blocks (k), and these encoded 16 data fragments (n) are stored in different locations. Even if up to m data blocks are lost or damaged, the original data can still be recovered from the remaining data blocks.

[0004] In practical applications, erasure codes are widely used in distributed storage systems, object storage, cloud computing, and large-scale data transmission.

[0005] In erasure coding technology, if a file is partially modified, the entire file may need to be re-encoded, which may incur large computing and network overhead. Therefore, erasure codes are generally suitable for scenarios where data modifications are infrequent. This application is intended to expand the applicable scenarios of erasure codes and reduce related computing and network overhead. Erasure Codes (EC) is a data protection method that is mainly used in data storage and network transmission to ensure data reliability and fault tolerance. It encodes data by splitting it into fragments and creating redundant data blocks. These encoded data fragments are stored in different locations, such as different disks, storage nodes, or other geographical locations. The core technologies of erasure coding include polynomial interpolation or oversampling, which enable the system to recover the original data when some data is lost.

[0006] The erasure code system can be expressed as "n = k + m", where "k" represents the number of original data blocks, "m" represents the number of added check blocks (redundant data blocks), and "n" is the total number of encoded data blocks. For example, in the "EC 10 / 16" configuration, 6 additional check blocks (m) are added to 10 original data blocks (k), and these encoded 16 data fragments (n) are stored in different locations. Even if up to m data blocks are lost or damaged, the original data can still be recovered from the remaining data blocks.

[0007] In practical applications, erasure codes are widely used in distributed storage systems, object storage, cloud computing, and large-scale data transmission.

[0008] In erasure coding technology, if a file is partially modified, the entire file may need to be re-encoded, which may result in large computational and network overhead. Therefore, erasure coding is generally suitable for scenarios where data modification is infrequent. This application is intended to expand the applicable scenarios of erasure coding and reduce related computational and network overhead. Summary of the invention

[0009] In view of this, the present application discloses a distributed storage method based on data modification frequency. The method may include obtaining a data partition contained in a data file to be stored, and a target encoding strategy corresponding to the modification frequency level of the data partition; wherein different data modification frequency levels correspond to different erasure code encoding strategies, and the higher the modification frequency level, the smaller the number of data blocks used for encoding corresponding to the encoding strategy indication; according to the first number included in the target encoding strategy, the multiple data blocks included in the data partition are divided into at least one data stripe, and the data stripe includes the first number of data blocks; for each of the data stripes, the first number of data blocks contained therein are encoded to obtain the second number of check blocks included in the target encoding strategy, and the first number of data blocks and the second number of check blocks are distributed and stored in different storage nodes.

[0010] In some embodiments, the modification frequency level includes at least two levels; the method for generating a coding strategy includes: obtaining a first coding strategy corresponding to the lowest level, the first coding strategy including the number of data blocks involved in coding in the erasure code technology and the number of check blocks generated by the coding; according to the distance between other levels and the lowest level, reducing the number of data blocks involved in coding in the first coding strategy to obtain a second coding strategy corresponding to the other levels.

[0011] In some embodiments, reducing the number of data blocks involved in encoding in the first encoding strategy according to the distance between other levels and the lowest level includes: reducing the number of data blocks involved in encoding in the first encoding strategy in equal proportion according to the distance; or reducing the number of data blocks involved in encoding in the first encoding strategy in equal increments according to the distance.

[0012] In some embodiments, the method also includes: updating the modification frequency level of the data partition; wherein, in response to the time length between the generation time of the data in the data partition and the current time being within a preset time length, the modification frequency level of the data partition is updated to a higher level; in response to the time length between the generation time of the data in the data partition and the current time exceeding the preset time length, the modification frequency level of the data partition is updated to a lower level; in response to the frequency of the data in the data partition being operated exceeding the preset frequency, the modification frequency level of the data partition is updated to a higher level; in response to the frequency of the data in the data partition being operated within the preset frequency, the modification frequency level of the data partition is updated to a lower level; in response to the modification frequency level of the data partition being adjusted, at the idle time of the distributed storage system, the data of the data partition that has been stored is obtained and distributedly stored according to the encoding strategy corresponding to the adjusted modification frequency level.

[0013] In some embodiments, the data file is a data table, and a fixed length is set for at least one column field of the data table to allocate a fixed storage space according to the fixed length to store the field content of the column field, so that modifying the field content will not change the size of the storage space occupied by the field, and will not change the order of the data blocks; the method also includes: establishing a correspondence between the data block and the field content contained in the data block.

[0014] In some embodiments, the method includes: in response to a modification operation on the target field content, obtaining a target data block corresponding to the target field content according to the corresponding relationship; updating the target data block; and updating a target check block corresponding to the target data block according to the updated target data block.

[0015] In some embodiments, updating the target check block corresponding to the target data block according to the updated target data block includes: obtaining the target check block corresponding to the target data block; obtaining an intermediate value through a matrix inverse operation based on the target check block and the target data block before updating; and obtaining an updated target check block through a matrix operation based on the intermediate value and the updated target data block.

[0016] In some embodiments, the method further includes: for at least one of the column fields, counting the longest field length and the frequency of occurrence of the longest field; when the frequency of occurrence of the longest field reaches a preset number of times, updating the fixed length of the column field to the longest field length; when the longest field does not reach the set initial fixed length and the longest field does not change for a preset time length, updating the fixed length of the column field to the longest field length; after updating the fixed length of the column field to the longest field length, in response to the column field receiving a target field content greater than the longest field length, comparing the length of the target field content with the initial fixed length of the column field, if the length of the target field content is greater than the initial fixed length, discarding the target field content, if the length of the target field content is less than or equal to the initial fixed length, updating the fixed length of the column field to the length of the target field content.

[0017] The present application also proposes a distributed storage device based on data modification frequency, the device comprising: an acquisition module, acquiring a data partition contained in a data file to be stored, and a target encoding strategy corresponding to the modification frequency level of the data partition; wherein different data modification frequency levels correspond to different erasure code encoding strategies, and the higher the modification frequency level, the smaller the number of data blocks used for encoding indicated by the corresponding encoding strategy; a division module, dividing the multiple data blocks included in the data partition into at least one data stripe according to the first number included in the target encoding strategy, the data stripe including the first number of data blocks; a distributed storage module, encoding the first number of data blocks contained in each data stripe to obtain the second number of check blocks included in the target encoding strategy, and distributing and storing the first number of data blocks and the second number of check blocks to different storage nodes.

[0018] The present application also proposes an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor implements the distributed storage method based on data modification frequency as described in any of the aforementioned embodiments by running the executable instructions.

[0019] In the scheme recorded in any of the aforementioned embodiments, different erasure code encoding strategies can be configured for different modification frequency levels, wherein the higher the modification frequency level, the smaller the number of data blocks used for encoding indicated by the corresponding encoding strategy, and the modification frequency level is marked in units of data partitions included in the data file, so that when a data file is distributedly stored, an adaptive erasure code encoding strategy can be adopted for the data partition according to the modification frequency, wherein if the data modification frequency is high, the number of encoded data blocks adopted in the encoding strategy is small to reduce the computing resources occupied and network overhead occupied when the data blocks are modified; if the data modification frequency is low, the number of encoded data blocks adopted in the encoding strategy is large to improve the storage space utilization.

[0020] Therefore, compared with a one-size-fits-all distributed storage solution for a data file, the solution described in this application improves the flexibility of the distributed storage system and improves the performance of the distributed storage system.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the drawings required for use in the embodiments or related technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions.

[0024] Figure 1 A method flow chart of a distributed storage method based on data modification frequency shown in the present application.

[0025] Figure 2 A schematic flow chart of a method for generating a coding strategy shown in the present application.

[0026] Figure 3 This is a flow chart of a method for updating a check block shown in the present application.

[0027] Figure 4 This is a flow chart of a method for updating a check block shown in the present application.

[0028] Figure 5 A schematic diagram of the structure of a distributed storage device based on data modification frequency shown in an embodiment of the present application.

[0029] Figure 6 A schematic diagram of the hardware structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0031] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used in this article, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0032] In the related art, if a data file is distributedly stored based on erasure codes, the number of data blocks k used in the coding and the number of check blocks m generated by the coding are input into the system. Then the data file is divided into multiple data stripes, each of which includes k data blocks. Then m redundant check blocks are calculated from these k data blocks, and finally these k+m data blocks are concurrently stored in different storage nodes.

[0033] Different k and m values ​​can be set according to the value and storage cost of each file, and the storage space utilization rate is k / (k+m). Generally speaking, the larger k is and the smaller m is, the higher the storage space utilization rate is, but the protection level is lower, and the more data needs to be read for reconstruction; conversely, the smaller k is and the larger m is, the lower the storage space utilization rate is, but the protection level is higher, and the reconstruction is faster.

[0034] In this solution, data files are distributedly stored using a one-size-fits-all erasure code encoding method. If a large k and small m strategy is used, the storage space utilization rate is high. However, for some scenarios where data is frequently modified, after a data block is modified, it is necessary to obtain other k-1 data blocks and recalculate m encoding blocks. The larger k is, the more computing resources and network overhead it will occupy. If a small k and large m strategy is used to reduce computing resource usage and network overhead, the storage space utilization rate will be reduced. It can be seen that this one-size-fits-all erasure code encoding method limits the flexibility of distributed storage and affects the performance of distributed storage systems.

[0035] In view of this, the present application proposes a distributed storage method based on data modification frequency. The method can configure different erasure code encoding strategies for different modification frequency levels, wherein the higher the modification frequency level, the smaller the number of data blocks used for encoding corresponding to the encoding strategy indication, and the modification frequency level is marked in units of data partitions included in the data file, so that when a data file is distributedly stored, an adaptive erasure code encoding strategy can be adopted for the data partition according to the modification frequency, wherein, if the data modification frequency is high, the number of encoded data blocks adopted in the encoding strategy is small, so as to reduce the occupation of computing resources and network overhead when the data blocks are modified, and if the data modification frequency is low, the number of encoded data blocks adopted in the encoding strategy is large, so as to improve the storage space utilization.

[0036] Therefore, compared with a one-size-fits-all distributed storage solution for a data file, the solution described in this application improves the flexibility of the distributed storage system and improves the performance of the distributed storage system.

[0037] The following is a description of the embodiments with reference to the accompanying drawings.

[0038] See also Figure 1 , Figure 1 The following is a flow chart of a distributed storage method based on data modification frequency shown in this application. Figure 1 As shown, the method may include S102-S106.

[0039] S102, obtaining data partitions contained in the data file to be stored, and target encoding strategies corresponding to modification frequency levels of the data partitions.

[0040] The data file may include any type of file, such as audio, image, document, data table, etc. The present application does not limit the type of data file.

[0041] The data partition is obtained by processing the data file according to the set partitioning rules, and may include horizontal partitioning, vertical partitioning, and mixed partitioning.

[0042] The horizontal partition refers to a data combination obtained by splitting the data in units of at least one row. Taking the data file as a data table as an example, table A has 1 million rows, and each 10,000 rows is a horizontal partition, so 100 horizontal partitions can be obtained.

[0043] The vertical partition refers to a data combination obtained by splitting data in units of at least one column. For example, data table A has 100 columns, each column is a vertical partition, and 100 vertical partitions can be obtained.

[0044] The hybrid partitioning refers to a partitioning that combines horizontal and vertical partitioning. For example, data table A can be first partitioned horizontally by row, and then partitioned vertically by column based on the horizontal partitioning.

[0045] It should be noted that the data partitioning strategy in this application is set according to business needs, and the specific partitioning strategy is not limited. In some scenarios, the original storage method of the data file is stored by row partitions, and this application is partitioned by column and the encoding strategy is set. Then in the solution recorded in this application, the partitioning method of the data file can be changed from horizontal partitioning to vertical partitioning, and then the data storage is completed.

[0046] It can be understood that the divided storage of data files in the present application is actually to store the data partitions contained in the data files one by one, and this step illustrates the process of storing a data partition of the data file.

[0047] Different from the related art, in the present application, the corresponding data modification frequency level can be marked for each data partition to indicate the modification frequency of the data partition and facilitate the subsequent adaptation of the corresponding erasure code encoding strategy.

[0048] The data modification frequency level includes at least two levels, which can be set according to business requirements. For example, it can include high frequency and low frequency, or high frequency, medium frequency and low frequency. For the convenience of explanation, this application is illustrated by taking two levels including high frequency and low frequency as an example.

[0049] In some methods, the levels of data files can be automatically or manually labeled during the partitioning process.

[0050] In the automatic labeling method, some attribute words corresponding to different levels can be maintained in advance, and then the data partitions can be labeled by judging the attributes of the data partitions. Taking the data table as an example, product names, phone numbers, etc. can correspond to low frequencies, and product prices, quantities, etc. can correspond to high frequencies. Through the mapping relationship between levels and attributes, the frequency level of data partitions can be automatically labeled and modified.

[0051] It should be noted that, in some embodiments, the above modification frequency level marking is a preliminary marking, and in subsequent embodiments, embodiments for dynamically updating the modification frequency level according to some properties of the data partition will be proposed.

[0052] In the present application, different data modification frequency levels correspond to different erasure code encoding strategies, and the higher the modification frequency level, the smaller the number of data blocks used for encoding in the corresponding encoding strategy indication.

[0053] The coding strategy may include the number of data blocks k involved in the coding during an erasure code encoding process, and the number of check blocks m obtained by encoding. In these erasure code encoding strategies, the higher the modification frequency level, the smaller k. It should be noted that the number of check blocks m in different coding strategies can be the same or different, and can be set according to requirements. In some embodiments, the higher the modification frequency level, in addition to the smaller k, the ratio of k to m is also smaller, which improves the efficiency of reconstructing the check code.

[0054] For example, for the low frequency level, k is 8 and m is 2 in the coding strategy, and for the high frequency level, k is 4 and m is 2 in the coding strategy. For another example, for the low frequency level, k is 8 and m is 2 in the coding strategy, and for the high frequency level, k is 4 and m is 1 in the coding strategy.

[0055] The coding strategy can be manually configured or automatically generated. In the manual configuration mode, an interface can be provided for the user to configure the coding strategy corresponding to different modification frequency levels. The automatic generation scheme is introduced in the subsequent embodiments.

[0056] In this step, after obtaining the data partition to be stored, a corresponding target encoding strategy can be obtained from a bunch of encoding strategies according to the modification frequency level marked on the data partition.

[0057] S104: Divide the multiple data blocks included in the data partition into at least one data stripe according to the first number included in the target encoding strategy, where the data stripe includes the first number of data blocks.

[0058] The data block refers to a data combination of a preset size. For example, 1M or 1K of data is 1 data block.

[0059] The data stripe may be understood as a combination of a certain number of data blocks.

[0060] In the present application, the number k of data blocks involved in the encoding process of erasure coding can be obtained from the target coding strategy. Then, based on k, the data blocks included in the data partition are grouped to obtain multiple data stripes. The data stripes include the first number of data blocks.

[0061] For example, the data partition is 100M, the preset data block size is 1M, and k in the target encoding strategy is 4, then the data partition can be divided into 25 data stripes including 4 data blocks.

[0062] S106: For each of the data stripes, encode the first number of data blocks contained therein to obtain the second number of check blocks included in the target encoding strategy, and store the first number of data blocks and the second number of check blocks in a distributed manner to different storage nodes.

[0063] In this step, for each data stripe, encoding can be performed using array erasure codes (Array Code: RAID5, RAID6, etc.), RS (Reed-Solomon) erasure codes, and LDPC (Low Density Parity Check Code) erasure codes.

[0064] Taking RS (Reed-Solomon) erasure code as an example, for k data blocks (Data block) D1, D2...Dk, the k data blocks can be combined into a vector D and point-multiplied to generate a matrix B to obtain a data vector, which is composed of k data blocks and m check blocks. If a data block is lost, the lost data block can be restored through a series of calculations. RS (k, m) can tolerate the loss of up to m blocks (including data blocks and check blocks). RS encoding and decoding involves matrix inversion, using Gaussian elimination method, and the relevant technology will not be described in detail here.

[0065] Take the erasure code with redundancy level k=8 and m=2 as an example. Arrange the 8 source data blocks D1~D8 into vector D by columns, and then construct an (n+m)n matrix B, which is called the distribution matrix. There is a requirement for matrix B: any of its k row vectors are independent of each other, that is, the kk matrix composed of these k row vectors is reversible. Perform matrix-vector multiplication B*D to obtain 2 check blocks C1~Cm. Among them, if no more than 2 data blocks or coding blocks are lost, the remaining data blocks or coding blocks can be used to restore the data.

[0066] Therefore, when a data file is distributedly stored, an adaptive erasure coding strategy can be used for different data partitions included in the data file based on the modification frequency, rather than a one-size-fits-all approach.

[0067] For example, low-frequency data partitions can be encoded with k=8 and m=2, and high-frequency data partitions can be encoded with k=4 and m=2. In this way, when modifying data blocks for high-frequency data, only 4 data blocks are needed to complete the reconstruction. If a one-size-fits-all storage method is adopted, 8 data blocks are needed to complete the reconstruction like low-frequency data, which reduces the computing resource occupation and network overhead when modifying data blocks. For low-frequency data, the strategy of encoding 8 data blocks can be maintained to improve storage space utilization.

[0068] Therefore, compared with a one-size-fits-all distributed storage solution for a data file, the solution described in this application improves the flexibility of the distributed storage system and improves the performance of the distributed storage system.

[0069] In some embodiments, in order to simplify the user's work and configure a reasonable encoding strategy, the user only needs to configure the encoding strategy corresponding to the lowest level of modification frequency, and encoding strategies of other levels will be automatically generated.

[0070] See also Figure 2 , Figure 2 A schematic flow chart of a method for generating a coding strategy shown in the present application. Figure 2 The illustrated method is an illustration of a scheme for automatically generating a coding strategy. Figure 2 As shown, the method shown may include S202 - S204 .

[0071] S202, obtaining a first coding strategy corresponding to the lowest level.

[0072] The first coding strategy includes the number k of data blocks involved in coding and the number m of check blocks generated by coding in the erasure coding technology.

[0073] In this step, an interface may be provided for the user to input k and m corresponding to the lowest modification frequency level (eg, low frequency), for example, 8 and 2.

[0074] S204: reducing the number of data blocks involved in encoding in the first encoding strategy according to the distance between the other levels and the lowest level, to obtain a second encoding strategy corresponding to the other levels.

[0075] The distance is a quantifiable way to express the distance between other levels and the lowest level in order to facilitate the calculation of the number of data blocks. For example, if there are three levels of modification frequency, the low frequency can be represented as 1, the medium frequency can be represented as 2, and the high frequency can be represented as 3. Then the distance from the medium frequency to the low frequency is 1, and the distance from the high frequency to the low frequency is 2. If there are two levels of modification frequency, the low frequency can be represented as 1, and the high frequency can be represented as 2. Then the distance from the high frequency to the low frequency is 1.

[0076] In this step, at least two methods can be used to reduce the number of data blocks, namely, proportional reduction and arithmetic difference reduction.

[0077] Mode 1: according to the distance, the number of data blocks involved in encoding in the first encoding strategy is reduced arithmetic difference.

[0078] The formula is: b=as*t; s is the distance between other levels and the lowest level, t is the preset unit step length, a is the first number before reduction, and b is the first number after reduction.

[0079] If the modification frequency level is the lowest level, then s is 0, which is equivalent to not reducing the value of the first quantity.

[0080] If the frequency level is modified to be one level higher than the lowest level, then s is 1, which is equivalent to reducing the value of the first quantity.

[0081] Mode 2: reducing the number of data blocks involved in encoding in the first encoding strategy in equal proportion according to the distance.

[0082] The formula is: d=c / (p+1); p is the distance between other levels and the lowest level, and p is greater than or equal to 0. c is the first number before reduction, and d is the first number after reduction.

[0083] If the modification frequency level is the lowest level, then l is 0, which is equivalent to not reducing the value of the first quantity.

[0084] If the frequency level is modified to be one level higher than the lowest level, then s is 1, which is equivalent to reducing the value of the first quantity by one-half.

[0085] Through the scheme recorded in S202-S204, the user only needs to input the coding strategy corresponding to the lowest level, and other coding strategies will be automatically generated, reducing the workload of the user. In addition, the scheme recorded in S202-S204 can also keep m in different coding strategies the same, without changing the protection level for the same data file. Compared with the scheme of manually generating coding strategies, the generated coding strategies are more reasonable.

[0086] In some embodiments, in order to maintain the accuracy and rationality of the modification frequency of the data partitions and thereby ensure the adaptation of the correct encoding strategy to complete the distributed storage, the modification frequency level of the data partitions may be updated periodically or by setting probes.

[0087] In some embodiments, the following four situations may be included.

[0088] Scenario 1: In response to the time length between the generation time of the data in the data partition and the current time being within a preset time length, the modification frequency level of the data partition is updated to a higher level.

[0089] Scenario 2: In response to the time length between the generation time of the data in the data partition and the current time exceeding a preset time length, the modification frequency level of the data partition is updated to a lower level.

[0090] Scenario 3: In response to the frequency of data being operated on the data partition exceeding a preset frequency, the modification frequency level of the data partition is updated to a higher level.

[0091] Scenario 4: In response to the frequency of data being operated in the data partition being within the preset frequency, the modification frequency level of the data partition is updated to a lower level.

[0092] Situation 1 and Situation 2 are equivalent to periodically determining the length of time between the time when the data in the data partition is generated and the current time. If it is within the preset time (for example, 1 week or 5 days), it means that the data is relatively new and is likely to be modified, and the modification frequency level can be updated to high frequency. If it exceeds the preset time (for example, 1 week or 5 days), it means that the data is old and is less likely to be modified, and the modification frequency level can be updated to low frequency.

[0093] Situation 3 and Situation 4 can count the frequency of data operations in the data partition. The operations may include but are not limited to addition, deletion and modification. If the frequency of data operations in the data partition exceeds a preset number of times (for example, 3 or 5), it means that the data partition is often modified, and the modification frequency level can be updated to high frequency. If the frequency of data operations in the data partition is lower than the preset number of times (for example, 3 or 5), it means that the data partition is not often modified, and the modification frequency level can be updated to low frequency.

[0094] In some embodiments, after the level of the data partition changes, in addition to affecting the encoding strategy (storage strategy) of subsequent incremental data of the data table, it will also affect the storage strategy of the data that has been stored.

[0095] Specifically, in response to the modification frequency level of the data partition being adjusted, during the idle time of the distributed storage system, the data of the data partition that has been stored is obtained and distributedly stored according to the encoding strategy corresponding to the adjusted modification frequency level.

[0096] For example, a data partition changes from a high-frequency level to a low-frequency level. In this example, the idle rate of the distributed storage system is periodically counted. When the idle rate reaches a preset threshold (e.g., 70% or 80%), the data of the data partition can be obtained and encoded and distributedly stored according to the encoding strategy corresponding to the low-frequency level.

[0097] This not only allows the data storage strategy to be changed synchronously after the level change, but also does not affect the performance of the distributed storage system.

[0098] In the related technology, especially in the data table storage scenario, the modification of the length of one or some fields is often involved. In this case, if the field content changes, the storage space size will change, resulting in a change in the data block size. Therefore, it may be necessary to divide the entire data table into data blocks and re-encode the storage, which will affect the storage performance.

[0099] In order to solve the above problems, in some embodiments, a fixed length is set for at least one column field of the data table that is in demand, so that a fixed storage space is allocated according to the fixed length to store the field content of the column field, so that modifying the field content will not change the size of the storage space occupied by the field, and will not change the order of the data blocks, thereby establishing a corresponding relationship between the data block and the field content contained in the data block. Therefore, when a certain field is subsequently modified, it is only necessary to update the data block corresponding to the field and recalculate the check block, without having to re-store the entire data table, thereby reducing the amount of data calculation and improving storage performance.

[0100] For example, a fixed length of 2 bytes is set for the column field, that is, 2 bytes of storage space is allocated to column field A. Assuming that after the data partition is divided, field A belongs to data block B, then the correspondence between field A and data block B can be established. No matter how the content of field A is modified, field A always corresponds to data block B. When field A is subsequently modified, only data block B needs to be modified.

[0101] See also Figure 3 , Figure 3 This is a flow chart of a method for updating a check block shown in the present application. Figure 3 The illustrated steps are an illustration of how to update the check block when the column field length is fixed. Figure 3 As shown, the method may include S302-S306.

[0102] S302: In response to a modification operation on the target field content, a target data block corresponding to the target field content is acquired according to the corresponding relationship.

[0103] In this step, the correspondence between the field content and the data block is maintained when the data block is segmented. In this step, the target data block below which the target field to be modified is lower than can be obtained based on the correspondence.

[0104] S304: Update the target data block.

[0105] Since the field content has changed, the target data block stored in the storage node needs to be updated.

[0106] S306: Update the target check block corresponding to the target data block according to the updated target data block.

[0107] In this step, the data stripe described by the target data block can be obtained, and other data blocks included in the data stripe can be obtained, and then re-encoded based on the corresponding encoding strategy to obtain a redundant check block to complete subsequent distributed storage.

[0108] Therefore, through the solution recorded in S302-S306, when a field is subsequently modified, it is only necessary to update the data block corresponding to the field and recalculate the check block. There is no need to re-store the entire data table, which reduces the amount of data calculation and improves storage performance.

[0109] In some embodiments, the amount of system calculation after the field content is modified can be further reduced to improve storage performance. Figure 4 , Figure 4 This is a flow chart of a method for updating a check block shown in the present application. Figure 4 The illustrated steps are for explaining some embodiments of S306. Figure 4 As shown, the method may include S402-S406.

[0110] S402: Acquire a target check block corresponding to the target data block.

[0111] When encoding the data block, the corresponding relationship between the data block and the check block can be maintained. In this step, the target check block corresponding to the target data block can be obtained according to the corresponding relationship. After the field content is modified, the target check block is updated, which is equivalent to completing data redundant storage.

[0112] S404: Obtain an intermediate value through a matrix inverse operation according to the target check block and the target data block before updating.

[0113] In this step, matrix E can be generated based on the target check block group, and matrix A can be generated based on the target data block before updating. The inverse matrix of matrix A can be obtained through matrix inverse operation, and then the inverse matrix of matrix A is multiplied with matrix E to obtain the intermediate value matrix F.

[0114] S406: Obtain an updated target check block through matrix operation according to the intermediate value and the updated target data block.

[0115] In this step, a matrix C can be generated based on the updated target data block, and the updated target check block can be obtained by multiplying the matrix C with the intermediate value matrix F. After that, the original target check block can be replaced.

[0116] Through the scheme recorded in S402-S406, the updated target check block can be obtained based on the matrix inverse operation, only based on the target data block before the update, the target data block after the update, and the target check block before the update. Compared with the traditional scheme of obtaining other data blocks in the same data stripe and recalculating the check block, there is no need to obtain other data blocks, thus reducing the amount of network data transmission and network occupancy, which helps to improve the performance of distributed storage systems. Especially in scenarios where large amounts of data are stored, saving network occupancy is of great significance to improving system performance.

[0117] In some embodiments, setting a fixed length for a column field depends on manual experience, which may cause the fixed length to be set too large, resulting in a waste of storage space. In order to solve the above problem, a solution of automatically updating the fixed length may be adopted. Specifically, for at least one of the column fields, the longest field length and the frequency of occurrence of the longest field may be counted.

[0118] In this step, for at least one column field that needs to update the fixed length, a special identifier may be assigned to count the longest field length and the frequency of occurrence of the longest field within a period of time.

[0119] When the frequency of occurrence of the longest field reaches a preset number of times, the fixed length of the column field is updated to the longest field length.

[0120] The preset number is set based on experience, for example, 2 or 3. If the longest field appears at a preset number of times, it means that the longest field is the longest field, so the fixed length of the column field can be updated to the longest field length.

[0121] In the case that the longest field does not reach the set initial fixed length and the longest field does not change after reaching a preset time length, the fixed length of the column field is updated to the longest field length.

[0122] The preset time length is not set according to the demand, for example, it can be 10 days or 15 days. If the field reaches the longest field and the maintenance time exceeds the preset time length, it means that the field may not change, and the fixed length corresponding to the field can be shortened.

[0123] After the fixed length of the column field is updated to the longest field length, in response to the column field receiving a target field content greater than the longest field length, the length of the target field content is compared with the initial fixed length of the column field; if the length of the target field content is greater than the initial fixed length, the target field content is discarded; if the length of the target field content is less than or equal to the initial fixed length, the fixed length of the column field is updated to the length of the target field content.

[0124] For example, the initial fixed length of a column field in a data table is 10 bytes, and the field length after adjustment by the previous rules is 6 bytes. If the column receives 8 bytes of field content later, it means that the adjusted field length is unreasonable, and the field length needs to be increased and the field content needs to be retained. If the column receives 11 bytes of field content, there is no need to adjust the field length and discard the unreasonable content. In this way, the field length can be dynamically adjusted to achieve the effect of reducing space occupancy and not lose reasonable data. Because the initial fixed length is generally set by experienced personnel, it is unreasonable to output dirty data for field content exceeding this length, and it is reasonable to output field content less than this length. In this embodiment, the fixed length corresponding to the column field can be dynamically updated to reasonably optimize the field length of the column field. When new data needs to be stored later, storage space can be saved and system performance can be optimized.

[0125] Corresponding to any of the above embodiments, the present application also proposes a distributed storage device based on data modification frequency. Figure 5 , Figure 5 FIG. 1 is a schematic diagram of a distributed storage device based on data modification frequency according to an embodiment of the present application. Figure 5 As shown, the distributed storage device based on data modification frequency includes:

[0126] The acquisition module 510 acquires the data partitions included in the data file to be stored, and the target encoding strategy corresponding to the modification frequency level of the data partitions; wherein different data modification frequency levels correspond to different erasure coding strategies, and the higher the modification frequency level, the smaller the number of data blocks used for encoding corresponding to the encoding strategy indication;

[0127] A partitioning module 520, which partitions the plurality of data blocks included in the data partition into at least one data stripe according to the first number included in the target encoding strategy, wherein the data stripe includes the first number of data blocks;

[0128] The distributed storage module 530 encodes the first number of data blocks contained in each data stripe to obtain the second number of check blocks included in the target encoding strategy, and distributes the first number of data blocks and the second number of check blocks to different storage nodes.

[0129] In some embodiments, the modification frequency level includes at least two levels; the apparatus 500 further includes a module for generating a coding strategy;

[0130] The generated coding strategy module:

[0131] Obtaining a first coding strategy corresponding to the lowest level, wherein the first coding strategy includes the number of data blocks involved in coding and the number of check blocks generated by coding in the erasure coding technology;

[0132] According to the distance between other levels and the lowest level, the number of data blocks involved in encoding in the first encoding strategy is reduced to obtain a second encoding strategy corresponding to the other levels.

[0133] In some embodiments, the module for generating a coding strategy further comprises:

[0134] According to the distance, the number of data blocks involved in encoding in the first encoding strategy is reduced in equal proportion; or,

[0135] According to the distance, the number of data blocks involved in encoding in the first encoding strategy is reduced arithmetic difference.

[0136] In some embodiments, the device 500 further includes a module for updating and modifying a frequency level:

[0137] The update modifies the frequency level module:

[0138] updating a modification frequency level of the data partition;

[0139] wherein, in response to the time length between the generation time of the data in the data partition and the current time being within a preset time length, updating the modification frequency level of the data partition to a higher level;

[0140] In response to the time length between the generation time of the data in the data partition and the current time exceeding a preset time length, updating the modification frequency level of the data partition to a lower level;

[0141] In response to a frequency of data being operated in the data partition exceeding a preset frequency, updating a modification frequency level of the data partition to a higher level;

[0142] In response to the frequency of data being operated in the data partition being within the preset frequency, updating the modification frequency level of the data partition to a lower level;

[0143] In response to the modification frequency level of the data partition being adjusted, during an idle moment of the distributed storage system, the data of the data partition that has been stored is obtained and distributedly stored according to the encoding strategy corresponding to the adjusted modification frequency level.

[0144] In some embodiments, the data file is a data table, and a fixed length is set for at least one column field of the data table, so that a fixed storage space is allocated according to the fixed length to store the field content of the column field, so that modifying the field content does not change the size of the storage space occupied by the field, and does not change the order of the data blocks;

[0145] The apparatus 500 further includes a building module; the building module:

[0146] A correspondence between a data block and the field contents contained in the data block is established.

[0147] In some embodiments, the apparatus 500 includes a module for updating a check block;

[0148] The update check block module:

[0149] In response to a modification operation on the target field content, acquiring a target data block corresponding to the target field content according to the corresponding relationship;

[0150] Updating the target data block;

[0151] The target check block corresponding to the target data block is updated according to the updated target data block.

[0152] In some embodiments, the updating check block module updates the check block module, further:

[0153] Acquire a target check block corresponding to the target data block;

[0154] Obtaining an intermediate value through a matrix inverse operation according to the target check block and the target data block before updating;

[0155] An updated target check block is obtained through matrix operation according to the intermediate value and the updated target data block.

[0156] In some embodiments, the apparatus 500 further includes a module for updating field length;

[0157] The update field length module:

[0158] For at least one of the column fields, counting the longest field length and the frequency of occurrence of the longest field;

[0159] When the frequency of occurrence of the longest field reaches a preset number of times, updating the fixed length of the column field to the longest field length;

[0160] If the longest field does not reach the set initial fixed length and the longest field does not change after reaching the preset time length, updating the fixed length of the column field to the longest field length;

[0161] After the fixed length of the column field is updated to the longest field length, in response to the column field receiving a target field content greater than the longest field length, the length of the target field content is compared with the initial fixed length of the column field; if the length of the target field content is greater than the initial fixed length, the target field content is discarded; if the length of the target field content is less than or equal to the initial fixed length, the fixed length of the column field is updated to the length of the target field content.

[0162] In the scheme recorded in the aforementioned device embodiment, different erasure code encoding strategies can be configured for different modification frequency levels, wherein the higher the modification frequency level, the smaller the number of data blocks used for encoding indicated by the corresponding encoding strategy, and the modification frequency level is marked in units of data partitions included in the data file, so that when a data file is distributedly stored, an adaptive erasure code encoding strategy can be adopted for the data partition according to the modification frequency, wherein if the data modification frequency is high, the number of encoded data blocks adopted in the encoding strategy is small to reduce the computing resources occupied and network overhead occupied when the data blocks are modified; if the data modification frequency is low, the number of encoded data blocks adopted in the encoding strategy is large to improve the storage space utilization.

[0163] Therefore, compared with a one-size-fits-all distributed storage solution for a data file, the solution described in this application improves the flexibility of the distributed storage system and improves the performance of the distributed storage system.

[0164] The embodiment of the distributed storage device based on data modification frequency shown in the present application can be applied to electronic devices. Accordingly, the present application discloses an electronic device, which may include: a processor.

[0165] Memory used to store processor-executable instructions.

[0166] The processor is configured to call the executable instructions stored in the memory to implement the distributed storage method based on data modification frequency shown in any of the aforementioned embodiments.

[0167] See also Figure 6 , Figure 6 A schematic diagram of the hardware structure of an electronic device shown in an embodiment of the present application.

[0168] like Figure 6 As shown, the electronic device may include a processor for executing instructions, a network interface for network connection, a memory for storing operation data for the processor, and a non-volatile memory for storing instructions corresponding to a distributed storage device based on a data modification frequency.

[0169] The embodiments of the device can be implemented by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, in addition to Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device in the embodiment is located may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0170] It is understandable that, in order to improve the processing speed, the corresponding instructions of the distributed storage device based on the data modification frequency can also be directly stored in the memory, which is not limited here.

[0171] The present application proposes a computer-readable storage medium, which stores a computer program. The computer program can be used to enable a processor to execute the distributed storage method based on data modification frequency shown in any of the aforementioned embodiments.

[0172] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CDROM, optical storage, etc.) containing computer-usable program code.

[0173] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".

[0174] Each embodiment in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0175] The above describes a specific embodiment of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0176] The subject matter described in this application and the embodiments of the functional loading can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that can include the structure disclosed in this application and its structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, that is, one or more modules in the computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or control the loading of a data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0177] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by loading according to input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special-purpose logic circuits.

[0178] A computer suitable for executing a computer program may include, for example, a general and / or special purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or the computer will be loadably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0179] Computer readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0180] Although the application includes many specific implementation details, these should not be interpreted as limiting any disclosed scope or the scope of protection claimed, but are mainly used to describe the features of specific disclosed specific embodiments. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work in some combinations as described and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0181] Similarly, although loads are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these loads be executed in the particular order shown or executed sequentially, or requiring that all illustrated loads be executed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0182] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0183] The above are only preferred embodiments of one or more embodiments of the present application and are not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of one or more embodiments of the present application.

Claims

1. A distributed storage method based on data modification frequency, characterized in that: The method comprises: Obtaining data partitions contained in the data file to be stored, and target encoding strategies corresponding to modification frequency levels of the data partitions; wherein different data modification frequency levels correspond to different erasure coding strategies, and the higher the modification frequency level, the smaller the number of data blocks used for encoding corresponding to the encoding strategy indication; According to a first number included in the target encoding strategy, the plurality of data blocks included in the data partition are divided into at least one data stripe, wherein the data stripe includes the first number of data blocks; For each of the data stripes, the first number of data blocks contained therein are encoded to obtain the second number of check blocks included in the target encoding strategy, and the first number of data blocks and the second number of check blocks are distributed and stored in different storage nodes.

2. The distributed storage method based on data modification frequency according to claim 1, characterized in that: The modification frequency level includes at least two levels; and the method for generating the coding strategy includes: Obtaining a first coding strategy corresponding to the lowest level, wherein the first coding strategy includes the number of data blocks involved in coding and the number of check blocks generated by coding in the erasure coding technology; According to the distance between other levels and the lowest level, the number of data blocks involved in encoding in the first encoding strategy is reduced to obtain a second encoding strategy corresponding to the other levels.

3. The distributed storage method based on data modification frequency according to claim 2, characterized in that: The reducing the number of data blocks involved in encoding in the first encoding strategy according to the distances between other levels and the lowest level comprises: According to the distance, the number of data blocks involved in encoding in the first encoding strategy is reduced in equal proportion; or, According to the distance, the number of data blocks involved in encoding in the first encoding strategy is reduced arithmetic difference.

4. The distributed storage method based on data modification frequency according to claim 1, characterized in that: The method further comprises: updating a modification frequency level of the data partition; wherein, in response to the time length between the generation time of the data in the data partition and the current time being within a preset time length, updating the modification frequency level of the data partition to a higher level; In response to the time length between the generation time of the data in the data partition and the current time exceeding a preset time length, updating the modification frequency level of the data partition to a lower level; In response to a frequency of data being operated in the data partition exceeding a preset frequency, updating a modification frequency level of the data partition to a higher level; In response to the frequency of data being operated in the data partition being within the preset frequency, updating the modification frequency level of the data partition to a lower level; In response to the modification frequency level of the data partition being adjusted, during an idle moment of the distributed storage system, the data of the data partition that has been stored is obtained and distributedly stored according to the encoding strategy corresponding to the adjusted modification frequency level.

5. The distributed storage method based on data modification frequency according to claim 1, characterized in that: The data file is a data table, and a fixed length is set for at least one column field of the data table, so that a fixed storage space is allocated according to the fixed length to store the field content of the column field, so that modifying the field content will not change the size of the storage space occupied by the field, and will not change the order of the data blocks; The method further comprises: A correspondence between a data block and the field contents contained in the data block is established.

6. The distributed storage method based on data modification frequency according to claim 5, characterized in that: The method comprises: In response to a modification operation on the target field content, acquiring a target data block corresponding to the target field content according to the corresponding relationship; Updating the target data block; The target check block corresponding to the target data block is updated according to the updated target data block.

7. The distributed storage method based on data modification frequency according to claim 6, characterized in that: The updating of the target check block corresponding to the target data block according to the updated target data block comprises: Acquire a target check block corresponding to the target data block; Obtaining an intermediate value through a matrix inverse operation according to the target check block and the target data block before updating; An updated target check block is obtained through matrix operation according to the intermediate value and the updated target data block.

8. The distributed storage method based on data modification frequency according to claim 5, characterized in that: The method further comprises: For at least one of the column fields, counting the longest field length and the frequency of occurrence of the longest field; When the frequency of occurrence of the longest field reaches a preset number of times, updating the fixed length of the column field to the longest field length; If the longest field does not reach the set initial fixed length and the longest field does not change after reaching the preset time length, updating the fixed length of the column field to the longest field length; After the fixed length of the column field is updated to the longest field length, in response to the column field receiving a target field content greater than the longest field length, the length of the target field content is compared with the initial fixed length of the column field; if the length of the target field content is greater than the initial fixed length, the target field content is discarded; if the length of the target field content is less than or equal to the initial fixed length, the fixed length of the column field is updated to the length of the target field content.

9. A distributed storage device based on data modification frequency, characterized in that: The device comprises: An acquisition module is used to acquire data partitions contained in the data file to be stored, and a target encoding strategy corresponding to the modification frequency level of the data partition; wherein different data modification frequency levels correspond to different erasure code encoding strategies, and the higher the modification frequency level, the smaller the number of data blocks used for encoding corresponding to the encoding strategy indication; a partitioning module, which partitions the plurality of data blocks included in the data partition into at least one data stripe according to the first number included in the target encoding strategy, wherein the data stripe includes the first number of data blocks; The distributed storage module encodes the first number of data blocks contained in each data stripe to obtain the second number of check blocks included in the target encoding strategy, and distributes the first number of data blocks and the second number of check blocks to different storage nodes.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the distributed storage method based on data modification frequency as described in any one of claims 1-8 by running the executable instructions.

Images