A method and device for distributed storage of stream data

Abstract

The present application relates to the field of database technology, and specifically to a method and device for distributed storage of streaming data. On the one hand, the method can adjust the size of the data block according to the write rate of the streaming data, and the adjusted data block matches the write rate, so that the data block of the size matching the write rate can be pre-allocated, avoiding the problem of excessive storage resource occupation due to the pre-allocated data block being too large, and this solution does not require a large amount of cached streaming data, reducing the occupation of cache resources and the risk of data loss; on the other hand, the adjusted data blocks can be gradually merged until the size of the merged data block is restored to the default size, thereby alleviating the problem of excessive storage resource occupation, and will not increase the number of data blocks, the number of file directories will not increase, and there is no need to affect client services.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a method and device for distributed storage of stream data. Background Art

[0002] Erasure Codes (EC) is a data protection method mainly used in data storage and network transmission to ensure data reliability and fault tolerance. It encodes data by splitting it into fragments and creating redundant data blocks. These encoded data fragments are stored in different locations, such as different disks, storage nodes, or other geographical locations. The core technologies of erasure coding include polynomial interpolation or oversampling, which enable the system to recover the original data when some data is lost.

[0003] The erasure code system can be expressed as "n = k + m", where "k" represents the number of original data blocks, "m" represents the number of added check blocks (redundant data blocks), and "n" is the total number of encoded data blocks. For example, in the "EC 10 / 16" configuration, 6 additional check blocks (m) are added to 10 original data blocks (k), and these encoded 16 data fragments (n) are stored in different locations. Even if up to m data blocks are lost or damaged, the original data can still be recovered from the remaining data blocks.

[0004] For streaming data, there may be a scenario where the amount of data written in real time may be small and not enough to be stored through the erasure code calculation process. In this scenario, the following solutions may be used in related technologies, but these solutions will have some technical problems:

[0005] First, cache the received streaming data, and when the cache size is large enough, store it through erasure coding calculation. This method requires more cache resources. In addition, if the cache is in memory, data may be lost when power failure or other abnormal conditions occur.

[0006] Second, k data blocks of default size are allocated in advance and filled with default data, and then the default data is gradually replaced during the process of writing streaming data. Since k data blocks are allocated in advance, the conditions for storage through erasure code calculation are met. However, for the previous scenario, due to the small amount of data, allocating data blocks of default size in advance will cause excessive storage resources to be occupied, and when updating the default data, it is necessary to perform erasure code update operations many times, which occupies too many computing resources.

[0007] Third, based on the second solution, we can think of reducing the default size of the data block. This operation can alleviate the problem of excessive storage resource usage to a certain extent, but it will also lead to a significant increase in the number of data blocks and an excessive number of file directories. Summary of the invention

[0008] In view of this, the present application discloses a method for distributed storage of stream data. The method may include: adjusting the size of the data block according to the write rate of the stream data and the preset number of data blocks participating in the erasure code calculation; the adjusted size of the data block is greater than or equal to the write rate and the difference with the write rate is not greater than a preset value; performing distributed storage of the stream data according to the data block; wherein the distributed storage of the stream data according to the data block includes repeatedly performing the following steps: allocating the number of the adjusted data blocks to write to the stream data and storing them through the erasure code calculation; in response to the number of the data blocks being full, when the size of the data block is less than the default size In this case, a merge operation is performed to obtain a merged data block, and a remaining number of merged data blocks are applied for to continue writing into the stream data and stored through erasure code calculation, where the remaining number is the number minus 1; after the applied merged data block is full, if the size of the merged data block is still smaller than the default size, the merge is continued and the merged data block is applied for to write into the stream data and stored through erasure code calculation, and so on, until the size of the merged data block is equal to the default size, and stored through erasure code calculation; or, in the case where the size of the data block is equal to the default size, stored through erasure code calculation.

[0009] In some embodiments, adjusting the size of the data block according to the write rate of the streaming data and the preset number of data blocks participating in the erasure code calculation includes: periodically obtaining the write rate of the streaming data and the current segmentation strategy; the current segmentation strategy is used to determine the size and number of data blocks participating in the erasure code calculation, including the number and a first current size; adjusting the first current size according to the number to obtain a second current size; the second current size is greater than or equal to the write rate and the difference between the second current size and the write rate is not greater than a preset value.

[0010] In some embodiments, adjusting the first current size according to the quantity to obtain the second current size includes: when the first current size is greater than or equal to the write rate and the difference between the first current size and the write rate is not greater than a preset value, determining the first current size as the second current size; when the first current size is greater than or equal to the write rate and the difference between the first current size and the write rate is greater than a preset value, reducing the first current size at least once according to the quantity until the reduced data block size is greater than or equal to the write rate and the difference between the first current size and the write rate is not greater than a preset value, and determining the reduced data block size as the second current size.

[0011] In some embodiments, adjusting the first current size according to the quantity to obtain the second current size also includes: when the first current size is less than the write rate, increasing the first current size at least once according to the quantity until the increased data block size is greater than or equal to the write rate and the difference with the write rate is not greater than a preset value, and determining the increased data block size as the second current size.

[0012] In some embodiments, the storage through erasure code calculation includes: in response to any target data block being full, performing erasure code calculation on the target data blocks of the number to obtain a preset number of erasure code blocks; the target data blocks include the data blocks and any merged data blocks; and storing the target data blocks of the number and the preset number of erasure code blocks.

[0013] In some embodiments, a method for merging data blocks includes: when the storage locations of the number of data blocks are continuous, updating metadata node information so that the metadata node information indicates that the data storage location of the merged data block includes the minimum storage location and the maximum storage location in the data block; and deleting the erasure code of the data block before the merge.

[0014] In some embodiments, the method for merging data blocks further includes: when the storage locations of the number of data blocks are discontinuous, applying for continuous storage locations according to the size of the merged data blocks, and moving the contents of the number of data blocks to the applied continuous storage locations to achieve data block merging; or, keeping the storage locations of the data blocks before merging unchanged, updating the metadata node information of the merged data blocks so that the metadata node information indicates that the data storage location of the merged data blocks is obtained by mapping the storage locations of each of the data blocks; and deleting the erasure code of the data blocks before merging.

[0015] In some embodiments, the method for acquiring the write rate of the stream data includes: calculating the amount of data within a statistical period; and obtaining the write rate according to the amount of data and the duration of the period.

[0016] In some embodiments, the method for obtaining the write rate of the stream data includes: obtaining the historical write rate of the stream data within a preset time length; inputting the historical write rate into a write rate prediction model to obtain the write rate; the write rate prediction model is an LSTM model trained with multiple training samples; the training samples include the write rate sequence of the preset time length, and the marked write rate true value.

[0017] The present application also proposes a distributed storage device for streaming data. The device includes: an adjustment module, which is used to adjust the size of the data block according to the writing rate of the streaming data and the preset number of data blocks participating in the erasure code calculation; the size of the adjusted data block is greater than or equal to the writing rate and the difference with the writing rate is not greater than a preset value; a storage module, which is used to perform distributed storage of the streaming data according to the data blocks; wherein the storage module includes: an allocation module, which is used to allocate the number of the resized data blocks to write into the streaming data and store them through erasure code calculation; a storage submodule, which is used to allocate the number of the data blocks to be written full when the size of the data block is less than the preset value; In the case of a default size, a merge operation is performed to obtain a merged data block, and a remaining number of merged data blocks are applied for to continue writing into the stream data and stored through erasure code calculation, where the remaining number is the number minus 1; after the applied merged data block is full, if the size of the merged data block is still smaller than the default size, the merge is continued and the merged data block is applied for to write into the stream data and stored through erasure code calculation, and so on, until the size of the merged data block is equal to the default size, and the data block is stored through erasure code calculation; or, in the case that the size of the data block is equal to the default size, the data block is stored through erasure code calculation.

[0018] In the scheme recorded in any of the aforementioned embodiments, on the one hand, the size of the data block can be adjusted according to the write rate of the streaming data, and the adjusted data block matches the write rate, so that the data block of the size matching the write rate can be pre-allocated, avoiding the problem of excessive storage resource occupation due to the pre-allocated data block being too large, and this scheme does not require a large amount of cached streaming data, reduces the occupation of cache resources, and reduces the risk of data loss; on the other hand, the adjusted data blocks can be gradually merged until the size of the merged data block is restored to the default size, thereby alleviating the problem of excessive storage resource occupation, and will not increase the number of data blocks or the number of file directories, and there is no need to affect client business.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the drawings required for use in the embodiments or related technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions.

[0022] Figure 1 A method flow chart of a method for distributed storage of stream data shown in the present application.

[0023] Figure 2 This is a flow chart of a method for distributed storage of the stream data according to the data blocks shown in the present application.

[0024] Figure 3 This is a flow chart of a method for periodically adjusting the size of a data block according to the present application.

[0025] Figure 4 A flowchart of a method for merging data blocks is shown in the present application.

[0026] Figure 5 This is a flow chart of a method for acquiring a write rate shown in the present application.

[0027] Figure 6 A schematic diagram of the structure of a distributed storage device for stream data shown in an embodiment of the present application.

[0028] Figure 7 A schematic diagram of the hardware structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0030] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used in this article, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0031] The present application proposes a method for distributed storage of streaming data. On the one hand, the method can adjust the size of the data block according to the writing rate of the streaming data, and the adjusted data block matches the writing rate, so that the data block of the size matching the writing rate can be pre-allocated, avoiding the problem of excessive storage resource occupation due to the pre-allocated data block being too large, and this solution does not require a large amount of cached streaming data, which reduces the occupation of cache resources and reduces the risk of data loss; on the other hand, the adjusted data blocks can be gradually merged until the size of the merged data block is restored to the default size, thereby alleviating the problem of excessive storage resource occupation, and will not increase the number of data blocks, the number of file directories will not increase, and there is no need to affect client services.

[0032] The following is a description of the embodiments with reference to the accompanying drawings.

[0033] See also Figure 1 , Figure 1 The following is a flow chart of a method for distributed storage of stream data shown in this application. Figure 1 As shown, the method may include S102 and S104.

[0034] S102: Adjust the size of the data block according to the writing rate of the stream data and the preset number of data blocks participating in erasure code calculation.

[0035] In the process of writing stream data, you can count the writing rate of the stream data, that is, the data flow per unit time. The unit can be set according to the needs, for example, it can be MB / h, KB / min, KB / s, etc. By counting the data flow within a period of time, you can get the writing rate.

[0036] The data block refers to the data block involved in the erasure code calculation. The number of data blocks involved in the erasure code calculation (ie, the k value) and the size of the data block can be configured in the partitioning strategy.

[0037] For example, if k in the adopted segmentation strategy is 4, if the size of the data to be segmented is 40MB, that is, the size of the data block is 10MB. It should be noted that a default segmentation strategy may be set in the distributed storage system, and k in the default segmentation strategy is the k value in this application, and the data block size is the default size of this application.

[0038] Regardless of the adjustment strategy, the data block size adjusted by S102 matches the write rate. The matching condition set in the present application is that the size of the data block after adjustment is greater than or equal to the write rate and the difference with the write rate is not greater than a preset value. The preset value can be set according to demand.

[0039] When comparing the data block with the write rate, the unit of the write rate can be converted into a first preset unit. The first preset unit can be set according to the needs, such as KB / min. The size of the data block can also be converted into a second preset unit, such as KB. Then the write rate can be multiplied by the preset time and compared with the size of the data block. The preset time can be 1min, 2min, 5min, or 1h, etc. It can be understood that the unit conversion of the write rate and the data block size is to allow the two to be compared numerically. If the units of the two are of the same magnitude, there is no need to perform unit conversion. For example, the preset time is 1min, the write rate is 100KB / min, the preset value is 100KB, and the adjusted size can be 160KB.

[0040] The adjustment strategy can be set according to demand, such as a strategy of increasing or decreasing arithmetic differences or a strategy of increasing or decreasing geometrically. In some embodiments, in order to facilitate calculation and subsequent data block merging operations, a strategy of geometric resizing is adopted. The number of segmentations can be used as a ratio to increase or decrease the data block size geometrically. The size thus adjusted always has an integer multiple relationship of the number of segmentations with the default size, and subsequently when merging data blocks, the data blocks can be merged to the default size. For example, the default size is 10MB, k is 4, the write rate is 100KB / min, and the preset value is 100KB. Using 4 as a ratio, the adjusted size of 160KB can be obtained after 4 adjustments.

[0041] S104: Distributed storage of the stream data is performed according to the data blocks.

[0042] After the data block size is adjusted, the stream data can be stored. Figure 2 , Figure 2 This is a flow chart of a method for distributed storage of the stream data according to the data blocks shown in the present application. Figure 2 The illustrated solution is an explanation of S104. In S104, the following steps can be repeated: Figure 2 The steps shown in the figure complete the storage of stream data. Figure 2 As shown, the method shown may include S202 and S204.

[0043] S202, allocating the number of resized data blocks to write into the stream data and storing them through erasure code calculation.

[0044] This step can be understood as an operation of pre-allocating data blocks.

[0045] For example, if k is 4 and the adjusted data block size is 160KB, four 160KB data blocks filled with default data are requested in advance and stored through erasure code calculation. Later, in the process of writing stream data, the default data is gradually replaced to realize the storage of real data.

[0046] S204, in response to the number of data blocks being full, if the size of the data block is smaller than the default size, performing a merge operation to obtain a merged data block, and applying for a remaining number of merged data blocks to continue writing the stream data and storing them through erasure code calculation, where the remaining number is the number minus 1; after the applied merged data blocks are full, if the size of the merged data blocks is still smaller than the default size, continuing to merge and applying for the merged data blocks to write into the stream data and storing them through erasure code calculation, and so on, until the size of the merged data blocks is equal to the default size, and storing them through erasure code calculation;

[0047] Alternatively, when the size of the data block is equal to the default size, storage is performed through erasure code calculation.

[0048] The data block is full means that the default values ​​in the allocated data block are all replaced by real data. If 4 data blocks are allocated, it means that the default values ​​in the allocated 4 data blocks are all replaced by real data. If 3 data blocks are allocated during the data block merging process, it means that the default values ​​in the allocated 3 data blocks are all replaced by real data.

[0049] The merging operation refers to the process of merging k small blocks into a large block. For example, k is 4, the small data block is 160KB, and the merged data block is 640KB.

[0050] The remaining number refers to the number minus 1. For example, k is 4. If one of the data blocks has been obtained by merging, then it is necessary to continue to apply for 4-1 to write data for the three data blocks.

[0051] After the pre-allocated 4 data blocks are filled, it can be determined whether the data block size is consistent with the default size. If consistent, the 4 data blocks are stored after erasure coding calculation. Then continue to execute S202 and S204.

[0052] If they are inconsistent, a merge operation needs to be performed until the merged data block size is consistent with the default size, and then three data blocks of the merged size are allocated, and after they are filled, the erasure code calculation is completed and stored. Then, S202 and S204 are continued.

[0053] For example, if the data block size adjusted by S102 is exactly 10 MB, then the erasure code calculation is completed after the four data blocks are fully written, and four more 10 MB data blocks are applied to write data and repeat.

[0054] If the data block size after adjustment by S102 is 160KB, the number k is 4. If the 4 160KB data blocks are full, because 160KB is less than 10MB, the 4 160KB data blocks need to be merged into 1 640KB data block, and then 4-1=3 640KB data blocks are applied to continue writing streaming data. If the newly applied 3 640KB data blocks are full, because 640KB is still less than 10MB, the 4 640KB data blocks need to be merged to obtain 2.5MB data blocks, and then 3 2.5MB data blocks are applied to write streaming data, and so on, until the 4 2.5MB data blocks are full and continue to be merged to obtain 10MB data blocks, and then 3 10MB data blocks are applied. When the 4 10MB data blocks are full, a round of storage will end. S202 and S204 will be repeated later, that is, 4 160KB data blocks are applied to write streaming data.

[0055] Through the streaming data storage method such as S102-S104 and S202-S204, on the one hand, in the case of low traffic, the size of the data block can be dynamically adjusted according to the write rate to match the rate. Compared with the scheme using the default size, it avoids the problem of excessive storage resource usage due to the pre-allocated data block being too large, and this scheme does not require a large amount of cached streaming data, which reduces the occupation of cache resources and the risk of data loss. On the other hand, no matter how the adjusted data block size is, the data block can be restored to the default size data block through the data block merging strategy, which not only alleviates the problem of excessive storage resource usage, but also does not increase the number of data blocks and the number of file directories, and does not need to affect client services.

[0056] In some embodiments, the size of the data block can be dynamically adjusted periodically based on the write rate to increase the adaptability of the distributed storage solution. Figure 3 , Figure 3This is a flow chart of a method for periodically adjusting the size of a data block according to the present application. Figure 3 The illustrated scheme is an illustration of S102. Figure 3 As shown, the method may include S302-S304.

[0057] S302, periodically obtaining the writing rate of the stream data and the current segmentation strategy; the current segmentation strategy is used to determine the size and number of data blocks involved in erasure code calculation, including the number and a first current size.

[0058] The periodicity can be set as required, for example, by hour, by day, or by week. In some ways, the write rate can be obtained based on the amount of data within a period of time. Other solutions for obtaining the write rate are also disclosed in other embodiments of the present application.

[0059] The newly received data cannot obtain the streaming data write rate, so the default value can be used. When the streaming data is first stored, the current segmentation strategy is the default segmentation strategy.

[0060] S304, adjusting the first current size according to the quantity to obtain a second current size; the second current size is greater than or equal to the write rate and the difference between the second current size and the write rate is not greater than a preset value.

[0061] In some implementations, different adjustment methods may be used for the three situations.

[0062] Scenario 1: when the first current size is greater than or equal to the write rate and the difference between the first current size and the write rate is not greater than a preset value, the first current size is determined as the second current size;

[0063] Scenario two: when the first current size is greater than or equal to the write rate and the difference between the first current size and the write rate is greater than a preset value, the first current size is reduced at least once according to the quantity until the reduced data block size is greater than or equal to the write rate and the difference between the first current size and the write rate is not greater than a preset value, and the reduced data block size is determined as the second current size.

[0064] Scenario three: When the first current size is smaller than the write rate, the first current size is increased at least once according to the quantity until the increased data block size is greater than or equal to the write rate and the difference between the increased data block size and the write rate is not greater than a preset value, and the increased data block size is determined as the second current size.

[0065] For example, scenario 1: the first current size ≥ the writing rate and the difference ≤ the preset value. Assumption: the first current size = 160 KB, the writing rate = 100 KB / min, the preset value = 100, and the number k = 4.

[0066] The current size 160 KB is greater than or equal to the write rate 100 KB / min, and the difference (160-100=60) is less than the preset value 100. Therefore, there is no need to adjust the first current size, and the first current size is the second current size.

[0067] Case 2: First current size ≥ writing rate and difference > preset value. Assumption: First current size = 2.5 MB, writing rate = 100 KB / min, preset value = 100, quantity k = 4.

[0068] The current size 2.5MB is greater than or equal to the write rate 100, and the difference (2500-100=2400) is greater than the preset value 100. This meets the conditions of the second case. The second current size can be 160KB by performing three reduction operations in a geometric reduction manner with 4 as the ratio.

[0069] Scenario 3: First current size < write rate. Assume: first current size = 60KB, write rate = 100KB / min, preset value = 100, quantity k = 4. The initial first current size of 60KB is less than the write rate of 100KB / min. You can use geometric enlargement to take 4 as the ratio and perform one enlargement operation to get the second current size of 160KB.

[0070] In this way, the data block size can be adjusted periodically according to the current write rate, so that the data block size better matches the write rate, making the storage of stream data more reasonable.

[0071] In the related technology of using pre-allocated data blocks for storage, the erasure code is updated every time new data is written to the data block. This operation requires too many calculations and occupies too many computing resources. In order to solve this problem, in some embodiments, the erasure code can be updated in response to any data block being full. Specifically, in response to any target data block being full, the erasure code calculation can be performed on the target data blocks of the number to obtain a preset number of erasure code blocks; the target data blocks include the data blocks and any merged data blocks; the target data blocks of the number and the preset number of erasure code blocks are stored.

[0072] In this application, whether it is a data block allocated according to the adjusted size or a data block newly allocated during the merging process, the erasure code can be performed in the manner described in this embodiment. In order to avoid duplication of content, the data block and any merged data block are collectively referred to as the target data block.

[0073] Assume that the erasure code is 6=4+2. 4 is the data block and 2 is the erasure code block. In the related art, after new stream data is written into any data block, which may be only 10KB of data, the erasure code block will also be updated. Such an operation will take up too many computing resources. In this application, the erasure code block will only be updated after any data block is full, and the number of updates is significantly reduced. Moreover, since the data block size matches the write rate, the size will not be very large and will not take up too many cache resources.

[0074] In some embodiments, data blocks need to be merged in S204. This example proposes corresponding merging strategies for two merging situations to improve the adaptability of the solution.

[0075] See also Figure 4 , Figure 4 This is a flow chart of a method for merging data blocks shown in this application. Figure 4 As shown, the method may include S402-S408.

[0076] S402, when the storage locations of the data blocks of the number are continuous, update the metadata node information so that the metadata node information indicates that the data storage location of the merged data block includes the minimum storage location and the maximum storage location in the data block.

[0077] For example, suppose there are 4 consecutive data blocks, and their storage locations are as follows: location of data block 1: 1000–1999; location of data block 2: 2000–2999; location of data block 3: 3000–3999; location of data block 4: 4000–4999. In this case, the metadata node information can be updated to indicate that the data storage location of the merged data block includes the minimum storage location and the maximum storage location in the data block. The minimum storage location is the starting location of data block 1: 1000, and the maximum storage location is the ending location of data block 4: 4999. The storage location of the merged data block will be from 1000 to 4999.

[0078] S404, deleting the erasure code of the data block before merging.

[0079] The erasure code of the original data block before the merge is no longer useful and needs to be deleted.

[0080] S406, in the case where the storage locations of the said number of data blocks are not continuous, applying for continuous storage locations according to the size of the merged data blocks, and moving the contents of the said number of data blocks to the applied continuous storage locations, so as to realize data block merging; or, updating the metadata node information so that the metadata node information indicates that the data storage location of the merged data block is obtained by mapping the storage locations of each of the said data blocks.

[0081] For example, suppose there are 4 data blocks, and their storage locations are not continuous: data block 1: location 1000–1999; data block 2: location 3000–3999; data block 3: location 5000–5999; data block 4: location 7000–7999. In this case, the size of the merged data block can be calculated linearly: the size of each data block is 1000 (for example, each data block occupies 1000 units of storage space), so the total size of the 4 data blocks is: size of the merged data block = 4*1000 = 4000. Then, in order to merge these 4 data blocks, you can apply for a continuous storage location to accommodate the merged data. Assume that the system allocates a new continuous storage location from 10000 to 13999, and the size of this continuous area is 4000 (which matches the size of the merged data block we calculated). Then, the contents of the 4 discontinuous data blocks can be moved to this new continuous storage location. The migration process is as follows: move the content of data block 1 from location 1000-1999 to new location 10000–10999; move the content of data block 2 from location 3000-3999 to new location 11000–11999; move the content of data block 3 from location 5000-5999 to new location 12000–12999; move the content of data block 4 from location 7000-7999 to new location 13000–13999. The updated metadata node information indicates the new storage location of the merged data block.

[0082] Or in another solution, suppose there are the following four original data blocks and their storage locations: data block 1: location 2000–2999; data block 2: location 5000–5999; data block 3: location 8000–8999; data block 4: location 11000–11999. During the merging process, these data blocks can be merged into one logical data block, but the storage locations of the original data blocks remain unchanged.

[0083] The steps to update the metadata node information are as follows: Create a new entry, such as "merged_block", to represent the merged data block. List the storage locations of the original data blocks in the metadata node information corresponding to the new entry: location 2000–2999; location 5000–5999; location 8000–8999; location 11000–11999. In this way, we keep the storage location of the original data block unchanged, and at the same time, by updating the metadata node information, the new metadata can correctly indicate the storage location of the merged data block. In this way, the system can find the correct storage location based on the metadata when accessing the merged data block.

[0084] S408, deleting the erasure code of the data block before merging.

[0085] The erasure code of the original data block before the merge is no longer useful and needs to be deleted.

[0086] The scheme described in S402 - S408 can complete the merging of data blocks in two situations.

[0087] In some embodiments, a method for acquiring a write rate of stream data includes: calculating a data volume within a statistical period; and obtaining the write rate according to the data volume and a duration of the period.

[0088] For example, if the cycle is 1 day, the total traffic volume of the previous day can be calculated as 144,000KB, and then divided by 1440 minutes to get a write rate of 100KB / min. It can be understood that this process is equivalent to the write rate of the previous day to the write rate of the current day in the future, which is a prediction. This prediction method is relatively simple, but the prediction is not very accurate and will affect the performance of distributed storage of streaming data to a certain extent.

[0089] In order to improve the performance of distributed storage of streaming data, the accuracy of prediction of write rate can be improved. In some embodiments, the prediction of write rate can be achieved by means of long short-term memory network LSTM. The model can be used to interpret the characteristics of time series data and more accurately predict the future write rate based on historical data.

[0090] See also Figure 5 , Figure 5 FIG. 1 is a flow chart of a method for obtaining a write rate shown in the present application. Figure 5 As shown, the method may include S502-S504.

[0091] S502: Obtain a historical writing rate of the stream data within a preset time period.

[0092] The preset time length can be set according to the demand, for example, it can be 1 week. The historical writing rate can be obtained by counting the amount of data per unit time length, which is the real historical writing rate in the past week.

[0093] S504, input the historical write rate into a write rate prediction model to obtain the write rate; the write rate prediction model is an LSTM model trained with multiple training samples; the training samples include the write rate sequence of the preset time length and the marked true value of the write rate.

[0094] The write rate prediction model is an LSTM model. The Long Short-Term Memory (LSTM) network is a time recurrent neural network that can extract information useful for predicting future data from historical data, making the prediction effect better.

[0095] When training the network, you need to build training samples first. When building training samples, you can first obtain the historical write rate for a period of time, with each group of 8 days, where the write quantity of the first 7 days (a week) is the input data, and the write data of the 8th day is the true value of the write rate of the group. In this way, multiple groups of training samples can be obtained, which can include positive and negative samples, and can be divided into training samples and test samples. For example, 365 days of data can be obtained for streaming data, where each 8 days can form a group of data, for example, January 1 to January 8 is a group, January 2 to January 9 is a group, January 3 to January 10 is a group, and so on. In each group, the first 7 days are input data, and the 8th day is the true value data.

[0096] After obtaining the training samples, the write rate prediction model can be trained in a supervised manner so that the model has the ability to predict the write rate.

[0097] In S504, the data of the past 7 days obtained in S502 can be input into the model to obtain the write rate of the next day, thereby completing the acquisition of the write rate. It can be understood that this example shows that the periodicity is days for illustration. In actual situations, the preset duration can be adjusted according to the duration of the periodicity, which is not explained here.

[0098] Through the solution recorded in S502-S504, the LSTM can be used to interpret the characteristics of time series data and more accurately predict the future write rate based on historical data, thereby obtaining more accurate data blocks and improving the distributed storage performance of streaming data.

[0099] Corresponding to any of the above embodiments, the present application also proposes a distributed storage device for stream data. Figure 6 , Figure 6 This is a schematic diagram of the structure of a distributed storage device for stream data shown in an embodiment of the present application. Figure 6 As shown, the stream data distributed storage device 600 includes:

[0100] An adjustment module 610 is used to adjust the size of the data block according to the writing rate of the stream data and the preset number of data blocks participating in erasure code calculation; the size of the data block after adjustment is greater than or equal to the writing rate and the difference with the writing rate is not greater than a preset value;

[0101] A storage module 620, configured to perform distributed storage of the stream data according to the data blocks;

[0102] The storage module 620 includes:

[0103] An allocating module 621, configured to allocate the number of the resized data blocks to be written into the stream data and to store them through erasure code calculation;

[0104] The storage submodule 622 is configured to respond to the number of data blocks being full,

[0105] In the case where the size of the data block is smaller than the default size, a merge operation is performed to obtain a merged data block, and a remaining number of merged data blocks are applied for to continue writing into the stream data and stored through erasure code calculation, where the remaining number is the number minus 1; after the applied merged data block is fully written, if the size of the merged data block is still smaller than the default size, the merge operation is continued and the merged data block is applied for to write into the stream data and stored through erasure code calculation, and so on, until the size of the merged data block is equal to the default size, and the data block is stored through erasure code calculation;

[0106] Alternatively, when the size of the data block is equal to the default size, storage is performed through erasure code calculation.

[0107] In some embodiments, the adjustment module 610 further:

[0108] Periodically obtaining a write rate and a current segmentation strategy for the stream data; the current segmentation strategy is used to determine the size and number of data blocks involved in erasure code calculation, including the number and a first current size;

[0109] The first current size is adjusted according to the quantity to obtain a second current size; the second current size is greater than or equal to the write rate and the difference between the second current size and the write rate is not greater than a preset value.

[0110] In some embodiments, the adjustment module 610 further:

[0111] In a case where the first current size is greater than or equal to the write rate and the difference between the first current size and the write rate is not greater than a preset value, determining the first current size as the second current size;

[0112] In a case where the first current size is greater than or equal to the write rate and the difference between the first current size and the write rate is greater than a preset value, the first current size is reduced at least once according to the quantity until the reduced data block size is greater than or equal to the write rate and the difference between the first current size and the write rate is not greater than a preset value, and the reduced data block size is determined as the second current size.

[0113] In some embodiments, the adjustment module 610 further:

[0114] In the case where the first current size is smaller than the write rate, the first current size is increased at least once according to the quantity until the increased data block size is greater than or equal to the write rate and the difference with the write rate is not greater than a preset value, and the increased data block size is determined as the second current size.

[0115] In some embodiments, the storage submodule 622 further:

[0116] In response to any target data block being full, performing erasure code calculation on the target data blocks of the number to obtain a preset number of erasure code blocks; the target data blocks include the data blocks and any merged data blocks;

[0117] The target data blocks of the number and the preset number of erasure code blocks are stored.

[0118] In some embodiments, the storage submodule 622 further:

[0119] In a case where the storage locations of the data blocks of the number are continuous, updating the metadata node information so that the metadata node information indicates that the data storage location of the merged data block includes the minimum storage location and the maximum storage location in the data block;

[0120] Delete the erasure code of the data block before the merge.

[0121] In some embodiments, the storage submodule 622 further:

[0122] In the case where the storage locations of the data blocks of the number are not continuous, applying for continuous storage locations according to the size of the merged data blocks, and moving the contents of the data blocks of the number to the applied continuous storage locations, so as to realize data block merging; or, keeping the storage locations of the data blocks before merging unchanged, updating the metadata node information of the merged data blocks so that the metadata node information indicates that the data storage location of the merged data blocks is obtained by mapping the storage locations of each of the data blocks;

[0123] Delete the erasure code of the data block before the merge.

[0124] In some embodiments, the adjustment module 610 further:

[0125] The amount of data within the statistical period;

[0126] The write rate is obtained according to the data amount and the duration of the cycle.

[0127] In some embodiments, the adjustment module 610 further:

[0128] Obtaining a historical writing rate of the stream data within a preset time period;

[0129] The historical write rate is input into a write rate prediction model to obtain the write rate; the write rate prediction model is an LSTM model trained with multiple training samples; the training samples include a write rate sequence of the preset time length and a marked true value of the write rate.

[0130] In the scheme described in the above device embodiment, on the one hand, in a low-traffic scenario, the size of the data block can be dynamically adjusted according to the write rate to match the rate. Compared with the scheme using the default size, the problem of excessive storage resource occupation due to the pre-allocated data block being too large is avoided, and this scheme does not require a large amount of cached streaming data, which reduces the occupation of cache resources and reduces the risk of data loss. On the other hand, regardless of the adjusted data block size, the data block can be restored to the default size data block through the data block merging strategy, which not only alleviates the problem of excessive storage resource occupation, but also does not increase the number of data blocks, the number of file directories, and does not need to affect client services.

[0131] The embodiment of the stream data distributed storage device shown in the present application can be applied to electronic devices. Accordingly, the present application discloses an electronic device, which may include: a processor.

[0132] Memory used to store processor-executable instructions.

[0133] The processor is configured to call the executable instructions stored in the memory to implement the distributed storage method for stream data shown in any of the aforementioned embodiments.

[0134] See also Figure 7 , Figure 7 A schematic diagram of the hardware structure of an electronic device shown in an embodiment of the present application.

[0135] like Figure 7As shown, the electronic device may include a processor for executing instructions, a network interface for network connection, a memory for storing operation data for the processor, and a non-volatile memory for storing instructions corresponding to the stream data distributed storage device.

[0136] The embodiments of the device can be implemented by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, in addition to Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device in the embodiment is located may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0137] It is understandable that, in order to improve the processing speed, the corresponding instructions of the stream data distributed storage device can also be directly stored in the memory, which is not limited here.

[0138] The present application proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can be used to enable a processor to execute the distributed storage method for stream data shown in any of the aforementioned embodiments.

[0139] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CDROM, optical storage, etc.) containing computer-usable program code.

[0140] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".

[0141] Each embodiment in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0142] The above describes a specific embodiment of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0143] The subject matter described in this application and the embodiments of the functional loading can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that can include the structure disclosed in this application and its structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, that is, one or more modules in the computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or control the loading of a data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0144] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by loading according to input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special-purpose logic circuits.

[0145] A computer suitable for executing a computer program may include, for example, a general and / or special purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or the computer will be loadably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0146] Computer readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0147] Although the application includes many specific implementation details, these should not be interpreted as limiting any disclosed scope or the scope of protection claimed, but are mainly used to describe the features of specific disclosed specific embodiments. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work in some combinations as described and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0148] Similarly, although loads are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these loads be executed in the particular order shown or executed sequentially, or requiring that all illustrated loads be executed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0149] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0150] The above are only preferred embodiments of one or more embodiments of the present application and are not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of one or more embodiments of the present application.

Claims

1. A method for distributed storage of stream data, characterized in that: The method comprises: According to the writing rate of the stream data and the preset number of data blocks participating in erasure code calculation, the size of the data block is adjusted; the adjusted size of the data block is greater than or equal to the writing rate and the difference between the size of the data block and the writing rate is not greater than a preset value; Performing distributed storage of the stream data according to the data blocks; The distributed storage of the stream data according to the data blocks includes repeatedly performing the following steps: Allocating the number of the resized data blocks to write into the stream data and storing through erasure coding calculation; In response to said number of said data blocks being filled, In the case where the size of the data block is smaller than the default size, a merge operation is performed to obtain a merged data block, and a remaining number of merged data blocks are applied for to continue writing into the stream data and stored through erasure code calculation, where the remaining number is the number minus 1; after the applied merged data block is fully written, if the size of the merged data block is still smaller than the default size, the merge operation is continued and the merged data block is applied for to write into the stream data and stored through erasure code calculation, and so on, until the size of the merged data block is equal to the default size, and the data block is stored through erasure code calculation; When the size of the data block is equal to the default size, storage is performed through erasure code calculation.

2. The method for distributed storage of stream data according to claim 1, characterized in that: The adjusting the size of the data block according to the writing rate of the stream data and the preset number of data blocks participating in erasure code calculation includes: Periodically obtaining a write rate and a current segmentation strategy for the stream data; the current segmentation strategy is used to determine the size and number of data blocks involved in erasure code calculation, including the number and a first current size; The first current size is adjusted according to the quantity to obtain a second current size; the second current size is greater than or equal to the write rate and the difference between the second current size and the write rate is not greater than a preset value.

3. The method for distributed storage of stream data according to claim 2, characterized in that: The adjusting the first current size according to the quantity to obtain a second current size includes: In a case where the first current size is greater than or equal to the write rate and the difference between the first current size and the write rate is not greater than a preset value, determining the first current size as the second current size; In a case where the first current size is greater than or equal to the write rate and the difference between the first current size and the write rate is greater than a preset value, the first current size is reduced at least once according to the quantity until the reduced data block size is greater than or equal to the write rate and the difference between the first current size and the write rate is not greater than a preset value, and the reduced data block size is determined as the second current size.

4. The method for distributed storage of stream data according to claim 3, characterized in that: The method further includes: adjusting the first current size according to the quantity to obtain a second current size; In the case where the first current size is smaller than the write rate, the first current size is increased at least once according to the quantity until the increased data block size is greater than or equal to the write rate and the difference with the write rate is not greater than a preset value, and the increased data block size is determined as the second current size.

5. The method for distributed storage of stream data according to claim 1, characterized in that: The storing by erasure code calculation includes: In response to any target data block being full, performing erasure code calculation on the target data blocks of the number to obtain a preset number of erasure code blocks; the target data blocks include the data blocks and any merged data blocks; The target data blocks of the number and the preset number of erasure code blocks are stored.

6. The method for distributed storage of stream data according to claim 1, characterized in that: The method for merging data blocks includes: In a case where the storage locations of the data blocks of the number are continuous, updating the metadata node information so that the metadata node information indicates that the data storage location of the merged data block includes the minimum storage location and the maximum storage location in the data block; Delete the erasure code of the data block before the merge.

7. The method for distributed storage of stream data according to claim 6, characterized in that: The method for merging data blocks further includes: In the case where the storage locations of the data blocks of the number are not continuous, applying for continuous storage locations according to the size of the merged data blocks, and moving the contents of the data blocks of the number to the applied continuous storage locations, so as to realize data block merging; or, keeping the storage locations of the data blocks before merging unchanged, updating the metadata node information of the merged data blocks so that the metadata node information indicates that the data storage location of the merged data blocks is obtained by mapping the storage locations of each of the data blocks; Delete the erasure code of the data block before the merge.

8. The method for distributed storage of stream data according to claim 1, characterized in that: The method for acquiring the writing rate of the stream data includes: The amount of data within the statistical period; The write rate is obtained according to the data amount and the duration of the cycle.

9. The method for distributed storage of stream data according to claim 8, characterized in that: The method for acquiring the writing rate of the stream data includes: Obtaining a historical writing rate of the stream data within a preset time period; The historical write rate is input into a write rate prediction model to obtain the write rate; the write rate prediction model is an LSTM model trained with multiple training samples; the training samples include a write rate sequence of the preset time length and a marked true value of the write rate.

10. A distributed storage device for stream data, characterized in that: The device comprises: An adjustment module, configured to adjust the size of the data block according to a writing rate of the stream data and a preset number of data blocks participating in erasure code calculation; the size of the data block after adjustment is greater than or equal to the writing rate and the difference between the size of the data block and the writing rate is not greater than a preset value; A storage module, used for performing distributed storage of the stream data according to the data blocks; Wherein, the storage module includes: An allocating module, configured to allocate the number of the resized data blocks to write into the stream data and store them through erasure code calculation; a storage submodule, configured to respond to the number of data blocks being full, In the case where the size of the data block is smaller than the default size, a merge operation is performed to obtain a merged data block, and a remaining number of merged data blocks are applied for to continue writing into the stream data and stored through erasure code calculation, where the remaining number is the number minus 1; after the applied merged data block is fully written, if the size of the merged data block is still smaller than the default size, the merge operation is continued and the merged data block is applied for to write into the stream data and stored through erasure code calculation, and so on, until the size of the merged data block is equal to the default size, and the data block is stored through erasure code calculation; When the size of the data block is equal to the default size, storage is performed through erasure code calculation.

Images