An Incremental Data Writing Method and System Based on Incremental Erasure Code

Abstract

This application relates to the field of database technologies, and specifically relates to a method and system for writing incremental data based on incremental erasure codes. By collecting the write position (offset value) and data length of the incremental data, calculating the incremental erasure code based on the incremental change data with a length of the data length starting from the write position for each data block, and updating the check data at the corresponding position of the check block, this method realizes the calculation and update of the incremental erasure code during the incremental data writing process. Thus, in a distributed storage scenario, it is not necessary to obtain the full amount of data of the data block for erasure code calculation, reducing the occupation of data transmission and computing resources.

Description

Technical Field

[0001] This application relates to the field of database technology, and particularly to an incremental data writing method and system based on incremental erasure codes. Background Art

[0002] Erasure Codes (EC) is a data protection method mainly used in data storage and network transmission to ensure data reliability and fault tolerance. It encodes data by splitting it into fragments and creating redundant data blocks, and these encoded data fragments are stored at different locations, such as different disks, storage nodes, or other geographical locations. The core technologies of erasure codes include polynomial interpolation or oversampling, which enable the system to recover the original data in case of partial data loss.

[0003] The erasure code system can be expressed as "n = k + m", where "k" represents the number of original data blocks, "m" represents the number of added parity blocks (redundant data blocks), and "n" is the total number of encoded data blocks. For example, in the "EC 10 / 16" configuration, 6 additional parity blocks (m) are added to 10 original data blocks (k), and these 16 encoded data fragments (n) are scattered and stored at different locations. Even if up to m data blocks are lost or damaged, the original data can still be recovered from the remaining data blocks.

[0004] For the scenario of frequent updates in the data lake, the MOR strategy is often adopted to write the changes into a log file, and then merge these changes into the base data file according to the configured strategy. Due to the frequent data updates in the data lake scenario, it leads to frequent writing of log files in distributed storage, that is, frequent erasure code calculations are required.

[0005] In the related technology, full erasure code calculations are performed based on the complete data of each data block, and this full erasure code calculation requires obtaining all the data of the data block, consuming data transmission and computing resources. Summary of the Invention

[0006] In view of this, the present application discloses an incremental data writing method based on an incremental erasure code. In this method, an incremental log storage data space is created in advance. The storage data space includes a first number of data blocks for storing incremental data, and a second number of parity blocks for storing parity data for the incremental data. The first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of parity blocks set in the erasure code calculation rule. The method includes: generating a writing instruction for target incremental data in a target incremental log; the writing instruction is used to write the target incremental data into a target data block included in the storage data space; the writing instruction includes the data length of the target incremental data; according to the writing position of the target incremental data in the target data block and the data length, obtain the target data starting from the writing position and having a length of the data length in each data block included in the storage data space; determine an incremental erasure code according to the target data in each data block; according to the incremental erasure code, update the target parity data starting from the writing position and having a length of the data length in each parity block included in the storage data space.

[0007] In some embodiments, the method further includes: in response to detecting a preset trigger condition, merging the data in each data block included in the storage data space with the base data to complete log merging.

[0008] In some embodiments, the base data is distributedly stored in a distributed system in the form of data blocks; each data block included in the storage data space is distributedly stored in the distributed system; the distributed system includes multiple storage nodes; the method for determining the target data block includes: in response to each data block included in the storage data space being empty or having a default value, determining the current data block currently participating in the merging in the base data; using the data block allocated for the storage data space in the current storage node corresponding to the current data block as the target data block; the target data block is the first data block into which incremental data is written.

[0009] In some embodiments, in response to completing the log merging, the method further includes: clearing the data in each data block included in the storage data space; the storage data space is still retained and not released.

[0010] In some embodiments, the first quantity of data blocks has an order, where the data block that is first written with incremental data is the first data block; the method for writing incremental data includes: in response to receiving a write instruction for the target incremental data, if the currently written data block is the first data block, generating a first matrix based on the target incremental data, multiplying it by a preset encoding matrix to determine the current erasure code; updating the target data in the target data block starting from the write position and with a length equal to the data length based on the target incremental data, and updating the target parity data in the parity block starting from the write position and with a length equal to the data length based on the current erasure code.

[0011] In some embodiments, before writing incremental data, the method further includes: obtaining the amount of incremental data merged from two adjacent historical logs according to historical data; comparing the amount of incremental data with the size of a data block in the storage data space; in a case where the amount of incremental data is smaller than the size of a data block in the storage data space, the method for writing incremental data includes: in response to receiving a write instruction for the target incremental data, generating a first matrix based on the target incremental data, multiplying it by a preset encoding matrix to determine the current erasure code; updating the target data in the target data block starting from the write position and with a length equal to the data length based on the target incremental data, and updating the target parity data in the parity block starting from the write position and with a length equal to the data length based on the current erasure code.

[0012] In some embodiments, the historical data includes the average amount of incremental data merged from two adjacent historical logs statistically calculated for each time period within a day; obtaining the amount of incremental data merged from two adjacent historical logs according to historical data includes: querying the historical data according to the current time to determine the amount of incremental data; the method further includes: in a case where the amount of incremental data is smaller than the size of a data block in the storage data space, within the time period corresponding to the current time, performing the method for writing incremental data.

[0013] In some embodiments, the duration of the time period is within the range of 0.5 hours to 1.5 hours.

[0014] In some embodiments, the size of the storage data space is the same as the preset incremental log size.

[0015] The present application also provides an incremental data writing system based on incremental erasure code. In this system, an incremental log storage data space is created in advance. The storage data space includes a first number of data blocks for storing incremental data and a second number of parity blocks for storing parity data for the incremental data. The first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of parity blocks set in the erasure code calculation rule. The system includes: a generation module for generating a writing instruction for target incremental data in a target incremental log. The writing instruction is used to write the target incremental data into a target data block included in the storage data space. The writing instruction includes the data length of the target incremental data. An acquisition module for acquiring, according to the writing position of the target incremental data in the target data block and the data length, target data starting from the writing position and having a length of the data length in each data block included in the storage data space. A determination module for determining an incremental erasure code according to the target data in each data block. An update module for updating, according to the incremental erasure code, target parity data starting from the writing position and having a length of the data length in each parity block included in the storage data space.

[0016] In the solution described in any of the foregoing embodiments, by collecting the writing position (offset value) and data length of the incremental data, calculating the incremental erasure code based on the incremental change data with a length of the data length starting from the writing position in each data block, and updating the parity data at the corresponding position of the parity block, the calculation and update of the incremental erasure code during the incremental data writing process are realized. Thus, in a distributed storage scenario, it is not necessary to obtain the full amount of data in the data block for erasure code calculation, reducing the occupation of data transmission and computing resources.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the following will briefly introduce the drawings required for the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments described in one or more embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] The following will briefly introduce the drawings required for the description of the embodiments or related technologies.

[0020] Figure 1Schematic flow chart of a method for writing incremental data based on incremental erasure code shown in this application.

[0021] Figure 2 Schematic flow chart of a method for determining a target data block shown in this application.

[0022] Figure 3 Schematic flow chart of a method for writing incremental data shown in this application.

[0023] Figure 4 Schematic structure diagram of a system for writing incremental data based on incremental erasure code shown in this application. Detailed implementation manners

[0024] Exemplary embodiments will be described in detail below, and examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with this application. On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0025] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "the", and "said" used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items. It should also be understood that the word "if" used herein can be interpreted as "when", "while", or "in response to determining" depending on the context.

[0026] This application proposes a method for writing incremental data based on incremental erasure code. By collecting the write position (offset value) and data length of the incremental data, calculating the incremental erasure code according to the incremental change data of the length of the data block starting from the write position for the data length, and updating the check data at the corresponding position of the check block, the calculation and update of the incremental erasure code in the incremental data writing process are realized. Thus, in a distributed storage scenario, it is not necessary to obtain the full amount of data of the data block for erasure code calculation, reducing the occupation of data transmission and computing resources.

[0027] The following are embodiments described in conjunction with the drawings. s

[0028] Please refer to Figure 1 , Figure 1 Schematic flow chart of a method for writing incremental data based on incremental erasure code shown in this application. Figure 1The schematic method is applied to the client and / or server side of the database, and the client corresponds to the server side. The database involved in this application can be a relational database, a non-relational database, etc., without special restrictions. This article can be executed on the client side, and the database is taken as a MySQL database for illustration.

[0029] An incremental log storage data space can be pre-created on the client side. The storage data space includes a first number of data blocks for storing incremental data and a second number of parity blocks for storing parity data for the incremental data; the first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of parity blocks set in the erasure code calculation rule. In some embodiments, the size of the storage data space is the same as the preset incremental log size. This avoids waste of storage space.

[0030] For example, if the distributed storage system adopts an erasure code calculation rule of 4+2, then the first number is 4 and the second number is 2. The distributed storage system can include many storage nodes, such as node 1 to node 10, and the data blocks and parity blocks included in the storage data space can be stored in different storage nodes respectively. When there is no real data, these data blocks and parity blocks can be understood as in an empty state or filled with default values.

[0031] Before the log under the MOR policy is merged with the base data, the log data will be stored in the incremental log storage data space. By pre-creating the incremental log storage data space, it is possible not to intrude into the existing distributed storage logic and add a new space for storing and processing incremental logs.

[0032] As Figure 1 shown, the method may include S102-S108.

[0033] S102, generate a write instruction for the target incremental data in the target incremental log.

[0034] The write instruction is used to write the target incremental data into the target data block included in the storage data space; the write instruction includes the data length of the target incremental data.

[0035] The target incremental log refers to the data log generated under the MOR policy and needs to be written into the distributed storage system. The data in the target incremental log is called target incremental data. The MOR policy (Merge-On-Read) is a data update policy, and its core idea is to merge when reading data.

[0036] The write instruction is generated by the client based on the trigger of storing incremental data. The trigger is consistent with the related technology and will not be elaborated here. The generated write instruction may include the data length of the target incremental data. The server or the client of the database can determine the target data block to be written. The relevant information of the target data block can also be stored in the write instruction, and the server can write the target incremental data into the corresponding target data block based on this instruction. It should be noted that the rule for determining the target data block can be executed on the client or the server. The relevant rule can be understood as the replacement of the default value of the data block in the incremental log storage data space, and can be updated from the first data block to the fourth data block according to the number of data blocks.

[0037] S104, according to the writing position of the target incremental data in the target data block and the data length, obtain the target data with the length of the data length starting from the writing position in each data block included in the storage data space.

[0038] The client obtains the writing position and the data length from the write instruction. The writing position refers to the initial position of the target incremental data in the target data block, and the data length refers to the length of the target incremental data.

[0039] In this step, the client can obtain the target data with the length of the data length starting from the writing position in each data block from each storage node of the distributed storage system.

[0040] For example, the writing position may be the 1000th byte of the target data block, and the data length may be 10 bytes. The number of data blocks is 4, that is, the target data of 10 bytes starting from the 1000th byte in 4 data blocks can be obtained. Suppose it is represented by labels 、 、 、 This step does not obtain the full amount of data of the data block, reducing the occupancy of data transmission compared with the related technology.

[0041] S106, determine the incremental erasure code according to the target data in each data block.

[0042] This step can refer to the related erasure code calculation technology. In the related erasure code calculation technology, the following formula 1 can be used:

[0043] .

[0044] In the formula 1, m is a preset encoding matrix, D is the data in the data block, and C is the calculated result. Among them, by setting the encoding matrix, C0 to C3 can be made the same as D0 to D3, and C4 and C5 are the calculated erasure codes. For example, the preset encoding matrix can be: ;

[0045] The previous formula 1 can be transformed into formula 2: .

[0046] Among them, D is the data in the data block, and C0 and C1 are the calculated erasure codes.

[0047] In this step, the above preset encoding matrix can be adopted, and using formula 3: , the calculation of the incremental erasure code is performed. Among them, and are the incremental erasure codes.

[0048] S108. According to the incremental erasure code, update the target check data starting from the writing position and having a length of the data length in each check block included in the storage data space.

[0049] In this step, the incremental erasure code calculated by S106 can be used to replace the data at the corresponding position in the check data block to achieve the update of the erasure code. For example, the writing position can be the 1000th byte of the target data block, and the data length can be 10 bytes. Then, can be used to replace the 10-byte data starting from the 1000th byte data in the first check block, and can be used to replace the 10-byte data starting from the 1000th byte data in the second check block to achieve the update of the erasure code.

[0050] Through the solution described in S102 - S108, by collecting the writing position (offset value) and data length of the incremental data, calculating the incremental erasure code according to the incremental change data of each data block starting from the writing position and having a length of the data length, and updating the check data at the corresponding position of the check block, the calculation and update of the incremental erasure code during the incremental data writing process are realized. Thus, in the distributed storage scenario, it is not necessary to obtain the full amount of data of the data block for erasure code calculation, reducing the occupation of data transmission and computing resources.

[0051] In some embodiments, the method further includes: in response to detecting a preset trigger condition, merging the data in each data block included in the storage data space with the base data to complete log merging.

[0052] The setting of the trigger condition is related to the MOR policy used, and the MOR policy may include:

[0053] Time-based Compaction Strategy (DayBasedCompactionStrategy): Compresses data according to dates and is applicable to scenarios where recent partitions are frequently updated.

[0054] Partition-based Compaction Strategy (BoundedPartitionAwareCompactionStrategy): Selects specific partitions for compaction, usually targeting recently updated partitions.

[0055] IO-based Compaction Strategy (BoundedIOCompactionStrategy): Controls the compaction plan by restricting the size of read / write IOs to avoid performance impacts caused by overly large IO operations.

[0056] Log File Size-based Compaction Strategy (LogFileSizeBasedCompactionStrategy): Triggers compaction when the size of the log file reaches a certain threshold. This is the default compaction strategy in Hudi.

[0057] Correspondingly, corresponding trigger conditions can be set, and after detecting that the trigger conditions are met, the incremental data and the base data can be merged.

[0058] For example, in a distributed storage system, to optimize performance, the MOR (Merge Once Read) strategy is adopted. The trigger condition is that when the size of the system log reaches a set threshold (e.g., 10 MB), data merging is initiated. The log data is stored in chunks on multiple nodes. Once the log size reaches 10 MB, the system reads all relevant data chunks, merges this data with the base information, and generates new merged data chunks. Through this process, the system can effectively organize and compress data, improving storage efficiency and read performance.

[0059] In some embodiments, the log data can be preferentially stored in the storage nodes that need to be merged, thereby triggering local merging as much as possible and reducing the transmission of data across nodes, which would otherwise occupy transmission resources.

[0060] In the distributed storage system, the base data is distributedly stored in the distributed system in the form of data chunks; each data chunk included in the storage data space is distributedly stored in the distributed system; the distributed system includes multiple storage nodes.

[0061] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of a method for determining a target data chunk shown in this application.

[0062] S202. In response to each data block included in the storage data space being empty or having a default value, determine the current data block currently participating in merging in the basic data.

[0063] It can be understood that each data block being empty or having a default value may correspond to multiple situations. For example, just completing a log merge and the data in the data block being cleared. Another example is that when starting to store data, the data block is empty or has a default value. This application does not enumerate the above situations.

[0064] In this step, the data in the basic data that needs to participate in merging will be marked. For example, the basic data is divided into many data blocks and stored in different storage nodes respectively. A label is attached to each data block that needs to participate in merging each time. By identifying the label, the current data block that needs to participate in merging can be determined.

[0065] S204. Use the data block allocated for the storage data space in the current storage node corresponding to the current data block as the target data block.

[0066] The distributed storage system includes many nodes, and each data block included in the storage data space is also allocated in different nodes. After determining the current data block in S202, the current storage node corresponding to the current data block can be determined, and then the data block allocated for the storage data space in the current storage node is used as the target data block. It can be understood that in the case where each data block being empty or having a default value may correspond to multiple situations, the target data block is the first data block of the data blocks that participate in the erasure code calculation and are written with incremental data.

[0067] Through S202 - S204, the data block in the same storage node as the basic data block participating in merging can be used as the target data block to store the log data. Subsequently, when triggering merging, data merging can be implemented locally, reducing cross-node data transmission.

[0068] For example, in a distributed storage system, assume that the log storage data space is divided into 4 data blocks (D0, D1, D2, D3), which are stored in different storage nodes (node A, node B, node C, node D) respectively. When the system starts, these data blocks may be empty or have default values. The basic data is divided into 10 data blocks (E0 to E9), which are stored in different storage nodes respectively. The system can mark the current data block that needs to participate in merging among the 10 data blocks. Assume that it is identified that E0 is the current data block that needs to participate in merging.

[0069] The system detects that each data block is empty. For example, when starting to store data, D0, D1, D2, and D3 are all empty.

[0070] After the system determines that the current data block is E0 by querying the identifier, it can find the current storage node where E0 is located (assumed to be node A).

[0071] The system searches for the data block related to the log data block in node A and uses it as the target data block. For example, if there is a stored block D0 in node A, then D0 is the target data block, which is used to preferentially store log data.

[0072] The system outputs the log data to the target data block D0 of node A for subsequent merge operations.

[0073] Doing so can ensure that related data is all within the same node, avoiding cross-node transmission and reducing resource occupancy.

[0074] When both the target data block D0 in node A and the current data block E0 are ready, the system triggers a data merge operation locally. After the merge is completed, the data in D0 is updated, and since it is a local operation, network bandwidth is saved.

[0075] Through the above steps, the system effectively utilizes the local storage node for log storage and data merge, reducing the need for cross-node data transmission, thereby improving the overall performance and resource utilization rate.

[0076] In some embodiments, in response to completing the log merge, the method further includes: clearing the data in each data block included in the storage data space; the storage data space is still retained without being released. In the related art, the storage space is often released after completing the log merge task, so when storing log data subsequently, it is necessary to reallocate the storage space, resulting in low storage efficiency. In this example, after completing the log merge task, only the data is cleared, but the space is not released, improving the rate of writing incremental logs.

[0077] For example, the log storage data space includes four data blocks D0 to D4. After triggering a log merge operation once, the data in D0 to D4 can be cleared, for example, filled with default values, but this storage space is still retained for subsequent log data storage. By this method, after the system completes the log merge, it only clears the data block but retains the storage space, which not only optimizes the use of storage resources but also improves the writing rate of subsequent log data. This strategy reduces the repeated application of storage space and improves the overall system performance.

[0078] In some scenarios, log merge may be frequently triggered. In such scenarios, the calculation frequency of erasure codes is very high. In some embodiments, for such scenarios, the data acquisition volume can be further reduced, the occupancy of data transmission channels, and the occupancy of computing resources can be reduced. The present application proposes two ways to further reduce the data acquisition volume, the occupancy of data transmission channels, and the occupancy of computing resources.

[0079] Method 1:

[0080] The first quantity of data blocks has an order, where the data block that is first written with incremental data is the No. 1 data block;

[0081] The method of writing incremental data includes:

[0082] In response to receiving a write instruction for the target incremental data, if the currently written data block is the No. 1 data block, generate a first matrix based on the target incremental data, multiply it by a preset encoding matrix, and determine the current erasure code;

[0083] Based on the target incremental data, update the target data in the target data block starting from the write position and having a length of the data length, and based on the current erasure code, update the target parity data in the parity block starting from the write position and having a length of the data length.

[0084] In Method 1, when writing incremental data, it is judged whether it is writing to the No. 1 data block. If so, it means that the data in other data blocks is useless. Thus, only the target incremental data to be written currently can be used for erasure code calculation and update, without obtaining the data of other data blocks, thereby further reducing the data acquisition volume, reducing the occupation of the data transmission channel, and reducing the occupation of computing resources. The method of using the target incremental data for erasure code calculation can refer to the introduction in Method 2.

[0085] Method 2:

[0086] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of a method for writing incremental data shown in this application. As Figure 3 shown, the method may include S302 - S308.

[0087] S302, before writing incremental data, obtain the incremental data volume of the combination of the adjacent two historical logs according to historical data.

[0088] In some methods, the incremental data volume can be calculated by calculating the average value. The historical data includes the incremental data volume written in each combination of adjacent two logs. By calculating the average value, the incremental quantity can be obtained.

[0089] S304, compare the incremental data volume with the size of a data block in the storage data space.

[0090] S306. In the case where the size of the incremental data amount is smaller than the size of a data block in the storage data space, the method for writing incremental data includes: in response to receiving a write instruction for the target incremental data, generating a first matrix based on the target incremental data, multiplying it by a preset encoding matrix, and determining the current erasure code.

[0091] The fact that the size of the incremental data amount is smaller than the size of a data block in the storage data space indicates that before log merging, the incremental data may only be written to the first data block, and the data in other data blocks is meaningless. Therefore, only the erasure code calculation and update are performed based on the target incremental data.

[0092] In this step, the format of the first matrix can be preset, and the target incremental data can be filled into the matrix. The preset encoding matrix can be: .

[0093] Referring to Formula 3, is the target incremental data, , , can all be filled with 0 to obtain Formula 4:

[0094] ;

[0095] Then the calculation of the current erasure code can be simplified, and there is no need to obtain the real data of , , , further reducing the data volume transmission and calculation process.

[0096] S308. Based on the target incremental data, update the target data in the target data block starting from the write position and with a length of the data length, and based on the current erasure code, update the target parity data in the parity block starting from the write position and with a length of the data length.

[0097] This step can write into the target data block and write the erasure code into the parity block respectively.

[0098] In some ways, the accuracy of determining the incremental data amount can be improved, thereby improving the correctness of data writing. The historical data includes the average value of the incremental data amounts of two adjacent historical log mergings statistically calculated for each time period within a day. The day can be divided into several fixed time periods. For example, each time period is 6 hours, so there are 4 time periods in a day, and then the average value of the incremental data amounts of two adjacent historical log mergings within each time period is calculated respectively.

[0099] For example, time period 1 (00:00 - 06:00): incremental data volume = 1MB; time period 2 (06:00 - 12:00): incremental data volume = 2MB; time period 3 (12:00 - 18:00): incremental data volume = 3MB; time period 4 (18:00 - 24:00): incremental data volume = 5MB.

[0100] Obtaining the incremental data volume of two adjacent historical log merges according to the historical data includes:

[0101] Querying the historical data according to the current time to determine the incremental data volume;

[0102] The method further includes:

[0103] In the case where the size of the incremental data volume is smaller than the size of a data block in the storage data space, within the time period corresponding to the current time, execute the method of writing incremental data.

[0104] Taking the above time periods as an example, assuming the current time is 7 am, it can be determined that it falls within time period 2, and the incremental data volume is obtained as 2MB. Assuming the size of one data block is 2.5MB, then it can be determined that the incremental data volume generated by two adjacent log merges within time period 2 cannot fill one data block. Then, within time period 2, a method such as formula 4 can be used to simplify the erasure code calculation process, that is, only the log data is required, and the data of other data blocks is not required to complete the erasure code calculation, reducing data transmission and calculation volume, and only performing this simplified operation within the corresponding time period to avoid data writing errors caused by the simplified operation. It can be understood that the granularity of dividing time periods has a certain impact on data writing accuracy. Generally, the smaller the time period granularity, the higher the accuracy, but if the granularity is too small, it will be meaningless. In some embodiments, the duration of the above time periods is within the range of 0.5 hours to 1.5 hours.

[0105] This application also proposes an incremental data writing system based on incremental erasure codes. Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of an incremental data writing system based on incremental erasure codes shown in this application. In this system, an incremental log storage data space is pre-created. The storage data space includes a first number of data blocks for storing incremental data and a second number of parity blocks for storing parity data for the incremental data; the first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of parity blocks set in the erasure code calculation rule;

[0106] As Figure 4 shown, the system 400 includes:

[0107] A generating module 410, configured to generate a write instruction for the target incremental data in the target incremental log; the write instruction is used to write the target incremental data into a target data block included in the storage data space; the write instruction includes the data length of the target incremental data.

[0108] An obtaining module 420, configured to obtain, according to the writing position of the target incremental data in the target data block and the data length, target data with the length of the data length starting from the writing position in each data block included in the storage data space.

[0109] A determining module 430, configured to determine an incremental erasure code according to the target data in each data block.

[0110] An updating module 440, configured to update, according to the incremental erasure code, the target check data with the length of the data length starting from the writing position in each check block included in the storage data space.

[0111] In some embodiments, the system 400 further includes:

[0112] A merging module, configured to, in response to detecting a preset trigger condition, merge the data in each data block included in the storage data space with the base data to complete log merging.

[0113] In some embodiments, the base data is distributedly stored in the distributed system 400 in the form of data blocks; each data block included in the storage data space is distributedly stored in the distributed system 400; the distributed system 400 includes multiple storage nodes; the system 400 further includes a target data block determining module, configured to:

[0114] In response to each data block included in the storage data space being empty or having a default value, determine a current data block currently participating in the merging in the base data.

[0115] Use the data block allocated for the storage data space in the current storage node corresponding to the current data block as the target data block; the target data block is the first data block into which incremental data is written.

[0116] In some embodiments, the system 400 further includes:

[0117] An emptying module, configured to, in response to completing the log merging, empty the data in each data block included in the storage data space; the storage data space is still retained without being released.

[0118] In some embodiments, the first quantity of data blocks has an order, where the data block into which the incremental data is first written is the No. 1 data block; the system 400 further includes a first writing module for:

[0119] In response to receiving a write instruction for the target incremental data, if the currently written data block is the No. 1 data block, generate a first matrix based on the target incremental data, multiply it by a preset encoding matrix, and determine the current erasure code;

[0120] Based on the target incremental data, update the target data in the target data block starting from the write position and having a length of the data length, and based on the current erasure code, update the target parity data in the parity block starting from the write position and having a length of the data length.

[0121] In some embodiments, the system 400 further includes a comparison module for:

[0122] Before writing the incremental data, obtain the amount of incremental data merged by adjacent two historical logs according to historical data;

[0123] Compare the amount of incremental data with the size of a data block in the storage data space;

[0124] A second writing module for:

[0125] In a case where the size of the amount of incremental data is less than the size of a data block in the storage data space, in response to receiving a write instruction for the target incremental data, generate a first matrix based on the target incremental data, multiply it by a preset encoding matrix, and determine the current erasure code;

[0126] Based on the target incremental data, update the target data in the target data block starting from the write position and having a length of the data length, and based on the current erasure code, update the target parity data in the parity block starting from the write position and having a length of the data length.

[0127] In some embodiments, the historical data includes the average value of the amount of incremental data merged by adjacent two historical logs statistically according to each time period within a day;

[0128] The comparison module, further:

[0129] According to the current time, query the historical data to determine the amount of incremental data;

[0130] The writing module, further:

[0131] In the case where the size of the incremental data amount is smaller than the size of a data block in the storage data space, the method of writing incremental data is executed within the time period corresponding to the current moment.

[0132] In some embodiments, the duration of the time period is within the range of 0.5 hours to 1.5 hours.

[0133] In some embodiments, the size of the storage data space is the same as the preset incremental log size.

[0134] For the description of related embodiments, reference may be made to the previous embodiments.

[0135] Those skilled in the art should understand that one or more embodiments of the present application may be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include, but are not limited to, disk storage, CDROM, optical storage, etc.) containing computer-usable program code.

[0136] "And / or" in the present application means at least one of the two. For example, "A and / or B" may include three scenarios: A, B, and "A and B".

[0137] Each embodiment in the present application is described in a progressive manner. For the same or similar parts between the embodiments, reference may be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiment of the data processing device, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, reference may be made to the partial description of the method embodiment.

[0138] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0139] The subject matter and the embodiments of the functional loading described in this application can be implemented in the following: digital electronic circuits, tangible computer software or firmware, computer hardware that may include the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this application can be implemented as one or more computer programs, that is, one or more modules in computer program instructions encoded on a tangible non-transitory program carrier for being executed by a data processing device or controlling the data processing device to perform the loading. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0140] The processes and logical flows described in this application can be executed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by performing the loading according to the input data and generating the output. The processes and logical flows can also be executed by dedicated logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuits.

[0141] Computers suitable for executing computer programs can include, for example, general and / or special microprocessors, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer can include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also can include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks or optical disks, etc., or the computer will be loadably coupled to such mass storage devices to receive data from them or transmit data to them, or both. However, a computer is not necessarily required to have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, just to name a few.

[0142] A computer-readable medium suitable for storing computer program instructions and data can include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0143] Although this application contains many specific implementation details, these should not be construed as limiting the scope of any disclosure or the scope of what is claimed, but are mainly used to describe the features of specific embodiments of a particular disclosure. Certain features described in multiple embodiments in this application can also be implemented in combination in a single embodiment. On the other hand, various features described in a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may act in certain combinations and are even initially claimed as such, one or more features from the claimed combination can in some cases be removed from the combination, and the claimed combination can be directed to a sub-combination or a variation of the sub-combination.

[0144] Similarly, although the loading is depicted in a specific order in the drawings, this should not be construed as requiring that these loadings be performed in the specific order shown or sequentially, or that all of the illustrated loadings be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0145] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the acts recited in the claims can be performed in a different order and still achieve the desired result. In addition, the processes depicted in the drawings are not necessarily in the specific order or sequential order shown, to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0146] The above are only the preferred embodiments of one or more embodiments of this application, and are not intended to limit one or more embodiments of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of this application shall be included within the scope protected by one or more embodiments of this application.

Claims

1. A method for writing incremental data based on incremental erasure codes, characterized in that: Pre-creating an incremental log storage data space, the storage data space including a first number of data blocks for storing incremental data and a second number of check blocks for storing check data for the incremental data; The first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of check blocks set in the erasure code calculation rule; The method comprises: Generate a write instruction for target incremental data in a target incremental log; the write instruction is used to write the target incremental data into a target data block included in the storage data space; the write instruction includes the data length of the target incremental data; According to the write position of the target incremental data in the target data block and the data length, obtaining target data starting from the write position and having a length equal to the data length in each data block included in the data storage space; determining an incremental erasure code according to the target data in each data block; updating, in each check block included in the storage data space, target check data starting from the write position and having a length equal to the data length, according to the incremental erasure code; In response to detecting a preset trigger condition, merging the data in each data block included in the storage data space with the basic data to complete the log merging; Before writing incremental data, obtain the incremental data volume of two adjacent historical log merges based on historical data; comparing the incremental data amount with a size of a data block in the data storage space; In a case where the size of the incremental data amount is smaller than the size of a data block in the data storage space, the method for writing the incremental data includes: In response to receiving a write instruction for the target incremental data, generating a first matrix based on the target incremental data, and multiplying the first matrix by a preset encoding matrix to determine a current erasure code; Based on the target incremental data, the target data in the target data block starting from the write position and having a length of the data length is updated, and based on the current erasure code, the target check data in the check block starting from the write position and having a length of the data length is updated.

2. The incremental data writing method based on incremental erasure coding according to claim 1, characterized in that: The basic data is distributed and stored in the distributed system in the form of data blocks; the data blocks included in the data storage space are distributed and stored in the distributed system; The distributed system includes a plurality of storage nodes; The method for determining the target data block includes: In response to each data block included in the storage data space being empty or having a default value, determining a current data block currently participating in merging in the basic data; The data block allocated to the data storage space in the current storage node corresponding to the current data block is used as the target data block; the target data block is data block No. 1 to which the incremental data is first written.

3. The incremental data writing method based on incremental erasure coding according to claim 1, characterized in that: In response to completing the log merging, the method further includes: The data in each data block included in the data storage space is cleared; the data storage space is still reserved and not released.

4. The incremental data writing method based on incremental erasure coding according to claim 1, characterized in that: The first number of data blocks are arranged in an order, wherein the data block to which the incremental data is first written is data block number one; Methods for writing incremental data include: In response to receiving a write instruction for the target incremental data, if the data block currently being written is the first data block, generating a first matrix based on the target incremental data, and multiplying the first matrix by a preset encoding matrix to determine a current erasure code; Based on the target incremental data, the target data in the target data block starting from the write position and having a length of the data length is updated, and based on the current erasure code, the target check data in the check block starting from the write position and having a length of the data length is updated.

5. The incremental data writing method based on incremental erasure coding according to claim 1, characterized in that: The historical data includes the average of the incremental data volume of two adjacent historical logs combined according to each time period within each day; The method of obtaining the incremental data amount of two adjacent historical log merges based on the historical data includes: According to the current time, query the historical data to determine the incremental data amount; The method further comprises: In a case where the size of the incremental data amount is smaller than the size of a data block in the data storage space, the method of writing incremental data is executed within a time period corresponding to the current moment.

6. The incremental data writing method based on incremental erasure coding according to claim 5, characterized in that: The duration of the time period is in the range of 0.5 hours to 1.5 hours.

7. The incremental data writing method based on incremental erasure coding according to claim 1, characterized in that: The size of the storage data space is the same as the preset incremental log size.

8. An incremental data writing system based on incremental erasure coding, characterized in that: Pre-creating an incremental log storage data space, the storage data space including a first number of data blocks for storing incremental data and a second number of check blocks for storing check data for the incremental data; The first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of check blocks set in the erasure code calculation rule; The system comprises: A generating module, configured to generate a write instruction for target incremental data in a target incremental log; the write instruction is configured to write the target incremental data into a target data block included in the storage data space; the write instruction includes a data length of the target incremental data; an acquisition module, configured to acquire, according to a write position of the target incremental data in the target data block and the data length, target data starting from the write position and having a length equal to the data length in each data block included in the storage data space; a determination module, configured to determine an incremental erasure code according to the target data in each data block; an updating module, configured to update, in each check block included in the storage data space, target check data starting from the write position and having a length equal to the data length, according to the incremental erasure code; a merging module, configured to merge the data in each data block included in the storage data space with the basic data in response to detecting a preset trigger condition, thereby completing log merging; The comparison module is used to obtain the incremental data volume of two adjacent historical log merges based on historical data before writing incremental data; comparing the incremental data amount with a size of a data block in the data storage space; The second writing module is configured to write incremental data when the size of the incremental data is smaller than the size of a data block in the data storage space. The method includes: In response to receiving a write instruction for the target incremental data, generating a first matrix based on the target incremental data, and multiplying the first matrix by a preset encoding matrix to determine a current erasure code; Based on the target incremental data, the target data in the target data block starting from the write position and having a length of the data length is updated, and based on the current erasure code, the target check data in the check block starting from the write position and having a length of the data length is updated.

Images