Incremental data writing method and system based on incremental erasure codes

Abstract

The invention relates to the technical field of databases, in particular to an incremental data writing method and system based on incremental erasure codes. The method comprises the following steps: acquiring a write-in position (offset value) and a data length of incremental data, calculating an incremental erasure code according to incremental change data of each data block starting from the write-in position and having the length of the data length, and updating verification data at a corresponding position of a verification block; according to the method, calculation and updating of the incremental erasure codes in the incremental data writing process are achieved, and therefore in a distributed storage scene, it is not needed to obtain full data of data blocks for erasure code calculation, and occupation of data transmission and calculation resources is reduced.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a method and system for writing incremental data based on incremental erasure codes. Background Art

[0002] Erasure Codes (EC) is a data protection method mainly used in data storage and network transmission to ensure data reliability and fault tolerance. It encodes data by splitting it into fragments and creating redundant data blocks. These encoded data fragments are stored in different locations, such as different disks, storage nodes, or other geographical locations. The core technologies of erasure coding include polynomial interpolation or oversampling, which enable the system to recover the original data when some data is lost.

[0003] The erasure code system can be expressed as "n = k + m", where "k" represents the number of original data blocks, "m" represents the number of added check blocks (redundant data blocks), and "n" is the total number of encoded data blocks. For example, in the "EC 10 / 16" configuration, 6 additional check blocks (m) are added to 10 original data blocks (k), and these encoded 16 data fragments (n) are stored in different locations. Even if up to m data blocks are lost or damaged, the original data can still be recovered from the remaining data blocks.

[0004] For scenarios where data lakes are frequently updated, the MOR strategy is often used to write changes to log files, and then merge these changes into the basic data files according to the configured strategy. Since data in data lake scenarios is frequently updated, log files are also frequently written to distributed storage, which requires frequent erasure code calculations.

[0005] In the related art, full erasure code calculation is performed based on the complete data of each data block. The full erasure code calculation needs to obtain all the data of the data block, which consumes data transmission and calculation resources. Summary of the invention

[0006] In view of this, the present application discloses a method for writing incremental data based on incremental erasure codes. In the method, an incremental log storage data space is pre-created, the storage data space includes a first number of data blocks for storing incremental data, and a second number of check blocks for storing check data for the incremental data; the first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of check blocks set in the erasure code calculation rule; the method includes: generating a write instruction for the target incremental data in the target incremental log; the write instruction is used to write the target incremental data into the target data block included in the storage data space; the write instruction includes the data length of the target incremental data; according to the write position and the data length of the target incremental data in the target data block, obtaining the target data starting from the write position and having a length of the data length in each data block included in the data storage space; determining the incremental erasure code according to the target data in each data block; and updating the target check data starting from the write position and having a length of the data length in each check block included in the data storage space according to the incremental erasure code.

[0007] In some embodiments, the method further includes: in response to detecting a preset trigger condition, merging the data in each data block included in the storage data space with the basic data to complete the log merge.

[0008] In some embodiments, the basic data is distributed and stored in a distributed system in the form of data blocks; the data blocks included in the storage data space are distributed and stored in the distributed system; the distributed system includes multiple storage nodes; the method for determining the target data block includes: in response to the data blocks included in the storage data space being empty or default values, determining the current data block currently participating in the merge in the basic data; taking the data block allocated to the storage data space in the current storage node corresponding to the current data block as the target data block; the target data block is the number one data block to which the incremental data is first written.

[0009] In some embodiments, in response to completing the log merge, the method further includes: clearing data in each data block included in the storage data space; the storage data space is still reserved and not released.

[0010] In some embodiments, the first number of data blocks have an order, wherein the data block to which the incremental data is first written is data block number one; the method for writing incremental data comprises: in response to receiving a write instruction for the target incremental data, if the data block currently being written is data block number one, generating a first matrix based on the target incremental data, multiplying it with a preset encoding matrix, and determining a current erasure code; based on the target incremental data, updating the target data in the target data block starting from the write position and having a length of the data length, and based on the current erasure code, updating the target check data in the check block starting from the write position and having a length of the data length.

[0011] In some embodiments, before writing the incremental data, the method further includes: obtaining the incremental data amount of two adjacent historical logs merged based on the historical data; comparing the incremental data amount with the size of a data block in the data storage space; in the case where the size of the incremental data amount is smaller than the size of a data block in the data storage space, the method of writing the incremental data includes: in response to receiving a write instruction for the target incremental data, generating a first matrix based on the target incremental data, multiplying it with a preset encoding matrix, and determining a current erasure code; based on the target incremental data, updating the target data in the target data block starting from the write position and having a length of the data length, and based on the current erasure code, updating the target check data in the check block starting from the write position and having a length of the data length.

[0012] In some embodiments, the historical data includes an average of the incremental data volume of two adjacent historical logs merged according to statistics in each time period of each day; obtaining the incremental data volume of two adjacent historical logs merged based on the historical data includes: querying the historical data according to the current moment to determine the incremental data volume; the method also includes: when the size of the incremental data volume is smaller than the size of a data block in the data storage space, executing the method of writing incremental data within the time period corresponding to the current moment.

[0013] In some embodiments, the time period is between 0.5 hours and 1.5 hours.

[0014] In some embodiments, the size of the storage data space is the same as a preset incremental log size.

[0015] The present application also proposes an incremental data writing system based on incremental erasure code. In the system, an incremental log storage data space is pre-created, and the storage data space includes a first number of data blocks for storing incremental data, and a second number of check blocks for storing check data for the incremental data; the first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of check blocks set in the erasure code calculation rule; the system includes: a generation module for generating a write instruction for the target incremental data in the target incremental log; the write instruction is used to write the target incremental data into the target data block included in the storage data space; the write instruction includes the data length of the target incremental data; an acquisition module for acquiring the target data starting from the write position and having a length of the data length in each data block included in the data storage space according to the write position and the data length of the target incremental data in the target data block; a determination module for determining the incremental erasure code according to the target data in each data block; an update module for updating the target check data starting from the write position and having a length of the data length in each check block included in the data storage space according to the incremental erasure code.

[0016] In the scheme recorded in any of the aforementioned embodiments, by collecting the write position (offset value) and data length of the incremental data, calculating the incremental erasure code based on the incremental change data of each data block starting from the write position with a length of the data length, and updating the check data at the corresponding position of the check block, a calculation and update of the incremental erasure code in the incremental data writing process is implemented, so that in a distributed storage scenario, there is no need to obtain the full amount of data of the data block for erasure code calculation, thereby reducing the occupancy of data transmission and computing resources.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the drawings required for use in the embodiments or related technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions.

[0020] Figure 1A method flow chart of an incremental data writing method based on incremental erasure codes shown in the present application.

[0021] Figure 2 This is a flow chart of a method for determining a target data block shown in the present application.

[0022] Figure 3 A flowchart of a method for writing incremental data is shown in the present application.

[0023] Figure 4 A schematic diagram of the structure of an incremental data writing system based on incremental erasure codes shown in the present application. DETAILED DESCRIPTION

[0024] The exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0025] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used in this article, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0026] The present application proposes a method for writing incremental data based on incremental erasure codes. The method collects the writing position (offset value) and data length of the incremental data, calculates the incremental erasure code according to the incremental change data of each data block starting from the writing position with a length of the data length, and updates the check data at the corresponding position of the check block, thereby realizing the calculation and update of the incremental erasure code in the incremental data writing process, so that in a distributed storage scenario, it is not necessary to obtain the full amount of data of the data block for erasure code calculation, which reduces the occupation of data transmission and computing resources.

[0027] The following is a description of the embodiments with reference to the accompanying drawings.

[0028] See also Figure 1 , Figure 1 A method flow chart of an incremental data writing method based on incremental erasure codes shown in the present application. Figure 1The illustrated method is applied to the client and / or server of the database, and the client corresponds to the server. The database involved in this application can be a relational database, a non-relational database, etc., without special restrictions. This article can be executed on the client, and the database is a MySQL database as an example for illustration.

[0029] An incremental log storage data space may be pre-created on the client, the storage data space including a first number of data blocks for storing incremental data, and a second number of check blocks for storing check data for the incremental data; the first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of check blocks set in the erasure code calculation rule. In some embodiments, the size of the storage data space is the same as the preset incremental log size. This avoids waste of storage space.

[0030] For example, if the distributed storage system adopts the 4+2 erasure code calculation rule, the first number is 4 and the second number is 2. The distributed storage system may include many storage nodes, such as nodes 1 to 10, and the data blocks and check blocks included in the storage data space may be stored in different storage nodes respectively. When there is no real data, these data blocks and check blocks may be understood as empty or filled with default values.

[0031] Before the log under the triggering MOR policy is merged with the basic data, the log data will be stored in the incremental log storage data space. By pre-creating the incremental log storage data space, a new space can be added for the storage and processing of incremental logs without intruding the existing distributed storage logic.

[0032] like Figure 1 As shown, the method may include S102-S108.

[0033] S102, generating a write instruction for the target incremental data in the target incremental log.

[0034] The write instruction is used to write the target incremental data into the target data block included in the storage data space; the write instruction includes the data length of the target incremental data.

[0035] The target incremental log refers to the data log generated under the MOR strategy, which needs to be written to the distributed storage system. The data in the target incremental log is called the target incremental data. The MOR strategy (Merge-On-Read) is a data update strategy, the core idea of ​​which is to merge when reading data.

[0036] The write instruction is generated by the client based on the storage trigger of the incremental data. The trigger is consistent with the relevant technology and will not be described in detail here. The generated write instruction may include the data length of the target incremental data. The server or client of the database can determine the target data block to be written. The relevant information of the target data block can also be stored in the write instruction, and the server can write the target incremental data to the corresponding target data block based on the instruction. It should be noted that the rules for determining the target data block can be executed on the client or the server, and the relevant rules can be understood as the replacement of the default value of the data block in the incremental log storage data space, which can be updated from the first data block to the fourth data block according to the number of the data block.

[0037] S104, according to the write position of the target incremental data in the target data block and the data length, obtaining target data starting from the write position and having a length equal to the data length in each data block included in the data storage space.

[0038] The client obtains the write position and the data length from the write instruction. The write position refers to the initial position of the target incremental data in the target data block, and the data length refers to the length of the target incremental data.

[0039] In this step, the client can obtain target data starting from the write position and having a length equal to the data length in each data block from each storage node of the distributed storage system.

[0040] For example, the write position may be the 1000th byte of the target data block, and the data length may be 10 bytes. The number of data blocks is 4, that is, 10 bytes of target data starting from the 1000th byte in 4 data blocks can be obtained. Assume that the target data is labeled D0. ′ 、D1 ′ 、D2 ′ 、D3 ′ This step does not obtain the full amount of data in the data block, which reduces the occupancy of data transmission compared to related technologies.

[0041] S106: Determine an incremental erasure code according to the target data in each data block.

[0042] This step may refer to the relevant erasure code calculation technology, in which the following formula 1 may be used:

[0043]

[0044] In Formula 1, m is a preset coding matrix, D is data in a data block, and C is a calculated result, wherein C0 to C3 can be made the same as D0 to D3 by setting the coding matrix, and C4 and C5 are calculated erasure codes. For example, the coding matrix can be preset as:

[0045] The previous formula 1 can be transformed into formula 2:

[0046] Where D is the data in the data block, and C0 and C1 are the calculated erasure codes.

[0047] In this step, the above preset coding matrix can be used, using Formula 3: Calculate the incremental erasure code. Among them, C0 ′ and C1 ′ It is an incremental erasure code.

[0048] S108, updating target verification data starting from the write position and having a length equal to the data length in each verification block included in the data storage space according to the incremental erasure code.

[0049] In this step, the incremental erasure code calculated in S106 can be used to replace the data at the corresponding position in the check data block to update the erasure code. For example, the write position can be the 1000th byte of the target data block, and the data length can be 10 bytes. ′ Replace the 10 bytes starting from 1000 bytes in the first check block with C1 ′ Replace the 10-byte data starting from the 1000-byte data in the second check block to implement the update of the erasure code.

[0050] Through the scheme recorded in S102-S108, by collecting the write position (offset value) and data length of the incremental data, calculating the incremental erasure code according to the incremental change data of each data block starting from the write position with a length of the data length, and updating the check data at the corresponding position of the check block, a calculation and update of the incremental erasure code in the incremental data writing process is implemented, so that in a distributed storage scenario, there is no need to obtain the full amount of data of the data block for erasure code calculation, thereby reducing the occupancy of data transmission and computing resources.

[0051] In some embodiments, the method further includes: in response to detecting a preset trigger condition, merging the data in each data block included in the storage data space with the basic data to complete the log merge.

[0052] The setting of the trigger condition is related to the MOR strategy used, and the MOR strategy may include:

[0053] DayBasedCompactionStrategy: Compresses data by date, which is suitable for scenarios where recent partitions are frequently updated.

[0054] BoundedPartitionAwareCompactionStrategy: selects specific partitions for compaction, usually the most recently updated partitions.

[0055] BoundedIOCompactionStrategy: Controls the compaction plan by limiting the size of read / write IO to avoid excessive IO operations affecting performance.

[0056] LogFileSizeBasedCompactionStrategy: When the size of the log file reaches a certain threshold, compression is triggered. This is the default compression strategy of Hudi.

[0057] Corresponding trigger conditions can be set accordingly, and the incremental data and basic data can be merged after it is detected that the trigger conditions are met.

[0058] For example, in a distributed storage system, in order to optimize performance, the MOR (Merge Once Read) strategy is adopted. The trigger condition is that when the size of the system log reaches the set threshold (for example, 10MB), data merging is started. Log data is stored in blocks on multiple nodes. Once the log size reaches 10MB, the system will read all related data blocks, merge these data with basic information, and generate new merged data blocks. Through this process, the system can effectively organize and compress data, improve storage efficiency and reading performance.

[0059] In some embodiments, log data may be preferentially stored in storage nodes that need to be merged, thereby triggering local merging as much as possible and reducing the transmission of data across nodes, which would cause the occupation of transmission resources.

[0060] In the distributed storage system, the basic data is distributedly stored in the distributed system in the form of data blocks; the data blocks included in the storage data space are distributedly stored in the distributed system; and the distributed system includes multiple storage nodes.

[0061] See also Figure 2 , Figure 2 This is a flow chart of a method for determining a target data block shown in the present application.

[0062] S202: In response to each data block included in the storage data space being empty or having a default value, determine a current data block currently participating in merging in the basic data.

[0063] It is understandable that each data block being empty or having a default value may correspond to a variety of situations, for example, when a log merge is just completed, the data block data is cleared. For another example, when data is just beginning to be stored, the data block is empty or has a default value. This application does not exhaustively list the above situations.

[0064] In this step, the data that needs to be merged in the basic data will be marked. For example, the basic data is divided into many data blocks and stored in different storage nodes, and each data block that needs to be merged is labeled. By identifying the label, Jack determines the current data block that needs to be merged.

[0065] S204: Use the data block allocated to the data storage space in the current storage node corresponding to the current data block as the target data block.

[0066] The distributed storage system includes many nodes, and the data blocks included in the storage data space are also allocated in different nodes. After determining the current data block in S202, the current storage node corresponding to the current data block can be determined, and then the data block allocated to the storage data space in the current storage node is used as the target data block. It can be understood that in the case where each data block is empty or the default value may correspond to multiple situations, the target data block is the first data block of the data block that is first written into the incremental data participating in the erasure code calculation.

[0067] Through S202-S204, the data block in the same storage node as the basic data block involved in the merge can be used as the target data block to store log data. When the merge is triggered later, the data merge can be implemented locally to reduce data transmission across nodes.

[0068] For example, in a distributed storage system, assume that the log data storage space is divided into 4 data blocks (D0, D1, D2, D3), which are stored in different storage nodes (node ​​A, node B, node C, node D). When the system starts, these data blocks may be empty or default values. The basic data is divided into 10 data blocks (E0 to E9), which are stored in different storage nodes. The system can mark the current data block that needs to be merged among the 10 data blocks, assuming that E0 is identified as the current data block that needs to be merged.

[0069] The system detects that each data block is empty. For example, when data is first stored, D0, D1, D2, and D3 are all empty.

[0070] After the system determines that the current data block is E0 through the query identifier, it can find the current storage node where E0 is located (assuming it is node A).

[0071] The system searches for data blocks related to the log data blocks in node A and uses them as target data blocks. For example, if node A has a stored block D0, then D0 is the target data block and is used to store log data first.

[0072] The system outputs the log data to the target data block D0 of node A to facilitate subsequent merge operations.

[0073] Doing so can ensure that related data is in the same node, avoiding cross-node transmission and reducing resource usage.

[0074] When the target data block D0 and the current data block E0 in node A are ready, the system triggers a data merge operation locally. After the merge is completed, the data in D0 is updated, and since it is a local operation, network bandwidth is saved.

[0075] Through the above steps, the system effectively uses local storage nodes for log storage and data merging, reducing the need for cross-node data transmission, thereby improving overall performance and resource utilization.

[0076] In some embodiments, in response to completing the log merge, the method further includes: clearing the data in each data block included in the storage data space; the storage data space is still reserved and not released. In the related art, the storage space is often released after the log merge task is completed, so the storage space needs to be reallocated when the log data is subsequently stored, and the storage efficiency is low. In this example, after completing the log merge task, only the data is cleared, but the space is not released, which increases the rate of writing incremental logs.

[0077] For example, the log data storage space includes four data blocks D0 to D4. After a log merge operation is triggered, the data in D0 to D4 can be cleared, for example, filled with default values, but this storage space is still reserved for the subsequent log data storage. In this way, after the system completes the log merge, it only clears the data blocks but retains the storage space, which not only optimizes the use of storage resources, but also increases the write rate of subsequent log data. This strategy reduces the repeated application of storage space and improves the overall system performance.

[0078] In some scenarios, log merging may be triggered frequently. In such scenarios, the calculation frequency of erasure codes is very high. In some embodiments, for such scenarios, the amount of data acquisition can be further reduced, and the occupancy of data transmission channels and computing resources can be reduced. This application proposes two ways to further reduce the amount of data acquisition, the occupancy of data transmission channels, and the occupancy of computing resources.

[0079] Method 1:

[0080] The first number of data blocks are arranged in an order, wherein the data block to which the incremental data is first written is data block number one;

[0081] Methods for writing incremental data include:

[0082] In response to receiving a write instruction for the target incremental data, if the data block currently being written is the first data block, generating a first matrix based on the target incremental data, and multiplying the first matrix with a preset coding matrix to determine a current erasure code;

[0083] Based on the target incremental data, the target data in the target data block starting from the write position and having a length of the data length is updated, and based on the current erasure code, the target check data in the check block starting from the write position and having a length of the data length is updated.

[0084] In method one, when incremental data can be written, it is determined whether data block number one is to be written. If so, the data in other data blocks are useless. Therefore, the erasure code calculation and update can be performed only based on the target incremental data that needs to be written at present, without obtaining data from other data blocks, thereby further reducing the amount of data acquisition, reducing the occupancy of data transmission channels, and reducing the occupancy of computing resources. For the method of using the target incremental data to calculate the erasure code, please refer to the introduction in method two.

[0085] Method 2:

[0086] See also Figure 3 , Figure 3 This is a flow chart of a method for writing incremental data shown in this application. Figure 3 As shown, the method may include S302-S308.

[0087] S302, before writing the incremental data, obtaining the incremental data volume of two adjacent historical log merges based on the historical data.

[0088] In some methods, the incremental data volume can be calculated by calculating an average value. The historical data includes the incremental data volume written in every two adjacent log merges, and the incremental volume can be obtained by calculating the average value.

[0089] S304: Compare the incremental data amount with the size of a data block in the data storage space.

[0090] S306, in the case where the size of the incremental data amount is smaller than the size of a data block in the data storage space, the method of writing incremental data includes: in response to receiving a write instruction for the target incremental data, generating a first matrix based on the target incremental data, multiplying it with a preset coding matrix, and determining a current erasure code.

[0091] The size of the incremental data is smaller than the size of a data block in the data storage space, which means that before the log is merged, the incremental data may only be written into the first data block, and the data in other data blocks is meaningless. Therefore, the erasure code calculation and update can be performed only based on the target incremental data.

[0092] In this step, the format of the first matrix can be preset, and the target incremental data can be filled into the matrix. The preset encoding matrix can be:

[0093] Referring to formula 3, D0 ′ That is the target incremental data, D1 ′ 、D2 ′ 、D3 ′ Both can be filled with 0, and the formula 4 is obtained:

[0094]

[0095] Then the calculation of the current erasure code can be simplified, and there is no need to obtain D1 ′ 、D2 ′ 、D3 ′ The real data further reduces the data transmission and calculation process.

[0096] S308, based on the target incremental data, updating the target data in the target data block starting from the write position and having a length of the data length, and based on the current erasure code, updating the target check data in the check block starting from the write position and having a length of the data length.

[0097] In this step, D0 ′ Write the target data block and write the erasure code into the check block respectively.

[0098] In some ways, the accuracy of determining the amount of incremental data can be improved, thereby improving the correctness of data writing. The historical data includes the average of the amount of incremental data merged from two adjacent historical logs, which is counted according to each time period within each day. Each day can be divided into several fixed time periods, for example, each time period is 6 hours, so there are 4 time periods per day, and then the average of the amount of incremental data merged from two adjacent historical logs in each time period is calculated respectively.

[0099] For example, time period 1 (00:00-06:00): incremental data volume = 1MB; time period 2 (06:00-12:00): incremental data volume = 2MB; time period 3 (12:00-18:00): incremental data volume = 3MB; time period 4 (18:00-24:00): incremental data volume = 5MB.

[0100] The step of obtaining the incremental data amount of two adjacent historical log merges based on the historical data includes:

[0101] According to the current time, query the historical data to determine the incremental data amount;

[0102] The method further comprises:

[0103] In a case where the size of the incremental data amount is smaller than the size of a data block in the data storage space, the method of writing incremental data is executed within a time period corresponding to the current moment.

[0104] Taking the above time period as an example, assuming that the current time is 7 am, it can be determined that it falls within time period 2, and the incremental data volume is 2MB. Assuming that the size of a data block is 2.5MB, it can be determined that the incremental data volume generated by two adjacent log merges in time period 2 is less than one data block. Then, the erasure code calculation process can be simplified in time period 2 using a method such as formula 4, that is, only log data is needed, and the erasure code calculation can be completed without the data of other data blocks, which reduces data transmission and calculation. This simplified operation is only performed in the corresponding time period to avoid data writing errors caused by the simplified operation. It can be understood that the granularity of dividing the time period has a certain influence on the accuracy of data writing. Generally speaking, the smaller the time period granularity, the higher the accuracy, but it will be meaningless if the granularity is too small. In some embodiments, the length of the above time period is in the range of 0.5 hours to 1.5 hours.

[0105] This application also proposes an incremental data writing system based on incremental erasure coding. Figure 4 , Figure 4 This is a schematic diagram of the structure of an incremental data writing system based on incremental erasure code shown in the present application. In the system, an incremental log storage data space is pre-created, and the storage data space includes a first number of data blocks for storing incremental data, and a second number of check blocks for storing check data for the incremental data; the first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of check blocks set in the erasure code calculation rule;

[0106] like Figure 4 As shown, the system 400 includes:

[0107] A generating module 410 is used to generate a write instruction for target incremental data in a target incremental log; the write instruction is used to write the target incremental data into a target data block included in the storage data space; the write instruction includes a data length of the target incremental data;

[0108] An acquisition module 420 is used to acquire, according to the write position of the target incremental data in the target data block and the data length, target data starting from the write position and having a length equal to the data length in each data block included in the data storage space;

[0109] A determination module 430, configured to determine an incremental erasure code according to the target data in each data block;

[0110] The updating module 440 is used to update the target verification data starting from the writing position and having a length of the data length in each verification block included in the data storage space according to the incremental erasure code.

[0111] In some embodiments, the system 400 further includes:

[0112] The merging module is used to merge the data in each data block included in the storage data space with the basic data in response to detecting a preset trigger condition, so as to complete the log merging.

[0113] In some embodiments, the basic data is distributed and stored in the distributed system 400 in the form of data blocks; each data block included in the storage data space is distributed and stored in the distributed system 400; the distributed system 400 includes a plurality of storage nodes; the system 400 further includes a target data block determination module, which is used to:

[0114] In response to each data block included in the storage data space being empty or having a default value, determining a current data block currently participating in merging in the basic data;

[0115] The data block allocated for the data storage space in the current storage node corresponding to the current data block is used as the target data block; the target data block is the first data block into which the incremental data is first written.

[0116] In some embodiments, the system 400 further includes:

[0117] The clearing module is used to clear the data in each data block included in the storage data space in response to completing the log merging; the storage data space is still reserved and not released.

[0118] In some embodiments, the first number of data blocks are ordered, wherein the data block to which the incremental data is first written is data block number 1; the system 400 further includes a first writing module, which is used to:

[0119] In response to receiving a write instruction for the target incremental data, if the data block currently being written is the first data block, generating a first matrix based on the target incremental data, and multiplying the first matrix with a preset coding matrix to determine a current erasure code;

[0120] Based on the target incremental data, the target data in the target data block starting from the write position and having a length of the data length is updated, and based on the current erasure code, the target check data in the check block starting from the write position and having a length of the data length is updated.

[0121] In some embodiments, the system 400 further includes a comparison module configured to:

[0122] Before writing incremental data, obtain the incremental data volume of two adjacent historical log merges based on historical data;

[0123] Comparing the incremental data amount with the size of a data block in the data storage space;

[0124] The second writing module is used to:

[0125] In a case where the size of the incremental data amount is smaller than the size of a data block in the data storage space, in response to receiving a write instruction for the target incremental data, generating a first matrix based on the target incremental data, multiplying the first matrix with a preset coding matrix, and determining a current erasure code;

[0126] Based on the target incremental data, the target data in the target data block starting from the write position and having a length of the data length is updated, and based on the current erasure code, the target check data in the check block starting from the write position and having a length of the data length is updated.

[0127] In some embodiments, the historical data includes an average of the incremental data volume of two adjacent historical logs combined according to each time period within each day;

[0128] The comparison module further:

[0129] According to the current time, query the historical data to determine the incremental data amount;

[0130] The writing module further:

[0131] In a case where the size of the incremental data amount is smaller than the size of a data block in the data storage space, the method of writing incremental data is executed within a time period corresponding to the current moment.

[0132] In some embodiments, the time period is between 0.5 hours and 1.5 hours.

[0133] In some embodiments, the size of the storage data space is the same as a preset incremental log size.

[0134] The description of the relevant embodiments can refer to the previous embodiments.

[0135] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CDROM, optical storage, etc.) containing computer-usable program code.

[0136] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".

[0137] Each embodiment in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0138] The above describes a specific embodiment of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0139] The subject matter described in this application and the embodiments of the functional loading can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that can include the structure disclosed in this application and its structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, that is, one or more modules in the computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or control the loading of a data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0140] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by loading according to input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special-purpose logic circuits.

[0141] A computer suitable for executing a computer program may include, for example, a general and / or special purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or the computer will be loadably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0142] Computer readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0143] Although the application includes many specific implementation details, these should not be interpreted as limiting any disclosed scope or the scope of protection claimed, but are mainly used to describe the features of specific disclosed specific embodiments. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work in some combinations as described and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0144] Similarly, although loads are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these loads be executed in the particular order shown or executed sequentially, or requiring that all illustrated loads be executed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0145] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0146] The above are only preferred embodiments of one or more embodiments of the present application and are not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of one or more embodiments of the present application.

Claims

1. A method for writing incremental data based on incremental erasure coding, characterized in that: Pre-creating an incremental log storage data space, the storage data space comprising a first number of data blocks for storing incremental data, and a second number of check blocks for storing check data for the incremental data; The first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of check blocks set in the erasure code calculation rule; The method comprises: Generate a write instruction for the target incremental data in the target incremental log; the write instruction is used to write the target incremental data into the target data block included in the storage data space; the write instruction includes the data length of the target incremental data; According to the write position of the target incremental data in the target data block and the data length, obtaining the target data starting from the write position and having a length equal to the data length in each data block included in the data storage space; Determine an incremental erasure code according to the target data in each data block; According to the incremental erasure code, target verification data starting from the write position and having a length equal to the data length in each verification block included in the data storage space is updated.

2. The incremental data writing method based on incremental erasure coding according to claim 1, characterized in that: The method further comprises: In response to detecting a preset trigger condition, the data in each data block included in the storage data space is merged with the basic data to complete the log merge.

3. The incremental data writing method based on incremental erasure coding according to claim 2, characterized in that: The basic data is distributed and stored in the distributed system in the form of data blocks; each data block included in the storage data space is distributed and stored in the distributed system; The distributed system includes a plurality of storage nodes; The method for determining the target data block includes: In response to each data block included in the storage data space being empty or having a default value, determining a current data block currently participating in merging in the basic data; The data block allocated for the data storage space in the current storage node corresponding to the current data block is used as the target data block; the target data block is the first data block into which the incremental data is first written.

4. The incremental data writing method based on incremental erasure coding according to claim 2, characterized in that: In response to completing the log merging, the method further includes: The data in each data block included in the data storage space is cleared; the data storage space is still reserved and not released.

5. The incremental data writing method based on incremental erasure coding according to claim 2, characterized in that: The first number of data blocks are arranged in an order, wherein the data block to which the incremental data is first written is data block number one; Methods for writing incremental data include: In response to receiving a write instruction for the target incremental data, if the data block currently being written is the first data block, generating a first matrix based on the target incremental data, and multiplying the first matrix with a preset coding matrix to determine a current erasure code; Based on the target incremental data, the target data in the target data block starting from the write position and having a length of the data length is updated, and based on the current erasure code, the target check data in the check block starting from the write position and having a length of the data length is updated.

6. The incremental data writing method based on incremental erasure coding according to claim 2, characterized in that: Before writing the incremental data, the method further includes: Based on historical data, obtain the incremental data volume of two consecutive historical log merges; Comparing the incremental data amount with the size of a data block in the data storage space; In the case where the size of the incremental data amount is smaller than the size of a data block in the data storage space, the method for writing incremental data includes: In response to receiving a write instruction for the target incremental data, generating a first matrix based on the target incremental data, and multiplying the first matrix with a preset coding matrix to determine a current erasure code; Based on the target incremental data, the target data in the target data block starting from the write position and having a length of the data length is updated, and based on the current erasure code, the target check data in the check block starting from the write position and having a length of the data length is updated.

7. The incremental data writing method based on incremental erasure coding according to claim 6, characterized in that: The historical data includes the average of the incremental data volume of two adjacent historical logs combined according to each time period within each day; The step of obtaining the incremental data amount of two adjacent historical log merges based on the historical data includes: According to the current time, query the historical data to determine the incremental data amount; The method further comprises: In a case where the size of the incremental data amount is smaller than the size of a data block in the data storage space, the method of writing incremental data is executed within a time period corresponding to the current moment.

8. The incremental data writing method based on incremental erasure coding according to claim 7, characterized in that: The duration of the time period is in the range of 0.5 hours to 1.5 hours.

9. The incremental data writing method based on incremental erasure coding according to claim 1, characterized in that: The size of the storage data space is the same as the preset incremental log size.

10. An incremental data writing system based on incremental erasure coding, characterized in that: Pre-creating an incremental log storage data space, the storage data space comprising a first number of data blocks for storing incremental data, and a second number of check blocks for storing check data for the incremental data; The first number is the number of data blocks set in the erasure code calculation rule, and the second number is the number of check blocks set in the erasure code calculation rule; The system comprises: A generating module, used for generating a write instruction for target incremental data in a target incremental log; the write instruction is used for writing the target incremental data into a target data block included in the storage data space; the write instruction includes a data length of the target incremental data; An acquisition module, configured to acquire, according to a write position of the target incremental data in the target data block and the data length, target data starting from the write position and having a length equal to the data length in each data block included in the data storage space; A determination module, configured to determine an incremental erasure code according to the target data in each data block; An update module is used to update target verification data starting from the write position and having a length of the data length in each verification block included in the data storage space according to the incremental erasure code.

Images