Data processing method, electronic equipment and storage medium

Abstract

The invention discloses a data processing method, electronic equipment and a storage medium, and relates to the technical field of data processing, a plurality of check bit nodes and a plurality of data bit nodes are determined according to the data block size and the data block offset of a data block, then the check bit nodes and the data bit nodes are used for caching data of the data block, and the data processing efficiency is improved. And then, the erasure code is directly calculated in situ through the check bit node, so that the data forwarding flow between the distributed storage system and the cache node and the data node is prevented from being indirectly erased, the links of data block processing are reduced, the write-in amount of the whole data is reduced, and the processing performance of the full-flash storage is improved. The technical problem of how to improve the data erasure correction efficiency can be solved, and the technical effects of reducing data block processing links and improving the data erasure correction efficiency are achieved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a method for data processing, an electronic device, and a storage medium. Background Art

[0002] Storage system redundancy technologies mainly include Redundant Array of Independent Disks (RAID) and erasure codes. RAID mainly distributes and replicates data on multiple disks through striping and mirroring technologies to provide different levels of fault tolerance and performance improvement. Erasure codes are technologies that generate additional parity blocks through algorithms to protect data. When some data is lost, the remaining original data blocks and parity blocks can be used to recover the data.

[0003] When processing small data blocks through erasure codes, an indirect erasure method is usually adopted, that is, after caching and aggregating the data into strip data, erasure code calculation is performed, and each calculation is based on a complete strip data. However, in the existing data erasure methods, since the data needs to be written into the buffer area first, then the complete strip data is read, and after the calculation is completed, it is written into a new address space, the number of data read and write operations and the number of cross-node interaction operations need to be increased, resulting in low data erasure efficiency. Summary of the Invention

[0004] This application provides a method for data processing, an electronic device, and a storage medium, so as to at least solve the problem of low data erasure efficiency in related technologies.

[0005] This application provides a method for data processing, including:

[0006] Determine data nodes according to the data block size and data block offset of the data blocks in the data to be processed, so as to obtain multiple parity bit nodes corresponding to the data to be processed and multiple data bit nodes corresponding to each of the multiple data blocks, where the data to be processed includes multiple data blocks;

[0007] When the data block size meets the preset size condition, store the first data block into the data bit node corresponding to the first data block and multiple parity bit nodes respectively, where the first data block is the first data block in the data to be processed;

[0008] When the data block size does not meet the preset size condition, perform data splitting processing on the first data block to obtain split data blocks, and store the split data blocks into the data bit node corresponding to the first data block and multiple parity bit nodes respectively;

[0009] Until all data blocks are stored in their respective corresponding data bit nodes and multiple parity bit nodes, data erasure calculation processing is performed on the multiple parity bit nodes respectively to obtain multiple parity data blocks corresponding to the data to be processed.

[0010] The present application also provides a data processing device, including:

[0011] A first determination unit, configured to determine data nodes according to the data block size and data block offset in the data to be processed, so as to obtain multiple parity bit nodes corresponding to the data to be processed and multiple data bit nodes corresponding to each of the multiple data blocks, where the data to be processed includes multiple data blocks; [[ID=,8]]

[0012] A first storage unit, configured to store the first data block in the corresponding data bit node of the first data block and multiple parity bit nodes respectively when the data block size meets the preset size condition, where the first data block is the first data block in the data to be processed;

[0013] A splitting unit, configured to perform data splitting processing on the first data block when the data block size does not meet the preset size condition to obtain split data blocks;

[0014] A second storage unit, configured to store the split data blocks in the corresponding data bit node of the first data block and multiple parity bit nodes respectively;

[0015] An erasure unit, configured to perform data erasure calculation processing on the multiple parity bit nodes respectively until all data blocks are stored in their respective corresponding data bit nodes and multiple parity bit nodes, so as to obtain multiple parity data blocks corresponding to the data to be processed.

[0016] The present application also provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any one of the above data processing methods when executing the computer program.

[0017] The present application also provides a computer-readable storage medium, in which a computer program is stored, where the computer program implements the steps of any one of the above data processing methods when executed by a processor.

[0018] The present application also provides a computer program product, including a computer program, where the computer program implements the steps of any one of the above data processing methods when executed by a processor.

[0019] A method, an electronic device, and a storage medium for data processing according to the present application determine a plurality of parity bit nodes and a plurality of data bit nodes based on the data block size and the data block offset of a data block, and then use the parity bit nodes and the data bit nodes to cache the data block data. After that, the erasure code is directly calculated in place through the parity bit nodes to avoid the indirect erasure of the data forwarding traffic between the cache nodes and the data nodes in the distributed storage system, reduce the links of data block processing, reduce the overall data write volume, and improve the processing performance of the all-flash storage. Therefore, the technical problem of how to improve the efficiency of data erasure can be solved, and the technical effect of reducing the links of data block processing and improving the efficiency of data erasure can be achieved. Description of the Drawings

[0020] To more clearly illustrate the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 A flowchart of a method for data processing provided by an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the principle of a method for data processing provided by an embodiment of the present application;

[0023] Figure 3 An example diagram of a method for data processing provided by an embodiment of the present application;

[0024] Figure 4 A schematic diagram of the structure of a device for data processing provided by an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of another device for data processing provided by an embodiment of the present application. Detailed Embodiments

[0026] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts belong to the protection scope of the present application.

[0027] It should be noted that in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device including a series of elements not only includes those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects and not to describe a specific order or sequence.

[0028] To enable those skilled in the art of this technical field to better understand the solution of this application, the following further detailed description of this application will be given in conjunction with the accompanying drawings and specific embodiments.

[0029] Figure 1 A flowchart of a data processing method provided for an embodiment of this application is shown. In combination with the execution process of the data processing method, the method will be described in detail.

[0030] As Figure 1 shown, the data processing method includes:

[0031] Step 101, determine data nodes according to the data block size and data block offset in the data to be processed, to obtain a plurality of check bit nodes corresponding to the data to be processed and a plurality of data bit nodes corresponding to each of the plurality of data blocks, where the data to be processed includes a plurality of data blocks.

[0032] In the embodiments of this application, the data block size of a data block refers to the actual storage capacity size of the data block, which is determined according to the actual situation of the data block. For example: 4 kilobytes (kb), 6 kb, etc. The data block offset refers to the logical position coordinate of the data block in the complete file or data stream, which is determined according to the actual situation of the data block. Specifically, this application does not limit the data block size and the data block offset.

[0033] It should be noted that in the data to be processed, the data block size and the data block offset of each data block are the same.

[0034] A data bit node refers to a node that stores the data block of the data to be processed. Each data block is mapped to a specific data bit node according to the data block offset; a check bit node corresponds to a dedicated node that stores the subsequent generated erasure check data, and its quantity matches the number of check blocks in the erasure code redundancy strategy.

[0035] To facilitate understanding of the implementation process of this application, this application provides a schematic diagram of the principle of a data processing method. As Figure 2 shown, where the data to be processed includes M data blocks, there are M data bit nodes and N check bit nodes, and the check block is the check data block.

[0036] Determination of parity bit nodes and data bit nodes specifically refers to that the user writes to determine the storage nodes of M data bit nodes and N parity bit nodes of the erasure code according to the data block offset and data block size of the data block. When the data block size of the write request meets the preset size condition, N parity data blocks are directly calculated, and then the data block is written to the data bit node, and the parity block is written to the parity bit node;

[0037] Specifically, the determination of parity bit nodes and data bit nodes can be implemented by, but not limited to, the following method: Use data block offset analysis to calibrate the positions of data blocks in the data to be processed. By parsing the data block offset and the preset erasure code strip mapping table, multiple data bit nodes and multiple parity bit nodes corresponding to each data block are determined, where the preset erasure code strip mapping table is a metadata set of the pre-set storage node topology structure and data distribution rules.

[0038] Step 102, when the data block size meets the preset size condition, store the first data block to the data bit node corresponding to the first data block and multiple parity bit nodes respectively, where the first data block is the first data block in the data to be processed.

[0039] In the embodiment of the present application, the preset size condition is a custom-set condition. For example: whether it meets less than the preset data bit storage size (preset strip size), etc. Specifically, the present application does not limit the preset size condition.

[0040] When it is detected that the data block size is less than or equal to the preset data bit storage size (that is, the preset size condition is met), enter the direct write processing mode. Taking the first data block (the first data block) of the data to be processed as an example, the first data block will be written to the corresponding data bit node and all associated parity bit nodes at the same time. The storage operation here can be, but not limited to: the replica write mechanism, creating an original data replica at the data bit node, and establishing a temporary cache replica at the parity bit node. This dual-path write strategy ensures data redundancy and security while establishing a data basis for subsequent local calculations.

[0041] Specifically, the steps of this application may refer to: when the data block of the write request is smaller than the erasure stripe data, that is, the data block size meets the preset size condition, the data block is directly written to the corresponding data bit nodes according to the data block offset, and copies are written to the parity bit nodes in the form of N + 1 copies. The data block is redundantly stored in the form of copies to ensure data reliability. The writing method can be main data bit forwarding (that is, forwarded to multiple parity bit nodes through the data bit nodes), or it may be directly written to the data bit nodes and multiple parity bit nodes. At the same time, during the process of writing the data block, that is, during the process of storing the first data block to the data bit nodes and multiple parity bit nodes respectively, the read request can directly read the data in the data bit nodes or parity bit nodes; and the fault handling is processed according to the N + 1 copy redundancy, and the data can be guaranteed to have no single point of failure loss through the copy mechanism.

[0042] Regarding the description of writing copies to the parity bit nodes in the N + 1 copy mode, taking the first data block as an example, it means that when writing the first data block to the corresponding data bit nodes and multiple parity bit nodes, multiple copy data consistent with the first data block are generated, and each copy data is respectively stored in all the parity bit nodes. The first data block is written to the data bit node corresponding to the first data block as the original data. If the number of parity bit nodes is N, then at this time, the first data block is equivalent to having N identical data, plus the original data of the first data block in the data bit node itself, which is equivalent to having N + 1 first data blocks.

[0043] Step 103, in the case where the data block size does not meet the preset size condition, perform data splitting processing on the first data block to obtain split data blocks, and store the split data blocks to the data bit nodes corresponding to the first data block and multiple parity bit nodes respectively.

[0044] In the embodiments of this application, for data blocks that do not meet the preset size condition, that is, data blocks larger than the preset data bit storage size (preset stripe size), the adaptive sharding mechanism is started. When performing data splitting processing, the following methods can be used but are not limited to: logically reorganize in combination with the erasure code stripe boundary, split the data block larger than the preset stripe size to generate split data blocks that meet the strip alignment requirements. The split data units (split data blocks) still maintain the physical continuity of the original data, and the dynamic aggregation ability with subsequent data blocks is achieved through metadata marking. The reorganized data blocks also perform multi-node copy writing to ensure seamless participation in erasure calculation in the subsequent processing stage.

[0045] For the convenience of understanding the embodiments of this application, refer to Figure 2, a further explanation is given. Among them, the preset size condition is set to whether the data block size is greater than the preset data bit storage size (preset striping size). The preset data bit storage size is the actual storage size of the data bit node, which is a preset size, such as 4kb, 8kb, etc. When the data block size of the data block in the data to be processed is 4kb and the preset data bit storage size is 4kb, since the data block size is not greater than the preset data bit storage size, at this time, it can be considered that the data block size meets the preset size condition; when the data block size of the data block in the data to be processed is 6kb and the preset data bit storage size is 4kb, since the data block size is greater than the preset data bit storage size, at this time, it can be considered that the data block size does not meet the preset size condition, and the data block needs to be split. For example: the first data block (data block 1) is split into a 4kb-sized data block and a 2kb-sized data block, that is, the data block is split, and then the 4kb-sized data block is directly stored in the corresponding data bit node (data bit node 1), and the 2kb-sized data block is aggregated with the second data block (data block 2) to obtain an 8kb second data block, and then the second data block is split again, and the split second data block is stored in the data bit node and the parity bit node.

[0046] Step 104. Until all data blocks are stored in their respective corresponding data bit nodes and multiple parity bit nodes, data erasure calculation processing is performed on each of the multiple parity bit nodes to obtain multiple parity data blocks corresponding to the data to be processed.

[0047] In the embodiment of the present application, when all data blocks are written to the nodes, the local erasure calculation processing of the parity bit nodes is started. The local erasure calculation processing refers to directly using the local computing resources of the nodes to perform encoding operations without collecting data across nodes. Each parity bit node performs finite field operations on the cached replica data (all data blocks) based on a pre-loaded encoding matrix (such as the generating matrix of the Reed-Solomon algorithm) to generate parity data blocks that match the erasure code configuration. The parity data blocks are mathematical redundancy units for data reconstruction.

[0048] Until all data blocks are stored in their respective corresponding data bit nodes and multiple parity bit nodes means that the data blocks are written in the N+1 replica mode. The data bit nodes are calculated according to the offset, and the replica data is directly written into the parity bit nodes. The data blocks are sent to different data bit nodes in sequence, and the replica data of the data blocks is sent to multiple parity bit nodes in parallel. After multiple storages, the data blocks gradually accumulate in the parity bit nodes until the data length size is equal to the size of the data to be processed, and then the parity bit nodes trigger local erasure calculation to generate parity data blocks.

[0049] A method, an electronic device, and a storage medium for data processing according to the present application determine a plurality of parity bit nodes and a plurality of data bit nodes based on the data block size and data block offset of a data block, and then use the parity bit nodes and data bit nodes to cache the data block data. Then, the erasure code is directly calculated in place through the parity bit nodes, so as to avoid the indirect erasure of the data forwarding traffic between the cache nodes and data nodes in the distributed storage system, reduce the data block processing links, reduce the overall data write volume, and improve the processing performance of the all-flash storage. Therefore, the technical problem of how to improve the efficiency of data erasure can be solved, and the technical effect of reducing the data block processing links and improving the efficiency of data erasure can be achieved.

[0050] In an implementable manner of the embodiment of the present application, when determining whether the data block size meets the preset size condition, the following manner can also be adopted but is not limited to: when the data block size is less than or equal to the preset striping size, it is determined that the data block size meets the preset size condition, where the preset striping size is the storage data size of the data bit node set in advance; when the data block size is greater than the preset striping size, it is determined that the data block size does not meet the preset size condition.

[0051] In the embodiment of the present application, the preset striping size is the standard data capacity of a single storage operation of the predefined data bit node, that is, the actual storage size of the data bit node, and is used to judge the storage form adaptability of the data block. When it is detected that the data block size is less than or equal to the preset striping size (that is, the preset size condition is met), it is determined that the data block can be directly stored. At this time, the data block is directly and completely written into the corresponding data bit node, and a temporary copy is generated in the parity bit node. This mechanism ensures the write efficiency of the data block in the distributed storage environment.

[0052] For data blocks larger than the preset striping size (that is, the case where the preset size condition is not met), a data splitting processing engine will be started, for example: a calculation module that realizes intelligent splitting of data blocks. The data block is decomposed into multiple split data blocks according to the preset striping size. The split data blocks include multiple split units, that is, the first split data block, the second split data block, etc. Each split unit inherits the logical offset attribute of the original data block. The split units will perform a dual write operation in parallel, storing both in the data bit node and in the parity bit node process.

[0053] By directly adopting the standard write path for data blocks smaller than or equal to the preset striping size, the processing delay can be reduced; for data blocks larger than the preset striping size, the storage form can be standardized through intelligent splitting. The utilization rate of storage resources can be optimized, the waste of unused storage space can be avoided, and the cross-node data reorganization operation can be reduced, reducing the computational complexity and network transmission overhead of the distributed storage system.

[0054] In an implementable manner of an embodiment of the present application, when storing the first data block in the corresponding data bit nodes and multiple parity bit nodes respectively, the following manner can be adopted but is not limited thereto: when the size of the data block is equal to the preset strip size, store the first data block in the data bit node corresponding to the first data block, and generate multiple first copy data according to the first data block, and store the multiple first copy data in the multiple parity bit nodes respectively, wherein the number of the first copy data is the same as the number of the parity bit nodes; when the size of the data block is less than the preset strip size, store the first data block in the data bit node corresponding to the first data block, and generate multiple first copy data according to the first data block, and store the multiple first copy data in the multiple parity bit nodes respectively; determine the first difference between the preset strip size and the data block size, and determine the first repair data block according to the subsequent data blocks in the data to be processed and the first difference, wherein the size of the first repair data block is the same as the first difference, and the subsequent data blocks are the data blocks that are continuously adjacent after the first data block in the data to be processed; store the first repair data block in the data bit node corresponding to the first data block and the multiple parity bit nodes respectively.

[0055] In an embodiment of the present application, when the size of the data block is equal to the preset strip size, the standard copy writing process can be directly started, and the first data block will be directly written into the corresponding data bit node. At the same time, multiple first copy data (the number is equal to the total number of parity bit nodes, ensuring that each parity bit node obtains a complete copy) are generated according to the preset redundancy policy (i.e., the N + 1 copy mechanism), and the first copy data are written into all parity bit nodes in parallel. The atomic write protocol is used to ensure data consistency, so that the original data block and the copy data form instant redundancy protection in the distributed storage environment.

[0056] When the size of the data block is less than the preset strip size, a data compensation mechanism needs to be introduced on the basis of the standard copy writing. First, the first data block is still written into the corresponding data bit node according to the established rules, and an equal amount of first copy data are generated and distributed to each parity bit node. Subsequently, calculate the first difference between the preset strip size and the data block size (i.e., the vacancy between the standard capacity and the actual capacity of the data bit node), and start dynamic data repair, extract compensation data from the subsequent data blocks of the data to be processed, and generate the first repair data block (the first repair data block is a supplementary data unit whose size can exactly match the first difference). The first repair data block inherits the logical offset attribute of the original data block and also performs a dual writing operation: both supplementing and writing into the data bit node to form a complete strip structure, and generating corresponding copies at the parity bit node.

[0057] Further, referring to Figure 2For example, if the data block size of the data block in the data to be processed is 2 kb and the preset data bit storage size (preset striping size) is 4 kb, the first data block (data block 1) can be directly stored in the corresponding data bit node (data bit node 1). At the same time, the first difference can be determined to be 2 kb. Then, the first repair data block is determined according to the subsequent data blocks of the first data block (data block 1). Since the data block size is 2 kb, the second data block (data block 2) as a whole can be used as the first repair data block and stored in the data bit node (data bit node 1) corresponding to the first data block;

[0058] If the data block size of the data block in the data to be processed is 3 kb and the preset data bit storage size (preset striping size) is 4 kb, the first data block (data block 1) can also be directly stored in the corresponding data bit node (data bit node 1). At the same time, the first difference can be determined to be 1 kb. Then, the first repair data block is determined according to the subsequent data blocks of the first data block (data block 1). Since the data block size is 3 kb, 1 kb of data can be extracted from the second data block (data block 2) as the first repair data block and stored in the data bit node (data bit node 1) corresponding to the first data block;

[0059] If the data block size of the data block in the data to be processed is 1 kb and the preset data bit storage size (preset striping size) is 4 kb, the first data block (data block 1) can also be directly stored in the corresponding data bit node (data bit node 1). At the same time, the first difference can be determined to be 3 kb. Then, the first repair data block is determined according to the subsequent data blocks of the first data block (data block 1). Since the data block size is 1 kb, the second data block (data block 2) plus the third data block (data block 3) plus the fourth data block (data block 4) need to be used as the first repair data block and stored in the data bit node (data bit node 1) corresponding to the first data block.

[0060] Furthermore, it should be noted that the generation process of the first repair data block can be implemented by, but not limited to: using the sliding window algorithm. When writing the current data block, the system pre-reads the header data of the subsequent data block (with a length equal to the first difference) to form a compensation data unit. The data access latency is hidden through the pipeline operation by the forward data interception mechanism, and at the same time, the copy-on-write technology is used to ensure the integrity of the original data. When the subsequent data block is officially processed, the data segment that has been compensated and used will be automatically skipped to achieve seamless connection of the storage space.

[0061] The storage form standardization of incomplete data blocks is achieved through a dynamic compensation mechanism. The generation of patched data blocks does not involve filling invalid data but intelligently intercepting subsequent valid data, which not only avoids waste of storage space but also maintains data continuity. The compensation operation is synchronized with the replica writing to ensure that the data redundancy before erasure code conversion always meets the N+1 replica protection requirement. After receiving the patched data, the data bit nodes automatically reconstruct the complete stripe, laying a data foundation for subsequent local erasure calculation. It can complete the data form conversion without interrupting the writing process, significantly reducing the latency jitter caused by cache waiting in traditional solutions.

[0062] In an implementable manner of the embodiment of the present application, when performing data splitting on the first data block, the following manner can also be adopted but is not limited to: splitting the first data block according to a preset strip size to obtain a first split data block and a second split data block; wherein, the size of the first split data block is equal to the preset strip size, and the split data blocks include the first split data block and the second split data block, and the second split data block is the adjacent data block after the first split data block in the first data block.

[0063] In the embodiment of the present application, regarding the splitting of the first data block, it can be achieved by but not limited to: the boundary alignment algorithm for splitting, accurately dividing the first data block into two logically continuous data segments, namely the first split data block and the second split data block. Among them, the first split data block is a standardized data unit with a size equal to the preset strip size, and the second split data block is a continuous data unit containing the remaining data of the first data block. During the data splitting process, the following can also be adopted but is not limited to: the sliding window technique, performing byte-level boundary calibration based on the preset strip size to ensure that the first split data block completely fills the corresponding data bit nodes of the first data block, and the second split data block enters the subsequent processing flow as a continuation unit to be processed.

[0064] Refer to Figure 2 For example, if the data block size of the data block in the data to be processed is 6 kb and the preset data bit storage size (preset strip size) is 4 kb, the first data block (data block 1) can also be directly stored in the corresponding data bit node (data bit node 1). The first data block (data block 1) is split into a 4 kb data block (the first split data block) and a 2 kb data block (the second split data block). The first split data block is directly stored in the corresponding data bit node (data bit node 1), and the second split data block enters the subsequent processing flow as a continuation unit to be processed.

[0065] If the data block size of the data block in the data to be processed is 9 kb and the preset data bit storage size (preset striping size) is 4 kb, the first data block (data block 1) can also be directly stored in the corresponding data bit node (data bit node 1). The first data block (data block 1) is split into a data block with a size of 4 kb (the first split data block) and a data block with a size of 5 kb (the second split data block). The first split data block is directly stored in the corresponding data bit node (data bit node 1), and the second split data block enters the subsequent processing flow as a continuation unit to be processed.

[0066] By standardizing the splitting, it is ensured that each data unit strictly conforms to the format requirements of the erasure code striping process, creating an ideal data structure for subsequent local erasure calculation; the dynamic routing processing of the continuous data block (the second split data block) realizes the elastic allocation of storage resources, avoiding storage fragmentation caused by fluctuations in the data block size. It reduces the splitting processing overhead of large data blocks, making it exhibit excellent performance in high-throughput storage scenarios.

[0067] In an implementable manner of the embodiment of the present application, when storing the split data blocks in the corresponding data bit nodes and multiple parity bit nodes respectively, the following methods can be adopted but are not limited to: storing the first split data block in the data bit node corresponding to the first data, generating multiple second copy data according to the first split data block, and storing the multiple second copy data in the multiple parity bit nodes respectively; in the case where the size of the second split data block is less than or equal to the preset striping size, storing the second split data block in the data bit node corresponding to the second data block and the multiple parity bit nodes respectively, where the second data block is the data block adjacent to the first data block in the data to be processed; in the case where the size of the second split data block is greater than the preset striping size, performing data splitting processing on the second split data block according to the preset striping size to obtain a third split data block, and storing the third split data block in the data bit node corresponding to the second data block and the multiple parity bit nodes respectively.

[0068] In the embodiment of the present application, the processing of the split data blocks is divided into two stages: the first split data block processing stage and the second split data block processing stage. Regarding the first split data block processing stage, it includes but is not limited to the following methods: writing the standardized split first split data block into the data bit node corresponding to the first data block, and synchronously generating multiple second copy data (the number is the same as the total number of parity bit nodes) according to the first split data block. The second copy data is distributed to all parity bit nodes through a parallel write protocol to form immediate redundancy protection. The second copy data can be written into the parity bit node by using but not limited to the write-time verification technology to ensure the bit-level consistency between the first split data block and the second copy data, creating a trusted data source for subsequent erasure code conversion.

[0069] Regarding the second split data block processing stage, it includes but is not limited to the following methods:

[0070] When it is detected that the size of the second split data block is less than or equal to the preset stripe size, the second split data block can be identified as an independent storage unit, and dual writing can be directly executed. The second split data block is directly written to the data bit nodes, and the synchronously generated replica data is distributed to the parity bit nodes.

[0071] When it is detected that the size of the second split data block is greater than the preset stripe size, recursive splitting processing is performed. With the preset stripe size as the cutting granularity, the second split data block is split again into a third split data block and a new continuation unit. The third split data block immediately triggers the storage routing calculation and executes the replica writing process. The continuation unit then enters the next round of splitting judgment loop until all data units meet the storage specification requirements (i.e., are all less than or equal to the preset stripe size).

[0072] Reference Figure 2 For example, if the preset data bit storage size (preset stripe size) is 4 kb and the second split data block is a 2 - kb - sized data block, at this time, the second split data block can directly execute dual writing. The second split data block is directly written to the data bit node corresponding to the second data block (data bit node 2), and the synchronously generated replica data is distributed to all parity bit nodes.

[0073] If the preset data bit storage size (preset stripe size) is 4 kb and the second split data block is a 5 - kb - sized data block, the second split data block is split into a 4 - kb - sized data block (the third split data block) and a 1 - kb - sized data block (the new continuation unit). The third split data block is directly stored in the corresponding data bit node (data bit node 1), and the new continuation unit enters the subsequent processing flow as the continuation unit to be processed.

[0074] Through multi - level adaptive splitting and storage path planning, data streams of any size are transformed into a sequence of standardized storage units. The rigid limit on the data block size can be eliminated through the recursive processing mechanism to achieve storage unboundedness; the storage distribution is always optimally matched with the cluster state through the dynamic routing strategy to improve the scalability of large - scale storage systems; and the recomputation overhead is minimized while ensuring the logical integrity of the data through the chained metadata management.

[0075] In an implementable manner of the embodiment of the present application, after obtaining the verification data block, the following methods may be adopted but are not limited to: performing information detection processing between multiple data bit nodes and multiple parity bit nodes to obtain information detection data; in the case where it is determined according to the information detection data that any data bit node and / or any parity bit node has a fault, performing fault recovery processing on the faulty data bit node and / or the faulty parity bit node; in the case where it is determined according to the information detection data that none of the multiple data bit nodes and multiple parity bit nodes has a fault, deleting the copy data stored in the multiple parity bit nodes.

[0076] In the embodiment of the present application, the process of performing information detection processing may be implemented in the following ways but is not limited to: after the parity bit node completes the local erasure calculation and obtains the verification data block, asynchronously sending a first message (such as: local erasure data ready) to the main data node; after the main data node receives all the first messages from the parity bit nodes, sending a second message (such as: copy to erasure state) to each data bit node. If the status of each data bit node is complete and normal, that is, there is no fault, recovery, reconstruction, etc. status, each data bit node replies with a third message (such as: ready); if the data status of each data bit node is abnormal, after it returns to normal, it replies with the third message. After the main data node receives the third messages from all the data bit nodes, it updates the redundant ratio record metadata information and sends a fourth message (such as: cache release) to the parity bit node. At this time, the parity bit node deletes the copy data stored.

[0077] Among them, the information detection data at least includes the response status of the message. For example: if there are message responses from all data bit nodes and parity bit nodes in the information detection data, it is determined that none of the multiple data bit nodes and multiple parity bit nodes has a fault; if there is an abnormal message response from any data bit node and parity bit node in the information detection data, it is determined that the data bit node and / or the parity bit node has a fault.

[0078] Through the closed-loop operation and maintenance management design, the reliability of the erasure code storage system is improved from the simple data protection level to the system-level fault tolerance dimension. Through the state perception and policy switching mechanism, the distributed storage system can achieve all-round elastic guarantee from the data layer to the service layer when dealing with abnormal situations such as hardware failures and network partitions, providing key technical support for building an autonomously evolving intelligent storage infrastructure.

[0079] In an implementable manner of the embodiment of the present application, when performing fault recovery processing on a data bit node with a fault and / or a check data block with a fault, the following manner can also be adopted but is not limited to: when it is determined according to information detection data that any data bit node has a fault, determine the first target data stored in the data bit node with a fault, and generate a replacement data bit node according to the data block size and the data block offset; determine the first target data according to the replica data stored in multiple check bit nodes, and store the first target data into the replacement data bit node, where the first target data is the data block stored in the data bit node with a fault; when it is determined according to information detection data that any check bit node has a fault, determine the second target data stored in the check bit node with a fault, and generate a replacement check bit node according to the data block size and the data block offset, where the second target data is multiple replica data stored in the check bit node with a fault; determine the second target data according to the replica data stored in other check bit nodes and the data blocks stored in multiple data bit nodes, and store the second data into the replacement data bit node, where the other check bit nodes are the check bit nodes other than the check bit node with a fault among the multiple check bit nodes.

[0080] In the embodiment of the present application, parse the abnormal signals in the information detection data (such as: node heartbeat timeout, data verification failure, etc.). When it is determined that a data bit node fails, based on the data block offset and the data block size, dynamically generate a replacement data bit node (preferably select an available node outside the same fault domain and follow the disaster recovery policy of the storage cluster).

[0081] After that, retrieve all the stored first target data (i.e., the original data block lost by the faulty data bit node) from the set of check bit nodes, and determine the data source by adopting but not limited to the replica consistency arbitration algorithm (verifying the validity of replicas based on the majority principle). The selected replica data is written into the replacement data bit node through a secure transmission channel to complete the seamless reconstruction of the data bit topology structure.

[0082] When it is detected that a check bit node fails, first parse the second target data (the replica data set stored in the faulty check bit node), generate a replacement check bit node based on the data block offset. During the recovery process, build a distributed data aggregation pipeline, obtain partial replica data from other normal check bit nodes, and at the same time extract the original data from the associated data bit nodes, and recover the complete replica set through but not limited to the hybrid data source reconstruction algorithm.

[0083] Through the design of the dual recovery engine, the minute-level self-healing ability of storage node faults is achieved, and it is also possible to continuously analyze the abnormal patterns in the information detection data (such as: the growth trend of disk bad sectors, network latency fluctuations, etc.), and start preventive data migration before the hardware completely fails to minimize the impact of the repair window period on the business.

[0084] In an implementable manner of the embodiment of the present application, after deleting the replicated data stored in multiple parity nodes, the following methods may be adopted but are not limited to: performing a fault detection process between multiple data nodes and multiple parity nodes to obtain fault detection data; when it is determined that any data node has a fault according to the fault detection data, generating a replacement data node according to the data block size and the data block offset; performing data erasure correction processing on multiple parity data blocks to determine the first target data, and storing the first target data in the replacement data node; when it is determined that any parity node has a fault according to the information detection data, generating a replacement parity node according to the data block size and the data block offset; performing data erasure correction processing on other parity data blocks and the data blocks stored in multiple data nodes to determine the second target data, and storing the second data in the replacement data node, where the other parity data blocks are the parity data blocks stored in the parity nodes other than the faulty parity node among the multiple parity nodes.

[0085] In the embodiment of the present application, since the replicated data in the parity nodes has been deleted and only the parity data blocks are retained, it is impossible to perform data recovery of the faulty node through simple data replication. At this time, when a data node fault is detected, the following methods may be adopted but are not limited to the algebraic reconstruction recovery process:

[0086] Based on the data block offset and the data block size, a replacement data node is generated. All parity data blocks are obtained from normal parity nodes, and combined with the data of normal data nodes, the first target data (the original data block lost by the faulty data node) is reconstructed through, but not limited to, finite field inverse operations. The recovered first target data is written into the replacement data node through an encrypted channel.

[0087] When a parity node fails, the following methods may be adopted but are not limited to the collaborative parity regeneration mechanism:

[0088] A replacement parity node is created, and the other normal parity data blocks (valid parity units from normal parity nodes) are combined with the data of all data nodes to re - execute the erasure code encoding operation. The process of re - executing the erasure code encoding operation may adopt, but is not limited to, the streaming computing mode. The newly generated parity block is written into the replacement parity node.

[0089] By deeply integrating the mathematical redundancy characteristics of erasure codes with the elastic architecture of distributed systems, an intelligent storage network with self - healing capabilities is constructed. Breaking the dependence of traditional recovery mechanisms on replicated data, early warning of faulty nodes and minute - level fault recovery can still be achieved in the pure erasure code form.

[0090] To facilitate understanding of the implementation process of this application, this application provides an example diagram of a data processing method, as follows Figure 3 shown. Among them, the user cache data is the data to be processed, and the data to be processed includes 4 data blocks, 4 data bit nodes, and 2 parity bit nodes.

[0091] Specifically, the method for data processing may include but is not limited to the following steps:

[0092] The user writes to determine the storage nodes of 4 data bits and 2 parity bits according to the data block offset and data block size. When the data block size of the write request meets the preset size condition, 2 parity data blocks are directly calculated, and then the data blocks are written to the data bit nodes, and the parity blocks are written to the parity bit nodes;

[0093] (2) When the data block size of the write request is less than the preset striping size, the data block is directly written to the corresponding data bit node according to the offset, and copies are written to other parity bit nodes in the 2+1 replica manner. The data is redundantly stored in the replica manner to ensure data reliability (the write method can be master data bit forwarding or directly writing to the data bit node and 2 parity bit nodes). At the same time, during the process of writing the data block, the read request can directly read the data in the data bit node. When a node failure occurs, it is processed with 2+1 replica redundancy, and the data can be guaranteed to be free from single-point failure loss through the replica mechanism;

[0094] (3) The data block is written in the 2+1 replica manner. The data bit nodes are calculated according to the offset, and the replica data is directly written into the parity bit nodes. The data blocks are sequentially sent to different data bit nodes. When the data of the data block gradually accumulates in the parity bit node to a data length that meets the data to be processed, the parity bit node triggers local erasure calculation to generate a parity data block, and the parity data block is saved to the storage node;

[0095] (4) After the parity bit node completes the local erasure calculation and obtains the parity data block, it asynchronously sends a first message (such as: local erasure data ready) to the master data node; the master data node collects all the first messages from the parity bit nodes and sends a second message (such as: replica to erasure status) to each data bit node. If the status of each data bit node is complete and normal, that is, there are no faults, recoveries, reconstructions, etc., each data bit node replies with a third message (such as: ready); if the data status of each data bit node is abnormal, after it returns to normal, it replies with the third message.

[0096] (5) After the master data node receives the third messages from all data bit nodes, it updates the redundancy ratio record metadata information and sends a fourth message (e.g., cache release) to the parity bit nodes. At this time, the parity bit nodes delete the stored replica data. After the parity bit nodes delete the stored replica data, if a faulty node appears, erasure code fault recovery will be adopted. After the parity bit nodes delete the stored replica data and a fault occurs, the master data bit still uses the replica mechanism to recover the data;

[0097] (6) After receiving the cache release message, the parity bit nodes delete the replica data used for calculating the erasure code and only retain the parity data blocks.

[0098] In summary, the present application can achieve the following technical effects:

[0099] 1. The data processing method, electronic device, and storage medium of the present application determine multiple parity bit nodes and multiple data bit nodes through the data block size and data block offset of the data block. Then, the parity bit nodes and data bit nodes are used to cache the data block data. After that, the parity bit nodes directly calculate the erasure code in place to avoid the indirect erasure cache data forwarding traffic between the nodes in the distributed storage system, reduce the data block processing links, reduce the overall data write volume, and improve the processing performance of the all-flash storage. Therefore, it can solve the technical problem of how to improve the efficiency of data erasure and achieve the technical effect of reducing the data block processing links and improving the efficiency of data erasure.

[0100] 2. The present application directly writes the data block to the erasure destination location, and at the same time writes the replica data of the data block to the parity bit nodes. When all the data to be processed is aggregated completely and the parity bit nodes cache all the data, local erasure calculation is performed to generate parity blocks. After all the calculations are completed, the temporarily cached replica data is deleted. The data is divided into data bit nodes and parity bit nodes, and local erasure is performed on the parity bit nodes to cache the data blocks, avoiding the east-west data forwarding across nodes in the distributed storage.

[0101] 3. The present application performs M N-replica writes, caches M data blocks at the parity bit nodes, converts from replicas to erasure calculation, and uses local erasure calculation to generate parity blocks, avoiding cross-node data forwarding; the process of generating the M+N erasure code is divided into the processes of M N-replica writes, local erasure of M data blocks to generate N parity blocks, selection of parity blocks, negotiation of data block status, and formation of a new erasure group, realizing the process of first writing replicas and then converting to an erasure group for the data blocks in the distributed storage.

[0102] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. However, in many cases, the former is a better implementation manner.

[0103] An embodiment of the present application further provides a data processing device. Figure 4 It is a schematic structural diagram of a data processing device provided by the present application, as Figure 4 shown, including:

[0104] A first determination unit 41, configured to determine data nodes according to the data block size and data block offset in the data to be processed, so as to obtain a plurality of check bit nodes corresponding to the data to be processed and a plurality of data bit nodes corresponding to each of the plurality of data blocks, wherein the data to be processed includes a plurality of data blocks;

[0105] A first storage unit 42, configured to store the first data block in the data bit nodes corresponding to the first data block and a plurality of check bit nodes respectively when the data block size meets the preset size condition, wherein the first data block is the first data block in the data to be processed;

[0106] A splitting unit 43, configured to perform data splitting processing on the first data block when the data block size does not meet the preset size condition, so as to obtain split data blocks;

[0107] [[ID=ID=18]]A second storage unit 44, configured to store the split data blocks in the data bit nodes corresponding to the first data block and a plurality of check bit nodes respectively;

[0108] An erasure correction unit 45, configured to perform data erasure correction calculation processing on the plurality of check bit nodes respectively until all data blocks are stored in the data bit nodes corresponding to them and the plurality of check bit nodes, so as to obtain a plurality of check data blocks corresponding to the data to be processed.

[0109] In an embodiment of the present application, as Figure 5 shown, the data processing device further includes:

[0110] A second determination unit 46, configured to determine that the data block size meets the preset size condition when the data block size is less than or equal to the preset striping size, wherein the preset striping size is the storage data size of the data bit nodes set in advance; <ID=31>

[0111] The second determination unit 46 is further configured to determine that the data block size does not meet the preset size condition when the data block size is greater than the preset striping size.

[0112] In an embodiment of the present application, the first storage unit 42 is further configured to:

[0113] When the data block size is equal to the preset strip size, store the first data block in the data bit node corresponding to the first data block, generate multiple first replica data according to the first data block, and store the multiple first replica data in multiple parity bit nodes respectively, where the number of the first replica data is the same as the number of the parity bit nodes;

[0114] When the data block size is less than the preset strip size, store the first data block in the data bit node corresponding to the first data block, generate multiple first replica data according to the first data block, and store the multiple first replica data in multiple parity bit nodes respectively;

[0115] Determine the first difference between the preset strip size and the data block size, and determine the first repair data block according to the subsequent data blocks in the data to be processed and the first difference, where the size of the first repair data block is the same as the first difference, and the subsequent data blocks are the consecutive adjacent data blocks after the first data block in the data to be processed;

[0116] Store the first repair data block in the data bit node corresponding to the first data block and multiple parity bit nodes respectively.

[0117] In an embodiment of the present application, the splitting unit 43 is further configured to:

[0118] Perform data splitting processing on the first data block according to the preset strip size to obtain a first split data block and a second split data block;

[0119] Wherein, the size of the first split data block is equal to the preset strip size, the split data block includes the first split data block and the second split data block, and the second split data block is the adjacent data block after the first split data block in the first data block.

[0120] In an embodiment of the present application, the second storage unit 44 is further configured to:

[0121] Store the first split data block in the data bit node corresponding to the first data, generate multiple second replica data according to the first split data block, and store the multiple second replica data in multiple parity bit nodes respectively;

[0122] When the size of the second split data block is less than or equal to the preset strip size, store the second split data block in the data bit node corresponding to the second data block and multiple parity bit nodes respectively, where the second data block is the adjacent data block after the first data block in the data to be processed;

[0123] When the size of the second split data block is greater than the preset strip size, the second split data block is subjected to data splitting processing according to the preset strip size to obtain a third split data block, and the third split data block is respectively stored in the data bit nodes corresponding to the second data block and multiple parity bit nodes.

[0124] In an embodiment of the present application, as Figure 5 shown, the data processing device further includes:

[0125] A processing unit 47, configured to perform information detection processing between multiple data bit nodes and multiple parity bit nodes to obtain information detection data;

[0126] The processing unit 47 is further configured to perform a fault recovery process on the faulty data bit node and / or the faulty parity bit node when it is determined according to the information detection data that any data bit node and / or any parity bit node has a fault;

[0127] The processing unit 47 is further configured to delete the copy data stored in the multiple parity bit nodes when it is determined according to the information detection data that none of the multiple data bit nodes and the multiple parity bit nodes have a fault.

[0128] In an embodiment of the present application, the processing unit 47 is further configured to:

[0129] When it is determined according to the information detection data that any data bit node has a fault, determine the first target data stored in the faulty data bit node, and generate a replacement data bit node according to the data block size and the data block offset;

[0130] Determine the first target data according to the copy data stored in the multiple parity bit nodes, and store the first target data in the replacement data bit node, where the first target data is the data block stored in the faulty data bit node;

[0131] When it is determined according to the information detection data that any parity bit node has a fault, determine the second target data stored in the faulty parity bit node, and generate a replacement parity bit node according to the data block size and the data block offset, where the second target data is multiple copy data stored in the faulty parity bit node;

[0132] Determine the second target data according to the copy data stored in the other parity bit nodes and the data blocks stored in the multiple data bit nodes, and store the second data in the replacement data bit node, where the other parity bit nodes are the parity bit nodes other than the faulty parity bit node among the multiple parity bit nodes.

[0133] In an embodiment of the present application, the processing unit 47 is further configured to:

[0134] Perform fault detection processing between multiple data bit nodes and multiple check bit nodes to obtain fault detection data;

[0135] When it is determined that any data bit node has a fault according to the fault detection data, generate a replacement data bit node according to the data block size and data block offset;

[0136] Perform data erasure correction processing on multiple check data blocks to determine the first target data, and store the first target data in the replacement data bit node;

[0137] When it is determined that any check bit node has a fault according to the information detection data, generate a replacement check bit node according to the data block size and data block offset;

[0138] Perform data erasure correction processing on other check data blocks and the data blocks stored in multiple data bit nodes to determine the second target data, and store the second data in the replacement data bit node, where the other check data blocks are the check data blocks stored in the check bit nodes other than the faulty check bit node among the multiple check bit nodes.

[0139] For the description of the features in the corresponding embodiments of the data processing device, reference can be made to the relevant descriptions in the corresponding embodiments of the data processing method, which will not be elaborated here one by one.

[0140] An embodiment of the present application further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments of data processing.

[0141] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any one of the above method embodiments of data processing when running.

[0142] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drive, read-only memory (abbreviated as ROM), random access memory (abbreviated as RAM), mobile hard disk, magnetic disk or optical disc, etc., various media that can store computer programs.

[0143] An embodiment of the present application further provides a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any one of the above method embodiments of data processing.

[0144] Embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments of data processing are implemented.

[0145] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0146] The above has introduced in detail a data processing method, an electronic device, and a storage medium provided by the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for data processing, characterized in that, Including: Determine data nodes according to the data block size and data block offset in the data to be processed, to obtain a plurality of parity bit nodes corresponding to the data to be processed and a plurality of data bit nodes corresponding to each of the plurality of data blocks, wherein the data to be processed includes a plurality of the data blocks; When the data block size meets the preset size condition, store the first data block into the data bit node corresponding to the first data block and the plurality of parity bit nodes respectively, wherein the first data block is the first data block in the data to be processed; When the data block size does not meet the preset size condition, perform data splitting processing on the first data block to obtain split data blocks, and store the split data blocks into the data bit node corresponding to the first data block and the plurality of parity bit nodes respectively; Until all the data blocks are stored into the data bit nodes corresponding to them and the plurality of parity bit nodes respectively, perform data erasure calculation processing on the plurality of parity bit nodes respectively to obtain a plurality of check data blocks corresponding to the data to be processed.

2. The method for data processing according to claim 1, wherein After determining the data nodes according to the data block size and data block offset in the data to be processed, to obtain a plurality of parity bit nodes corresponding to the data to be processed and a plurality of data bit nodes corresponding to each of the plurality of data blocks, the method further includes: When the data block size is less than or equal to the preset strip size, determine that the data block size meets the preset size condition, wherein the preset strip size is the storage data size of the data bit node set in advance; When the data block size is greater than the preset strip size, determine that the data block size does not meet the preset size condition.

3. The method for data processing according to claim 2, wherein The step of, when the data block size meets the preset size condition, storing the first data block into the corresponding data bit node and the plurality of parity bit nodes respectively includes: When the data block size is equal to the preset strip size, store the first data block into the data bit node corresponding to the first data block, generate a plurality of first copy data according to the first data block, and store the plurality of first copy data into the plurality of parity bit nodes respectively, wherein the number of the first copy data is the same as the number of the parity bit nodes; When the data block size is less than the preset strip size, store the first data block into the data bit node corresponding to the first data block, generate the plurality of first copy data according to the first data block, and store the plurality of first copy data into the plurality of parity bit nodes respectively; Determine the first difference between the preset strip size and the data block size, and determine a first repair data block according to the subsequent data blocks in the data to be processed and the first difference, wherein the size of the first repair data block is the same as the first difference, and the subsequent data blocks are the data blocks that are continuously adjacent after the first data block in the data to be processed; Store the first repaired data block in the data bit nodes corresponding to the first data block and the multiple parity bit nodes respectively.

4. The method for data processing according to claim 3, wherein When the size of the data block does not meet the preset size condition, the data splitting process is performed on the first data block, and the obtained split data blocks include: Perform data splitting on the first data block according to the preset strip size to obtain a first split data block and a second split data block; Among them, the size of the first split data block is equal to the preset strip size, the split data blocks include the first split data block and the second split data block, and the second split data block is the data block adjacent to the first split data block after the first split data block in the first data block.

5. The method for data processing according to claim 4, wherein The storing the split data blocks in the corresponding data bit nodes and the multiple parity bit nodes respectively includes: Store the first split data block in the data bit node corresponding to the first data, generate multiple second copy data according to the first split data block, and store the multiple second copy data in the multiple parity bit nodes respectively; When the size of the second split data block is less than or equal to the preset strip size, store the second split data block in the data bit nodes corresponding to the second data block and the multiple parity bit nodes respectively, where the second data block is the data block adjacent to the first data block in the data to be processed; When the size of the second split data block is greater than the preset strip size, perform data splitting on the second split data block according to the preset strip size to obtain a third split data block, and store the third split data block in the data bit nodes corresponding to the second data block and the multiple parity bit nodes respectively.

6. The method for data processing according to claim 1, characterized in that After performing data erasure calculation processing on the multiple parity bit nodes respectively to obtain the parity data block corresponding to the data to be processed, the method further includes: Perform information detection processing between the multiple data bit nodes and the multiple parity bit nodes to obtain information detection data; When it is determined according to the information detection data that any one of the data bit nodes and / or any one of the parity bit nodes has a fault, perform fault recovery processing on the faulty data bit node and / or the faulty parity bit node; When it is determined according to the information detection data that none of the multiple data bit nodes and the multiple parity bit nodes has a fault, delete the copy data stored in the multiple parity bit nodes.

7. The method for data processing according to claim 6, wherein The performing fault recovery processing on the faulty data bit node and / or the faulty parity data block includes: When it is determined according to the information detection data that any one of the data bit nodes has a fault, determine the first target data stored in the faulty data bit node, and generate a replacement data bit node according to the data block size and the data block offset; Determine the first target data according to the replicated data stored in the multiple check bit nodes, and store the first target data in the replacement data bit node, where the first target data is the data block stored in the faulty data bit node; When it is determined according to the information detection data that any one of the check bit nodes is faulty, determine the second target data stored in the faulty check bit node, and generate a replacement check bit node according to the data block size and the data block offset, where the second target data is the multiple replicated data stored in the faulty check bit node; Determine the second target data according to the replicated data stored in other check bit nodes and the data blocks stored in the multiple data bit nodes, and store the second target data in the replacement data bit node, where the other check bit nodes are the check bit nodes in the multiple check bit nodes except the faulty check bit node.

8. The method for data processing according to claim 7, wherein After deleting the replicated data stored in the multiple check bit nodes, the method further includes: Perform a fault detection process between the multiple data bit nodes and the multiple check bit nodes to obtain fault detection data; When it is determined according to the fault detection data that any one of the data bit nodes is faulty, generate the replacement data bit node according to the data block size and the data block offset; Determine the first target data through data erasure correction processing based on the multiple check data blocks, and store the first target data in the replacement data bit node; When it is determined according to the information detection data that any one of the check bit nodes is faulty, generate the replacement check bit node according to the data block size and the data block offset; Determine the second target data through data erasure correction processing based on other check data blocks and the data blocks stored in the multiple data bit nodes, and store the second data in the replacement data bit node, where the other check data blocks are the check data blocks stored in the check bit nodes in the multiple check bit nodes except the faulty check bit node.

9. An electronic device, characterized in that, including: A memory for storing a computer program; A processor for implementing the steps of the data processing method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, where the computer program implements the steps of the data processing method according to any one of claims 1 to 8 when executed by a processor.

Images