Data processing method and device in erasure code storage system

Abstract

The application provides a data processing method in an erasure code storage system, comprising: processing a stripe, the processing comprising reading the stripe from a plurality of storage units or writing the stripe into the plurality of storage units; and in a case where a preset condition is triggered in the process of processing the stripe, marking the stripe as a to-be-repaired stripe. In the technical solution of the application, in a case where a preset condition is triggered in the process of processing a stripe, the stripe is marked as a to-be-repaired stripe, so that the repair of the to-be-repaired stripe is started, and the data blocks on the faulty storage unit associated with the stripe are repaired onto a healthy storage unit. Therefore, triggering the entry into the repair process in units of stripes can reduce the influence of a single faulty storage unit on the overall read-write performance of the stripe, and maintain good read-write performance.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer device, and computer-readable storage medium in an erasure coding storage system. Background Technology

[0002] With the development of computer technology, the massive storage and reading / writing of various types of data has become a hot topic. To improve the fault tolerance and security of storage, the industry has provided erasure coding technology. Erasure coding is a forward error correction technique that can be applied in scenarios such as distributed storage systems to achieve data redundancy and improve storage reliability.

[0003] Using erasure coding, an object can be divided into multiple stripes and stored in different storage units. However, in a distributed storage system, if a storage unit experiences slow read / write speeds, it can affect the overall read / write performance of the data.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0005] The purpose of this application is to provide a data processing method, apparatus, computer device, and computer-readable storage medium in an erasure coding storage system to solve or alleviate the above-mentioned problems.

[0006] One aspect of this application provides a data processing method in an erasure coding storage system, the method comprising:

[0007] The stripe is processed, the processing including reading the stripe from a plurality of storage cells or writing the stripe into the plurality of storage cells;

[0008] If a preset condition is triggered during the processing of the strip, the strip is marked as a strip to be repaired.

[0009] Optionally, the preset condition includes: the processing delay for the stripe reaches a first threshold; wherein the first threshold is the time required to read the stripe using erasure coding.

[0010] Optionally, marking the strip as a strip to be repaired includes:

[0011] A first identifier is applied to the strip, the first identifier being used to indicate that the strip is a strip to be repaired;

[0012] A second identifier is assigned to the physical block containing each data block in the stripe, and the second identifier indicates a read / write failure of the corresponding physical block.

[0013] Optionally, each storage unit includes multiple physical blocks, each storage unit is used to store multiple data blocks of the stripe, each stripe includes multiple data blocks, and multiple data blocks of the same stripe are stored in physical blocks of different storage units; the method further includes:

[0014] In the case where multiple stripes are marked as stripes to be repaired, the position of the physical block associated with each stripe is recorded;

[0015] A target storage cell is determined among the plurality of storage cells, wherein the number of physical blocks at the recorded location in the target storage cell is greater than a second threshold; and

[0016] Set the target storage unit to read-only.

[0017] Optionally, the strip to be repaired is repaired based on automated operations; wherein, the automated operations include:

[0018] Set up a first task and multiple second tasks, with each second task corresponding to a storage unit;

[0019] The storage unit topology map is updated periodically by the first task, and the storage unit topology map includes information about multiple storage units;

[0020] When storage cells are added or removed in the storage cell topology diagram, the corresponding second task is added or disabled.

[0021] The multiple second tasks periodically detect whether there are stripes to be repaired in the multiple storage units, and if there are stripes to be repaired, pull the stripe information of the corresponding stripe to be modified to generate a repair task identifier.

[0022] Repair the stripes and their physical blocks that have a repair task identifier, and delete the corresponding repair identifier after successful repair.

[0023] Optionally, the method further includes:

[0024] A state machine is configured for the strip to be repaired, so as to control the state transition of the strip to be repaired.

[0025] Optionally, the method further includes:

[0026] Obtain a read request, which is used to request the reading of the target stripe;

[0027] When the target strip is marked as a strip to be repaired, the target data block is read by erasure coding decoding.

[0028] Optionally, the method further includes:

[0029] Obtain a read request, the read request being used to request the reading of the target stripe;

[0030] If the target strip is not marked as a strip to be repaired, the plurality of target physical blocks are determined based on the target strip;

[0031] If the triggering preset condition is triggered during the process of reading data from one or more target physical blocks among multiple target physical blocks, the target stripe is marked as a stripe to be repaired, and the target data block is read by erasure coding decoding;

[0032] If the target data block is obtained from the plurality of target physical blocks, the target data block is returned.

[0033] Optionally, the method further includes:

[0034] Obtain a read request, the read request being used to request the reading of the target stripe;

[0035] In the case where the storage unit associated with the target stripe is a faulty device, the target data block is read by erasure coding decoding.

[0036] Another aspect of this application provides a data processing apparatus in an erasure coding storage system, the apparatus comprising:

[0037] A processing module is used to process stripes, the processing including reading the stripes from a plurality of storage units or writing the stripes into the plurality of storage units;

[0038] The marking module is used to mark the strip as a strip to be repaired when a preset condition is triggered during the processing of the strip.

[0039] Another aspect of this application provides a computer device, including:

[0040] At least one processor; and

[0041] A memory that is communicatively connected to the at least one processor;

[0042] Wherein: the memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0043] Another aspect of this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described above.

[0044] The data processing method, apparatus, computer equipment, and computer-readable storage medium in the erasure coding storage system provided in this application, on a stripe-by-strip basis, mark a stripe as a stripe to be repaired when preset conditions are triggered during the processing of a stripe, thereby initiating the repair of the stripe to be repaired and repairing the data blocks on the faulty storage unit associated with the stripe to the healthy storage unit. Therefore, triggering the repair process on a stripe-by-strip basis can reduce the impact of a single faulty storage unit on the overall read and write performance of the stripe and maintain excellent overall read and write performance. Attached Figure Description

[0045] Figure 1 The diagram illustrates the operating environment of the data processing method in the erasure coding storage system according to Embodiment 1 of this application;

[0046] Figure 2 A flowchart illustrating a data processing method in an erasure coding storage system according to Embodiment 1 of this application is shown in the schematic diagram.

[0047] Figure 3 This schematically illustrates the relationship between stripes and physical blocks;

[0048] Figures 4 to 7 The flowcharts illustrating the newly added steps of the data processing method in the erasure coding storage system according to Embodiment 1 of this application are shown in the illustrations.

[0049] Figure 8 This illustration illustrates a slow disk processing flow in an exemplary application;

[0050] Figure 9 A block diagram illustrating a data processing apparatus in an erasure coding storage system according to Embodiment 2 of this application is shown schematically; and

[0051] Figure 10 A schematic diagram of the hardware architecture of a computer device according to Embodiment 3 of this application is shown. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0053] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0054] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0055] First, a definition of the terminology used in this application is provided:

[0056] Erasure coding (EC) is a data protection method that can tolerate data loss. It's an encoding technique that segments data, expands and encodes redundant data blocks, and stores them in different locations. It can be used in network transmission to prevent packet loss, and storage systems use it to improve storage reliability. Compared to multiple replication, erasure coding achieves higher data reliability with less data redundancy.

[0057] Stripe: A method of dividing continuous data into data blocks of the same size and writing each segment of data to different disks in an array.

[0058] State machine: A mathematical computational model of a finite number of states and the transitions and actions between these states.

[0059] IO (Input Output) refers to communication between an information processing system and the external world.

[0060] Secondly, to facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below:

[0061] In large-scale distributed erasure coding storage clusters, the failure of a single storage unit disk can impact the overall read and write performance of the corresponding stripe. A general and efficient process is needed to quickly restore data from the failed node to healthy nodes, stripe by stripe, using erasure coding decoding algorithms.

[0062] 1. There is no clear definition of a faulty disk, and no clear repair method for slow I / O in some sectors of the disk.

[0063] 2. From disk failure to successful repair, the system lacks an overall control strategy.

[0064] 3. Some processes require terminal operation and lack automated repair methods.

[0065] To address this, this application provides a novel disk processing solution. First, read / write latency is calculated on a stripe-by-strip basis. If the latency does not meet expectations and reaches a threshold, a stripe repair process is triggered. Second, when an EIO (read / write error) occurs on the disk, a callback function marks the disk as faulty. After system verification, the repair process begins. Subsequent read requests directly use erasure coding decoding, avoiding further I / O with the faulty disk until repair is complete. Third, if slow disk I / O affects the read performance of some stripes, and the overhead of reading the slow disk exceeds the overhead of erasure coding decoding, the system verifies and marks the stripe as faulty, initiating a stripe repair process. Subsequent read requests use decoding, and reads of this stripe no longer generate I / O with the slow disk until repair is complete. Fourth, the triggering conditions and thresholds for disk and stripe repair processes are configured in the configuration center and can be dynamically adjusted. The repair process is automated through state machine transitions.

[0066] Figure 1 The diagram schematically illustrates the operational environment of a data processing method in an erasure coding storage system according to an embodiment of this application. For example... Figure 1 As shown in the diagram, the environment includes an erasure coding storage system 2, a network 4, and an electronic device 6, wherein:

[0067] Erasure coding storage system 2, as a storage platform, can consist of multiple computing devices. These multiple computing devices can include virtualized computing instances. Virtualized computing instances can include virtual machines, such as emulations of computer systems, operating systems, servers, etc. The computing devices can load virtual machines based on virtual images and / or other data that define specific software used for emulation (e.g., operating systems, dedicated applications, servers). As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device.

[0068] Erasure coding storage system 2 can provide storage, reading, writing, and query services.

[0069] Erasure coding storage system 2 can be configured to communicate with electronic devices 6 via network 4.

[0070] Electronic device 6 can provide a user access page for controlling erasure coding storage system 2 or issuing requests, etc.

[0071] The following describes the technical implementation scheme provided in this application through multiple embodiments, using erasure coding storage system 2 as the main implementation entity.

[0072] Example 1

[0073] Figure 2 A flowchart illustrating a data processing method in an erasure coding storage system according to Embodiment 1 of this application is shown. Figure 2 As shown, the data processing method in this erasure coding storage system may include steps S200 to S202, wherein:

[0074] Step S200 involves processing the stripe, which includes reading the stripe from a plurality of storage cells or writing the stripe into the plurality of storage cells.

[0075] Step S202: If a preset condition is triggered during the processing of the strip, the strip is marked as a strip to be repaired.

[0076] Erasure coding storage system 2 can consist of multiple storage units, forming a distributed structure. The distributed storage system can employ various networking methods, with each storage unit providing computing and storage services. The number of storage units can be configured according to actual needs. Storage units can be disks or other non-volatile storage media. Taking disks as an example, a storage unit can be a single disk or a disk array composed of multiple disks.

[0077] Erasure coding storage system 2 employs erasure coding technology. Compared to multiple replicas, erasure coding technology can achieve higher data reliability with less data redundancy. The following section provides a more detailed introduction to erasure coding technology.

[0078] During storage, erasure coding divides the data to be stored into n original data blocks and calculates m checksum data blocks. It can then reconstruct the original n data blocks from any n data blocks out of the n+m original data blocks. That is, if 1 to m data blocks are corrupted, the corrupted data can be recovered through erasure coding decoding. Theoretically, the n+m data blocks can be stored on n+m storage units, thus tolerating the failure of any m storage units.

[0079] like Figure 3 As shown, erasure coding provides two dimensions: a logical concept based on stripes and a physical concept based on storage units. A stripe is a logical concept, defined as a series of n+m consecutive data blocks. Therefore, a stripe contains n+m data blocks. These data blocks are stored in physical blocks within different storage units. A physical block can be a partition or a sector within a storage unit.

[0080] When a user needs to read or write a stripe, it means reading or writing to multiple data blocks. In other words, it means writing or reading multiple data blocks from multiple physical blocks in multiple storage units.

[0081] However, if a storage cell fails during the write or read process, it will affect the overall read and write performance of that stripe. In this embodiment, a new definition method for identifying storage cell failures is proposed, namely, judging the read and write performance of storage nodes on a stripe-by-strip basis, and triggering a stripe repair process when preset conditions are met.

[0082] The data processing method provided in this embodiment operates on a stripe-by-strip basis. When preset conditions are triggered during the processing of a stripe, the stripe is marked as a stripe to be repaired, thereby initiating the repair process for that stripe and transferring data blocks from the faulty storage unit associated with the stripe to the healthy storage unit. Therefore, triggering the repair process on a stripe-by-strip basis reduces the impact of a single faulty storage unit on the overall read / write performance of the stripe.

[0083] It should be noted that the preset conditions in step S202 can be dynamically set. For example, a configuration interface can be provided, allowing users to add or delete items in the preset conditions, or adjust the parameters in the items, thereby achieving dynamic control.

[0084] In an optional embodiment, the preset condition includes: the processing latency for the stripe reaches a first threshold. That is, read / write latency is calculated on a stripe-by-strip basis; if the latency does not meet expectations and reaches the first threshold, a stripe repair process is triggered. The first threshold is the time required to read the stripe using erasure coding decoding. Erasure coding decoding can bypass faulty physical blocks (physical blocks that cannot be read or have slow I / O) and decode the data blocks to be read from data blocks in other physical blocks (part of the original data blocks and checksum data blocks in the stripe).

[0085] In an exemplary application, the existence of slow I / O physical blocks can be determined by the processing latency of the stripe. Simultaneously, the first threshold for identifying slow I / O physical blocks is set to "the time required to read the stripe using erasure coding decoding". If the read / write overhead of a physical block exceeds the erasure coding decoding read overhead, it is identified as a slow I / O physical block. This setting enables scheduling that minimizes system read overhead while ensuring efficient stripe reading. That is, under the system read overhead minimization scheduling logic, either a direct read mode or an erasure coding decoding read mode is selected. It should be noted that the above overhead refers to time overhead. However, in other embodiments, the above overhead can also be the sum of time overhead and hardware resource (such as CPU) overhead. Decoding reads consume CPU resources, while direct reads consume almost none.

[0086] In an optional embodiment, step S202, "marking the strip as a strip to be repaired," may include:

[0087] A first identifier is applied to the strip, the first identifier being used to indicate that the strip is a strip to be repaired;

[0088] A second identifier is assigned to the physical block containing each data block in the stripe, and the second identifier indicates a read / write failure of the corresponding physical block.

[0089] A first identifier is assigned to the stripe, indicating that the stripe is a stripe to be repaired. This first identifier can also be called a slow I / O identifier. The first identifier allows for quick identification and statistical analysis of stripes requiring modification. A second identifier can locate physical blocks with read / write errors or slow read / write operations, allowing erasure coding decoding to be performed without accessing these physical blocks. It should be noted that the first and second identifiers can be the same or different identifiers. In some embodiments, the second identifier can exist in the form of a logic switch.

[0090] In optional embodiments, read / write settings for one or more storage cells can be set based on the determination of the stripe to be repaired. As described above, and in conjunction with... Figure 3 Each storage unit includes multiple physical blocks, and each storage unit is used to store multiple data blocks of the stripe. Each stripe includes multiple data blocks, and multiple data blocks of the same stripe are stored in physical blocks of different storage units. Figure 4 As shown, the method may further include:

[0091] Step S400: In the case where multiple stripes are marked as stripes to be repaired, record the position of the physical block associated with each stripe;

[0092] Step S402: Determine a target storage unit among the plurality of storage units, wherein the number of physical blocks at the recorded location in the target storage unit is greater than a second threshold; and

[0093] Step S404: Set the target storage unit to read-only.

[0094] For example, if disk A has N physical blocks, and the locations of M of these blocks are recorded (i.e., the stripes corresponding to the data blocks stored in these M blocks are stripes to be repaired), and the ratio of M / N is greater than a second threshold, then disk A is considered to have poor read / write performance and is unsuitable for handling write requests. Therefore, disk A is marked as read-only. This setting minimizes the writing of data to disk A with poor read / write performance, ensuring efficient subsequent data read / write operations. It should be noted that the second threshold can be determined based on actual requirements.

[0095] The above text introduced preset conditions and how to mark the stripes that trigger preset conditions.

[0096] Next, after the strip is marked as a strip to be repaired (with the first identifier), the following two issues still need to be addressed:

[0097] The first question: How do I enter the repair process?

[0098] The second question: How to handle read requests.

[0099] Regarding the first question: the repair process can be initiated manually or automatically.

[0100] Specifically as follows:

[0101] In an optional embodiment, after a stripe is marked as a target to be repaired, the marked stripe to be repaired can be repaired based on automated operations without terminal operation. As shown in Figure 5, the automated operation includes: step S500, setting a first task and multiple second tasks, each second task corresponding to a storage unit; step S502, updating the storage unit topology map periodically through the first task, the storage unit topology map including information of multiple storage units; step S504, adding or closing the corresponding second task when storage units are added or removed in the storage unit topology map; step S506, periodically detecting whether there are stripes to be repaired in the multiple storage units through the multiple second tasks, and when there are stripes to be repaired, retrieving the stripe information of the corresponding stripe to be modified to generate a repair task identifier; step S508, repairing the stripe to be repaired with the repair task identifier and its physical block, and deleting the corresponding repair identifier after successful repair.

[0102] In an example application, automated operations can be performed as follows:

[0103] 1. Start two scheduled tasks for offline service;

[0104] 2. The first task is to periodically retrieve the storage unit topology diagram;

[0105] 3. The second task is to start a coroutine for each storage unit to periodically pull the slow I / O stripe information of the stripe to be repaired;

[0106] 4. If the first task detects that a storage unit is online or offline, then add or close the second task corresponding to that online or offline storage unit;

[0107] 5. The second task generates a unique repair task ID (identifier) ​​based on the slow IO stripe information and the corresponding positions of multiple physical blocks. If the repair task ID already exists, it is skipped; otherwise, the state machine process is entered.

[0108] 6. After the second task is completed, the corresponding slow I / O information (such as slow I / O flags) will be deleted, so the next scheduled task will not fetch it again.

[0109] As in the example application above, the first task manages the second task, which periodically monitors the stripe information of each stripe. If a stripe to be repaired (the stripe marked with the first identifier) ​​is detected, the stripe information is automatically retrieved and the repair process is automatically initiated. That is, stripe management and repair of stripes to be repaired are both automatic and can be achieved without user intervention.

[0110] In an optional embodiment, to improve the effective control of automated operations, the method further includes:

[0111] A state machine is configured for the strip to be repaired, so as to control the state transition of the strip to be repaired.

[0112] Regarding the second question, namely how to handle read requests, the specific solution is as follows.

[0113] In optional embodiments, such as Figure 6 As shown, the method may further include:

[0114] Step S600: Obtain a read request, the read request being used to request the reading of the target strip;

[0115] In step S602, when the target stripe is marked as a stripe to be repaired, the target data block is read by erasure coding decoding.

[0116] Based on the above embodiments, when it is determined that the target stripe has been marked with the first identifier, considering the principle of minimizing system read overhead and read efficiency, the erasure coding decoding read process is directly followed to obtain the target data block. Of course, if the target stripe is not marked as a stripe to be repaired, the target data block is read directly, as follows: A read request is obtained, which requests to read the target stripe; if the target stripe is not marked as a stripe to be repaired, the plurality of target physical blocks are determined based on the target stripe; if the trigger preset condition is triggered during the process of reading data from one or more of the plurality of target physical blocks, the target stripe is marked as a stripe to be repaired, and the target data block is obtained through erasure coding decoding; if the target data block is obtained from the plurality of target physical blocks, the target data block is returned. When direct reading is possible, direct reading is prioritized to minimize read overhead.

[0117] The above embodiments use the target stripe as a reference to determine the method of reading the target data block, such as whether to follow the erasure coding decoding reading process.

[0118] However, considering that data cannot be read due to storage unit damage, erasure coding decoding can also be used to read data in the event of storage unit damage.

[0119] In optional embodiments, such as Figure 7 As shown, the method further includes:

[0120] Step S700: Obtain a read request, the read request being used to request the reading of the target strip;

[0121] In step S702, if the storage unit associated with the target stripe is a faulty device, the target data block is read by erasure coding decoding.

[0122] When a read error / obstacle occurs in a storage unit, a callback function can be used to mark the storage unit as bad. After the system verifies the error, it enters the repair process. Subsequent read requests will directly use erasure coding decoding, avoiding I / O with the marked bad storage unit until it is repaired, ensuring effective reads.

[0123] To make this application easier to understand, the following is combined with... Figure 8 An example application is provided.

[0124] The S800 SDK (Software Development Kit) receives read requests from users for stripes, which correspond to multiple data blocks.

[0125] S802, determine the multiple physical blocks corresponding to the strip based on the strip identifier of the strip.

[0126] S804, determine whether any of the plurality of physical blocks are identified as slow I / O physical blocks.

[0127] If a physical block is identified as having slow I / O, proceed to step S806 if it does not exist, and proceed to step S810 if it does exist.

[0128] S806, directly read the target data block corresponding to the stripe from the plurality of physical blocks; and

[0129] Determine whether the reading delay of the strip reaches a first threshold during the reading process.

[0130] If the read latency does not reach the first threshold, the target data block is read directly from the plurality of physical blocks and a response is returned.

[0131] If the read latency reaches the first threshold, proceed to step S810.

[0132] The S810 reads the target data block through erasure coding and returns a response.

[0133] S812, mark the strip with a first identifier (slow IO identifier) ​​to mark the strip as a strip to be repaired.

[0134] S814, the offline service periodically generates a unique repair task ID (identifier) ​​based on the slow IO stripe information and the corresponding locations of multiple physical blocks. If the repair task ID already exists, skip this step; otherwise, proceed to step S816.

[0135] S816, Configure a state machine for the strip to be repaired, so as to control the state transition of the strip to be repaired through the state machine.

[0136] A state machine can include multiple states, and the transitions between these states can be as follows: Repair task initialization → Obtain stripe node distribution → Reassign faulty nodes → Decode and read stripe data → Write data to newly assigned nodes → Update stripe-node mapping relationship → Record reverse mapping from disk to stripe → Repair complete.

[0137] S818, after the repair is completed, the slow IO flag of the stripe is deleted, so the next scheduled task will not fetch it again.

[0138] Example 2

[0139] Figure 9 The diagram schematically illustrates a data processing apparatus in an erasure coding storage system according to Embodiment 2 of this application. This data processing apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in these embodiments are a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment. For example... Figure 9 As shown, the data processing device 900 in the erasure coding storage system may include a processing module 910 and a tagging module 920, wherein:

[0140] The processing module 910 is used to process the stripe, the processing including reading the stripe from a plurality of storage cells or writing the stripe into the plurality of storage cells;

[0141] The marking module 920 is used to mark the strip as a strip to be repaired when a preset condition is triggered during the processing of the strip.

[0142] In an optional embodiment, the preset condition includes: the processing latency for the stripe reaches a first threshold; wherein the first threshold is the time required to read the stripe using erasure coding.

[0143] In an optional embodiment, the marking module 920 is further configured to:

[0144] A first identifier is applied to the strip, the first identifier being used to indicate that the strip is a strip to be repaired;

[0145] A second identifier is assigned to the physical block containing each data block in the stripe, and the second identifier indicates a read / write failure of the corresponding physical block.

[0146] In an optional embodiment, each storage unit includes multiple physical blocks, each storage unit is used to store multiple data blocks of the stripes, each stripe includes multiple data blocks, and multiple data blocks of the same stripe are stored in physical blocks of different storage units; the apparatus may further include a recording module (unidentified), used for:

[0147] In the case where multiple stripes are marked as stripes to be repaired, the position of the physical block associated with each stripe is recorded;

[0148] A target storage cell is determined among the plurality of storage cells, wherein the number of physical blocks at the recorded location in the target storage cell is greater than a second threshold; and

[0149] Set the target storage unit to read-only.

[0150] In an optional embodiment, the strip to be repaired is repaired based on automated operations; wherein, the automated operations include:

[0151] Set up a first task and multiple second tasks, with each second task corresponding to a storage unit;

[0152] The storage unit topology map is updated periodically by the first task, and the storage unit topology map includes information about multiple storage units;

[0153] When storage cells are added or removed in the storage cell topology diagram, the corresponding second task is added or disabled.

[0154] The multiple second tasks periodically detect whether there are stripes to be repaired in the multiple storage units, and if there are stripes to be repaired, pull the stripe information of the corresponding stripe to be modified to generate a repair task identifier.

[0155] Repair the stripes and their physical blocks that have a repair task identifier, and delete the corresponding repair identifier after successful repair.

[0156] In an optional embodiment, the device may further include a configuration module (unidentified) for:

[0157] A state machine is configured for the strip to be repaired, so as to control the state transition of the strip to be repaired.

[0158] In an optional embodiment, the device may further include a reading module (unidentified) for:

[0159] Obtain a read request, which is used to request the reading of the target stripe;

[0160] When the target strip is marked as a strip to be repaired, the target data block is read by erasure coding decoding.

[0161] In an optional embodiment, the device may further include a reading module (unidentified) for:

[0162] Obtain a read request, the read request being used to request the reading of the target stripe;

[0163] If the target strip is not marked as a strip to be repaired, the plurality of target physical blocks are determined based on the target strip;

[0164] If the triggering preset condition is triggered during the process of reading data from one or more target physical blocks among multiple target physical blocks, the target stripe is marked as a stripe to be repaired, and the target data block is read by erasure coding decoding;

[0165] If the target data block is obtained from the plurality of target physical blocks, the target data block is returned.

[0166] In an optional embodiment, the device may further include a reading module (unidentified) for:

[0167] Obtain a read request, the read request being used to request the reading of the target stripe;

[0168] In the case where the storage unit associated with the target stripe is a faulty device, the target data block is read by erasure coding decoding.

[0169] Example 3

[0170] Figure 10 This diagram schematically illustrates the hardware architecture of a computer device 10000 suitable for implementing a data processing method in an erasure coding storage system according to Embodiment 3 of this application. The computer device 10000 can be part of the erasure coding storage system 2. In this embodiment, the computer device 10000 can be a rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers), etc. Figure 10As shown, the computer device 10000 includes, but is not limited to, a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked to each other via a system bus. Wherein:

[0171] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for data processing methods in an erasure coding storage system. Furthermore, the memory 10010 can also be used to temporarily store various types of data that have already been output or will be output.

[0172] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0173] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0174] It should be pointed out that, Figure 10 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0175] In this embodiment, the data processing method in the erasure coding storage system stored in memory 10010 can be further divided into one or more program modules and executed by one or more processors (processor 10020 in this embodiment) to complete the embodiment of this application.

[0176] Example 4

[0177] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the data processing method in the erasure coding storage system described in the embodiment.

[0178] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the data processing method in the erasure coding storage system of the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0179] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0180] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A data processing method in an erasure coding storage system, characterized in that, The method includes: The stripe is processed, the processing including reading the stripe from a plurality of storage cells or writing the stripe into the plurality of storage cells; If a preset condition is triggered during the processing of the strip, the strip is marked as a strip to be repaired; The step of marking the strip as a strip to be repaired includes: A first identifier is applied to the strip, the first identifier being used to indicate that the strip is a strip to be repaired; A second identifier is assigned to the physical block containing each data block in the stripe, and the second identifier indicates a read / write failure of the corresponding physical block. The preset conditions include: the processing latency for the stripe reaches a first threshold; wherein the first threshold is the time required to read the stripe using erasure coding decoding; Each storage unit includes multiple physical blocks, each storage unit is used to store multiple data blocks of the stripe, each stripe includes multiple data blocks, and multiple data blocks of the same stripe are stored in physical blocks of different storage units; the method further includes: In the case where multiple stripes are marked as stripes to be repaired, the position of the physical block associated with each stripe is recorded; A target storage cell is determined among the plurality of storage cells, wherein the number of physical blocks at the recorded location in the target storage cell is greater than a second threshold; and Set the target storage unit to read-only.

2. The method according to claim 1, characterized in that, The strip to be repaired is repaired based on automated operations; wherein, the automated operations include: Set up a first task and multiple second tasks, with each second task corresponding to a storage unit; The storage unit topology map is updated periodically by the first task, and the storage unit topology map includes information about multiple storage units. When storage cells are added or removed in the storage cell topology diagram, the corresponding second task is added or disabled. The multiple second tasks periodically detect whether there are stripes to be repaired in the multiple storage units, and if there are stripes to be repaired, pull the stripe information of the corresponding stripe to be modified to generate a repair task identifier. Repair the stripes and their physical blocks that have a repair task identifier, and delete the corresponding repair identifier after successful repair.

3. The method according to claim 2, characterized in that, The method further includes: A state machine is configured for the strip to be repaired, so as to control the state transition of the strip to be repaired.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain a read request, which is used to request the reading of the target stripe; When the target strip is marked as a strip to be repaired, the target data block is read by erasure coding decoding.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain a read request, which is used to request the reading of the target stripe; If the target strip is not marked as a strip to be repaired, multiple target physical blocks are determined based on the target strip; If the triggering preset condition is triggered during the process of reading data from one or more target physical blocks among multiple target physical blocks, the target stripe is marked as a stripe to be repaired, and the target data block is read by erasure coding decoding; If the target data block is obtained from the plurality of target physical blocks, the target data block is returned.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain a read request, which is used to request the reading of the target stripe; In the case where the storage unit associated with the target stripe is a faulty device, the target data block is read by erasure coding decoding.

7. A data processing apparatus in an erasure coding storage system, characterized in that, The device includes: A processing module is used to process stripes, the processing including reading the stripes from a plurality of storage units or writing the stripes into the plurality of storage units; The marking module is used to mark the strip as a strip to be repaired when a preset condition is triggered during the processing of the strip; The marking module is further used for: A first identifier is applied to the strip, the first identifier being used to indicate that the strip is a strip to be repaired; A second identifier is assigned to the physical block containing each data block in the stripe, and the second identifier indicates a read / write failure of the corresponding physical block. The preset conditions include: the processing latency for the stripe reaches a first threshold; wherein the first threshold is the time required to read the stripe using erasure coding decoding; Each storage unit includes multiple physical blocks, each storage unit is used to store multiple data blocks of the stripe, each stripe includes multiple data blocks, and multiple data blocks of the same stripe are stored in physical blocks of different storage units; the device may further include a recording module for: In the case where multiple stripes are marked as stripes to be repaired, the position of the physical block associated with each stripe is recorded; A target storage cell is determined among the plurality of storage cells, wherein the number of physical blocks at the recorded location in the target storage cell is greater than a second threshold; and Set the target storage unit to read-only.

8. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-6.

Images