A data reconstruction method and device, electronic equipment and storage medium
Patent Information
- Application Number
- CN202210893150.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-27
- Publication Date
- 2026-09-22
- Estimated Expiration
- 2042-07-27
AI Technical Summary
[0003]对于当前分布式集群来说，一个对象存储设备上一般承载的归置组数量是100-200个之间，所以当一块对象存储设备故障osd out之后，会给这些归置组重新选择新成员进行数据重构，对于纠删k+m来说，恢复一份数据要从k个对象存储设备上读取数据，但是一个对象存储设备参与恢复的归置组的个数是不均衡的，这样就会造成重构时对象存储设备上的读盘压力不均衡，当重构速度逐渐上升的时候，有些对象存储设备的磁盘压力就持续变为100％，成为性能的瓶颈点
[0039]本申请在接收到数据重构请求之后，确定异常对象存储设备。若异常对象存储设备为目标对象存储设备的从设备，则确定纠删算法中所有用于恢复所述异常对象存储设备中数据的备选对象存储设备，进而利用备选对象存储设备生成恢复数据。本申请基于各个备选对象存储设备中归置组的数量确定k个参与数据恢复的待读对象存储设备，k个待读对象存储设备中所有预设归置组的平均数量小于所有所述备选对象存储设备中所有预设归置组的平均数量。通过上述方式可以使归置组数量较大的备选对象存储设备参与数据恢复的概率较低，进而提高了分布式存储集群的数据重构效率。本申请同时还提供了一种数据重构装置、一种存储介质和一种电子设备，具有上述有益效果，在此不再赘述。
Smart Images
Figure CN115167780B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed storage technology, and in particular to a data reconstruction method, apparatus, electronic device and storage medium. Background Technology
[0002] During the operation of a distributed storage cluster, if a disk fails, a new object storage device will be selected for data reconstruction. To ensure that the new object storage device can complete the data reconstruction quickly and minimize the impact of reconstruction on front-end business, in a 4+2 erasure scenario, four object storage devices need to be selected from the acting members of this placement group (PG) to read the data, perform encoding and decoding calculations, and then send the data to the new object storage device for data writing.
[0003] For current distributed clusters, an object storage device typically supports 100-200 placement groups. Therefore, when an object storage device fails and its OSD goes out, new members are selected for these placement groups to reconstruct the data. For erasure k+m, recovering a piece of data requires reading data from k object storage devices. However, the number of placement groups participating in the recovery of an object storage device is uneven. This will cause uneven disk read pressure on the object storage devices during reconstruction. As the reconstruction speed gradually increases, the disk pressure of some object storage devices will continuously reach 100%, becoming a performance bottleneck.
[0004] Therefore, how to improve the data reconstruction efficiency of distributed storage clusters is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention
[0005] The purpose of this application is to provide a data reconstruction method, apparatus, electronic device, and storage medium that can improve the data reconstruction efficiency of distributed storage clusters.
[0006] To address the aforementioned technical problems, this application provides a data reconstruction method applied to a target object storage device in a distributed storage cluster. The data reconstruction method includes:
[0007] Receive a data reconstruction request and determine the storage device for the abnormal object based on the data reconstruction request;
[0008] If the abnormal object storage device is a slave device of the target object storage device, then a candidate object storage device corresponding to the abnormal object storage device is determined; wherein, the candidate object storage device is all object storage devices used in the erasure algorithm to recover data in the abnormal object storage device;
[0009] Determine the number of preset placement groups in all the candidate object storage devices; wherein, the preset placement group is the placement group participating in data recovery;
[0010] Based on the number of the preset placement groups, k object storage devices to be read are selected from all the candidate object storage devices; wherein, the average number of all preset placement groups in the k object storage devices to be read is less than the average number of all preset placement groups in all the candidate object storage devices;
[0011] The erasure algorithm is used to perform encoding and decoding calculations on the data in the k object storage devices to be read, the recovered data is obtained, and the recovered data is sent to a new object storage device.
[0012] Optionally, k object storage devices to be read are selected from all the candidate object storage devices according to the number of the preset placement groups, including:
[0013] Calculate the average number A of all preset placement groups in all the candidate storage devices; where the total number of all the candidate storage devices is k+m-1;
[0014] A lottery is conducted to select m-1 disabled object storage devices from the candidate object storage devices whose number in the preset placement group is greater than A.
[0015] Set k alternative object storage devices, excluding the disabled object storage device, as the object storage device to be read.
[0016] Optionally, a lottery is conducted to select m-1 disabled object storage devices from the candidate object storage devices whose number in the preset placement group is greater than A, including:
[0017] According to preset rules, the candidate object storage devices with a number greater than A in the preset placement group are randomly selected to obtain m-1 disabled object storage devices.
[0018] The preset rule is that the number of preset placement groups is positively correlated with the probability that the candidate object storage device is selected as a disabled object storage device.
[0019] Optionally, k object storage devices to be read are selected from all the candidate object storage devices according to the number of the preset placement groups, including:
[0020] The candidate storage devices are sorted in ascending order of the number of the preset placement groups;
[0021] Set the storage devices of the top k candidate objects in the sorted list as the storage devices of the objects to be read.
[0022] Optionally, determining the number of preset placement groups in all the candidate object storage devices includes:
[0023] Receive response information sent by the monitor of the distributed storage cluster, and determine the number of preset placement groups among all the candidate object storage devices based on the response information;
[0024] The monitor is used to receive placement group status information sent by all object storage devices and to summarize all the placement group status information to obtain the response information.
[0025] Optionally, determining the number of preset placement groups among all the candidate object storage devices based on the response information includes:
[0026] The number of preset placement groups in all the candidate object storage devices is determined based on the response information most recently sent by the monitor.
[0027] Optionally, before performing the erasure algorithm encoding / decoding calculations on the data in the k storage devices of the objects to be read, the method further includes:
[0028] If the target object storage device is the object storage device to be read, then a read request is sent to the other k-1 object storage devices to be read in order to receive data from the other k-1 object storage devices to be read;
[0029] If the target object storage device is not the object storage device to be read, then read requests are sent to k object storage devices to be read in order to receive data from the k object storage devices.
[0030] This application also provides a data reconstruction apparatus applied to a target object storage device in a distributed storage cluster, the data reconstruction apparatus comprising:
[0031] An abnormal device determination module is used to receive a data reconstruction request and determine the storage device of the abnormal object based on the data reconstruction request.
[0032] The alternative device determination module is used to determine alternative object storage devices corresponding to the abnormal object storage device if the abnormal object storage device is a slave device of the target object storage device; wherein, the alternative object storage devices are all object storage devices used in the erasure algorithm to recover data in the abnormal object storage device;
[0033] The module for determining the number of placement groups is used to determine the number of preset placement groups in all the candidate object storage devices; wherein, the preset placement groups are the placement groups participating in data recovery;
[0034] A filtering module is used to select k object storage devices to be read from all the candidate object storage devices according to the number of the preset placement groups; wherein, the average number of all preset placement groups in the k object storage devices to be read is less than the average number of all preset placement groups in all the candidate object storage devices;
[0035] The recovery module is used to perform erasure algorithm encoding and decoding calculations on the data in the k object storage devices to be read, obtain the recovered data, and send the recovered data to a new object storage device.
[0036] This application also provides a storage medium on which a computer program is stored, wherein the computer program, when executed, implements the steps of the above-described data reconstruction method.
[0037] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor invokes the computer program in the memory to implement the steps of the above-described data reconstruction method.
[0038] This application provides a data reconstruction method applied to a target object storage device in a distributed storage cluster, comprising: receiving a data reconstruction request and determining an abnormal object storage device based on the data reconstruction request; if the abnormal object storage device is a slave device of the target object storage device, determining a candidate object storage device corresponding to the abnormal object storage device; wherein, the candidate object storage devices are all object storage devices used in the erasure algorithm to recover data in the abnormal object storage device; determining the number of preset placement groups in all the candidate object storage devices; wherein, the preset placement groups are placement groups participating in data recovery; selecting k object storage devices to be read from all the candidate object storage devices according to the number of preset placement groups; wherein, the average number of all preset placement groups in the k object storage devices to be read is less than the average number of all preset placement groups in all the candidate object storage devices; performing erasure algorithm encoding and decoding calculations on the data in the k object storage devices to be read to obtain recovered data, and sending the recovered data to a new object storage device.
[0039] Upon receiving a data reconstruction request, this application identifies the abnormal object storage device. If the abnormal object storage device is a slave device of the target object storage device, it determines all candidate object storage devices in the erasure algorithm for recovering data from the abnormal object storage device, and then uses these candidate object storage devices to generate recovery data. This application determines k candidate object storage devices to participate in data recovery based on the number of placement groups in each candidate object storage device. The average number of all preset placement groups in the k candidate object storage devices is less than the average number of all preset placement groups in all candidate object storage devices. This method reduces the probability of candidate object storage devices with a large number of placement groups participating in data recovery, thereby improving the data reconstruction efficiency of the distributed storage cluster. This application also provides a data reconstruction apparatus, a storage medium, and an electronic device, which have the above-mentioned beneficial effects, and will not be elaborated further here. Attached Figure Description
[0040] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 A flowchart illustrating a data reconstruction method provided in an embodiment of this application;
[0042] Figure 2 This is a schematic diagram illustrating the data reconstruction principle of a distributed file system provided in an embodiment of this application;
[0043] Figure 3 A flowchart illustrating an optimization scheme for data reconstruction pressure balancing in a distributed file system, as provided in an embodiment of this application.
[0044] Figure 4 This is a schematic diagram of the structure of a data reconstruction device provided in an embodiment of this application. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0046] Please see below. Figure 1 , Figure 1This is a flowchart of a data reconstruction method provided in an embodiment of this application.
[0047] Specific steps may include:
[0048] S101: Receive a data reconstruction request and determine the storage device for the abnormal object based on the data reconstruction request;
[0049] This embodiment can be applied to the target object storage device in a distributed storage cluster. There can be multiple object storage devices (OSDs) in a distributed storage cluster. In the distributed storage system, data can be stored according to the k+m erasure algorithm, where k is the number of data blocks and m is the number of check blocks. Each data block or check block has its corresponding object storage device. An object storage device stores only one data block or one check block of the original data file.
[0050] When an object storage device in a distributed storage cluster malfunctions (i.e., the disk fails), the storage system monitor can initiate a data reconstruction request. After receiving the data reconstruction request, the target object storage device can identify the malfunctioning object storage device based on the data reconstruction request.
[0051] S102: If the abnormal object storage device is a slave device of the target object storage device, then determine the alternative object storage device corresponding to the abnormal object storage device;
[0052] In the k+m erasure algorithm, the object storage device storing the data blocks and checksum blocks of the original data file forms an acting member group. This acting member group includes a master device and slave devices. The master device is used to generate recovery data after a slave device fails. If the failed object storage device is the master device of the acting member group, a reselection of the master device can be performed before this step. Furthermore, in this embodiment, other object storage devices in the acting member group besides the failed object storage device can be set as candidate object storage devices. These candidate object storage devices are all the object storage devices used in the erasure algorithm to recover data from the failed object storage device; that is, data from the failed object storage device can be recovered using data from k candidate object storage devices.
[0053] S103: Determine the number of preset placement groups in all the candidate object storage devices;
[0054] Each candidate object storage device contains multiple preset placement groups, which are the placement groups participating in data recovery. The number of preset placement groups varies across candidate object storage devices, and the target object storage device can determine the number of preset placement groups in each candidate object storage device. A placement group (PG) is a logical unit for data distribution.
[0055] As a feasible implementation, the number of preset placement groups in all candidate object storage devices can be determined as follows: receiving response information sent by the monitor of the distributed storage cluster, and determining the number of preset placement groups in all candidate object storage devices based on the response information; wherein, the monitor is used to receive placement group status information sent by all object storage devices, and summarize all placement group status information to obtain the response information. Specifically, in this step, the number of preset placement groups in all candidate object storage devices can be determined based on the most recent response information sent by the monitor.
[0056] S104: Select k object storage devices to be read from all the candidate object storage devices according to the number of the preset placement groups;
[0057] In the k+m erasure algorithm, data recovery can be achieved using k data blocks and check blocks. This application can select k storage devices to be read from all the candidate storage devices to participate in data recovery. Specifically, in this embodiment, the storage devices to be read can be selected according to the number of preset placement groups, so that the average number of preset placement groups in the k storage devices to be read is less than the average number of preset placement groups in all the candidate storage devices. Specifically, the average number of preset placement groups for each storage device to be read is P1, and the average number of preset placement groups in all candidate storage devices is P2, where P1 < P2.
[0058] S105: Perform erasure algorithm encoding and decoding calculations on the data in the k object storage devices to be read, obtain the recovered data, and send the recovered data to the new object storage device.
[0059] This step involves aggregating data from k object storage devices to be read in the target object storage device, and performing erasure algorithm encoding and decoding calculations on the data in the k object storage devices to obtain the recovered data. Prior to this step, there may be an operation to determine a new object storage device from the distributed storage cluster, and then send the recovered data obtained above to the new object storage device to complete the data reconstruction of the distributed file storage cluster.
[0060] In this embodiment, after receiving a data reconstruction request, an abnormal object storage device is identified. If the abnormal object storage device is a slave device of the target object storage device, all candidate object storage devices for recovering data in the abnormal object storage device are determined in the erasure algorithm, and then recovery data is generated using the candidate object storage devices. This embodiment determines k read-to-read object storage devices to participate in data recovery based on the number of placement groups in each candidate object storage device. The average number of all preset placement groups in the k read-to-read object storage devices is less than the average number of all preset placement groups in all candidate object storage devices. This method reduces the probability of candidate object storage devices with a large number of placement groups participating in data recovery, thereby improving the data reconstruction efficiency of the distributed storage cluster.
[0061] As for Figure 1 A further description of the corresponding embodiment suggests that k object storage devices to be read can be selected in the following manner: Calculate the average number A of all preset placement groups among all the candidate object storage devices; randomly select m-1 disabled object storage devices from the candidate object storage devices whose number of preset placement groups is greater than A; and set the k candidate object storage devices other than the disabled object storage devices as the object storage devices to be read. In the k+m erasure algorithm, the same acting member group includes k+m object storage devices. When an abnormal object storage device appears in the acting member group, the total number of all candidate object storage devices is k+m-1.
[0062] Furthermore, m-1 disabled object storage devices can be selected by lottery in the following way: according to a preset rule, candidate object storage devices with a number greater than A in the preset placement group are selected by lottery to obtain m-1 disabled object storage devices; wherein, the preset rule is: the number of preset placement groups is positively correlated with the probability that a candidate object storage device is selected as a disabled object storage device.
[0063] As for Figure 1 In a further description of the corresponding embodiment, k storage devices to be read can also be selected in the following way: sort the candidate storage devices in ascending order of the number of the preset placement groups; and set the top k candidate storage devices as the storage devices to be read.
[0064] As for Figure 1 In a further description of the corresponding embodiment, before performing erasure algorithm encoding and decoding calculations on the data in the k object storage devices to be read, there may be an operation to determine whether the target object storage device, which is the master device, is an object storage device to be read participating in data recovery. Specifically, the data in the object storage device to be read may be received in the following manner:
[0065] Method 1: If the target object storage device is the object storage device to be read, then send read requests to the other k-1 object storage devices to be read in order to receive data from the other k-1 object storage devices to be read;
[0066] Method 2: If the target object storage device is not the object storage device to be read, then send read requests to k object storage devices to be read in order to receive data from the k object storage devices.
[0067] The process described in the above embodiments is illustrated below through examples in practical applications.
[0068] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the data reconstruction principle of a distributed file system provided in an embodiment of this application. Figure 2 In the K+M erasure algorithm, when old1 to old6 and new2 are object storage devices, when old2 is damaged, the data in old3, old4 and old5 can be read into old1, and then the calculated recovery data can be written into new2.
[0069] When an object storage device fails and the distributed storage cluster begins to reconstruct, the following two problems will arise as the reconstruction speed increases: (1) The reconstruction speed of some object storage devices slows down, causing reconstruction to lag and the reconstruction completion time to increase. (2) Due to the disk pressure being 100% continuously, the read and write latency of the front end increases, affecting the front end business.
[0070] To address the aforementioned issues, this invention proposes an optimization scheme for balancing data recovery pressure. The MON (monitor, a monitor of the distributed storage cluster) counts the number of placement groups participating in data recovery for each object storage device and pushes this information to the object storage devices. During data reconstruction, object storage devices with higher weights are excluded from reading based on the weight of their participation in placement groups for data recovery. This balances read / write pressure among object storage devices, improves reconstruction speed, and reduces the impact on front-end services. In this embodiment, upon receiving a data reconstruction request, an abnormal object storage device is identified. If the abnormal object storage device is a slave device of the target object storage device, all candidate object storage devices for recovering data from the abnormal object storage device are determined using the erasure algorithm. Recovery data is then generated using these candidate object storage devices. This embodiment determines k object storage devices to participate in data recovery based on the number of placement groups in each candidate object storage device. The average number of all preset placement groups in these k candidate object storage devices is less than the average number of all preset placement groups in all candidate object storage devices. The above method can reduce the probability of storage devices with a large number of alternative objects participating in data recovery, thereby improving the data reconstruction efficiency of the distributed storage cluster.
[0071] Please see Figure 3 , Figure 3 This flowchart illustrates an optimization scheme for data reconstruction pressure balancing in a distributed file system, as provided in this embodiment. During cluster reconstruction, the MON (Monitoring Node) counts the number of PGs participating in the recovery of OSDs and pushes this information to the OSDs. During data reconstruction, OSDs with higher weights are de-emphasized based on their participation in the recovery of PGs, thus balancing read / write pressure among OSDs, improving reconstruction speed, and reducing the impact on front-end services. The above process may include the following steps:
[0072] S301: The OSD periodically sends stats messages to the MON, thereby reporting the placement group status information to the MON.
[0073] S302: After receiving the placement group status information, MON will summarize the information of each OSD, calculate the number of placement groups in which each OSD participates in the recovery, and then send a response message to the OSD, sending the information of the number of placement groups in which each OSD participates in the recovery to the OSD.
[0074] S303: After receiving the message returned by MON, the OSD saves the number of reset groups in which each OSD participated in the recovery.
[0075] S304: Data reconstruction by the OSD is triggered by the master OSD. The OSD will select k members from the acting members to perform data read operations. When selecting members, the average weight of these acting members in the number of PGs to be restored is calculated. Acting members with a weight greater than the average weight participate in a random lottery algorithm for selection. Members with a higher weight are more likely to be selected. The selected member will not participate in this read operation. After selecting k members, read requests are sent to these k members.
[0076] S305: Read data from the selected shard member, perform erasure encoding / decoding, and send the data to the newly selected member. Upon receiving the read request, the master OSD performs erasure encoding / decoding calculations, then sends the data to the newly selected member for data writing, thus ending the recovery process.
[0077] When performing data recovery, the cluster prioritizes OSDs with fewer participating PGs when selecting acting members to read data. Based on this, the advantages of this patent are: balancing the load across OSDs during data recovery, improving reconstruction speed, reducing front-end business latency, and enhancing user experience.
[0078] Please see Figure 4 , Figure 4 This is a schematic diagram of a data reconstruction device provided in an embodiment of this application, applied to a target object storage device in a distributed storage cluster. The data reconstruction device includes:
[0079] An abnormal device determination module 401 is used to receive a data reconstruction request and determine the abnormal object storage device based on the data reconstruction request.
[0080] The alternative device determination module 402 is used to determine alternative object storage devices corresponding to the abnormal object storage device if the abnormal object storage device is a slave device of the target object storage device; wherein, the alternative object storage devices are all object storage devices used in the erasure algorithm to recover data in the abnormal object storage device;
[0081] The placement group quantity determination module 403 is used to determine the number of preset placement groups in all the candidate object storage devices; wherein, the preset placement group is the placement group participating in data recovery;
[0082] The filtering module 404 is used to select k object storage devices to be read from all the candidate object storage devices according to the number of the preset placement groups; wherein, the average number of all preset placement groups in the k object storage devices to be read is less than the average number of all preset placement groups in all the candidate object storage devices.
[0083] The recovery module 405 is used to perform erasure algorithm encoding and decoding calculations on the data in the k object storage devices to be read, obtain the recovered data, and send the recovered data to a new object storage device.
[0084] This embodiment can be applied to the target object storage device in a distributed storage cluster. There can be multiple object storage devices (OSDs) in a distributed storage cluster. In the distributed storage system, data can be stored according to the k+m erasure algorithm, where k is the number of data blocks and m is the number of check blocks. Each data block or check block has its corresponding object storage device. An object storage device stores only one data block or one check block of the original data file.
[0085] When an object storage device in a distributed storage cluster malfunctions (i.e., the disk fails), the storage system monitor can initiate a data reconstruction request. After receiving the data reconstruction request, the target object storage device can identify the malfunctioning object storage device based on the data reconstruction request.
[0086] In the k+m erasure algorithm, the object storage device storing the data blocks and checksum blocks of the original data file is an acting member group. The acting member group includes a master device and slave devices. The master device is used to generate recovery data after a slave device fails. If the failed object storage device is the master device of the acting member group, a reselection operation to determine the master device of the acting member group can occur before this step. Furthermore, in this embodiment, other object storage devices in the acting member group besides the failed object storage device can be set as candidate object storage devices. These candidate object storage devices are all the object storage devices used in the erasure algorithm to recover data from the failed object storage device; that is, data from the k candidate object storage devices can be used to recover data from the failed object storage device.
[0087] Each candidate object storage device contains multiple preset placement groups, which are the placement groups participating in data recovery. The number of preset placement groups varies across candidate object storage devices, and the target object storage device can determine the number of preset placement groups in each candidate object storage device. A placement group (PG) is a logical unit for data distribution.
[0088] As a feasible implementation, the number of preset placement groups in all candidate object storage devices can be determined as follows: receiving response information sent by the monitor of the distributed storage cluster, and determining the number of preset placement groups in all candidate object storage devices based on the response information; wherein, the monitor is used to receive placement group status information sent by all object storage devices, and summarize all placement group status information to obtain the response information. Specifically, in this step, the number of preset placement groups in all candidate object storage devices can be determined based on the most recent response information sent by the monitor.
[0089] In the k+m erasure algorithm, data recovery can be achieved using k data blocks and check blocks. This application can select k storage devices to be read from all the candidate storage devices to participate in data recovery. Specifically, in this embodiment, the storage devices to be read can be selected according to the number of preset placement groups, so that the average number of preset placement groups in the k storage devices to be read is less than the average number of preset placement groups in all the candidate storage devices. Specifically, the average number of preset placement groups for each storage device to be read is P1, and the average number of preset placement groups in all candidate storage devices is P2, where P1 < P2.
[0090] This embodiment can aggregate data from k object storage devices to be read in the target object storage device, and perform erasure algorithm encoding and decoding calculations on the data in the k object storage devices to obtain the recovered data. Before this step, there may be an operation to determine a new object storage device from the distributed storage cluster, and then send the recovered data obtained above to the new object storage device to complete the data reconstruction of the distributed file storage cluster.
[0091] In this embodiment, after receiving a data reconstruction request, an abnormal object storage device is identified. If the abnormal object storage device is a slave device of the target object storage device, all candidate object storage devices for recovering data in the abnormal object storage device are determined in the erasure algorithm, and then recovery data is generated using the candidate object storage devices. This embodiment determines k read-to-read object storage devices to participate in data recovery based on the number of placement groups in each candidate object storage device. The average number of all preset placement groups in the k read-to-read object storage devices is less than the average number of all preset placement groups in all candidate object storage devices. This method reduces the probability of candidate object storage devices with a large number of placement groups participating in data recovery, thereby improving the data reconstruction efficiency of the distributed storage cluster.
[0092] Furthermore, the process by which the filtering module 404 selects k object storage devices to be read from all the candidate object storage devices according to the number of the preset placement groups includes: calculating the average number A of all preset placement groups in all the candidate object storage devices; wherein the number of all the candidate object storage devices is k+m-1; randomly selecting the candidate object storage devices whose number of preset placement groups is greater than A to obtain m-1 disabled object storage devices; and setting the k candidate object storage devices other than the disabled object storage devices as the object storage devices to be read.
[0093] Furthermore, the process by which the screening module 404 selects m-1 disabled object storage devices from the candidate object storage devices whose number in the preset placement group is greater than A by lottery according to a preset rule includes: selecting m-1 disabled object storage devices from the candidate object storage devices whose number in the preset placement group is greater than A by lottery according to a preset rule; wherein, the preset rule is that the number of preset placement groups is positively correlated with the probability that a candidate object storage device is selected as a disabled object storage device.
[0094] Furthermore, the process by which the filtering module 404 selects k candidate object storage devices from all the candidate object storage devices according to the number of the preset placement groups includes: sorting the candidate object storage devices in ascending order of the number of the preset placement groups; and setting the top k candidate object storage devices as the candidate object storage devices to be read.
[0095] Furthermore, the process by which the placement group quantity determination module 403 determines the number of preset placement groups among all the candidate object storage devices includes: receiving response information sent by the monitor of the distributed storage cluster, and determining the number of preset placement groups among all the candidate object storage devices based on the response information; wherein, the monitor is used to receive placement group status information sent by all object storage devices, and to summarize all the placement group status information to obtain the response information.
[0096] Furthermore, the process by which the placement group number determination module 403 determines the number of preset placement groups in all the candidate object storage devices based on the response information includes: determining the number of preset placement groups in all the candidate object storage devices based on the response information most recently sent by the monitor.
[0097] Furthermore, it also includes:
[0098] The data reading module is configured to, before performing erasure algorithm encoding / decoding calculations on the data in the k storage devices to be read, send read requests to the other k-1 storage devices to be read if the target storage device is the storage device to be read, so as to receive data from the other k-1 storage devices to be read; and is also configured to, if the target storage device is not the storage device to be read, send read requests to the k storage devices to be read, so as to receive data from the k storage devices to be read.
[0099] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.
[0100] This application also provides a storage medium on which a computer program is stored, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0101] This application also provides an electronic device that may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the electronic device may also include various network interfaces, power supplies, and other components.
[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.
[0103] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
Claims
1. A data reconstruction method, characterized in that, The target object storage device is applied to a distributed storage cluster. The distributed storage cluster stores data using a k+m erasure algorithm, where k is the number of data blocks and m is the number of check blocks. The object storage devices storing the original data files' data blocks and check blocks constitute an acting member group. This acting member group includes a master device and slave devices, with the target object storage device being the master device. The data reconstruction method includes: Receive a data reconstruction request and determine the storage device for the abnormal object based on the data reconstruction request; If the abnormal object storage device is a slave device of the target object storage device, then alternative object storage devices corresponding to the abnormal object storage device are determined; wherein, the alternative object storage devices are all object storage devices used in the erasure algorithm to recover data in the abnormal object storage device; the alternative object storage devices are other object storage devices in the acting member group other than the abnormal object storage device; Determine the number of preset placement groups in all the candidate object storage devices; wherein, the preset placement group is the placement group participating in data recovery; Based on the number of the preset placement groups, k object storage devices to be read are selected from all the candidate object storage devices; wherein, the average number of all preset placement groups in the k object storage devices to be read is less than the average number of all preset placement groups in all the candidate object storage devices; The erasure algorithm is used to perform encoding and decoding calculations on the data in the k object storage devices to be read, to obtain the recovered data, and the recovered data is sent to a new object storage device; Specifically, selecting k object storage devices to be read from all the candidate object storage devices according to the number of the preset placement groups includes: Calculate the average number A of all preset placement groups in all the candidate storage devices; where the total number of all the candidate storage devices is k+m-1; According to a preset rule, candidate object storage devices with a number greater than A in the preset placement group are randomly selected to obtain m-1 disabled object storage devices; wherein, the preset rule is: the number of preset placement groups is positively correlated with the probability that a candidate object storage device is selected as a disabled object storage device; Set k alternative object storage devices, excluding the disabled object storage device, as the object storage device to be read.
2. The data reconstruction method according to claim 1, characterized in that, Based on the number of the preset placement groups, k object storage devices to be read are selected from all the candidate object storage devices, including: The candidate storage devices are sorted in ascending order of the number of the preset placement groups; The top k candidate object storage devices are set as the object storage devices to be read.
3. The data reconstruction method according to claim 1, characterized in that, Determining the number of preset placement groups in all the candidate object storage devices includes: Receive response information sent by the monitor of the distributed storage cluster, and determine the number of preset placement groups among all the candidate object storage devices based on the response information; The monitor is used to receive placement group status information sent by all object storage devices and to summarize all the placement group status information to obtain the response information.
4. The data reconstruction method according to claim 3, characterized in that, Determining the number of preset placement groups among all the candidate storage devices based on the response information includes: The number of preset placement groups in all the candidate object storage devices is determined based on the response information most recently sent by the monitor.
5. The data reconstruction method according to any one of claims 1 to 4, characterized in that, Before performing erasure algorithm encoding and decoding calculations on the data in the k storage devices of the objects to be read, the following steps are also included: If the target object storage device is the object storage device to be read, then a read request is sent to the other k-1 object storage devices to be read in order to receive data from the other k-1 object storage devices to be read; If the target object storage device is not the object storage device to be read, then read requests are sent to k object storage devices to be read in order to receive data from the k object storage devices.
6. A data reconstruction apparatus, characterized in that, A target object storage device applied to a distributed storage cluster, based on the data reconstruction method according to any one of claims 1 to 5, the data reconstruction apparatus comprising: An abnormal device determination module is used to receive a data reconstruction request and determine the storage device of the abnormal object based on the data reconstruction request. The alternative device determination module is used to determine alternative object storage devices corresponding to the abnormal object storage device if the abnormal object storage device is a slave device of the target object storage device; wherein, the alternative object storage devices are all object storage devices used in the erasure algorithm to recover data in the abnormal object storage device; The module for determining the number of placement groups is used to determine the number of preset placement groups in all the candidate object storage devices; wherein, the preset placement groups are the placement groups participating in data recovery; A filtering module is used to select k object storage devices to be read from all the candidate object storage devices according to the number of the preset placement groups; wherein, the average number of all preset placement groups in the k object storage devices to be read is less than the average number of all preset placement groups in all the candidate object storage devices; The recovery module is used to perform erasure algorithm encoding and decoding calculations on the data in the k object storage devices to be read, obtain the recovered data, and send the recovered data to a new object storage device.
7. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor, when calling the computer program in the memory, implements the steps of the data reconstruction method as described in any one of claims 1 to 5.
8. A storage medium, characterized in that, The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the data reconstruction method as described in any one of claims 1 to 5.