Data reconstruction method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a data reconstruction method and device, electronic equipment and a computer readable storage medium, the data reconstruction method and device are applied to main object storage equipment, the main object storage equipment is in communication connection with standby object storage equipment, and the method comprises the steps that when a data reconstruction request is received, a target reconstruction object is determined according to the data reconstruction request, adding an arrangement group lock for the target reconstruction object; determining object data information of the target reconstruction object, and forming a target key value pair by the target reconstruction object and the object data information; inputting the target key value pair into a data reconstruction queue, and releasing a collocated group lock for a target reconstruction object; performing data reconstruction on the target key value pairs according to the arrangement sequence of the key value pairs in the data reconstruction queue to obtain target reconstruction data; and sending the target reconstruction data to the standby object storage device, so that the standby object storage device performs data disk falling on the target reconstruction data. According to the technical scheme, the influence on the read-write service due to the addition of the arrangement group lock during the data reconstruction period can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data reconstruction method, and also to a data reconstruction device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the context of digital transformation, massive data is growing, and distributed storage systems have been widely used. Distributed storage systems usually consist of a master server, a storage server, and multiple clients. The essence of a distributed storage system is to evenly distribute a large number of files to multiple storage servers, which has the characteristics of high scalability and high reliability. At the same time, distributed storage systems can be applied to a variety of scenarios. During cluster operation, failures are inevitable. When a hard disk fails in a cluster, the redundancy of the cluster will be reduced. At the same time, data recovery caused by hard disk failures will inevitably affect the front-end business. Therefore, how to quickly complete data reconstruction and minimize the impact on user read and write business during reconstruction has become an inevitable requirement. Through research, it is found that the mutual exclusion between data reconstruction and read and write business mainly comes from the competition of PG (Placement Group) locks and object locks. Further analysis determines that the biggest impact is the competition of PG locks, that is, the impact of data reconstruction on read and write business is mainly caused by PG lock competition.

[0003] Therefore, how to effectively reduce the impact of adding PG locks on read and write services during data reconstruction is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0004] The purpose of the present application is to provide a data reconstruction method, which can effectively reduce the impact of adding a PG lock on read and write services during data reconstruction; another purpose of the present application is to provide a data reconstruction device, an electronic device, a computer-readable storage medium and a computer program product, all of which have the above-mentioned beneficial effects.

[0005] In a first aspect, the present application provides a data reconstruction method, which is applied to a primary object storage device, wherein the primary object storage device is communicatively connected to a backup object storage device, and the method comprises:

[0006] When a data reconstruction request is received, a target reconstruction object is determined according to the data reconstruction request, and a placement group lock is added to the target reconstruction object;

[0007] Determine the object data information of the target reconstruction object, and form a target key-value pair with the target reconstruction object and the object data information;

[0008] Input the target key-value pair into a data reconstruction queue, and release the placement group lock for the target reconstruction object;

[0009] Reconstruct the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data;

[0010] The target reconstructed data is sent to the standby object storage device, so that the standby object storage device writes the target reconstructed data to a disk.

[0011] Optionally, determining object data information of the target reconstruction object and combining the target reconstruction object and the object data information into a target key-value pair includes:

[0012] Determine the object data range, object data size, and object data version number of the target reconstruction object;

[0013] The target reconstruction object is used as a key, and the object data range, the object data size, and the object data version number are used as values ​​to generate the target key-value pair.

[0014] Optionally, reconstructing the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data includes:

[0015] Acquire the target key-value pair according to the arrangement order of the key-value pairs in the data reconstruction queue;

[0016] Setting an object lock, and obtaining target object data and metadata of the target object data corresponding to the target key-value pair;

[0017] Reconstructing the target object data and the metadata of the target object data to obtain the target reconstructed data;

[0018] Accordingly, sending the target reconstructed data to the standby object storage device so that the standby object storage device writes the target reconstructed data to a disk includes:

[0019] Calling a preset communication interface to send the target reconstruction data to the backup object storage device, so that the backup object storage device writes the target reconstruction data to a disk, and feeds back a data write completion message to the primary object storage device;

[0020] After sending the target reconstructed data to the standby object storage device so that the standby object storage device stores the target reconstructed data on a disk, the method further includes:

[0021] The object lock is released according to the data disk completion message.

[0022] Optionally, the data reconstruction method further includes:

[0023] When a read / write service request is received, a target read / write object is determined according to the read / write service request, and the placement group lock is added to the target read / write object;

[0024] If the target read / write object is in the data reconstruction queue, entering a waiting state until the target read / write object leaves the data reconstruction queue;

[0025] Setting an object lock, and executing the read / write service request according to the target read / write object to obtain a target read / write result;

[0026] The object lock is released, the target read and write result is fed back to the read and write service request end, and the placement group lock is released.

[0027] Optionally, if the read-write service request is a read service request, determining a target read-write object according to the read-write service request, and setting the placement group lock on the target read-write object includes:

[0028] Determine whether the current redundancy strategy is an erasure strategy;

[0029] If the current redundancy strategy is not the erasure strategy, determining a target read object according to the read service request, and setting the placement group lock on the target read object;

[0030] If the current redundancy strategy is the erasure strategy, a target read object is determined according to the read service request, and the placement group lock is not set on the target read object.

[0031] Optionally, determining whether the current redundancy strategy is an erasure strategy includes:

[0032] Determining whether a preset flag exists in the read service request;

[0033] If the preset flag exists in the read service request, determining that the current redundancy strategy is the erasure strategy;

[0034] If the preset flag does not exist in the read service request, it is determined that the current redundancy strategy is not the erasure strategy.

[0035] Optionally, before determining the object data information of the target reconstructed object and forming a target key-value pair with the target reconstructed object and the object data information, the method further includes:

[0036] Determine the current data reconstruction mode; wherein the data reconstruction mode includes a full reconstruction mode and an incremental reconstruction mode;

[0037] If the current data reconstruction mode is the full reconstruction mode, the step of determining the object data information of the target reconstruction object and forming a target key-value pair with the target reconstruction object and the object data information is performed.

[0038] In a second aspect, the present application further discloses a data reconstruction device, which is applied to a primary object storage device, wherein the primary object storage device is communicatively connected with a backup object storage device, and the device comprises:

[0039] A locking module, configured to, when receiving a data reconstruction request, determine a target reconstruction object according to the data reconstruction request, and set a placement group lock for the target reconstruction object;

[0040] A building module, used for determining the object data information of the target reconstruction object, and forming a target key-value pair with the target reconstruction object and the object data information;

[0041] An unlocking module, used for inputting the target key-value pair into a data reconstruction queue and releasing the placement group lock for the target reconstruction object;

[0042] A reconstruction module, used to reconstruct the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data;

[0043] The disk placement module is used to send the target reconstructed data to the backup object storage device, so that the backup object storage device can store the target reconstructed data to the disk.

[0044] In a third aspect, the present application further discloses an electronic device, comprising:

[0045] Memory for storing computer programs;

[0046] A processor is used to implement the steps of any one of the data reconstruction methods described above when executing the computer program.

[0047] In a fourth aspect, the present application further discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the data reconstruction methods described above are implemented.

[0048] In a fifth aspect, the present invention further discloses a computer program product, including a computer program / instruction, which implements the steps of any one of the data reconstruction methods described above when executed by a processor.

[0049] The present application provides a data reconstruction method, which is applied to a primary object storage device, wherein the primary object storage device is communicatively connected with a backup object storage device, and the method comprises: when a data reconstruction request is received, determining a target reconstruction object according to the data reconstruction request, and adding a placement group lock to the target reconstruction object; determining object data information of the target reconstruction object, and forming a target key-value pair with the target reconstruction object and the object data information; inputting the target key-value pair into a data reconstruction queue, and releasing the placement group lock for the target reconstruction object; reconstructing data of the target key-value pair according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data; and sending the target reconstructed data to the backup object storage device, so that the backup object storage device stores the target reconstructed data on a disk.

[0050] By applying the technical solution provided by the present application, in the process of data reconstruction using the primary and backup object storage devices, for the target reconstructed object, after setting a placement group lock on it, the object data information of the target reconstructed object is pulled into the data reconstruction queue, and the placement group lock is directly released to wait for the execution of the data reconstruction operation in the data reconstruction queue. Compared with the traditional technology of directly acquiring object data to achieve data reconstruction and then releasing the placement group lock after setting a placement group lock on the target reconstructed object, the present technical solution can effectively reduce the time for setting a placement group lock on the target reconstructed object in the data reconstruction process, and can further effectively reduce the impact of the placement group lock on the read and write business during the data reconstruction.

[0051] In one embodiment of the present application, the data reconstruction mode is further restricted, that is, the data reconstruction method provided by the present application is applied to the full reconstruction mode. It can be understood that the full reconstruction mode is a full data scan recovery, while the incremental reconstruction mode is an incremental data scan recovery. Since the incremental reconstruction mode needs to recover a small number of data objects, it is easy for the primary object storage device to miss data objects, resulting in data reconstruction failure. Therefore, limiting the data reconstruction mode to the full reconstruction mode can effectively ensure the successful execution of the data reconstruction operation.

[0052] The data reconstruction device, electronic device, computer-readable storage medium and computer program product provided in this application also have the above-mentioned technical effects, which will not be described in detail in this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the drawings required for use in the description of the prior art and the embodiments of the present application are briefly introduced below. Of course, the drawings described below in relation to the embodiments of the present application are only part of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work, and the obtained other drawings also belong to the protection scope of the present application.

[0054] Figure 1 A flowchart of a data reconstruction method provided in this application;

[0055] Figure 2 A flowchart of another data reconstruction method provided by this application;

[0056] Figure 3 A schematic diagram of the structure of a data reconstruction device provided by the present application;

[0057] Figure 4 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0058] The core of this application is to provide a data reconstruction method, which can effectively reduce the impact of adding a PG lock on the read and write services during data reconstruction; another core of this application is to provide a data reconstruction device, electronic device, computer-readable storage medium and computer program product, all of which have the above-mentioned beneficial effects.

[0059] In order to describe the technical solutions in the embodiments of the present application more clearly and completely, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0060] An embodiment of the present application provides a data reconstruction method.

[0061] Please refer to Figure 1 , Figure 1 This is a flow chart of a data reconstruction method provided in the present application. The data reconstruction method is applied to a primary object storage device. The primary object storage device is communicatively connected to a backup object storage device. The data reconstruction method may include the following S101 to S105.

[0062] S101: When a data reconstruction request is received, a target reconstruction object is determined according to the data reconstruction request, and a placement group lock is added to the target reconstruction object.

[0063] This step aims to determine the target reconstruction object and set the corresponding placement group lock (PG lock). Specifically, when a data reconstruction request is received, the target reconstruction object, that is, the data object that needs to be reconstructed, can be determined according to the data reconstruction request, and then the placement group lock is set for the data object to facilitate subsequent data reconstruction work.

[0064] It can be understood that the target reconstruction object is each data object on the object storage device that has failed in the current object storage system, and the object storage device fails because one or some hard disks therein fail. Furthermore, the primary object storage device refers to a new object storage device selected for each target reconstruction object, and the backup object storage device is used to implement a redundant backup function.

[0065] S102: Determine the object data information of the target reconstruction object, and combine the target reconstruction object and the object data information into a target key-value pair.

[0066] This step aims to achieve the acquisition of the object data information of the target reconstruction object and the construction of the corresponding target key-value pair. Specifically, after adding a placement group lock to the target reconstruction object, the object data information of the target reconstruction object can be obtained, such as the data range, data size, data version number and other information described in the following embodiments. These object data information are used to achieve data reconstruction of the target data object; further, the target reconstruction object and the object data information are combined into a target key-value pair so that it can be added to the data reconstruction queue to wait for the execution of the data reconstruction operation.

[0067] Wherein, determining the object data information of the target reconstruction object and forming a target key-value pair with the target reconstruction object and the object data information may include:

[0068] Determine the object data range, object data size, and object data version number of the target reconstruction object;

[0069] The target reconstructed object is used as the key, and the object data range, object data size, and object data version number are used as the value to generate the target key-value pair.

[0070] In one embodiment of the present application, before determining the object data information of the target reconstruction object and forming the target reconstruction object and the object data information into a target key-value pair, the following may also be included:

[0071] Determine the current data reconstruction mode; the data reconstruction mode includes the full reconstruction mode (backfill mode) and the incremental reconstruction mode (recovery mode);

[0072] If the current data reconstruction mode is the full reconstruction mode, the step of determining the object data information of the target reconstruction object and forming a target key-value pair with the target reconstruction object and the object data information is executed.

[0073] It should be noted that if the current data reconstruction mode is an incremental reconstruction mode, the data reconstruction process in the prior art can be directly entered.

[0074] It can be seen that the embodiment of the present application further restricts the data reconstruction mode, that is, the data reconstruction method provided by the present application is applied to the full reconstruction mode. It can be understood that the full reconstruction mode is a full data scan recovery, while the incremental reconstruction mode is an incremental data scan recovery. Since the incremental reconstruction mode needs to recover a small number of data objects, it is easy for the primary object storage device to miss data objects, resulting in data reconstruction failure. Therefore, limiting the data reconstruction mode to the full reconstruction mode can effectively ensure the successful execution of the data reconstruction operation.

[0075] Wherein, determining the current data reconstruction mode may include:

[0076] Determine whether the placement group lock includes a full reconstruction flag;

[0077] If the placement group lock includes a full reconstruction flag, the current data reconstruction mode is determined to be the full reconstruction mode;

[0078] If the placement group lock does not include a full reconstruction flag, it is determined that the current data reconstruction mode is an incremental reconstruction mode.

[0079] That is, in the full reconstruction mode, the placement group lock set by the primary object storage device for the target reconstruction object will include a full reconstruction flag, and in the incremental reconstruction mode, the placement group lock set by the primary object storage device for the target reconstruction object does not include a full reconstruction flag. Therefore, the current data reconstruction mode can be determined by judging whether there is a full reconstruction flag in the placement group lock.

[0080] S103: Input the target key-value pair into the data reconstruction queue, and release the placement group lock for the target reconstruction object.

[0081] This step is intended to input the target key-value pair into the data reconstruction queue and release the placement group lock. It is understandable that after the target key-value pair is input into the data reconstruction queue, the placement group lock can be released for the target reconstruction object, so that the target data object can wait in the data reconstruction object to perform the data reconstruction operation, without the need to directly obtain the object data to perform the data reconstruction operation after setting the placement group lock on the target reconstruction object, and then release the placement group lock after the data reconstruction is completed, as in the traditional technology, effectively reducing the time for setting the placement group lock on the target reconstruction object.

[0082] S104: reconstructing the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data.

[0083] This step aims to realize the data reconstruction operation. Specifically, for each key-value pair in the data reconstruction queue, the data reconstruction operation can be performed in sequence according to the order in which they are added to the data reconstruction queue, and the data reconstruction operation can be realized by relying on the object data information in each key-value pair, thereby obtaining the reconstructed data corresponding to each key-value pair.

[0084] S105: Send the target reconstructed data to the standby object storage device, so that the standby object storage device writes the target reconstructed data to a disk.

[0085] This step aims to achieve data storage after data reconstruction. Specifically, after completing data reconstruction of the target key-value pair to obtain the target reconstructed data, it can be sent to the backup object storage device for storage to achieve redundant backup function. At this point, the data reconstruction operation is completed.

[0086] In one embodiment of the present application, reconstructing the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain the target reconstructed data may include:

[0087] Obtain the target key-value pair according to the order of key-value pairs in the data reconstruction queue;

[0088] Set an object lock and obtain the target object data and metadata of the target object data corresponding to the target key-value pair;

[0089] Reconstructing the data using the target object data and the metadata of the target object data to obtain target reconstructed data;

[0090] Accordingly, sending the target reconstructed data to the backup object storage device so that the backup object storage device writes the target reconstructed data to the disk may include: calling a preset communication interface to send the target reconstructed data to the backup object storage device so that the backup object storage device writes the target reconstructed data to the disk, and feeding back a data write completion message to the primary object storage device;

[0091] After sending the target reconstructed data to the standby object storage device so that the standby object storage device stores the target reconstructed data on a disk, the method further includes: releasing the object lock according to a data storage completion message.

[0092] The embodiment of the present application provides a data reconstruction method based on key-value pairs. It is understandable that the data reconstruction process based on key-value pairs relies on the setting of object locks to effectively avoid the impact of other businesses on the target reconstruction object during the data reconstruction process, causing data object usage conflicts.

[0093] During the implementation process, after reading and obtaining the target key-value pair from the data reconstruction queue, an object lock can be set and a data reconstruction operation can be performed. During this process, data reconstruction can be achieved through the target object data in the target key-value pair and the metadata of the target object data; further, after completing the data reconstruction and obtaining the target reconstructed data and sending it to the backup object storage device for disk storage, the backup object storage device will feedback a data disk storage completion message to the primary object storage device, so that the primary object storage device releases the object lock based on the data disk storage completion message.

[0094] It can be seen that in the data reconstruction method provided in the embodiment of the present application, in the process of data reconstruction using the primary and backup object storage devices, for the target reconstructed object, after setting a placement group lock on it, the object data information of the target reconstructed object is pulled into the data reconstruction queue, and the placement group lock is directly released to wait for the execution of the data reconstruction operation in the data reconstruction queue. Compared with the traditional technology of directly acquiring object data to achieve data reconstruction and then releasing the placement group lock after setting a placement group lock on the target reconstructed object, the technical solution can effectively reduce the time for setting a placement group lock on the target reconstructed object in the data reconstruction process, and can further effectively reduce the impact of the placement group lock on the read and write business during the data reconstruction.

[0095] Based on the above embodiments:

[0096] In one embodiment of the present application, the data reconstruction method may further include:

[0097] When a read / write service request is received, a target read / write object is determined according to the read / write service request, and a placement group lock is added to the target read / write object;

[0098] If the target read / write object is in the data reconstruction queue, the system enters a waiting state until the target read / write object leaves the data reconstruction queue;

[0099] Set the object lock, and execute the read and write business request according to the target read and write object to obtain the target read and write results;

[0100] Release the object lock, feed back the target read and write results to the read and write service request end, and release the placement group lock.

[0101] The embodiment of the present application provides a method for processing read and write services during data reconstruction. The data reconstruction method provided by the embodiment of the present application can effectively reduce the impact of data reconstruction operations on front-end read and write services.

[0102] Specifically, when a read-write service request is received, the target read-write object can be determined according to the read-write service request, and a placement group lock can be set on it. At this time, the read-write service request is not directly executed, but it is determined whether the target read-write object is in the data reconstruction queue. It can be understood that if the target read-write object is in the data reconstruction queue, it means that the target read-write object is waiting to perform the data reconstruction operation. If the data read-write operation is performed directly at this time, it will inevitably only get a result of execution failure. Therefore, it can enter the waiting state until the target read-write object leaves the data reconstruction queue, that is, wait for its data reconstruction to be completed, and then perform the data read-write operation on it to ensure the success of the data read-write operation and obtain the target read-write result. Finally, release the object lock, feedback the target read-write result, and release the placement group lock in sequence.

[0103] If the read / write service request is a read service request, determining a target read / write object according to the read / write service request and setting a placement group lock on the target read / write object may include:

[0104] Determine whether the current redundancy strategy is an erasure strategy;

[0105] If the current redundancy strategy is not the erasure strategy, the target read object is determined according to the read service request, and a placement group lock is set on the target read object;

[0106] If the current redundancy strategy is the erasure strategy, the target read object is determined according to the read service request, and no placement group lock is set on the target read object.

[0107] The data reconstruction method provided in the embodiment of the present application can also be applied to the erasure scenario, which effectively improves the applicability of the technical solution. However, in the erasure scenario, the difference is that for the read service request, there is no need to add a placement group lock to the target read object, so as to effectively reduce the impact on the front-end business during the data reconstruction process in the erasure scenario. For other redundant strategy scenarios, such as the copy scenario, it is necessary to add a placement group lock to the target read object, and the above steps S101~S105 can be executed.

[0108] Furthermore, determining whether the current redundancy strategy is an erasure strategy may include:

[0109] Determine whether there is a preset flag in the read service request;

[0110] If there is a preset flag in the read service request, the current redundancy strategy is determined to be an erasure strategy;

[0111] If the preset flag does not exist in the read service request, it is determined that the current redundancy strategy is not an erasure strategy.

[0112] That is to say, in the erasure correction scenario, the read service request initiated by the front end will include the erasure correction redundancy flag (i.e., the above-mentioned preset flag), and in the non-erasure correction scenario, the read service request will not include the erasure correction redundancy flag. Therefore, the current redundancy strategy can be determined by judging whether there is an erasure correction redundancy flag in the read service request.

[0113] Based on the above embodiments, an embodiment of the present application provides another data reconstruction method.

[0114] Please refer to Figure 2 , Figure 2 This is a flow chart of another data reconstruction method provided by this application, and its implementation process is as follows:

[0115] 1. During business operation, the hard disk fails and the corresponding OSD (Object-based StorageDevice) goes down. After a period of time (the default is 10 minutes), the OSD goes out (permanently offline), and a new member (primary OSD) is selected for the PG of the OSD to start data reconstruction.

[0116] 2. Data reconstruction is triggered on the primary OSD. The primary OSD determines the data object to be recovered (target reconstruction object), sets a PG lock on it, and then starts preparations before reconstruction: calculate the data range and size that need to be reconstructed, obtain the version number of the data object to be recovered, etc., put this information into recovery_info, and then put the data object to be recovered and recovery_info into the background queue (data reconstruction queue). The way to put it in is to use the data pair to be recovered as the key and recovery_info as the value, and then release the PG lock.

[0117] 3. The background thread traverses the data reconstruction queue and starts data reconstruction if it finds data. It obtains the data object to be recovered and the recovery_info information from the data reconstruction queue, adds the object lock (read lock), and then starts to read the object data and its metadata from the disk, starts to build a push op (start data reconstruction), and then calls the interface to send the push message (target reconstruction data) to the standby OSD.

[0118] 4. After receiving the push message sent by the primary OSD, the backup OSD starts to download data to disk. After the download is successful, it responds to the primary OSD with a push replay message (data download completion message).

[0119] 5. After receiving the push replay message, the primary OSD releases the object lock and waits for the op to recover the object to re-enter the queue, and the recovery process ends.

[0120] 6. For write business requests, after the primary OSD receives the write business request, it starts to set the PG lock, and then starts to determine whether the write object is recovering. If the write object is in the data reconstruction queue, it means that the write object is waiting for recovery, and the write business request can be placed in the waiting queue for waiting; if the write object is not in the recovery state, it directly sets the object lock (write lock), builds the write transaction, and then downloads the transaction to the disk, and forwards the write transaction to the backup OSD, releasing the object lock. After all forwarded write business requests are responded to, the PG lock is released and the write process ends.

[0121] 7. For the read business process, the main OSD first adds a PG lock, then adds an object lock (read lock), and starts reading data. After the data is read, the object lock is released, a message is responded to the client, and then the PG lock is released.

[0122] It should be noted that the optimization of the above data reconstruction process is mainly aimed at replica scenarios, but it is also applicable to erasure scenarios. The difference is that when reading data, it is necessary to add a corresponding flag to the read business request to mark that the PG lock will no longer be set when reading data on the backup OSD, thereby reducing the impact of data reconstruction operations on front-end business in erasure scenarios.

[0123] In addition, for the data reconstruction process, there are two modes of data reconstruction, namely recovery mode and backfill mode. The former is incremental data recovery, and the latter is full data recovery. Since the number of data objects that need to be restored in the recovery mode is relatively small, it is easy for the primary object storage device to lose data objects, resulting in data reconstruction failure. Therefore, the above reconstruction optimization can be omitted for the recovery reconstruction scenario, that is, the above reconstruction optimization is only performed for the backfill reconstruction scenario. Furthermore, the recovery and backfill scenarios can distinguish whether to enter the reconstruction optimization process by adding corresponding flags to the PG lock.

[0124] It can be seen that in the data reconstruction method provided in the embodiment of the present application, in the process of data reconstruction using the primary and backup object storage devices, for the target reconstructed object, after setting a placement group lock on it, the object data information of the target reconstructed object is pulled into the data reconstruction queue, and the placement group lock is directly released to wait for the execution of the data reconstruction operation in the data reconstruction queue. Compared with the traditional technology of directly acquiring object data to achieve data reconstruction and then releasing the placement group lock after setting a placement group lock on the target reconstructed object, the technical solution can effectively reduce the time for setting a placement group lock on the target reconstructed object in the data reconstruction process, and can further effectively reduce the impact of the placement group lock on the read and write business during the data reconstruction.

[0125] An embodiment of the present application provides a data reconstruction device.

[0126] Please refer to Figure 3 , Figure 3 This is a structural schematic diagram of a data reconstruction device provided by the present application. The data reconstruction device is applied to a primary object storage device. The primary object storage device is communicatively connected with a backup object storage device. The data reconstruction device may include:

[0127] The locking module 1 is used for determining a target reconstruction object according to the data reconstruction request when receiving a data reconstruction request, and setting a placement group lock for the target reconstruction object;

[0128] A forming module 2 is used to determine the object data information of the target reconstruction object, and form a target key-value pair with the target reconstruction object and the object data information;

[0129] Unlocking module 3, used to input the target key-value pair into the data reconstruction queue and release the placement group lock for the target reconstruction object;

[0130] Reconstruction module 4, used to reconstruct the target key-value pair according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data;

[0131] The disk placement module 5 is used to send the target reconstructed data to the backup object storage device, so that the backup object storage device can store the target reconstructed data to the disk.

[0132] It can be seen that the data reconstruction device provided by the embodiment of the present application, in the process of data reconstruction using the primary and backup object storage devices, for the target reconstructed object, after setting the placement group lock on it, pulls the object data information of the target reconstructed object into the data reconstruction queue, and directly releases the placement group lock to wait for the execution of the data reconstruction operation in the data reconstruction queue. Compared with the traditional technology of directly acquiring object data to achieve data reconstruction and then releasing the placement group lock after setting the placement group lock on the target reconstructed object, the technical solution can effectively reduce the time for setting the placement group lock on the target reconstructed object in the data reconstruction process, and can further effectively reduce the impact of the placement group lock on the read and write business during the data reconstruction.

[0133] In one embodiment of the present application, the above-mentioned component module 2 can be specifically used to determine the object data range, object data size, and object data version number of the target reconstructed object; using the target reconstructed object as the key, and the object data range, object data size, and object data version number as the value, to generate a target key-value pair.

[0134] In one embodiment of the present application, the reconstruction module 4 can be specifically used to obtain a target key-value pair according to the key-value pair arrangement order in the data reconstruction queue; set an object lock, and obtain target object data and metadata of the target object data corresponding to the target key-value pair; use the target object data and metadata of the target object data to reconstruct data and obtain target reconstructed data;

[0135] Accordingly, the disk placement module 5 can be specifically used to call a preset communication interface to send the target reconstruction data to the backup object storage device, so that the backup object storage device can store the target reconstruction data on the disk and feedback a data storage completion message to the primary object storage device;

[0136] The disk placement module 5 may also be configured to release the object lock according to a data placement completion message after sending the target reconstructed data to the backup object storage device so that the backup object storage device places the target reconstructed data on the disk.

[0137] In one embodiment of the present application, the data reconstruction device may further include a read-write module, and the read-write module may include:

[0138] A determination unit, configured to determine a target read-write object according to the read-write service request when receiving the read-write service request, and to set a placement group lock on the target read-write object;

[0139] A waiting unit, used for, if the target read / write object is in the data reconstruction queue, entering a waiting state until the target read / write object leaves the data reconstruction queue;

[0140] The execution unit is used to set the object lock and execute the read-write service request according to the target read-write object to obtain the target read-write result;

[0141] The feedback unit is used to release the object lock, feed back the target read and write results to the read and write service request end, and release the placement group lock.

[0142] In one embodiment of the present application, the above-mentioned determination unit can be specifically used to determine whether the current redundancy strategy is an erasure strategy if the read-write business request is a read business request; if the current redundancy strategy is not an erasure strategy, determine the target read object according to the read business request, and set a placement group lock on the target read object; if the current redundancy strategy is an erasure strategy, determine the target read object according to the read business request, and do not set a placement group lock on the target read object.

[0143] In one embodiment of the present application, the above-mentioned determination unit can be specifically used to determine whether there is a preset flag in the read service request; if there is a preset flag in the read service request, it is determined that the current redundancy strategy is a correction and erasure strategy; if there is no preset flag in the read service request, it is determined that the current redundancy strategy is not a correction and erasure strategy.

[0144] In one embodiment of the present application, the data reconstruction device may also include a judgment module for determining the current data reconstruction mode before determining the object data information of the target reconstruction object and forming the target reconstruction object and the object data information into a target key-value pair; wherein the data reconstruction mode includes a full reconstruction mode and an incremental reconstruction mode; if the current data reconstruction mode is the full reconstruction mode, then executing the step of determining the object data information of the target reconstruction object and forming the target key-value pair with the target reconstruction object and the object data information.

[0145] For an introduction to the device provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0146] An embodiment of the present application provides an electronic device.

[0147] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application, and the electronic device may include:

[0148] A memory 11, used for storing computer programs;

[0149] The processor 10 can implement the steps of any one of the above-mentioned data reconstruction methods when being used to execute a computer program.

[0150] like Figure 4 , which is a schematic diagram of the composition structure of an electronic device, the electronic device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.

[0151] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0152] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in the embodiment of the data reconstruction method.

[0153] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0154] When a data reconstruction request is received, a target reconstruction object is determined according to the data reconstruction request, and a placement group lock is added to the target reconstruction object;

[0155] Determine the object data information of the target reconstruction object, and form a target key-value pair with the target reconstruction object and the object data information;

[0156] Reconstruct the target key-value pair input data into the queue and release the placement group lock for the target reconstruction object;

[0157] Reconstruct the target key-value pairs according to the order of the key-value pairs in the data reconstruction queue to obtain the target reconstructed data;

[0158] The target reconstructed data is sent to the standby object storage device, so that the standby object storage device writes the target reconstructed data to a disk.

[0159] In a possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0160] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0161] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0162] Of course, it should be noted that Figure 4 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include Figure 4 More or fewer components than shown, or combinations of certain components.

[0163] An embodiment of the present application provides a computer-readable storage medium.

[0164] The computer-readable storage medium provided in the embodiment of the present application stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned data reconstruction methods can be implemented.

[0165] Among them, the computer-readable storage medium can be any available medium that can be stored in a computer or a data storage device such as a server or data center that includes one or more available media integrated. For example, it can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape, etc.), an optical medium (such as a DVD) or a semiconductor medium (such as a solid-state hard disk) and other media that can store computer program codes.

[0166] For an introduction to the computer-readable storage medium provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0167] An embodiment of the present application provides a computer program product.

[0168] The computer program product provided in the embodiments of the present application includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of any of the above-mentioned data reconstruction methods can be implemented.

[0169] Specifically, in the above embodiments, all or part of them can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of them can be implemented in the form of a computer program product.

[0170] Among them, the computer program product may include one or more computer programs / instructions, and when the computer program / instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application may be generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line, etc.) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0171] For an introduction to the computer program product provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0172] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0173] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0174] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0175] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples herein, and the description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A data reconstruction method, characterized in that: Applied to a primary object storage device, the primary object storage device is communicatively connected with a backup object storage device, the method comprises: When a data reconstruction request is received, a target reconstruction object is determined according to the data reconstruction request, and a placement group lock is added to the target reconstruction object; Determine the object data information of the target reconstruction object, and form a target key-value pair with the target reconstruction object and the object data information; Input the target key-value pair into a data reconstruction queue, and release the placement group lock for the target reconstruction object; Reconstruct the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data; The target reconstructed data is sent to the standby object storage device, so that the standby object storage device writes the target reconstructed data to a disk.

2. The data reconstruction method according to claim 1, characterized in that: Determining object data information of the target reconstruction object, and combining the target reconstruction object and the object data information into a target key-value pair, including: Determine the object data range, object data size, and object data version number of the target reconstruction object; The target reconstruction object is used as a key, and the object data range, the object data size, and the object data version number are used as values ​​to generate the target key-value pair.

3. The data reconstruction method according to claim 1, characterized in that: Reconstructing the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data includes: Acquire the target key-value pair according to the arrangement order of the key-value pairs in the data reconstruction queue; Setting an object lock, and obtaining target object data and metadata of the target object data corresponding to the target key-value pair; Reconstructing the target object data and the metadata of the target object data to obtain the target reconstructed data; Accordingly, sending the target reconstructed data to the standby object storage device so that the standby object storage device writes the target reconstructed data to a disk includes: Calling a preset communication interface to send the target reconstruction data to the backup object storage device, so that the backup object storage device writes the target reconstruction data to a disk, and feeds back a data write completion message to the primary object storage device; After sending the target reconstructed data to the standby object storage device so that the standby object storage device stores the target reconstructed data on a disk, the method further includes: The object lock is released according to the data disk completion message.

4. The data reconstruction method according to claim 1, characterized in that: Also includes: When a read / write service request is received, a target read / write object is determined according to the read / write service request, and the placement group lock is added to the target read / write object; If the target read / write object is in the data reconstruction queue, entering a waiting state until the target read / write object leaves the data reconstruction queue; Setting an object lock, and executing the read / write service request according to the target read / write object to obtain a target read / write result; The object lock is released, the target read and write result is fed back to the read and write service request end, and the placement group lock is released.

5. The data reconstruction method according to claim 4, characterized in that: If the read / write service request is a read service request, determining a target read / write object according to the read / write service request, and setting the placement group lock on the target read / write object, including: Determine whether the current redundancy strategy is an erasure strategy; If the current redundancy strategy is not the erasure strategy, determining a target read object according to the read service request, and setting the placement group lock on the target read object; If the current redundancy strategy is the erasure strategy, a target read object is determined according to the read service request, and the placement group lock is not set on the target read object.

6. The data reconstruction method according to claim 5, characterized in that: Determine whether the current redundancy strategy is an erasure strategy, including: Determining whether a preset flag exists in the read service request; If the preset flag exists in the read service request, determining that the current redundancy strategy is the erasure strategy; If the preset flag does not exist in the read service request, it is determined that the current redundancy strategy is not the erasure strategy.

7. The data reconstruction method according to any one of claims 1 to 6, characterized in that: Before determining the object data information of the target reconstruction object and forming a target key-value pair with the target reconstruction object and the object data information, the method further includes: Determine the current data reconstruction mode; wherein the data reconstruction mode includes a full reconstruction mode and an incremental reconstruction mode; If the current data reconstruction mode is the full reconstruction mode, the step of determining the object data information of the target reconstruction object and forming a target key-value pair with the target reconstruction object and the object data information is performed.

8. A data reconstruction device, characterized in that: Applied to a primary object storage device, the primary object storage device is communicatively connected with a backup object storage device, the device comprises: A locking module, configured to, when receiving a data reconstruction request, determine a target reconstruction object according to the data reconstruction request, and set a placement group lock for the target reconstruction object; A building module, used for determining the object data information of the target reconstruction object, and forming a target key-value pair with the target reconstruction object and the object data information; An unlocking module, used for inputting the target key-value pair into a data reconstruction queue and releasing the placement group lock for the target reconstruction object; A reconstruction module, used to reconstruct the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data; The disk placement module is used to send the target reconstructed data to the backup object storage device, so that the backup object storage device can store the target reconstructed data to the disk.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data reconstruction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data reconstruction method according to any one of claims 1 to 7 are implemented.

Images