Data reconstruction method and device, electronic equipment and computer readable storage medium

Abstract

The application discloses a data reconstruction method and device, electronic equipment and a computer readable storage medium, which are applied to a main object storage device, the main object storage device is in communication connection with a backup object storage device, and the method comprises the following steps: when a data reconstruction request is received, a target reconstruction object is determined according to the data reconstruction request, and a placement group lock is added to the target reconstruction object; object data information of the target reconstruction object is determined, the target reconstruction object and the object data information are combined to form a target key-value pair; the target key-value pair is input into a data reconstruction queue, and the placement group lock of the target reconstruction object is released; data reconstruction is performed on the target key-value pair according to the arrangement order of the key-value pairs in the data reconstruction queue, target reconstruction data is obtained; and the target reconstruction data is sent to the backup object storage device, so that the backup object storage device performs data landing on the target reconstruction data. According to the technical scheme, the influence of the placement group lock on read-write services during data reconstruction can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data reconstruction method, and also to a data reconstruction device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the context of digital transformation, massive amounts of data are constantly growing, and distributed storage systems have become widely used. Distributed storage systems typically consist of a master server, storage servers, and multiple clients. Their essence is to evenly distribute large numbers of files across multiple storage servers, offering high scalability and reliability. Furthermore, distributed storage systems can be applied in a variety of scenarios. Failures are inevitable during cluster operation. When a hard drive fails, cluster redundancy is reduced. Furthermore, data recovery caused by the hard drive failure inevitably impacts front-end services. Therefore, how to quickly complete data reconstruction while minimizing the impact on user read and write services during the reconstruction period has become an unavoidable requirement. Research has found that the mutual exclusion between data reconstruction and read and write services primarily stems from contention for PG (Placement Group) locks and object locks. Further analysis confirms that PG lock contention is the most significant factor, indicating that the impact of data reconstruction on read and write services is primarily caused by PG lock contention.

[0003] Therefore, how to effectively reduce the impact of adding PG locks on read and write services during data reconstruction is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a data reconstruction method, which can effectively reduce the impact of adding PG locks on read and write services during data reconstruction; another purpose of this application is to provide a data reconstruction device, electronic device, computer-readable storage medium and computer program product, all of which have the above-mentioned beneficial effects.

[0005] In a first aspect, the present application provides a data reconstruction method, which is applied to a primary object storage device, wherein the primary object storage device is communicatively connected to a backup object storage device, and the method includes:

[0006] When a data reconstruction request is received, determining a target reconstruction object according to the data reconstruction request, and setting a placement group lock for the target reconstruction object;

[0007] Determine the object data information of the target reconstruction object, and form a target key-value pair with the target reconstruction object and the object data information;

[0008] Input the target key-value pair into a data reconstruction queue, and release the placement group lock for the target reconstruction object;

[0009] Reconstruct the target key-value pairs according to the order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data;

[0010] The target reconstructed data is sent to the standby object storage device, so that the standby object storage device writes the target reconstructed data to a disk.

[0011] Optionally, determining object data information of the target reconstruction object and combining the target reconstruction object and the object data information into a target key-value pair includes:

[0012] Determine the object data range, object data size, and object data version number of the target reconstructed object;

[0013] The target reconstructed object is used as a key, and the object data range, the object data size, and the object data version number are used as values ​​to generate the target key-value pair.

[0014] Optionally, reconstructing the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data includes:

[0015] Acquire the target key-value pair according to the arrangement order of the key-value pairs in the data reconstruction queue;

[0016] Setting an object lock, and obtaining target object data and metadata of the target object data corresponding to the target key-value pair;

[0017] Reconstructing the target object data and the metadata of the target object data to obtain the target reconstructed data;

[0018] Accordingly, sending the target reconstructed data to the backup object storage device so that the backup object storage device writes the target reconstructed data to a disk includes:

[0019] Invoking a preset communication interface to send the target reconstructed data to the backup object storage device, so that the backup object storage device writes the target reconstructed data to disk and feeds back a data write completion message to the primary object storage device;

[0020] After sending the target reconstructed data to the standby object storage device so that the standby object storage device writes the target reconstructed data to a disk, the method further includes:

[0021] The object lock is released according to the data disk completion message.

[0022] Optionally, the data reconstruction method further comprises:

[0023] When receiving the read-write service request, determining a target read-write object according to the read-write service request, and setting the disposition group lock on the target read-write object;

[0024] If the target read-write object is in the data reconstruction queue, entering a waiting state until the target read-write object leaves the data reconstruction queue;

[0025] Setting an object lock, and executing the read-write service request according to the target read-write object to obtain a target read-write result;

[0026] Releasing the object lock, feeding back the target read-write result to a read-write service request end, and releasing the disposition group lock.

[0027] Optionally, if the read-write service request is a read service request, determining a target read-write object according to the read-write service request, and setting the disposition group lock on the target read-write object, comprises:

[0028] Judging whether a current redundancy strategy is an erasure strategy;

[0029] If the current redundancy strategy is not the erasure strategy, determining a target read object according to the read service request, and setting the disposition group lock on the target read object;

[0030] If the current redundancy strategy is the erasure strategy, determining a target read object according to the read service request, and not setting the disposition group lock on the target read object.

[0031] Optionally, judging whether the current redundancy strategy is the erasure strategy, comprises:

[0032] Judging whether a preset flag exists in the read service request;

[0033] If the preset flag exists in the read service request, determining that the current redundancy strategy is the erasure strategy;

[0034] If the preset flag does not exist in the read service request, determining that the current redundancy strategy is not the erasure strategy.

[0035] Optionally, before determining object data information of the target reconstruction object, and grouping the target reconstruction object and the object data information into a target key-value pair, the method further comprises:

[0036] Determining a current data reconstruction mode; wherein the data reconstruction mode comprises a full reconstruction mode and an incremental reconstruction mode;

[0037] If the current data reconstruction mode is the full reconstruction mode, the step of determining the object data information of the target reconstruction object and forming a target key-value pair with the target reconstruction object and the object data information is performed.

[0038] In a second aspect, the present application further discloses a data reconstruction device, which is applied to a primary object storage device, wherein the primary object storage device is communicatively connected to a backup object storage device, and the device comprises:

[0039] a locking module configured to, upon receiving a data reconstruction request, determine a target reconstruction object according to the data reconstruction request and set a placement group lock for the target reconstruction object;

[0040] a forming module, configured to determine object data information of the target reconstruction object, and form a target key-value pair from the target reconstruction object and the object data information;

[0041] an unlocking module, configured to input the target key-value pair into a data reconstruction queue and release the placement group lock for the target reconstruction object;

[0042] A reconstruction module, configured to reconstruct the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data;

[0043] The disk write module is used to send the target reconstructed data to the backup object storage device, so that the backup object storage device writes the target reconstructed data to the disk.

[0044] In a third aspect, the present application further discloses an electronic device, comprising:

[0045] Memory for storing computer programs;

[0046] A processor is configured to implement the steps of any one of the data reconstruction methods described above when executing the computer program.

[0047] In a fourth aspect, the present application further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the data reconstruction methods described above are implemented.

[0048] In a fifth aspect, the present invention further discloses a computer program product, comprising a computer program / instruction, which implements the steps of any one of the data reconstruction methods described above when executed by a processor.

[0049] The present application provides a data reconstruction method, which is applied to a primary object storage device, wherein the primary object storage device is communicatively connected to a backup object storage device. The method comprises: when a data reconstruction request is received, determining a target reconstruction object according to the data reconstruction request, and setting a placement group lock for the target reconstruction object; determining object data information of the target reconstruction object, and forming a target key-value pair from the target reconstruction object and the object data information; inputting the target key-value pair into a data reconstruction queue, and releasing the placement group lock for the target reconstruction object; reconstructing data on the target key-value pair according to an arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data; and sending the target reconstructed data to the backup object storage device, so that the backup object storage device writes the target reconstructed data to a disk.

[0050] By applying the technical solution provided in the present application, in the process of data reconstruction using the primary and backup object storage devices, for the target reconstruction object, after setting a placement group lock on it, the object data information of the target reconstruction object is pulled into the data reconstruction queue, and the placement group lock is directly released to wait for the execution of the data reconstruction operation in the data reconstruction queue. Compared with the traditional technology of directly obtaining object data to realize data reconstruction and then releasing the placement group lock after setting a placement group lock on the target reconstruction object, the present technical solution can effectively reduce the time for setting a placement group lock on the target reconstruction object in the data reconstruction process, and further can effectively reduce the impact of setting a placement group lock on the read and write business during the data reconstruction period.

[0051] In one embodiment of the present application, the data reconstruction mode is further restricted, that is, the data reconstruction method provided by the present application is applied to the full reconstruction mode. It can be understood that the full reconstruction mode is a full data scan recovery, while the incremental reconstruction mode is an incremental data scan recovery. Since the incremental reconstruction mode needs to recover a small number of data objects, it is easy for the primary object storage device to miss data objects, resulting in data reconstruction failure. Therefore, limiting the data reconstruction mode to the full reconstruction mode can effectively ensure the successful execution of the data reconstruction operation.

[0052] The data reconstruction device, electronic device, computer-readable storage medium, and computer program product provided in this application also have the above-mentioned technical effects, and this application will not elaborate on them here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the following is a brief introduction to the drawings required for describing the prior art and the embodiments of the present application. Of course, the drawings described below in connection with the embodiments of the present application are only part of the embodiments of the present application. For those skilled in the art, other drawings can be obtained based on the provided drawings without inventive effort, and the obtained other drawings also fall within the scope of protection of the present application.

[0054] Figure 1 A flowchart of a data reconstruction method provided in this application;

[0055] Figure 2 A flowchart of another data reconstruction method provided in this application;

[0056] Figure 3 A schematic diagram of the structure of a data reconstruction device provided by this application;

[0057] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0058] The core of this application is to provide a data reconstruction method, which can effectively reduce the impact of adding PG locks on read and write services during data reconstruction; another core of this application is to provide a data reconstruction device, electronic device, computer-readable storage medium and computer program product, all of which have the above-mentioned beneficial effects.

[0059] In order to describe the technical solutions in the embodiments of the present application more clearly and completely, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0060] An embodiment of the present application provides a data reconstruction method.

[0061] Please refer to Figure 1 , Figure 1 This is a flow chart of a data reconstruction method provided in the present application. The data reconstruction method is applied to a primary object storage device, which is communicatively connected to a backup object storage device. The data reconstruction method may include the following S101 to S105.

[0062] S101: When a data reconstruction request is received, a target reconstruction object is determined according to the data reconstruction request, and a placement group lock is added to the target reconstruction object.

[0063] This step determines the target reconstruction object and sets a corresponding placement group lock (PG lock). Specifically, upon receiving a data reconstruction request, the target reconstruction object (i.e., the data object requiring reconstruction) is determined based on the request. A placement group lock is then set for the data object to facilitate subsequent data reconstruction.

[0064] It is understood that the target reconstruction objects are the data objects on the failed object storage devices within the current object storage system, and the object storage device failure is caused by the failure of one or more hard disks therein. Furthermore, the primary object storage device is the new object storage device selected for each target reconstruction object, and the backup object storage device is used to implement redundancy backup.

[0065] S102: Determine the object data information of the target reconstruction object, and combine the target reconstruction object and the object data information into a target key-value pair.

[0066] This step aims to obtain the object data information of the target reconstruction object and construct the corresponding target key-value pair. Specifically, after adding a placement group lock to the target reconstruction object, the object data information of the target reconstruction object can be obtained, such as the data range, data size, data version number, and other information described in the embodiments below. This object data information is used to achieve data reconstruction of the target data object; further, the target reconstruction object and the object data information are combined into a target key-value pair, so that it can be added to the data reconstruction queue to await the execution of the data reconstruction operation.

[0067] Determining the object data information of the target reconstruction object and forming a target key-value pair with the target reconstruction object and the object data information may include:

[0068] Determine the object data range, object data size, and object data version number of the target reconstruction object;

[0069] The target reconstructed object is used as the key, and the object data range, object data size, and object data version number are used as the value to generate the target key-value pair.

[0070] In one embodiment of the present application, before determining the object data information of the target reconstruction object and forming a target key-value pair with the target reconstruction object and the object data information, the following steps may also be included:

[0071] Determine the current data reconstruction mode; data reconstruction modes include full reconstruction mode (backfill mode) and incremental reconstruction mode (recovery mode);

[0072] If the current data reconstruction mode is the full reconstruction mode, the steps of determining the object data information of the target reconstruction object and forming a target key-value pair with the target reconstruction object and the object data information are executed.

[0073] It should be noted that if the current data reconstruction mode is the incremental reconstruction mode, the data reconstruction process in the prior art can be directly entered.

[0074] It can be seen that the embodiment of the present application further restricts the data reconstruction mode, that is, the data reconstruction method provided by the present application is applied to the full reconstruction mode. It can be understood that the full reconstruction mode is a full data scan and recovery, while the incremental reconstruction mode is an incremental data scan and recovery. Since the incremental reconstruction mode needs to recover a small number of data objects, it is easy for the primary object storage device to miss data objects, resulting in data reconstruction failure. Therefore, limiting the data reconstruction mode to the full reconstruction mode can effectively ensure the successful execution of the data reconstruction operation.

[0075] Determining the current data reconstruction mode may include:

[0076] Determine whether the placement group lock includes the full rebuild flag;

[0077] If the placement group lock includes the full reconstruction flag, the current data reconstruction mode is determined to be the full reconstruction mode;

[0078] If the placement group lock does not include a full reconstruction flag, the current data reconstruction mode is determined to be the incremental reconstruction mode.

[0079] That is, in full rebuild mode, the placement group lock set by the primary object storage device for the target object to be rebuilt will include the full rebuild flag. In incremental rebuild mode, the placement group lock set by the primary object storage device for the target object to be rebuilt will not include the full rebuild flag. Therefore, the current data reconstruction mode can be determined by determining whether the full rebuild flag is present in the placement group lock.

[0080] S103: Input the target key-value pair into the data reconstruction queue, and release the placement group lock for the target reconstruction object.

[0081] This step aims to input the target key-value pair into the data reconstruction queue and release the placement group lock. It will be appreciated that inputting the target key-value pair into the data reconstruction queue releases the placement group lock for the target reconstruction object, allowing the target data object to wait for the data reconstruction operation to be performed within the data reconstruction object. This eliminates the need for traditional techniques (which require directly acquiring object data to perform the data reconstruction operation after setting the placement group lock on the target reconstruction object, and then releasing the placement group lock after data reconstruction is complete), effectively reducing the time it takes to set the placement group lock on the target reconstruction object.

[0082] S104: reconstructing the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data.

[0083] This step aims to implement data reconstruction operations. Specifically, for each key-value pair in the data reconstruction queue, the data reconstruction operation can be performed in sequence according to the order in which they were added to the data reconstruction queue. This data reconstruction operation is based on the object data information in each key-value pair, and the reconstructed data corresponding to each key-value pair can be obtained.

[0084] S105: Send the target reconstructed data to the backup object storage device, so that the backup object storage device writes the target reconstructed data to a disk.

[0085] This step is to store the reconstructed data on disk. Specifically, after the target key-value pair is reconstructed and the target reconstructed data is obtained, it can be sent to the backup object storage device for storage to achieve redundant backup. This completes the data reconstruction operation.

[0086] In one embodiment of the present application, reconstructing target key-value pairs according to the order of key-value pairs in the data reconstruction queue to obtain target reconstructed data may include:

[0087] Obtain the target key-value pairs according to the order of key-value pairs in the data reconstruction queue;

[0088] Set an object lock and obtain the target object data and metadata corresponding to the target key-value pair;

[0089] Reconstructing the data using the target object data and the metadata of the target object data to obtain target reconstructed data;

[0090] Accordingly, sending the target reconstructed data to the backup object storage device so that the backup object storage device writes the target reconstructed data to disk may include: calling a preset communication interface to send the target reconstructed data to the backup object storage device so that the backup object storage device writes the target reconstructed data to disk, and feeding back a data write completion message to the primary object storage device;

[0091] After sending the target reconstructed data to the standby object storage device so that the standby object storage device writes the target reconstructed data to the disk, the method further includes: releasing the object lock according to a data write completion message.

[0092] The embodiment of the present application provides a data reconstruction method based on key-value pairs. It is understandable that the data reconstruction process based on key-value pairs relies on the setting of object locks to effectively avoid the impact of other businesses on the target reconstruction object during the data reconstruction process, causing data object usage conflicts.

[0093] During the implementation process, after reading the target key-value pair from the data reconstruction queue, an object lock can be set and the data reconstruction operation can be performed. During this process, data reconstruction can be achieved through the target object data and the metadata of the target object data in the target key-value pair. Furthermore, after completing the data reconstruction and obtaining the target reconstructed data and sending it to the backup object storage device for disk storage, the backup object storage device will feedback a data disk storage completion message to the primary object storage device, so that the primary object storage device releases the object lock based on the data disk storage completion message.

[0094] It can be seen that the data reconstruction method provided by the embodiment of the present application, in the process of data reconstruction using the primary and backup object storage devices, for the target reconstruction object, after setting the placement group lock on it, the object data information of the target reconstruction object is pulled into the data reconstruction queue, and the placement group lock is directly released to wait for the execution of the data reconstruction operation in the data reconstruction queue. Compared with the traditional technology of directly obtaining object data to realize data reconstruction and then releasing the placement group lock after setting the placement group lock on the target reconstruction object, this technical solution can effectively reduce the time for setting the placement group lock on the target reconstruction object in the data reconstruction process, and can further effectively reduce the impact of the placement group lock on the read and write business during the data reconstruction period.

[0095] Based on the above embodiment:

[0096] In one embodiment of the present application, the data reconstruction method may further include:

[0097] When a read / write service request is received, the target read / write object is determined according to the read / write service request, and a placement group lock is added to the target read / write object;

[0098] If the target read / write object is in the data reconstruction queue, it enters the waiting state until the target read / write object leaves the data reconstruction queue;

[0099] Set the object lock and execute the read and write business request according to the target read and write object to obtain the target read and write results;

[0100] Release the object lock, feed back the target read and write results to the read and write service request end, and release the placement group lock.

[0101] An embodiment of the present application provides a method for processing read and write services during data reconstruction. The data reconstruction method provided by the embodiment of the present application can effectively reduce the impact of data reconstruction operations on front-end read and write services.

[0102] Specifically, when receiving a read-write service request, a target read-write object can be determined according to the read-write service request, and a storage group lock is set for the target read-write object. At this time, the read-write service request is not directly executed, but it is determined whether the target read-write object is in the data reconstruction queue. It can be understood that if the target read-write object is in the data reconstruction queue, it means that the target read-write object is waiting for execution of a data reconstruction operation. At this time, if the data read-write operation is directly executed, it will inevitably only get a failed result. Therefore, a waiting state can be entered until the target read-write object leaves the data reconstruction queue, that is, waiting for its data reconstruction to be completed, and then performing a data read-write operation on it to ensure the success of the data read-write operation and obtain a target read-write result. Finally, the object lock is released, the target read-write result is fed back, and the storage group lock is released.

[0103] If the read-write service request is a read service request, the target read-write object is determined according to the read-write service request, and a storage group lock is set for the target read-write object. This can include:

[0104] It is determined whether the current redundancy strategy is a correction strategy.

[0105] If the current redundancy strategy is not a correction strategy, the target read object is determined according to the read service request, and a storage group lock is set for the target read object.

[0106] If the current redundancy strategy is a correction strategy, the target read object is determined according to the read service request, and no storage group lock is set for the target read object.

[0107] The data reconstruction method provided by the embodiment of the application can also be applied to a correction scenario, effectively improving the applicability of the technical solution. However, in the correction scenario, the difference is that for a read service request, the target read object does not need to be set with a storage group lock, so as to effectively reduce the influence of the data reconstruction process on the front-end service in the correction scenario. For other redundancy strategy scenarios, such as a copy scenario, the target read object needs to be set with a storage group lock, and the steps S101-S105 are executed.

[0108] Further, it is determined whether the current redundancy strategy is a correction strategy. This can include:

[0109] It is determined whether a preset flag exists in the read service request.

[0110] If the preset flag exists in the read service request, it is determined that the current redundancy strategy is a correction strategy.

[0111] If the preset flag does not exist in the read service request, it is determined that the current redundancy strategy is not a correction strategy.

[0112] In other words, in erasure scenarios, the read request initiated by the frontend will include the erasure redundancy flag (the aforementioned preset flag). In non-erasure scenarios, the read request will not include the erasure redundancy flag. Therefore, the current redundancy policy can be determined by determining whether the erasure redundancy flag is present in the read request.

[0113] Based on the above embodiments, an embodiment of the present application provides another data reconstruction method.

[0114] Please refer to Figure 2 , Figure 2 This is a flowchart of another data reconstruction method provided by this application, and its implementation process is as follows:

[0115] 1. During business operation, a hard disk fails and the corresponding OSD (Object-based StorageDevice) goes down. After a period of time (the default is 10 minutes), the OSD goes out (permanently offline), and a new member (primary OSD) is selected for the PG of the OSD to start data reconstruction.

[0116] 2. Data reconstruction is triggered on the primary OSD. The primary OSD determines the data object to be recovered (target reconstruction object), sets a PG lock on it, and then starts the preparation work before reconstruction: calculate the data range and size that needs to be reconstructed, obtain the version number of the data object to be recovered, etc., put this information into recovery_info, and then put the data object to be recovered and recovery_info into the background queue (data reconstruction queue). The way to put it in is to use the data pair to be recovered as the key and recovery_info as the value, and then release the PG lock.

[0117] 3. The background thread traverses the data reconstruction queue and begins data reconstruction if it finds data. It obtains the data object to be recovered and the recovery_info information from the data reconstruction queue, applies an object lock (read lock), reads the object data and metadata from the disk, constructs a push operation (starting data reconstruction), and calls an interface to send a push message (target reconstruction data) to the backup OSD.

[0118] 4. After receiving the push message sent by the primary OSD, the backup OSD starts to download the data to the disk. After the download is successful, it responds to the primary OSD with a push replay message (data download completion message).

[0119] 5. After receiving the push replay message, the primary OSD releases the object lock and waits for the object recovery op to re-enter the queue, and the recovery process ends.

[0120] 6. For write business requests, after the primary OSD receives the write business request, it starts to set the PG lock, and then starts to determine whether the write object is recovering. If the write object is in the data reconstruction queue, it means that the write object is waiting for recovery, and the write business request can be placed in the waiting queue for waiting; if the write object is not in the recovery state, it directly sets the object lock (write lock), builds the write transaction, and then downloads the transaction to the disk, and forwards the write transaction to the backup OSD, releasing the object lock. After all forwarded write business requests are responded to, the PG lock is released and the write process ends.

[0121] 7. For the read business process, the main OSD first adds a PG lock, then adds an object lock (read lock), and starts reading data. After the data is read, the object lock is released, a response message is sent to the client, and then the PG lock is released.

[0122] It should be noted that the optimization of the above data reconstruction process is mainly aimed at the replica scenario, but it is also applicable to the erasure scenario. The difference is that when reading data, a corresponding flag needs to be added to the read business request to mark that the PG lock will no longer be added when reading data on the backup OSD, thereby reducing the impact of data reconstruction operations on the front-end business in the erasure scenario.

[0123] In addition, the data reconstruction process has two modes: recovery mode and backfill mode. The former is incremental data recovery, and the latter is full data recovery. Because the number of data objects to be recovered in recovery mode is relatively small, it is easy for the primary object storage device to lose data objects, causing data reconstruction failure. Therefore, the above-mentioned reconstruction optimization is not required for the recovery reconstruction scenario, that is, it is only performed for the backfill reconstruction scenario. Furthermore, the recovery and backfill scenarios can be distinguished by adding corresponding flags to the PG lock to distinguish whether to enter the reconstruction optimization process.

[0124] It can be seen that the data reconstruction method provided by the embodiment of the present application, in the process of data reconstruction using the primary and backup object storage devices, for the target reconstruction object, after setting the placement group lock on it, the object data information of the target reconstruction object is pulled into the data reconstruction queue, and the placement group lock is directly released to wait for the execution of the data reconstruction operation in the data reconstruction queue. Compared with the traditional technology of directly obtaining object data to realize data reconstruction and then releasing the placement group lock after setting the placement group lock on the target reconstruction object, this technical solution can effectively reduce the time for setting the placement group lock on the target reconstruction object in the data reconstruction process, and can further effectively reduce the impact of the placement group lock on the read and write business during the data reconstruction period.

[0125] An embodiment of the present application provides a data reconstruction device.

[0126] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of a data reconstruction device provided by the present application. The data reconstruction device is applied to a primary object storage device. The primary object storage device is communicatively connected to a backup object storage device. The data reconstruction device may include:

[0127] Locking module 1 is used for determining a target reconstruction object according to the data reconstruction request when receiving a data reconstruction request, and setting a placement group lock for the target reconstruction object;

[0128] A forming module 2 is used to determine the object data information of the target reconstruction object and form a target key-value pair with the target reconstruction object and the object data information;

[0129] Unlocking module 3, used to input the target key-value pair into the data reconstruction queue and release the placement group lock for the target reconstruction object;

[0130] Reconstruction module 4, used to reconstruct the target key-value pairs according to the order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data;

[0131] The disk write module 5 is configured to send the target reconstructed data to the backup object storage device, so that the backup object storage device writes the target reconstructed data to the disk.

[0132] It can be seen that the data reconstruction device provided by the embodiment of the present application, in the process of data reconstruction using the primary and backup object storage devices, for the target reconstruction object, after setting the placement group lock on it, pulls the object data information of the target reconstruction object into the data reconstruction queue, and directly releases the placement group lock to wait for the execution of the data reconstruction operation in the data reconstruction queue. Compared with the traditional technology of directly obtaining object data to realize data reconstruction and then releasing the placement group lock after setting the placement group lock on the target reconstruction object, this technical solution can effectively reduce the time for setting the placement group lock on the target reconstruction object in the data reconstruction process, and can further effectively reduce the impact of the placement group lock on the read and write business during the data reconstruction period.

[0133] In one embodiment of the present application, the above-mentioned component module 2 can be specifically used to determine the object data range, object data size, and object data version number of the target reconstructed object; use the target reconstructed object as the key, and the object data range, object data size, and object data version number as the value to generate a target key-value pair.

[0134] In an embodiment of the present application, the reconstruction module 4 can be specifically configured to obtain a target key-value pair according to a data reconstruction order of the key-value pairs in the queue; set an object lock, and obtain target object data corresponding to the target key-value pair and metadata of the target object data; and perform data reconstruction on the target object data and the metadata of the target object data to obtain target reconstruction data.

[0135] Correspondingly, the landing module 5 can be specifically configured to call a preset communication interface to send the target reconstruction data to the backup object storage device, so that the backup object storage device performs data landing on the target reconstruction data, and feeds back a data landing completion message to the main object storage device.

[0136] The landing module 5 can also be configured to release the object lock according to the data landing completion message after the target reconstruction data is sent to the backup object storage device to perform data landing on the target reconstruction data.

[0137] In an embodiment of the present application, the data reconstruction device can further include a read-write module, which can include:

[0138] A determination unit configured to determine a target read-write object according to the read-write service request when the read-write service request is received, and set a placement group lock on the target read-write object;

[0139] A waiting unit configured to enter a waiting state until the target read-write object leaves the data reconstruction queue if the target read-write object is in the data reconstruction queue;

[0140] An execution unit configured to set an object lock, and execute the read-write service request according to the target read-write object to obtain a target read-write result;

[0141] A feedback unit configured to release the object lock, feed back the target read-write result to the read-write service request end, and release the placement group lock.

[0142] In an embodiment of the present application, the determination unit can be specifically configured to determine whether the current redundancy strategy is an erasure coding strategy if the read-write service request is a read service request; determine a target read object according to the read service request and set a placement group lock on the target read object if the current redundancy strategy is not the erasure coding strategy; and determine a target read object according to the read service request and not set a placement group lock on the target read object if the current redundancy strategy is the erasure coding strategy.

[0143] In an embodiment of the present application, the determination unit can be specifically configured to determine whether a preset flag exists in the read service request; determine that the current redundancy strategy is the erasure coding strategy if the preset flag exists in the read service request; and determine that the current redundancy strategy is not the erasure coding strategy if the preset flag does not exist in the read service request.

[0144] In an embodiment of the present application, the data reconstruction apparatus can further comprise a judging module for determining a current data reconstruction mode before determining the object data information of the target reconstruction object and before forming the target key-value pair by combining the target reconstruction object and the object data information; wherein the data reconstruction mode comprises a full-amount reconstruction mode and an incremental reconstruction mode; if the current data reconstruction mode is the full-amount reconstruction mode, the steps of determining the object data information of the target reconstruction object and forming the target key-value pair by combining the target reconstruction object and the object data information are executed.

[0145] The apparatus provided in the embodiments of the present application is introduced above with reference to the above method embodiments, and thus the present application will not make further description here.

[0146] The embodiments of the present application provide an electronic device.

[0147] Please refer to Figure 4 , Figure 4 The structure schematic diagram of the electronic device provided in the present application can comprise:

[0148] The memory 11 is used for storing a computer program;

[0149] The processor 10 is used for executing the computer program and can realize the steps of any one of the above data reconstruction methods.

[0150] As shown in Figure 4 , the composition structure schematic diagram of the electronic device can comprise a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 can complete mutual communication through the communication bus 13.

[0151] In the embodiments of the present application, the processor 10 can be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0152] The processor 10 can call the program stored in the memory 11, and specifically, the processor 10 can execute the operations in the embodiments of the data reconstruction method.

[0153] The memory 11 is used for storing one or more programs, and the program can comprise program code including computer operation instructions. In the embodiments of the present application, the memory 11 at least stores a program for realizing the following functions:

[0154] When receiving the data reconstruction request, the target reconstruction object is determined according to the data reconstruction request, and a group lock is set for the target reconstruction object;

[0155] Determine the object data information of the target reconstruction object, and form a target key-value pair with the target reconstruction object and the object data information;

[0156] Reconstruct the target key-value pair input data into the queue and release the placement group lock for the target reconstruction object;

[0157] Reconstruct the target key-value pairs according to the order of the key-value pairs in the data reconstruction queue to obtain the target reconstructed data;

[0158] The target reconstructed data is sent to the standby object storage device, so that the standby object storage device writes the target reconstructed data to a disk.

[0159] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0160] In addition, the memory 11 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0161] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0162] Of course, it needs to be explained that Figure 4 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include Figure 4 More or fewer components than shown, or combinations of certain components.

[0163] An embodiment of the present application provides a computer-readable storage medium.

[0164] The computer-readable storage medium provided in the embodiment of the present application stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned data reconstruction methods can be implemented.

[0165] Among them, computer-readable storage media can be any available medium that can be stored by a computer or a data storage device such as a server or data center that integrates one or more available media. For example, it can be magnetic media (such as floppy disks, hard disks, tapes, etc.), optical media (such as DVDs) or semiconductor media (such as solid-state drives) and other media that can store computer program codes.

[0166] For an introduction to the computer-readable storage medium provided in the embodiments of the present application, please refer to the above method embodiments, and this application will not elaborate on them here.

[0167] An embodiment of the present application provides a computer program product.

[0168] The computer program product provided in the embodiments of the present application includes a computer program / instruction, which, when executed by a processor, can implement the steps of any of the above-mentioned data reconstruction methods.

[0169] Specifically, in each of the above embodiments, all or part of the embodiments may be implemented through software, hardware, firmware, or any combination thereof. When implemented through software, all or part of the embodiments may be implemented in the form of a computer program product.

[0170] Among them, the computer program product may include one or more computer programs / instructions, which, when loaded and executed on a computer, may generate, in whole or in part, the processes or functions described in the embodiments of the present application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line, etc.) or wireless (e.g., infrared, wireless, microwave, etc.) method.

[0171] For an introduction to the computer program product provided in the embodiments of the present application, please refer to the above method embodiments, which will not be elaborated in this application.

[0172] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0173] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0174] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0175] The technical solution provided by the present application is described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications may be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A data reconstruction method, characterized in that: Applied to a primary object storage device, the primary object storage device being communicatively connected to a backup object storage device, the method includes: When a data reconstruction request is received, determining a target reconstruction object according to the data reconstruction request, and setting a placement group lock for the target reconstruction object; Determine the object data information of the target reconstruction object, and form a target key-value pair with the target reconstruction object and the object data information; Input the target key-value pair into a data reconstruction queue, and release the placement group lock for the target reconstruction object; Reconstruct the target key-value pairs according to the order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data; The target reconstructed data is sent to the standby object storage device, so that the standby object storage device writes the target reconstructed data to a disk.

2. The data reconstruction method according to claim 1, characterized in that: Determining object data information of the target reconstruction object, and combining the target reconstruction object and the object data information into a target key-value pair, including: Determine the object data range, object data size, and object data version number of the target reconstructed object; The target reconstructed object is used as a key, and the object data range, the object data size, and the object data version number are used as values ​​to generate the target key-value pair.

3. The data reconstruction method according to claim 1, characterized in that: Reconstructing the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data includes: Acquire the target key-value pair according to the arrangement order of the key-value pairs in the data reconstruction queue; Setting an object lock, and obtaining target object data and metadata of the target object data corresponding to the target key-value pair; Reconstructing the target object data and the metadata of the target object data to obtain the target reconstructed data; Accordingly, sending the target reconstructed data to the backup object storage device so that the backup object storage device writes the target reconstructed data to a disk includes: Invoking a preset communication interface to send the target reconstructed data to the backup object storage device, so that the backup object storage device writes the target reconstructed data to disk and feeds back a data write completion message to the primary object storage device; After sending the target reconstructed data to the standby object storage device so that the standby object storage device writes the target reconstructed data to a disk, the method further includes: The object lock is released according to the data disk completion message.

4. The data reconstruction method according to claim 1, characterized in that: Also includes: When a read / write service request is received, determining a target read / write object according to the read / write service request, and setting the placement group lock on the target read / write object; If the target read / write object is in the data reconstruction queue, then enter a waiting state until the target read / write object leaves the data reconstruction queue; Setting an object lock, and executing the read / write service request according to the target read / write object to obtain the target read / write result; The object lock is released, the target read / write result is fed back to the read / write service request end, and the placement group lock is released.

5. The data reconstruction method according to claim 4, characterized in that: If the read / write service request is a read service request, determining a target read / write object according to the read / write service request, and setting the placement group lock on the target read / write object, including: Determine whether the current redundancy strategy is an erasure strategy; If the current redundancy strategy is not the erasure strategy, determining a target read object according to the read service request, and setting the placement group lock on the target read object; If the current redundancy strategy is the erasure strategy, a target read object is determined according to the read service request, and the placement group lock is not set on the target read object.

6. The data reconstruction method according to claim 5, characterized in that: Determine whether the current redundancy strategy is an erasure strategy, including: Determining whether a preset flag exists in the read service request; If the preset flag exists in the read service request, determining that the current redundancy strategy is the erasure strategy; If the preset flag does not exist in the read service request, it is determined that the current redundancy strategy is not the erasure strategy.

7. The data reconstruction method according to any one of claims 1 to 6, characterized in that: Before determining the object data information of the target reconstruction object and forming a target key-value pair with the target reconstruction object and the object data information, the method further includes: Determine the current data reconstruction mode; wherein the data reconstruction mode includes a full reconstruction mode and an incremental reconstruction mode; If the current data reconstruction mode is the full reconstruction mode, the step of determining the object data information of the target reconstruction object and forming a target key-value pair with the target reconstruction object and the object data information is performed.

8. A data reconstruction device, characterized in that: Applied to a primary object storage device, the primary object storage device being communicatively connected to a backup object storage device, the apparatus comprises: a locking module configured to, upon receiving a data reconstruction request, determine a target reconstruction object according to the data reconstruction request and set a placement group lock for the target reconstruction object; a forming module, configured to determine object data information of the target reconstruction object, and form a target key-value pair from the target reconstruction object and the object data information; an unlocking module, configured to input the target key-value pair into a data reconstruction queue and release the placement group lock for the target reconstruction object; A reconstruction module, configured to reconstruct the target key-value pairs according to the arrangement order of the key-value pairs in the data reconstruction queue to obtain target reconstructed data; The disk write module is used to send the target reconstructed data to the backup object storage device, so that the backup object storage device writes the target reconstructed data to the disk.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data reconstruction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data reconstruction method according to any one of claims 1 to 7 are implemented.

Images