Data recovery method, device and equipment of distributed storage cluster and storage medium

Abstract

The application discloses a data recovery method and device of a distributed storage cluster, equipment and a storage medium, relates to the technical field of cloud computing distributed storage, and comprises the following steps: in the distributed storage cluster, a timeout configuration item is constructed, and a timeout timer is determined based on the timeout configuration item; when it is monitored that the state of a storage node changes, after a first homing group successfully applies for first resources of a first preset object storage device, triggering the operation of a second homing group applying for second resources of a second preset object storage device and triggering the timeout timer to start; within the time period represented by the timeout timer, it is monitored whether the second homing group successfully applies for the second resources, and a corresponding monitoring result is obtained; if the monitoring result indicates yes, a preset data migration mechanism is triggered through the first homing group and the second homing group, so as to complete the data recovery operation of the distributed storage cluster. Therefore, the application can optimize the data recovery and balance of the Ceph cluster.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing distributed storage, and particularly relates to a data recovery method and device of a distributed storage cluster, equipment and a storage medium. BACKGROUND

[0002] Under the promotion of cloud computing, big data and artificial intelligence technology, a distributed storage system is the core of modern information infrastructure. Ceph, a kind of distributed storage system, is an open source system. In actual operation and maintenance, it faces challenges such as node failure, capacity expansion and uneven data distribution, which affect the performance and stability of the cluster.

[0003] When a storage node in a Ceph cluster fails, the existing mechanism has limited self-healing capability. In a large-scale cluster, the recovery efficiency is low and resource consumption is large, which is difficult to meet the real-time response demand of a high-availability scenario and may cause data unavailability risk. Capacity expansion involves multiple links. Improper operation may easily cause uneven load, performance fluctuation and even service interruption. When there are many nodes, data migration and rebalancing are required, and it is difficult to reduce the impact on business. Unbalanced data distribution is caused by unreasonable initial strategy, and the load of hot nodes is too high. The automatic rebalancing mechanism of Ceph lacks intelligent perception and optimization capability in a complex dynamic environment, and it is difficult to achieve efficient dynamic load adjustment.

[0004] Therefore, how to optimize the data recovery and balancing of the Ceph cluster is a technical problem to be solved at present. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a data recovery method and device of a distributed storage cluster, equipment and a storage medium, which can optimize the data recovery and balancing of the Ceph cluster. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a data recovery method of a distributed storage cluster, comprising:

[0007] In the distributed storage cluster, a timeout configuration item is constructed, and a timeout timer is determined based on the timeout configuration item;

[0008] When it is monitored that the state of a storage node changes, after a first placement group successfully applies for a first resource of a first preset object storage device, an operation of a second placement group applying for a second resource of a second preset object storage device is triggered and the timeout timer is started; wherein the first placement group is located in the first preset object storage device and does not have a preset association relationship with the change in the state of the storage node; the second placement group is located in the second preset object storage device and has the preset association relationship with the change in the state of the storage node;

[0009] monitoring whether the second placement group successfully applies for the second resource within a time period represented by the timeout timer, and obtaining a corresponding monitoring result;

[0010] if the monitoring result indicates yes, triggering a preset data migration mechanism through the first placement group and the second placement group to complete a data recovery operation of the distributed storage cluster.

[0011] Optionally, when it is monitored that the state of the storage node changes, after the first placement group successfully applies for the first resource of the first preset object storage device, triggering an operation of the second placement group applying for the second resource of the second preset object storage device and triggering the timeout timer to start, comprising:

[0012] when it is monitored that the storage node fails or is expanded, adding the first placement group to a first resource application queue of the first preset object storage device, so as to apply for the first resource of the first preset object storage device through the first placement group;

[0013] if the first placement group has successfully applied for the first resource, while triggering the timeout timer to start, putting the second placement group into a second resource application queue of the second preset object storage device through a preset message mechanism, so as to apply for the second resource of the second preset object storage device through the second placement group.

[0014] Optionally, if the monitoring result indicates yes, triggering a preset data migration mechanism through the first placement group and the second placement group, comprising:

[0015] if the monitoring result indicates yes, closing the timeout timer;

[0016] traversing a target object in the first preset object storage device through the first placement group, and sending target data of the target object to the second placement group, so that the second placement group receives the target data.

[0017] Optionally, the obtaining a corresponding monitoring result further comprises:

[0018] if the monitoring result indicates no, releasing the first resource through the first placement group, and re-adding the first placement group to the first resource application queue;

[0019] prohibiting the second placement group from applying for the second resource of the second preset object storage device, triggering a new first placement group to apply for the first resource of the first preset object storage device, and adding the new first placement group to the first resource application queue.

[0020] Optionally, the data recovery method of the distributed storage cluster further comprises:

[0021] When the second placement group successfully applies for the second resource within the time period, a first message notification that the second placement group has acquired the second resource is generated, and the first message notification is sent to the first placement group;

[0022] When the second placement group fails to successfully apply for the second resource within the time period, a second message notification that the second placement group has not acquired the second resource is generated, and the second message notification is sent to the first placement group.

[0023] Optionally, the timeout configuration item is a configuration item that can be modified when the preset object storage device is in a running state.

[0024] In a second aspect, the present application provides a data recovery apparatus of a distributed storage cluster, comprising:

[0025] A timer determination module is configured to construct a timeout configuration item in the distributed storage cluster, and determine a timeout timer based on the timeout configuration item.

[0026] A resource application module is configured to, when a state change of a storage node is monitored, trigger a second placement group to apply for a second resource of a second preset object storage device and trigger the timeout timer to start after a first placement group successfully applies for a first resource of a first preset object storage device; wherein the first placement group is located in the first preset object storage device, and does not have a preset association relationship with the state change of the storage node; the second placement group is located in the second preset object storage device, and has the preset association relationship with the state change of the storage node.

[0027] A result monitoring module is configured to monitor whether the second placement group successfully applies for the second resource within a time period represented by the timeout timer, and obtain a corresponding monitoring result.

[0028] A data recovery module is configured to, if the monitoring result indicates that the second placement group successfully applies for the second resource, trigger a preset data migration mechanism through the first placement group and the second placement group to complete a data recovery operation of the distributed storage cluster.

[0029] In a third aspect, the present application provides an electronic device, comprising:

[0030] A memory is configured to save a computer program.

[0031] A processor is configured to execute the computer program to implement the data recovery method of the distributed storage cluster as described above.

[0032] In a fourth aspect, the present application provides a computer readable storage medium for storing a computer program, wherein the computer program is executed by a processor to implement the data recovery method of the distributed storage cluster.

[0033] In the present application, in the distributed storage cluster, a timeout configuration item is constructed, and a timeout timer is determined based on the timeout configuration item; when it is monitored that the state of a storage node changes, after a first homing group successfully applies for a first resource of a first preset object storage device, an operation of applying for a second resource of a second preset object storage device by a second homing group is triggered, and the timeout timer is started; wherein the first homing group is located in the first preset object storage device, and does not have a preset association relationship with the change of the state of the storage node; the second homing group is located in the second preset object storage device, and has the preset association relationship with the change of the state of the storage node; within a time period represented by the timeout timer, it is monitored whether the second homing group successfully applies for the second resource, and a corresponding monitoring result is obtained; if the monitoring result indicates yes, a preset data migration mechanism is triggered through the first homing group and the second homing group to complete the data recovery operation of the distributed storage cluster. As can be seen from the above, in the present application, in the distributed storage cluster, a timeout configuration item is first established, and then a timeout timer is determined according to the timeout configuration item; if it is detected that the state of the storage node changes, then after the first homing group successfully applies for the first resource of the first preset object storage device, the operation of applying for the second resource of the second preset object storage device by the second homing group is triggered, and the timeout timer is started; within the time period represented by the timeout timer, it is detected whether the second homing group successfully applies for the second resource, and a corresponding detection result is obtained; if the detection result is yes, a preset data migration mechanism is triggered through the first homing group and the second homing group, so as to complete the data recovery operation of the distributed storage cluster. In this way, the present application can solve the problem that the first homing group occupies the first resource for a long time without releasing due to the fact that the second homing group does not obtain the second resource, thereby optimizing the data recovery and balance of the Ceph cluster. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the provided drawings.

[0035] Figure 1 A data recovery method flow chart of a distributed storage cluster disclosed in the present application;

[0036] Figure 2A specific distributed storage cluster data recovery method flowchart disclosed by the present application is provided.

[0037] Figure 3 A distributed storage cluster data recovery device structure schematic diagram disclosed by the present application is provided.

[0038] Figure 4 A structure diagram of an electronic device disclosed by the present application is provided. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0040] Currently, if a storage node in a Ceph cluster fails, the existing mechanism has limited self-healing capability, and the recovery efficiency is low and resource consumption is large in a large-scale cluster, which is difficult to meet the real-time response demand of a high-availability scene and may cause data unavailability risk. Expansion involves multiple links, and improper operation may easily cause uneven load, performance fluctuation or even service interruption. When there are many nodes, data migration and rebalancing are required, and it is difficult to reduce the impact on business. Unbalanced data distribution is caused by unreasonable initial strategy, etc., and the load of hot nodes is too high. The Ceph automatic rebalancing mechanism lacks intelligent perception and optimization capability in a complex dynamic environment, and it is difficult to achieve efficient dynamic load adjustment. Therefore, the present application provides a distributed storage cluster data recovery method, device, equipment and storage medium, which can optimize the data recovery and balance of the Ceph cluster.

[0041] Referring to Figure 1 and Figure 2 The embodiments of the present application disclose a distributed storage cluster data recovery method, which comprises the following steps:

[0042] Step S11, in the distributed storage cluster, a timeout configuration item is constructed, and a timeout timer is determined based on the timeout configuration item.

[0043] In the embodiments, in the distributed storage cluster, a timeout configuration item is constructed, and a timeout timer is determined based on the constructed timeout configuration item. The timeout configuration item is a runtime type, that is, a configuration item that can be modified when a preset object storage device is in a running state. The timeout configuration item will be used to limit the resource application time length of the second placement group in the subsequent process, so as to avoid the recovery process blockage caused by resource competition.

[0044] Step S12, when the state of the storage node is changed, the operation of the second placement group applying for the second resource of the second preset object storage device is triggered and the timeout timer is started after the first placement group successfully applies for the first resource of the first preset object storage device; wherein the first placement group is located in the first preset object storage device and has no preset association relationship with the state change of the storage node; the second placement group is located in the second preset object storage device and has the preset association relationship with the state change of the storage node.

[0045] In the embodiment, when the state of the storage node is changed, that is, when the storage node failure, capacity expansion or data balancing is monitored, the first placement group (that is, the main PG, Placement Group, the basic unit of data distribution in Ceph) is added to the first resource application queue (that is, the LOCAL resource application queue) of the first preset object storage device (that is, the OSD, Object Storage Device), so as to apply for the first resource (that is, the LOCAL resource) of the first preset object storage device through the first placement group. The LOCAL resource has a maximum concurrent quantity limit, and the default is 2, that is, a single OSD has at most two main PGs to perform the BACKFILL process (that is, a data repair method in Ceph) concurrently. It should be emphasized that the first placement group is located in the first preset object storage device and has no preset association relationship with the state change of the storage node; the second placement group is located in the second preset object storage device and has the preset association relationship with the state change of the storage node.

[0046] In a specific embodiment, if the first placement group has successfully applied for the first resource, the second placement group (that is, the replica PG) is put into the second resource application queue of the second preset object storage device through a preset message mechanism at the same time when the timeout timer is started, so as to apply for the second resource (that is, the REMOTE resource) of the second preset object storage device through the second placement group. The REMOTE resource has a maximum concurrent quantity limit, and the default is 2, that is, a single OSD has at most two replica PGs to perform the BACKFILL process concurrently.

[0047] Step S13, within the time period represented by the timeout timer, whether the second placement group successfully applies for the second resource is monitored, and a corresponding monitoring result is obtained.

[0048] In this embodiment, it is needed to monitor whether the second placement group successfully applies for the second resource within the time period represented by the timeout timer. The monitoring result can be determined according to the resource allocation confirmation information returned by the second preset object storage device. If the confirmation information is received before the preset threshold of the timeout timer is reached, it is determined that the application is successful. Otherwise, it is determined that the application fails.

[0049] In a specific embodiment, if the monitoring result indicates no, i.e. the application fails, the first resource is released by the first placement group, and the first placement group is re-placed into the first resource application queue. In addition, the second placement group is prohibited from applying for the second resource of the second preset object storage device, a new first placement group is triggered to apply for the first resource of the first preset object storage device, and the new first placement group is placed into the first resource application queue to restart the resource application process.

[0050] When the second placement group successfully applies for the second resource within the time period, a first message notification that the second placement group has acquired the second resource is generated, and the first message notification is sent to the first placement group to synchronize the resource application state. The resource allocation details and a timestamp can be included in the first message notification. Correspondingly, when the second placement group fails to successfully apply for the second resource within the time period, a second message notification that the second placement group has not acquired the second resource is generated, and the second message notification is sent to the first placement group, which can provide a basis for subsequent resource application strategy adjustment.

[0051] In step S14, if the monitoring result indicates yes, a preset data migration mechanism is triggered by the first placement group and the second placement group to complete the data recovery operation of the distributed storage cluster.

[0052] In this embodiment, if the monitoring result indicates yes, the timeout timer is closed, and the timing operation is terminated. At the same time, the first placement group traverses the target objects in the first preset object storage device, and sends the target data of the target objects to the second placement group, so that the second placement group receives the target data to realize the data recovery of the distributed storage cluster.

[0053] From the above, in the present application, in the distributed storage cluster, first, the timeout configuration item is established, and then the timeout timer is determined according to the timeout configuration item; if it is detected that the storage node state changes, then after the first homing group successfully applies for the first resource of the first preset object storage device, the operation of applying for the second resource of the second preset object storage device by the second homing group is triggered, and the timeout timer is started; within the time period represented by the timeout timer, it is detected whether the second homing group successfully applies for the second resource, and then a corresponding detection result is obtained; if the detection result is yes, the preset data migration mechanism is triggered through the first homing group and the second homing group, so as to complete the data recovery operation of the distributed storage cluster. In this way, the present application can solve the problem that the first homing group occupies the first resource for a long time without releasing due to the fact that the second homing group does not obtain the second resource, thereby optimizing the data recovery and balance of the Ceph cluster.

[0054] The technical scheme of the embodiment of the present application will be specifically described below in the scene of expanding one storage node in the Ceph cluster.

[0055] In the original Ceph cluster, there are three storage nodes, and each of the three storage nodes storage01\02\03 has 10 OSDs, namely OSD.0-9, OSD.10-19 and OSD.20-29, and each OSD has 120 PGs. At this time, the original Ceph cluster is expanded, that is, a new storage node storage04 is added to the original Ceph cluster, and OSD.30-39 are deployed on the newly expanded storage04, and according to the PG quantity balance, each OSD has 90 PGs after expansion. Since it is a newly expanded storage node, all the PGs on the storage04 are replica PGs before the data balance is completed. Correspondingly, the PGs on each OSD in the storage01\02\03 are master PGs. Moreover, the maximum number of concurrent LOCAL resources and REMOTE resources is set to 1, so that the newly expanded storage04 has at most 10 replica PGs for data balance, and each OSD has one replica PG.

[0056] Taking the newly expanded OSD.30 as an example. After the first master PG (which can be a PG on the master OSD.0) on the OSD.30 sends the “replica PG applies for REMOTE resource” message, since the maximum number of concurrent REMOTE resources is set to 1, only the first replica PG applies for the REMOTE resource and informs the master PG to perform the subsequent BACKFILL process.

[0057] Without the technical solutions of the embodiments of the present application, for OSD.1-29, a total of 29 master PGs apply for LOCAL resources and send a message of "copy PG applying for REMOTE resources" to OSD.30. The master PGs on OSD.1-29 have applied for LOCAL resources, but none of them has applied for the REMOTE resources of the copy PG on OSD.30, and there is no timeout exit mechanism. Therefore, the 29 master PGs on OSD.1-29 occupy the LOCAL resources for a long time, and at this time, only one PG in the entire cluster performs the BACKFILL process, which seriously affects the expansion efficiency.

[0058] When the technical solutions of the embodiments of the present application are adopted, a new osd_pg_backfill_wait_rep_timeout configuration item (i.e., a timeout configuration item) is added, and the time period of the configuration item is 5 seconds by default. The timeout timer is determined through the configuration item.

[0059] For OSD.1-29, a total of 29 master PGs apply for LOCAL resources and send a message of "copy PG applying for REMOTE resources" to OSD.30. The master PGs on OSD.1-29 have applied for LOCAL resources, and the timeout exit mechanism is triggered through the timeout timer. However, none of them has applied for the REMOTE resources of the copy PG on OSD.30 within 5 seconds. The timeout exit is triggered, and the LOCAL resources are released, so that the new master PG has the opportunity to obtain the REMOTE resources of the copy PG on OSD.31. The second PG performs the BACKFILL process. After multiple rounds of timeout exit and release of the LOCAL resources mechanism, a total of 10 OSDs from OSD.30 to OSD.39 obtain the REMOTE resources of the copy PG, respectively. The 10 PGs perform the BACKFILL process in parallel, which greatly improves the efficiency of BACKFILL and shortens the expansion time.

[0060] Correspondingly, referring to FIG. 1, Figure 3 The data recovery device of the distributed storage cluster comprises:

[0061] The timer determination module 11 is configured to construct a timeout configuration item in the distributed storage cluster and determine a timeout timer based on the timeout configuration item.

[0062] The resource application module 12 is configured to, when it is monitored that the state of the storage node changes, trigger the operation of the second homing group applying the second resource of the second preset object storage device and trigger the start of the timeout timer after the first homing group successfully applies the first resource of the first preset object storage device; the first homing group is located in the first preset object storage device and does not have a preset association relationship with the change of the state of the storage node; the second homing group is located in the second preset object storage device and has the preset association relationship with the change of the state of the storage node.

[0063] The result monitoring module 13 is configured to monitor whether the second homing group successfully applies the second resource within the time period represented by the timeout timer and obtain a corresponding monitoring result.

[0064] The data recovery module 14 is configured to, if the monitoring result indicates yes, trigger a preset data migration mechanism through the first homing group and the second homing group to complete the data recovery operation of the distributed storage cluster.

[0065] In some embodiments, the resource application module 12 specifically includes:

[0066] The first resource application unit is configured to, when it is monitored that the storage node fails or is expanded, add the first homing group to the first resource application queue of the first preset object storage device to apply the first resource of the first preset object storage device through the first homing group.

[0067] The second resource application unit is configured to, if the first homing group has successfully applied the first resource, put the second homing group into the second resource application queue of the second preset object storage device through a preset message mechanism while triggering the start of the timeout timer, to apply the second resource of the second preset object storage device through the second homing group.

[0068] In some embodiments, the result monitoring module 13 specifically further includes:

[0069] The first homing group placement unit is configured to, if the monitoring result indicates no, release the first resource through the first homing group and re-add the first homing group to the first resource application queue.

[0070] The second homing group placement unit is configured to prohibit the second homing group from applying the second resource of the second preset object storage device, trigger a new first homing group to apply the first resource of the first preset object storage device, and add the new first homing group to the first resource application queue.

[0071] In some embodiments, the data recovery module 14 specifically includes:

[0072] a timer closing unit, configured to close the timeout timer if the monitoring result indicates that the timeout timer is to be closed;

[0073] a data receiving unit, configured to traverse a target object in the first preset object storage device through the first disposition group, and send target data of the target object to the second disposition group, so that the second disposition group receives the target data.

[0074] In some embodiments, the data recovery apparatus of the distributed storage cluster further comprises:

[0075] a first message sending unit, configured to, when the second disposition group successfully applies for the second resource within the time period, generate a first message notification that the second disposition group has acquired the second resource, and send the first message notification to the first disposition group;

[0076] a second message sending unit, configured to, when the second disposition group fails to successfully apply for the second resource within the time period, generate a second message notification that the second disposition group has not acquired the second resource, and send the second message notification to the first disposition group.

[0077] In some embodiments, the timeout configuration item is a configuration item that can be modified when the preset object storage device is in a running state.

[0078] Further, the embodiment of the present application further discloses an electronic device, Figure 4 is a structure diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to realize the related steps in the data recovery method of the distributed storage cluster disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the embodiment can be an electronic computer.

[0079] In the embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited here; the input / output interface 25 is used to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0080] In addition, the memory 22 can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc. as a carrier for storing resources, and the resources stored thereon can include an operating system 221, a computer program 222, etc. The storage mode can be temporary storage or permanent storage.

[0081] The operating system 221 is used to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the data recovery method of the distributed storage cluster executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0082] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the foregoing disclosed data recovery method of the distributed storage cluster. For the specific steps of the method, reference can be made to the corresponding content disclosed in the foregoing embodiments, which will not be described here.

[0083] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0084] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0085] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0086] Finally, it is to be understood that the phraseology or terminology such as "comprising", "including", "containing", or "consisting of" etc. used herein is merely open-ended, and does not exclude or deny the inherent coexistence of other non-recited, additional elements, methods, acts, objects, items, components or the like. In other words, the term "comprising", "including", "containing", or "consisting of" etc. does not exclude the presence of additional elements, methods, acts, objects, items, components or the like in the process, method, article, or apparatus. Furthermore, the word "comprising" does not exclude other elements being present in addition to those identified, nor does it exclude that an extra element might be present with respect to those identified. It is to be understood that the terminology used herein is for the purpose of describing specific embodiments only and is not intended to be limiting. The scope of the application should be given by the appended claims.

[0087] The above detailed description of the technical solutions provided by the present application has been described in detail, and the principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A data recovery method of a distributed storage cluster, characterized in that, The application comprises: In a distributed storage cluster, a timeout configuration item is constructed, and a timeout timer is determined based on the timeout configuration item; When a state change of a storage node is monitored, after a first preset object storage device successfully applies for a first resource of a first homing group, an operation of a second preset object storage device applying for a second resource of a second homing group is triggered, and the timeout timer is started; wherein the first homing group is located in the first preset object storage device, and there is no preset association relationship between the first homing group and the state change of the storage node; the second homing group is located in the second preset object storage device, and there is the preset association relationship between the second homing group and the state change of the storage node; Within a time period represented by the timeout timer, whether the second homing group successfully applies for the second resource is monitored, and a corresponding monitoring result is obtained; If the monitoring result indicates yes, a preset data migration mechanism is triggered through the first homing group and the second homing group to complete a data recovery operation of the distributed storage cluster.

2. The data recovery method of the distributed storage cluster according to claim 1, characterized in that, The operation of triggering the second homing group to apply for the second resource of the second preset object storage device and starting the timeout timer when the state change of the storage node is monitored, comprises: When a storage node failure or expansion is monitored, the first homing group is added to a first resource application queue of the first preset object storage device, so as to apply for the first resource of the first preset object storage device through the first homing group; If the first homing group has successfully applied for the first resource, the second homing group is put into a second resource application queue of the second preset object storage device through a preset message mechanism at the same time of starting the timeout timer, so as to apply for the second resource of the second preset object storage device through the second homing group. 3.The data recovery method of the distributed storage cluster of claim 1, wherein, If the monitoring result indicates yes, the timeout timer is closed; The first homing group traverses a target object in the first preset object storage device, and sends target data of the target object to the second homing group, so that the second homing group receives the target data. The corresponding monitoring result is obtained, further comprising:

4. The data recovery method of a distributed storage cluster according to claim 2, wherein, If the monitoring result indicates no, the first resource is released through the first homing group, and the first homing group is put into the first resource application queue again; The second homing group is prohibited from applying for the second resource of the second preset object storage device, a new first homing group is triggered to apply for the first resource of the first preset object storage device, and the new first homing group is put into the first resource application queue. Further comprising:

5. The data recovery method of the distributed storage cluster according to any one of claims 1 to 4, characterized in that, When the second homing group successfully applies for the second resource within the time period, a first message notification that the second homing group has obtained the second resource is generated, and the first message notification is sent to the first homing group; ​ When the second placement group fails to successfully apply for the second resource within the time period, a second message notification that the second placement group fails to obtain the second resource is generated and sent to the first placement group.

6. The data recovery method of a distributed storage cluster according to claim 1, wherein, The timeout configuration item is a configuration item that can be modified when the preset object storage device is in a running state.

7. A data recovery apparatus of a distributed storage cluster, characterized by comprising: The method comprises the steps of: a timer determination module configured to construct a timeout configuration item in a distributed storage cluster and determine a timeout timer based on the timeout configuration item; a resource application module configured to, when a state of a storage node is detected to change, trigger a second placement group to apply for a second resource of a second preset object storage device and start the timeout timer after a first placement group successfully applies for a first resource of a first preset object storage device; the first placement group is located in the first preset object storage device and does not have a preset association relationship with the change in the state of the storage node; the second placement group is located in the second preset object storage device and has the preset association relationship with the change in the state of the storage node; a result monitoring module configured to monitor whether the second placement group successfully applies for the second resource within a time period represented by the timeout timer and obtain a corresponding monitoring result; a data recovery module configured to, if the monitoring result indicates that the second placement group fails to successfully apply for the second resource, trigger a preset data migration mechanism through the first placement group and the second placement group to complete a data recovery operation of the distributed storage cluster.

8. The data recovery apparatus of a distributed storage cluster according to claim 7, wherein, The resource application module comprises: a first resource application unit configured to, when a storage node fault or expansion is detected, add a first placement group to a first resource application queue of a first preset object storage device to apply for a first resource of the first preset object storage device through the first placement group; a second resource application unit configured to, if the first placement group has successfully applied for the first resource, put a second placement group into a second resource application queue of a second preset object storage device through a preset message mechanism to apply for a second resource of the second preset object storage device through the second placement group while starting the timeout timer.

9. An electronic device, comprising: The method comprises the steps of: a memory configured to save a computer program; a processor configured to execute the computer program to implement the data recovery method of the distributed storage cluster according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, a memory configured to save a computer program; wherein the computer program is executed by a processor to implement the data recovery method of the distributed storage cluster according to any one of claims 1 to 6.

Images