A method, device and medium for adjusting the speed of data reconstruction according to a slow request

Abstract

The application discloses a method, device and medium for adjusting the speed of data reconstruction according to a slow request, and relates to the field of distributed storage systems. The method comprises the following steps: in the case that a slow request occurs in a distributed storage system, an object storage device where the slow request occurs is acquired; a storage pool where the slow request occurs is determined according to the object storage device; the data reconstruction speed percentage of the storage pool is adjusted according to the slow request; and the actual data reconstruction speed is determined according to a pre-set upper limit value of the data reconstruction speed and the adjusted data reconstruction speed percentage. As can be seen, in the method, the storage pool where the slow request occurs is determined, and then the data reconstruction speed percentage of the storage pool where the slow request occurs is adjusted according to the slow request, so that the speed of data reconstruction is adjusted according to the slow request, and the influence of data reconstruction on front-end business is reduced. In addition, the application also provides a device for adjusting the speed of data reconstruction according to a slow request and a computer readable storage medium, and the effects are the same.

Description

Technical Field

[0001] This application relates to the field of distributed storage systems, and in particular to a method, apparatus, and medium for adjusting the speed of data reconstruction based on slow requests. Background Technology

[0002] The file system manages storage server nodes and disks through a distributed storage cluster composed of service processes such as a cluster monitor, object-based storage devices (OSDs), and metadata servers (MDS). When a disk fails, data on the failed disk is restored to non-failed disks in the same failure domain using redundant data from other disks. This is a basic scenario of data recovery (or data reconstruction, data repair, recovery / backfill). During data recovery, speed is one of the most important factors for users. Data recovery speed is measured by the number of objects recovered per unit of time divided by the amount of data.

[0003] Existing solutions control the speed of data reconstruction based on set data reconstruction priorities, while others automatically adjust the speed based on current business pressure or system load. However, in production environments, distributed storage systems need to adapt to different specifications, scales, business models, and failures. Current solutions that automatically adjust data reconstruction speed may fail. For example, even with low data reconstruction priorities, it can still significantly impact front-end business operations, causing not only a decrease in distributed storage performance but also potentially resulting in slow requests lasting tens or hundreds of seconds, leading to business interruptions.

[0004] Therefore, how to adjust the data reconstruction speed according to slow requests is a technical problem that urgently needs to be solved by those in this field. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, and medium for adjusting the speed of data reconstruction based on slow requests, thereby reducing the impact of data reconstruction on front-end business.

[0006] To address the aforementioned technical problems, this application provides a method for adjusting the speed of data reconstruction based on slow requests, comprising:

[0007] In the event of a slow request in a distributed storage system, obtain the object storage device that caused the slow request;

[0008] The storage pool where the slow request occurred is determined based on the object storage device;

[0009] Adjust the data reconstruction speed percentage of the storage pool based on the slow requests;

[0010] The actual data reconstruction speed is determined based on the pre-set upper limit of the data reconstruction speed and the adjusted percentage of the data reconstruction speed.

[0011] Preferably, the step of acquiring the object storage device that caused the slow request includes:

[0012] The lifecycle data of all requests is monitored by the cluster monitor; wherein, the lifecycle data is the difference between the current time and the time when the request was initiated, and the lifecycle data is reported by the object storage device to the cluster monitor;

[0013] Obtain the maximum value from the lifecycle data;

[0014] If the maximum value is greater than the time threshold for the slow request, then it is determined that the slow request has occurred in the cluster of the distributed storage system.

[0015] Iterate through the lifecycle data of all requests for the object storage devices;

[0016] Determine whether the lifecycle data of the request for each of the object storage devices is greater than the time threshold;

[0017] If so, then when the lifecycle data is greater than the threshold, the root of the object storage device where the slow request occurs is determined;

[0018] Correspondingly, determining the storage pool where the slow request occurred based on the object storage device includes:

[0019] The cluster monitor iterates through all storage pools and determines the storage pool where the slow request occurred based on the root.

[0020] Preferably, adjusting the data reconstruction speed percentage of the storage pool based on the slow request includes:

[0021] From the start of determining whether the storage pool has a slow request until the first determination that the storage pool has a slow request, obtain the initial data reconstruction speed percentage of the storage pool;

[0022] Reduce the initial data reconstruction speed percentage to a preset lower limit of the data reconstruction speed percentage;

[0023] Correspondingly, determining the actual data reconstruction speed based on the preset upper limit of the data reconstruction speed and the adjusted percentage of the data reconstruction speed includes:

[0024] The current actual data reconstruction speed is calculated as the product of the preset upper limit of the data reconstruction speed and the preset lower limit of the data reconstruction speed percentage.

[0025] Preferably, after determining the current actual data reconstruction speed as the product of the preset upper limit of the data reconstruction speed and the preset lower limit of the data reconstruction speed percentage, the method further includes:

[0026] If the slow request does not disappear within a first preset time after the data reconstruction begins based on the current actual data reconstruction speed, then the data reconstruction will proceed based on the current actual data reconstruction speed.

[0027] Starting from the data reconstruction based on the current actual data reconstruction speed, if the slow request disappears within the first preset time and no new slow request appears within the second preset time, then the percentage of the data reconstruction speed is gradually increased from the lower limit value according to the first preset step size; the current actual data reconstruction speed is determined by multiplying the increased percentage of the data reconstruction speed by the upper limit value; and the data reconstruction is performed based on the current actual data reconstruction speed.

[0028] Preferably, when the percentage of the data reconstruction speed is gradually increased from the lower limit value according to the first preset step size, the method further includes:

[0029] During the process of increasing the data reconstruction speed percentage, it is determined whether a slow request occurs;

[0030] If so, the data reconstruction speed percentage is gradually reduced from the current data reconstruction speed percentage according to the second preset step size; the current actual data reconstruction speed is determined by multiplying the reduced data reconstruction speed percentage by the upper limit value; and the data reconstruction is performed according to the current actual data reconstruction speed.

[0031] Preferably, after determining the current actual data reconstruction speed based on the product of the reduced data reconstruction speed percentage and the upper limit value, the method further includes:

[0032] Get the current number of times the data reconstruction speed percentage has been adjusted;

[0033] Determine whether the current number of times is greater than the adjustment number threshold;

[0034] If so, then the data reconstruction is performed based on the current actual data reconstruction speed;

[0035] If not, return to the step of gradually increasing the percentage of data reconstruction speed from the lower limit value according to the first preset step size.

[0036] Preferably, after performing the data reconstruction based on the current actual data reconstruction speed, the method further includes:

[0037] Obtain the duration of data reconstruction based on the current actual data reconstruction speed;

[0038] Determine whether the duration is greater than the preset duration;

[0039] If so, the current count is cleared to zero, and the process returns to the step of gradually increasing the percentage of data reconstruction speed from the lower limit value according to the first preset step size.

[0040] To address the aforementioned technical problems, this application also provides an apparatus for adjusting the speed of data reconstruction based on slow requests, comprising:

[0041] The acquisition module is used to acquire the object storage device that caused the slow request in the case of a slow request in the distributed storage system.

[0042] The first determining module is used to determine the storage pool where the slow request occurs based on the object storage device;

[0043] The adjustment module is used to adjust the data reconstruction speed percentage of the storage pool based on the slow request;

[0044] The second determining module is used to determine the actual data reconstruction speed based on the preset upper limit of the data reconstruction speed and the adjusted percentage of the data reconstruction speed.

[0045] To address the aforementioned technical problems, this application also provides an apparatus for adjusting the speed of data reconstruction based on slow requests, comprising:

[0046] Memory, used to store computer programs;

[0047] A processor, used to implement the steps of the method described above for adjusting the speed of data reconstruction based on slow requests when executing the computer program.

[0048] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above for adjusting the speed of data reconstruction based on slow requests.

[0049] The method for adjusting the data reconstruction speed based on slow requests provided in this application includes: when a slow request occurs in a distributed storage system, obtaining the object storage device experiencing the slow request; determining the storage pool experiencing the slow request based on the object storage device; adjusting the data reconstruction speed percentage of the storage pool based on the slow request; and determining the actual data reconstruction speed based on a pre-set upper limit for the data reconstruction speed and the adjusted data reconstruction speed percentage. Therefore, this method, by identifying the storage pool experiencing the slow request and then adjusting the data reconstruction speed percentage of that storage pool, achieves the goal of adjusting the data reconstruction speed based on the slow request, thereby reducing the impact of data reconstruction on front-end business operations.

[0050] In addition, this application also provides an apparatus for adjusting the speed of data reconstruction based on slow requests and a computer-readable storage medium, which have the same or corresponding technical features as the method for adjusting the speed of data reconstruction based on slow requests mentioned above, and have the same effect. Attached Figure Description

[0051] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A flowchart illustrating a method for adjusting the speed of data reconstruction based on slow requests, provided in an embodiment of this application;

[0053] Figure 2 A structural diagram of an apparatus for adjusting the speed of data reconstruction based on slow requests, provided as an embodiment of this application;

[0054] Figure 3 A structural diagram of an apparatus for adjusting the speed of data reconstruction based on slow requests, provided as another embodiment of this application;

[0055] Figure 4 This is a flowchart illustrating a method for adjusting data recovery speed based on slow requests in a distributed storage system, as provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0057] The core of this application is to provide a method, apparatus, and medium for adjusting the speed of data reconstruction based on slow requests, thereby reducing the impact of data reconstruction on front-end business.

[0058] The file system manages storage server nodes and disks through a distributed storage cluster composed of service processes such as the Monitor daemon (MON), OSD, and MDS. Each disk is managed by a daemon process (OSD), and the OSD object storage device implements functions such as data read / write, replication, balancing, and recovery. The system implements a data protection mechanism through data redundancy generated by multiple replicas or erasure coding. When a disk fails, the data on the failed disk is restored to the non-failed disks in the same failure domain using redundant data on other disks. This is a basic scenario of data recovery (or data reconstruction, data repair, recovery / backfill). In addition, as a distributed storage system, besides disk failures, other types of cluster topology changes, such as cluster expansion / shrinkage, also require flexible object migration to achieve balanced data distribution. These scenarios can also be referred to as data recovery.

[0059] Data recovery speed can be measured by the number of objects recovered per unit of time divided by the amount of data recovered. The existing solution implements Quality of Service (QoS) for data recovery, allowing users to control recovery speed by setting four priorities: low, medium, high, and full recovery. This solution uses a 2-second recovery cycle and limits the recovery speed by restricting the maximum amount of data and the maximum number of objects that can be recovered within a single cycle. Each data recovery priority requires a set of maximum data volume and maximum number of objects to be configured. Users need to set data recovery priorities according to their needs. If users do not want data recovery to consume system resources and affect front-end business, they should lower the data recovery priority; if users want to complete data recovery as quickly as possible, they should raise the data recovery priority.

[0060] Besides controlling the data reconstruction speed based on a set data reconstruction priority, current solutions also automatically adjust the data recovery speed based on front-end business pressure or system pressure. When front-end business pressure increases or the system becomes busy, the data recovery speed is reduced; when front-end business pressure decreases or the system becomes idle, the data recovery speed is increased. However, in a production environment, distributed storage systems need to adapt to different specifications, scales, business models, and failures, and current automatic adjustment solutions for data reconstruction speed may fail. If data reconstruction has too much impact on front-end business, it will not only cause a decrease in distributed storage performance but may even generate slow requests lasting tens or hundreds of seconds, causing business interruption.

[0061] This application proposes a method for adjusting data recovery speed in a distributed storage system based on slow requests, as a supplement to adjusting data recovery speed according to business or system pressure. When a slow request occurs in an OSD of a storage pool, the data recovery speed of the corresponding storage pool is quickly adjusted to reduce the impact of data reconstruction on front-end services.

[0062] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 A flowchart illustrating a method for adjusting the speed of data reconstruction based on slow requests, as provided in this application embodiment, is shown below. Figure 1 As shown, the method includes:

[0063] S10: In the event of a slow request in a distributed storage system, retrieve the object storage device where the slow request occurred;

[0064] S11: Determine the storage pool where slow requests occur based on the object storage device;

[0065] S12: Adjust the data reconstruction speed percentage of the storage pool based on slow requests;

[0066] S13: Determine the actual data reconstruction speed based on the pre-set upper limit of the data reconstruction speed and the adjusted percentage of the data reconstruction speed.

[0067] Slow requests are read / write requests initiated by the frontend that take excessively long to complete, sometimes several seconds or even tens of seconds. This occurs when there is a hardware / software failure on the storage side, or when disk pressure is excessive. In the current solution, the OSD periodically tracks the lifecycle of all unfinished business requests—that is, the time difference between the current time and the request initiation time—and reports this to the MON cluster's monitoring daemon. MON analyzes the reports from all OSDs, and if a request's lifecycle exceeds a set threshold, it reports a slow request. The threshold is typically set to 30 seconds.

[0068] When both business pressure and data reconstruction speed are too fast, issues may arise such as prolonged data reconstruction time occupying the PG placement group (a carrier of objects), and business read / write requests being unable to acquire PG locks, resulting in slow business requests. Slow requests manifest on the client side as business lag, and in latency-sensitive applications, this can severely impact user experience. Therefore, it is necessary to promptly reduce the reconstruction speed to eliminate slow requests.

[0069] Before slow requests occur and before slow request elimination begins, this embodiment pre-sets an upper limit for data reconstruction speed. Once data reconstruction begins, this upper limit is used as the default speed. The default speed value is not limited and is determined based on actual conditions. When slow request elimination begins, the object storage device experiencing slow requests is first identified; then, the storage pool experiencing slow requests is located based on the object storage device experiencing the slow requests; the data reconstruction speed percentage of the storage pool experiencing slow requests is adjusted, and the product of the adjusted data reconstruction speed percentage and the pre-set upper limit for data reconstruction speed is used as the actual data reconstruction speed. Data reconstruction is then performed based on the actual data reconstruction speed.

[0070] There are no limitations on how the data reconstruction speed percentage for storage pools experiencing slow requests is adjusted; it can be determined based on the actual situation. For example, the data reconstruction speed percentage can be adjusted based on the time and number of slow requests. When slow requests occur, the data reconstruction speed percentage can be reduced to decrease the data reconstruction speed; when no slow requests occur, the data reconstruction speed percentage can be increased to increase the data reconstruction speed. In this embodiment, the data reconstruction speed percentage is set to 100% by default, and a lower limit can be set through configuration.

[0071] The method for adjusting the data reconstruction speed based on slow requests provided in this embodiment includes: when a slow request occurs in a distributed storage system, obtaining the object storage device experiencing the slow request; determining the storage pool experiencing the slow request based on the object storage device; adjusting the data reconstruction speed percentage of the storage pool based on the slow request; and determining the actual data reconstruction speed based on a pre-set upper limit for the data reconstruction speed and the adjusted data reconstruction speed percentage. Therefore, this method, by identifying the storage pool experiencing the slow request and then adjusting the data reconstruction speed percentage of that storage pool, achieves the goal of adjusting the data reconstruction speed based on the slow request, thereby reducing the impact of data reconstruction on front-end services.

[0072] To more accurately identify object storage devices experiencing slow requests, a preferred implementation method includes:

[0073] The cluster monitor monitors the lifecycle data of all requests; the lifecycle data is the difference between the current time and the time when the request was initiated, and the lifecycle data is reported to the cluster monitor by the object storage device.

[0074] Get the maximum value in the lifecycle data;

[0075] If the maximum value is greater than the slow request time threshold, then a slow request is determined to have occurred in the distributed storage system cluster.

[0076] Iterate through the lifecycle data of all requests for object storage devices;

[0077] Determine whether the lifecycle data of the requests for each object storage device exceeds the time threshold;

[0078] If so, then when the lifecycle data exceeds the threshold, a slow request occurs in the root of the corresponding object storage device;

[0079] Correspondingly, the storage pools experiencing slow requests, as determined based on the object storage device, include:

[0080] The cluster monitor iterates through all storage pools and identifies the storage pool experiencing slow requests based on the root node.

[0081] Set an upper limit for the data reconstruction speed. Once data reconstruction begins, this upper limit will be used as the default reconstruction speed. The cluster monitor periodically checks the request lifecycle statistics reported by OSDs. If the maximum value exceeds the slow request time threshold, it indicates that a slow request has occurred in the cluster. If no slow request has occurred, the current data reconstruction speed will be maintained. Otherwise, it iterates through the request lifecycle statistics of all OSDs to determine if any OSD has a request lifecycle greater than the slow request threshold. If an OSD exhibits a slow request, the root of that OSD is considered to have a slow request, and the number of slow requests occurring on the root is recorded. The cluster monitor then iterates through all storage pools, searching for their corresponding root. If a slow request exists on that root, the storage pool is considered to have a slow request, and all storage pools exhibiting slow requests are marked.

[0082] In the method for determining the object storage device where a slow request occurs provided in this embodiment, it is first determined whether a slow request occurs in the cluster, then the object storage resource where the slow request occurs is further determined, and then the storage pool corresponding to the root of the object storage resource where the slow request occurs is determined, so that the location of the slow request can be accurately located.

[0083] In the event of slow requests, in order to quickly reduce the data reconstruction speed and thus rapidly eliminate slow requests, a preferred implementation is to adjust the data reconstruction speed percentage of the storage pool based on the slow requests, including:

[0084] From the start of determining whether a slow request has occurred in the storage pool, until the initial determination that a slow request has occurred in the storage pool, obtain the initial data reconstruction speed percentage of the storage pool;

[0085] Reduce the initial data reconstruction speed percentage to the lower limit of the preset data reconstruction speed percentage;

[0086] Correspondingly, the actual data reconstruction speed is determined based on the pre-set upper limit of the data reconstruction speed and the adjusted percentage of the data reconstruction speed, including:

[0087] The current actual data reconstruction speed is calculated by multiplying the preset upper limit of the data reconstruction speed by the preset lower limit of the data reconstruction speed percentage.

[0088] The initial data reconstruction speed percentage is not limited and is determined based on the actual situation. In this embodiment, the initial data reconstruction speed percentage (corresponding to the upper limit of the data reconstruction speed) is set to 100%. Upon initial detection of a slow request, this embodiment directly reduces the data reconstruction speed percentage of the storage pool experiencing the slow request to the lower limit of the data reconstruction speed percentage. The lower limit of the data reconstruction speed percentage is not limited. For example, if the lower limit is 30%, then upon initial detection of a slow request in the storage pool, the data reconstruction speed percentage will directly decrease from 100% to 30%. Correspondingly, the current actual data reconstruction speed is the product of 30% and the upper limit of the data reconstruction speed.

[0089] In the method provided in this embodiment, if a slow request occurs in the storage pool, the data reconstruction speed percentage is directly changed to the lower limit, which can quickly reduce the reconstruction speed and eliminate the slow request.

[0090] Based on the above embodiments, after adjusting the data reconstruction speed percentage once, slow requests may not be completely eliminated, or new slow requests may reappear after a period of time after elimination. Therefore, in order to eliminate slow requests as much as possible, a preferred implementation method is that, after using the product of a preset upper limit for data reconstruction speed and a preset lower limit for data reconstruction speed percentage as the current actual data reconstruction speed, the method for adjusting the data reconstruction speed according to slow requests further includes:

[0091] If slow requests do not disappear within the first preset time after the data reconstruction begins based on the current actual data reconstruction speed, then the data reconstruction will proceed based on the current actual data reconstruction speed.

[0092] Starting from the data reconstruction based on the current actual data reconstruction speed, if slow requests disappear within a first preset time and no new slow requests appear within a second preset time, then the data reconstruction speed percentage is gradually increased from the lower limit value according to the first preset step size; the current actual data reconstruction speed is determined by multiplying the increased data reconstruction speed percentage by the upper limit value; and data reconstruction is performed based on the current actual data reconstruction speed.

[0093] The first preset time, second preset time, and first preset step size are not limited and can be determined based on the actual situation. In the event of a slow request, the data reconstruction speed percentage is adjusted once. If the slow request does not disappear, the percentage remains at the lower limit. If the slow request disappears and no new slow request appears after a period of time (e.g., 5 minutes), the reconstruction speed percentage is increased according to the configured step size, such as increasing by 1% every 10 seconds until it reaches 100%.

[0094] In the method provided in this embodiment, the slow request is eliminated by judging whether the slow request disappears within a first preset time and by reducing the data reconstruction speed percentage when the slow request reappears. That is, the impact of the slow request is eliminated according to the occurrence of the slow request.

[0095] During the process of gradually increasing the data reconstruction speed percentage from the lower limit in the above embodiments, new slow requests may occur. In order to eliminate slow requests, a preferred embodiment further includes, when gradually increasing the data reconstruction speed percentage from the lower limit according to a first preset step size, the method of adjusting the data reconstruction speed according to the slow requests:

[0096] During the process of increasing the data reconstruction speed percentage, determine whether slow requests occur;

[0097] If so, the data reconstruction speed percentage is gradually reduced from the current data reconstruction speed percentage according to the second preset step size; the current actual data reconstruction speed is determined by multiplying the reduced data reconstruction speed percentage by the upper limit value; and data reconstruction is performed according to the current actual data reconstruction speed.

[0098] If slow requests reappear during the upward adjustment of the data reconstruction speed percentage, the reconstruction speed percentage will be reduced by a second preset step size until it reaches the lower limit. The value of the second preset step size is not limited. The adjustment step size is proportional to the number of slow requests under the root of the previously recorded storage pool. The more slow requests there are, the larger the adjustment step size should be. If the slow requests disappear before reaching the lower limit, the reduction of the reconstruction percentage will stop. After a period of stability without slow requests, the data reconstruction speed percentage will continue to be gradually increased from the lower limit by a first preset step size.

[0099] The method provided in this embodiment offers a way to handle slow requests as the data reconstruction speed percentage gradually increases from the lower limit, thereby minimizing the impact of slow requests.

[0100] In practice, disk instability may occur, making it impossible to eliminate slow requests to the greatest extent even after multiple adjustments to the data reconstruction speed percentage. Therefore, based on the data reconstruction speed percentage adjustment in the above embodiment, this embodiment configures an upper limit on the number of adjustments. Specifically, after determining the current actual data reconstruction speed based on the product of the reduced data reconstruction speed percentage and the upper limit, the method for adjusting the data reconstruction speed according to slow requests further includes:

[0101] Get the current number of times the data reconstruction speed percentage has been adjusted;

[0102] Determine if the current count is greater than the adjustment count threshold;

[0103] If so, then reconstruct the data based on the current actual data reconstruction speed;

[0104] If not, return to the step of gradually increasing the percentage of data reconstruction speed from the lower limit according to the first preset step size.

[0105] There is no limit to the adjustment threshold; it is determined based on the actual situation. If the current number of adjustments is less than or equal to the adjustment threshold, the impact of slow requests is eliminated by following the steps described in the above embodiments to adjust the data reconstruction speed percentage of the storage pool based on slow requests. If the current number of adjustments is greater than the adjustment threshold, the reconstruction speed percentage is no longer increased, and data reconstruction is performed according to the current actual data reconstruction speed.

[0106] The method provided in this embodiment can avoid ineffective adjustments when the disk is unstable by setting a threshold for the number of adjustments.

[0107] In practice, the data recovery speed may remain stable at a certain value for an extended period, failing to eliminate the impact of slow requests. Therefore, after reconstructing the data based on the current actual data reconstruction speed, methods for adjusting the data reconstruction speed according to slow requests also include:

[0108] Get the time required for data reconstruction based on the current actual data reconstruction speed;

[0109] Determine if the duration exceeds the preset duration;

[0110] If so, the current count is reset to zero, and the process returns to the step of gradually increasing the data reconstruction speed percentage from the lower limit according to the first preset step size.

[0111] In the method provided in this embodiment, a time threshold, i.e., a preset duration, is set for the number of cleanup and adjustment cycles, such as 1 hour. If the data recovery speed remains stable at a certain value for a period exceeding this time threshold, the number of cleanup and adjustment cycles is increased, allowing the reconstruction speed percentage to be adjusted upwards further, thereby improving the data reconstruction speed.

[0112] In the above embodiments, the method for adjusting the speed of data reconstruction based on slow requests has been described in detail. This application also provides embodiments corresponding to an apparatus for adjusting the speed of data reconstruction based on slow requests. It should be noted that this application describes the embodiments of the apparatus from two perspectives: one based on functional modules and the other based on hardware.

[0113] Figure 2 A structural diagram of an apparatus for adjusting the speed of data reconstruction based on slow requests, provided as an embodiment of this application. This embodiment, from a functional module perspective, includes:

[0114] The acquisition module 10 is used to acquire the object storage device that is experiencing a slow request in the case of a slow request in the distributed storage system.

[0115] The first determining module 11 is used to determine the storage pool where slow requests occur based on the object storage device;

[0116] Adjustment module 12 is used to adjust the data reconstruction speed percentage of the storage pool based on slow requests;

[0117] The second determining module 13 is used to determine the actual data reconstruction speed based on the preset upper limit of the data reconstruction speed and the adjusted percentage of the data reconstruction speed.

[0118] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0119] The apparatus for adjusting the data reconstruction speed based on slow requests provided in this embodiment, when a slow request occurs in a distributed storage system, acquires the object storage device experiencing the slow request through an acquisition module; determines the storage pool experiencing the slow request through a first determination module based on the object storage device; adjusts the data reconstruction speed percentage of the storage pool based on the slow request through an adjustment module; and determines the actual data reconstruction speed through a second determination module based on a pre-set upper limit for the data reconstruction speed and the adjusted data reconstruction speed percentage. Therefore, by identifying the storage pool experiencing the slow request and then adjusting the data reconstruction speed percentage of that storage pool, the apparatus effectively adjusts the data reconstruction speed based on the slow request, reducing the impact of data reconstruction on front-end services.

[0120] Figure 3 This is a structural diagram of an apparatus for adjusting the speed of data reconstruction based on slow requests, as provided in another embodiment of this application. This embodiment is based on a hardware perspective, such as... Figure 3 As shown, the apparatus for adjusting the speed of data reconstruction based on slow requests includes:

[0121] Memory 20 is used to store computer programs;

[0122] Processor 21 is configured to implement, when executing a computer program, the steps of the method for adjusting the speed of data reconstruction based on slow requests as described in the above embodiments.

[0123] The device provided in this embodiment for adjusting the speed of data reconstruction based on slow requests may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0124] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0125] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the method for adjusting the speed of data reconstruction based on slow requests disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the aforementioned method for adjusting the speed of data reconstruction based on slow requests.

[0126] In some embodiments, the apparatus for adjusting the speed of data reconstruction based on slow requests may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0127] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the apparatus for adjusting the speed of data reconstruction based on slow requests, and may include more or fewer components than shown.

[0128] The apparatus for adjusting the speed of data reconstruction based on slow requests provided in this application includes a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: the method for adjusting the speed of data reconstruction based on slow requests, with the same effect as above.

[0129] This application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.

[0130] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] The computer-readable storage medium provided in this application includes the aforementioned method for adjusting the speed of data reconstruction based on slow requests, with the same effect.

[0132] To enable those skilled in the art to better understand the present application, the following description is provided in conjunction with the appendix. Figure 4 The present application will be further described in detail with reference to specific embodiments. Figure 4 A flowchart illustrating a method for adjusting data recovery speed based on slow requests in a distributed storage system, as provided in this application embodiment, is shown below. Figure 4 As shown, the method includes:

[0133] S14: Determine whether slow request protection for data reconstruction is enabled; if yes, proceed to step S15.

[0134] S15: Set the reconstruction speed percentage to the maximum of 100%;

[0135] S16: Determine if a slow request has occurred; if yes, proceed to step S17; if no, return to this step.

[0136] S17: Reduce the refactoring speed percentage to the lower limit;

[0137] S18: Determine if the slow request has disappeared; if not, return to this step; if yes, proceed to step S19.

[0138] S19: After stabilizing for a period of time, increase the reconstruction speed percentage by a certain step size;

[0139] S20: Determine if a slow request occurs again; if yes, proceed to step S21; if no, proceed to step S25.

[0140] S21: Decrease the reconstruction speed by a certain step size by a percentage;

[0141] S22: Determine if the slow request has disappeared; if not, return to step S17; if yes, proceed to step S23.

[0142] S23: Determine whether the number of slow requests has reached the set limit; if not, return to step S19; if yes, proceed to step S24.

[0143] S24: Maintain the current recovery rate until the time threshold is reached and return to step S19;

[0144] S25: Increase the reconstruction speed percentage to the maximum;

[0145] S26: Determine whether a slow request occurs within the threshold time; if yes, return to step S21; if no, proceed to step S27.

[0146] S27: Clear the count of slow requests and return to step S15.

[0147] The method for adjusting data recovery speed based on slow requests in a distributed storage system, as provided in this application embodiment, allows the distributed storage system to adjust its data recovery speed according to whether slow requests occur. During data reconstruction, if slow requests occur, the system identifies the storage pools affected by the object storage devices experiencing the slow requests and rapidly adjusts the data reconstruction speed of these storage pools to quickly eliminate the slow requests and reduce the impact on front-end services. This avoids service interruptions caused by excessive system resource consumption during data reconstruction.

[0148] The foregoing has provided a detailed description of a method, apparatus, and medium for adjusting the speed of data reconstruction based on slow requests, as provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0149] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method of adjusting the speed of data reconstruction from a slow request, characterized by, include: In the event of a slow request in a distributed storage system, obtain the object storage device that caused the slow request; The storage pool where the slow request occurred is determined based on the object storage device; Adjust the data reconstruction speed percentage of the storage pool based on the slow requests; The actual data reconstruction speed is determined based on the preset upper limit of the data reconstruction speed and the adjusted percentage of the data reconstruction speed. The step of adjusting the data reconstruction speed percentage of the storage pool based on the slow request includes: From the start of determining whether the storage pool has a slow request until the first determination that the storage pool has a slow request, obtain the initial data reconstruction speed percentage of the storage pool; Reduce the initial data reconstruction speed percentage to a preset lower limit of the data reconstruction speed percentage; Correspondingly, determining the actual data reconstruction speed based on the preset upper limit of the data reconstruction speed and the adjusted percentage of the data reconstruction speed includes: The current actual data reconstruction speed is the product of the preset upper limit of the data reconstruction speed and the preset lower limit of the data reconstruction speed percentage. After determining the current actual data reconstruction speed by multiplying the upper limit of the preset data reconstruction speed by the lower limit of the preset data reconstruction speed percentage, the method further includes: If the slow request does not disappear within a first preset time after the data reconstruction begins based on the current actual data reconstruction speed, then the data reconstruction will proceed based on the current actual data reconstruction speed. Starting from the data reconstruction based on the current actual data reconstruction speed, if the slow request disappears within the first preset time and no new slow request appears within the second preset time, then the percentage of the data reconstruction speed is gradually increased from the lower limit value according to the first preset step size; the current actual data reconstruction speed is determined by multiplying the increased percentage of the data reconstruction speed by the upper limit value; and the data reconstruction is performed based on the current actual data reconstruction speed.

2. The method of claim 1, wherein, The object storage device that receives the slow request includes: The lifecycle data of all requests is monitored by the cluster monitor; wherein, the lifecycle data is the difference between the current time and the time when the request was initiated, and the lifecycle data is reported by the object storage device to the cluster monitor; Obtain the maximum value from the lifecycle data; If the maximum value is greater than the time threshold for the slow request, then it is determined that the slow request has occurred in the cluster of the distributed storage system. Iterate through the lifecycle data of all requests for the object storage devices; Determine whether the lifecycle data of the request for each of the object storage devices is greater than the time threshold; If so, then when the lifecycle data is greater than the threshold, the root of the object storage device where the slow request occurs is determined; Correspondingly, determining the storage pool where the slow request occurred based on the object storage device includes: The cluster monitor iterates through all storage pools and determines the storage pool where the slow request occurred based on the root.

3. The method for adjusting the speed of data reconstruction based on slow requests according to claim 1, characterized in that, When the percentage of data reconstruction speed is gradually increased from the lower limit value according to the first preset step size, the method further includes: During the process of increasing the data reconstruction speed percentage, it is determined whether a slow request occurs; If so, the data reconstruction speed percentage is gradually reduced from the current data reconstruction speed percentage according to the second preset step size; the current actual data reconstruction speed is determined by multiplying the reduced data reconstruction speed percentage by the upper limit value; and the data reconstruction is performed according to the current actual data reconstruction speed.

4. The method for adjusting the speed of data reconstruction based on slow requests according to claim 3, characterized in that, After determining the current actual data reconstruction speed based on the product of the reduced data reconstruction speed percentage and the upper limit value, the method further includes: Get the current number of times the data reconstruction speed percentage has been adjusted; Determine whether the current number of times is greater than the adjustment number threshold; If so, then the data reconstruction is performed based on the current actual data reconstruction speed; If not, return to the step of gradually increasing the percentage of data reconstruction speed from the lower limit value according to the first preset step size.

5. The method for adjusting the speed of data reconstruction based on slow requests according to claim 4, characterized in that, After performing the data reconstruction based on the current actual data reconstruction speed, the method further includes: Obtain the duration of data reconstruction based on the current actual data reconstruction speed; Determine whether the duration is greater than the preset duration; If so, the current count is cleared to zero, and the process returns to the step of gradually increasing the percentage of data reconstruction speed from the lower limit value according to the first preset step size.

6. An apparatus for adjusting the speed of data reconstruction based on slow requests, characterized in that, include: The acquisition module is used to acquire the object storage device that caused the slow request in the case of a slow request in the distributed storage system. The first determining module is used to determine the storage pool where the slow request occurs based on the object storage device; The adjustment module is used to adjust the data reconstruction speed percentage of the storage pool based on the slow request; The second determining module is used to determine the actual data reconstruction speed based on the preset upper limit of the data reconstruction speed and the adjusted percentage of the data reconstruction speed; The adjustment module specifically includes: From the start of determining whether the storage pool has a slow request until the first determination that the storage pool has a slow request, obtain the initial data reconstruction speed percentage of the storage pool; Reduce the initial data reconstruction speed percentage to a preset lower limit of the data reconstruction speed percentage; Correspondingly, the second determining module specifically includes: The current actual data reconstruction speed is the product of the preset upper limit of the data reconstruction speed and the preset lower limit of the data reconstruction speed percentage. After determining the current actual data reconstruction speed by multiplying the upper limit of the preset data reconstruction speed by the lower limit of the preset data reconstruction speed percentage, the method further includes: If the slow request does not disappear within a first preset time after the data reconstruction begins based on the current actual data reconstruction speed, then the data reconstruction will proceed based on the current actual data reconstruction speed. Starting from the data reconstruction based on the current actual data reconstruction speed, if the slow request disappears within the first preset time and no new slow request appears within the second preset time, then the percentage of the data reconstruction speed is gradually increased from the lower limit value according to the first preset step size; the current actual data reconstruction speed is determined by multiplying the increased percentage of the data reconstruction speed by the upper limit value; and the data reconstruction is performed based on the current actual data reconstruction speed.

7. An apparatus for adjusting the speed of data reconstruction based on slow requests, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the method for adjusting the speed of data reconstruction based on slow requests as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for adjusting the speed of data reconstruction based on slow requests as described in any one of claims 1 to 5.

Images