Method, apparatus, and computer program product for managing a storage system

Abstract

Embodiments of the present disclosure relate to a method, an electronic device and a computer program product for managing a storage system. The method comprises determining, at a first device of the storage system, whether a load of a first accelerator resource of the first device exceeds a load threshold; sending, to a second device in a candidate device list, a job processing request to cause the second device to process a target job of the first device with a second accelerator resource of the second device if it is determined that the load exceeds the load threshold, the candidate device list indicating devices in the storage system that are available to assist the first device in processing jobs; receiving, from the second device, latency information related to a remote processing latency of processing the target job with the second accelerator resource; and updating the candidate device list based on the latency information. Embodiments of the present disclosure can optimize system performance.

Description

Technical Field

[0001] Embodiments of this disclosure generally relate to the field of data storage, and more specifically to methods, apparatus, and computer program products for managing storage systems. Background Technology

[0002] In storage systems, accelerator resources (such as coprocessors) are typically used to assist the central processing unit (CPU) in performing accelerated tasks. For example, a Quick Assist Technology (QAT) card is a type of coprocessor that can be used to accelerate computationally intensive tasks such as compression, encryption, and decryption. In storage systems with multiple devices (sometimes referred to as nodes), each device typically has its own accelerator resources, such as QAT cards. However, these accelerator resources on these devices are sometimes not utilized evenly. For example, in multi-device systems, although jobs can be distributed across devices, uneven distribution can still occur, causing some devices to perform more jobs and resulting in greater processing latency. Summary of the Invention

[0003] Embodiments of this disclosure provide methods, apparatus, and computer program products for managing storage systems.

[0004] In a first aspect of this disclosure, a method for managing a storage system is provided. The method includes, at a first device in the storage system, determining whether the load on a first accelerator resource of the first device exceeds a load threshold; if the load exceeds the load threshold, sending a job processing request to a second device in a candidate device list, such that the second device utilizes a second accelerator resource of the second device to process a target job of the first device, the candidate device list indicating devices in the storage system that can be used to assist the first device in processing jobs; receiving latency information from the second device related to remote processing latency for utilizing the second accelerator resource to process the target job; and updating the candidate device list based on the latency information.

[0005] In a second aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processing unit and at least one memory. The at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform an action, the action including, at a first device in a storage system, determining whether the load on a first accelerator resource of the first device exceeds a load threshold; if the load exceeds the load threshold, sending a job processing request to a second device in a candidate device list, causing the second device to utilize a second accelerator resource of the second device to process a target job of the first device, the candidate device list indicating devices in the storage system that can be used to assist the first device in processing jobs; receiving latency information from the second device related to remote processing latency for utilizing the second accelerator resource to process the target job; and updating the candidate device list based on the latency information.

[0006] In a third aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a non-transitory computer storage medium and includes machine-executable instructions. When executed by a device, the machine-executable instructions cause the device to perform any step of the method described in the first aspect of this disclosure.

[0007] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0008] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0009] Figure 1 A schematic diagram of an example system that can be implemented therein according to some embodiments of the present disclosure is shown;

[0010] Figure 2 A schematic block diagram of an example storage system according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A flowchart illustrating example methods of resource usage according to some embodiments of this disclosure is shown;

[0012] Figure 4 A schematic diagram of curves for determining a load threshold according to some embodiments of the present disclosure is shown;

[0013] Figure 5A schematic diagram is shown illustrating an example job processing request sent by a first device to a second device according to some embodiments of the present disclosure, and example latency information received by the first device from the second device;

[0014] Figure 6 A flowchart illustrating an example method for updating a list of candidate devices according to some embodiments of this disclosure is shown; and

[0015] Figure 7 A schematic block diagram of an example device that can be used to implement embodiments of the present disclosure is shown.

[0016] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0017] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0018] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] As mentioned above, nodes in a storage system may have accelerator resources, such as QAT cards, but there may be uneven utilization of accelerator resources during job execution, which can cause significant processing delays for jobs.

[0020] Traditionally, a simple solution to address insufficient accelerator resources is to upgrade the processing power of individual devices by deploying more accelerator resources. However, this increases overhead. Existing storage systems have job scheduling schemes, but these schemes primarily schedule jobs by comparing the processing latency of individual devices. Such schemes cannot adaptively schedule jobs across devices to solve the problem of insufficient accelerator resources. Furthermore, such schemes require obtaining the processing latency of each device, which incurs additional computing resources and processing time.

[0021] Embodiments of this disclosure propose a scheme for managing a storage system to address one or more of the aforementioned problems and other potential issues. In this scheme, when a first device in the storage system determines that its local first accelerator resources are overloaded, it sends a job processing request to a second device in a candidate device list, enabling the second device to utilize its second accelerator resources to process the target job of the first device. The scheme further includes updating the candidate device list based on latency information received from the second device related to the remote processing latency of the second accelerator resources processing the target job.

[0022] In this way, accelerator resource calls across nodes can be achieved by dynamically updating the candidate device list and selecting devices from it. This not only improves job processing efficiency but also enhances the overall utilization of system resources.

[0023] The basic principles and several exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0024] Figure 1 A schematic diagram of a storage system 100 in which embodiments of the present disclosure may be implemented is shown. The storage system 100 is used to provide data storage related tasks, including tasks such as storage, data access, and data protection (e.g., deduplication, backup, encryption, decryption, etc.). It should be understood that... Figure 1 The system shown is merely an example. In real-world applications, there may be many other devices and / or components within the storage system, or the devices and / or components shown may be arranged in other ways.

[0025] like Figure 1 As shown, storage system 100 includes multiple (e.g., N) nodes. Nodes can be implemented by one or more computing devices, computing systems, servers, mainframes, edge devices, etc. In this document, nodes are also referred to as devices. As shown, storage system 100 includes devices 110-1, 110-2, 110-3, ..., 110-N, where N is an integer greater than 1. In the following discussion, devices 110-1, 110-2, 110-3, ..., 110-N are sometimes collectively referred to or individually as device 110. Each device 110 is configured to perform various jobs of storage system 100.

[0026] In storage system 100, the jobs to be performed at device 110 include various types of data storage-related jobs. As some examples, jobs may include data backup jobs to perform backups of user data; data recovery jobs to recover corrupted user data; and garbage collection jobs to collect unused storage space in storage system 100. Of course, these are just some specific examples. Other types of jobs may exist in storage system 100.

[0027] Each device 110 is equipped with corresponding resources, including processing resources, storage resources, and network resources, to execute corresponding jobs. Typically, in addition to general-purpose processing equipment (not shown), each device 110 also includes accelerator resources that can be deployed. Figure 1 As shown, device 110-1 includes accelerator resource 112-1, device 110-2 includes accelerator resource 112-2, device 110-3 includes accelerator resource 112-3, ..., device 110-N includes accelerator resource 112-N. In the following discussion, for ease of discussion, accelerator resources 112-1, 112-2, 112-3, ..., 112-N are sometimes collectively referred to or individually as accelerator resource 112.

[0028] The use of accelerator resource 112 can improve the processing performance of individual devices 110, thereby increasing the overall throughput of storage system 100. In some embodiments, accelerator resource 112 can be one or more QAT cards. It should be understood that although QAT cards are used as examples of accelerator resources in some embodiments of this disclosure, accelerator resource 112 may also include other hardware processing devices with acceleration capabilities.

[0029] Depending on the storage system employed, in some embodiments, storage system 100 may include two devices (i.e., N=2) 110. In some embodiments, storage system 100 may include a scalable multi-node (multi-device) system comprising at least two (i.e., N≥2) devices 110. In such a system, jobs can be scheduled to be executed across the various devices 110. Each device 110 may be deployed with the same or different accelerator resources 112 for executing the corresponding job.

[0030] It should be understood that Figure 1 The storage system 100 shown is merely exemplary and not limiting. The storage system according to this disclosure may also have other forms or structures.

[0031] Figure 2 A schematic block diagram of an example storage system 200 according to some embodiments of the present disclosure is shown. For ease of discussion, in... Figure 2 Building upon the example storage system 200, we will continue to discuss how to implement accelerator resource usage according to embodiments of this disclosure. Figure 2As shown, storage system 200 includes devices 110-1, 110-2, 110-3, ... 110-N (collectively or individually referred to as device 110). Each device 110 is configured to perform various operations of storage system 100. Device 110-1 includes accelerator resource 112-1, device 110-2 includes accelerator resource 112-2, device 110-3 includes accelerator resource 112-3, ... devices 110-N include accelerator resource 112-N.

[0032] same Figure 1 Compared to storage system 100, Figure 2 A more detailed schematic block diagram of the storage system 200 is shown. (See attached diagram.) Figure 2 As shown, each device 110 of the storage system 200 is also equipped with schedulers 210-1, 210-2, 210-3, ... 210-N and Remote Procedure Call (RPC) interfaces 220-1, 220-2, 220-3, ... 220-N. In the following discussion, for ease of discussion, schedulers 210-1, 210-2, 210-3, ... 210-N are sometimes collectively referred to as scheduler 210 or referred to individually as RPC interface 220. The storage system 200 utilizes scheduler 210 and RPC interface 220 to implement cross-device accelerator resource calls.

[0033] Scheduler 210 in device 110 is configured to perform scheduling of jobs to be executed on device 110. During the execution of a scheduled job, according to embodiments of this disclosure, scheduler 210 is configured to determine whether to schedule local accelerator resource 112 or remote accelerator resource 112 to execute the job. RPC interface 220 in device 110 is configured to enable data access between devices 110 when it is determined that a remote accelerator resource should be used. The functionality of RPC interface 220 will be discussed in detail below.

[0034] In this document, "local accelerator resource" 112 refers to the accelerator resource deployed on the current device 110, and "remote accelerator resource" 112 refers to the accelerator resource deployed on other devices 110 for the current device 110. For example, for device 110-1, accelerator resource 112-1 is a local accelerator resource, while accelerator resources 112-2, 112-3, ..., 112-N are remote accelerator resources.

[0035] The inventors discovered through research that using remote accelerator resources to balance local processing pressure is feasible because data transfer rates between devices can be made faster, thereby reducing network latency between devices. In common accelerator resource deployments in some devices, if the system is busy and has many jobs to process, the processing latency for a unit of data, such as 128KB, can reach 1500μs. However, between devices connected via high-speed network interfaces such as Remote Direct Memory Access (RDMA), the round-trip time for transmitting a unit of data can be between 355μs and 135μs. That is, compared to the higher processing latency of accelerator resources, the network latency between devices may not have a significant impact on the overall processing latency of the job. Based on these findings, the inventors proposed improving job processing speed by calling accelerator resources across devices within the local device, thereby improving overall resource utilization.

[0036] In some embodiments, within the storage system 200, devices 110 capable of scheduling accelerator resources across devices are connected via high-speed network interfaces, such as RDMA interfaces. In some examples, the RDMA interface may enable data transfer between devices 110 via various protocols, such as RDMA over aggregated Ethernet (RoCE). Other high-speed network interfaces are also feasible. In some embodiments, to improve processing performance in cross-device scheduling scenarios, the round-trip time between devices 110 may be reduced to less than the local processing latency of a single device 110, which can be achieved by selecting appropriate network interfaces.

[0037] It should be understood that, although Figure 2 The diagram shows that each device 110 of the storage system 200 is equipped with a scheduler 210 and an RPC interface 220. Depending on actual needs, only some devices 110 may be equipped with the scheduler 210 and RPC interface 220 for implementing cross-device resource scheduling according to embodiments of this disclosure. It should be noted that, for clarity, Figure 2 The coupling between each RPC interface 220 and all other devices of system 100 is not shown; for example, the coupling between RPC interface 220-1 of device 110-1 and device 110-3 is not shown. However, such coupling may exist depending on actual needs.

[0038] The following will refer to Figure 3 This describes the cross-device resource call implemented at device 110. Figure 3A flowchart of a resource usage process 300 according to some embodiments of the present disclosure is shown. Process 300 may be implemented by device 110 of system 100, for example, by scheduler 210 of device 110. It should be understood that method 300 may also be performed by other suitable devices or apparatuses. Method 300 may include additional actions not shown and / or the actions shown may be omitted; the scope of the present disclosure is not limited in this respect. For ease of explanation, reference will be made to… Figure 2 Let's describe process 300. In the following text, for ease of discussion, we assume that... Figure 2 Process 300 is implemented at device 110-1 in storage system 200. It should be understood that, as needed, any device 110 in storage system 100 or storage system 200 may be configured to implement process 300.

[0039] In this text, the device 110 in which process 300 is implemented (e.g., device 110-1) is sometimes referred to as the "first device," and the accelerator resource 112 therein (e.g., accelerator resource 112-1) is sometimes referred to as the "first accelerator resource." Other devices 110 (e.g., devices 110-2, 110-3, ..., 110-N) are sometimes referred to as the "second device," and the accelerator resource 112 therein is sometimes referred to as the "second accelerator resource." The terms "first" and "second" are used herein only for ease of distinction.

[0040] like Figure 3 As shown, at point 310, the first device determines whether the load on its first accelerator resource exceeds a load threshold. Load may refer to the amount of work to be processed by the first accelerator resource, or it may refer to the amount of data to be processed by the first accelerator resource. The load threshold may be a predetermined or preset load threshold. For example, in some embodiments, the number of jobs to be processed by the first accelerator resource may be determined. If the number of jobs exceeds a predetermined number threshold, it is determined that the load on the first accelerator resource exceeds the load threshold.

[0041] In some embodiments, other methods may also be used to determine whether the load on the first accelerator resource exceeds a load threshold. For example, in some embodiments, the first device (e.g., device 110-1) determines whether the queue depth of pending requests for the first accelerator resource of the first device exceeds a queue depth threshold. The following will combine... Figure 4 Describe this process in more detail.

[0042] Figure 4 A graph is shown illustrating methods for determining a predetermined queue depth threshold according to some embodiments. Figure 4As shown, curve 401T = F(Q) can be a graph plotting the queue depth versus the local processing latency of the first accelerator resource, drawn based on data from the user manual of the first accelerator resource. In some embodiments, the desired latency 410 can be determined based on... Figure 4 The curve 401 is used to determine the queue depth threshold. Figure 4 In the example, the queue depth threshold can be determined as a queue depth of 410. Figure 4 Additionally, curve 402T = F(Q) + Δ is shown to represent the remote processing latency of the queue depth and other second accelerator resources of device 110. Curve 402 will be described in more detail below.

[0043] In some embodiments, the load threshold can also be determined based on experimental measurement data. For example, a load threshold that meets the expected delay can be obtained by averaging data measured from multiple experiments.

[0044] Return to Figure 3 If at 310 the first device determines that the load on the first accelerator resource exceeds a load threshold, then method 300 proceeds to 320. At 320, the first device sends a job processing request to a second device in a candidate device list, so that the second device utilizes its second accelerator resources to process the first device's target job. The candidate device list indicates devices 110 in storage system 100 that can be used to assist the first device in processing jobs. For example, the candidate device list may indicate that all devices 110 in storage system 100 other than the first device (e.g., device 110-1) can be used to assist the first device in processing jobs. As another example, the candidate device list may indicate that one or more devices 110 in storage system 100 other than the first device (e.g., device 110-1) (e.g., one or more of devices 110-2, 110-3, ..., 110-N) can be used to assist the first device in processing jobs.

[0045] In some embodiments, the second device may be any device selected from the list of candidate devices. Alternatively, a round-robin scheduling algorithm may be used to select the second device from the list of candidate devices. For example, a device 110 may be selected as the second device sequentially from devices 110-2, 110-3, ..., 110-N in a pre-set order.

[0046] refer to Figure 5 An example job processing request 500 sent from a first device to a second device according to some embodiments is described. Figure 5As shown, the job processing request 500 includes a message identifier (ID) 505, a source device ID 510, a target device ID 515, an expected processing latency 520, and a payload 525. For example, the source device ID 510 may refer to a first device (e.g., device 110-1). The target device ID 515 may refer to a second device selected by the first device (e.g., device 110-2). The expected processing latency 520 represents a desired processing latency threshold, such as 1500 μs. This is merely illustrative and does not limit the invention in any way; the expected processing latency threshold can be any suitable time length value. The payload 525 may include the target job of the first device to be processed by the second accelerator resources of the second device.

[0047] It should be understood that in some embodiments, other forms of job processing requests may be used. For example, a job processing request that only includes the target job and the request sending time may be used. Figure 5 The diagram also shows an example response received by the first device from the second device, which will be described in more detail below.

[0048] Return to Figure 3 Alternatively or concurrently, if the first device at 310 determines that the load on the first accelerator resource does not meet the load threshold, then the target operation continues to be performed by the first device. For example, the target operation continues to be performed by device 110-1.

[0049] At 330, the first device receives latency information from the second device related to the remote processing latency of processing the target job using the second accelerator resources. For example, device 110-1 receives from device 110-2 the latency information related to the remote processing latency of processing the target job (e.g., which may be included in the payload 525 of example job processing request 500) using the accelerator resources 112-2 of device 110-2.

[0050] In some embodiments, the latency information may indicate the time period between a first moment when the first device sends a job processing request and a second moment when the first device receives the latency information. Depending on the length of the time period between the first and second moments, the latency information may indicate whether the remote processing latency meets the requirements.

[0051] In some embodiments, latency information may use other information to indicate remote processing latency. Return to Figure 4 , Figure 4 The curve 402T=F(Q)+Δ is also shown for the remote processing latency of the queue depth and the second accelerator resources of other devices 110. Figure 4In this context, Δ represents the latency difference Δ between the remote processing latency incurred by the second accelerator resource in processing the target job and the local processing latency incurred by the first accelerator resource in processing the target job. In some embodiments, each device 110 uses the same type of accelerator resource 112, so the latency difference Δ can be a fixed value. For example, this latency difference Δ can be obtained through multiple offline experimental measurements or from the user manual of the accelerator resource 112.

[0052] according to Figure 4 As shown by curves 401 and 402, when the expected latency is 410, the queue depth threshold for the first accelerator resource is queue depth 420. Conversely, when the expected latency 410 is to be met, the queue depth threshold for the remote second accelerator resource is queue depth 430. Therefore, in some embodiments, the latency information received from the second device may be information about the queue depth of the second accelerator resource. This queue depth information can indicate whether the remote processing latency meets the expected latency. For example, if the queue depth indicated in the latency information exceeds queue depth 430, it indicates that the remote processing latency does not meet the expected latency 410. Conversely, if the queue depth indicated in the latency information does not exceed queue depth 430, it indicates that the remote processing latency meets the expected latency 410.

[0053] refer to Figure 5 Example delay information 550 received by a first device from a second device according to some embodiments is described. For example... Figure 5 As shown, the latency information 550 includes a message identifier (ID) 555, a source device ID 560, a target device ID 565, whether the expected processing latency is met 570, and the processed payload 575. For example, the source device ID 560 may refer to a first device (e.g., device 110-1). The target device ID 565 may refer to a second device selected by the first device (e.g., device 110-2). Whether the expected processing latency is met 570 indicates whether the remote processing latency incurred in processing the target job using the second accelerator resources meets the expected processing latency, for example, as... Figure 5 The expected processing delay 520 indicates the expected processing delay. For example, whether the expected processing delay is met 570 can be 1 if the expected processing delay is met, and 0 if the expected processing delay is not met. The processed payload 575 may include the target job of the first device that has been processed by the second accelerator resources of the second device.

[0054] According to some embodiments, the latency information is information generated by the second device and sent to the first device. For example, device 110-2 receives a job processing request (such as...) from device 110-1. Figure 5The job processing request 500). In response to receiving the job processing request, device 110-2 processes the target job included in the job processing request (e.g., Figure 5 The device 110-2 further determines and creates delay information (e.g., based on the time spent processing the target task, etc.) based on the payload 525 in the device. Figure 5 (Latency information 550). In addition, device 110-2 also sends latency information to the first device, such as device 110-1.

[0055] Return to Figure 3 At point 340, the first device updates the candidate device list based on latency information. For example, if device 110-1 updates the candidate device list based on latency information received from device 110-2 regarding the time period between the first moment when the first device sends a job processing request and the second moment when the first device receives the latency information, and if the latency information indicates that the time period between the first moment and the second moment is longer than the expected processing latency threshold, then device 110-2 is removed from the candidate device list. If the latency information indicates that the time period between the first moment and the second moment is shorter than the expected processing latency threshold, then device 110-2 is maintained in the candidate device list.

[0056] Additionally or alternatively, in some embodiments, other methods may be used to update the candidate device list. See the following references. Figure 6 The description will provide a more detailed account of the process of updating the candidate device list.

[0057] According to embodiments of this disclosure, when the load on the local first accelerator resources of the first device exceeds a load threshold, a second device can be selected from a list of candidate devices, and the second accelerator resources of the second device can handle the target job of the first device. This reduces the workload of the first accelerator resources of the first device, facilitating improvements in the overall performance of the storage system.

[0058] According to embodiments of this disclosure, latency thresholds and thus load thresholds are determined by using the accelerator resource user manual or by experimentally measuring the storage system. Furthermore, according to embodiments of this disclosure, whether a second device meets remote processing latency requirements is determined by receiving latency information from a second device. In this way, information from various remote devices can be collected to dynamically update the candidate device list.

[0059] Furthermore, embodiments of this disclosure can dynamically update the candidate device list. When a device in the candidate device list does not meet the latency requirements, it can be removed from the candidate device list. By dynamically updating the candidate device list, it is further ensured that the accelerator resources of the devices in the candidate device list can assist the first accelerator resources of the first device in working. Consequently, it is possible to better coordinate the various devices in the storage system, ensuring that the execution of jobs on each device is aligned with the service level agreement (SLA) requirements of the jobs. Consequently, the overall performance of the storage system can be improved.

[0060] In some embodiments, when the first device sends a job processing request to the second device, the scheduler 210 of the first device can invoke the RPC interface 220 to realize data transfer between the two devices, so as to execute the job using the selected remote second accelerator resource. The RPC interface 220 is used to dispatch data access (e.g., input / output I / O access) of the job to be executed to the remote accelerator resource 112.

[0061] For local device 110, when executing a job, the transactions to be performed include sending the data to be processed for the job to a remote device and asynchronously retrieving the processed data from the remote device via a callback function. For remote device 110, when executing a job, the transactions to be performed include receiving data to be processed from another device 110, processing the received data using accelerator resource 112, and returning the processed data to the other device 110. It should be understood that such cross-device data transmission and reception can be implemented in various ways, and no specific limitations are made here.

[0062] Connecting the various devices 110 via a high-speed network interface (e.g., RDMA) allows for very rapid data transmission and reception across devices. Therefore, significant latency is avoided due to data transfer between devices. This also allows the use of other devices 110 within the storage system 100 to assist the local device 110 in completing tasks, preventing excessive latency caused by too many tasks.

[0063] In some embodiments, such as Figure 6 The method shown in section 600 is used to update the candidate device list. The following section will combine... Figure 6 Several embodiments of updating the candidate device list are described in more detail.

[0064] Figure 6 A schematic diagram of an example method 600 for updating a list of candidate devices according to some embodiments of the present disclosure is shown. Method 600 can be considered as an example implementation of block 340 in method 300. Method 600 can be, for example, by... Figure 1The method 600 is executed by device 110 of the storage system 100 shown. For example, it can be implemented by scheduler 210 of device 110. It should be understood that method 600 can also be executed by other suitable devices or means. Method 600 may include additional actions not shown and / or the actions shown may be omitted, and the scope of this disclosure is not limited in this respect. For ease of explanation, reference will be made to... Figure 2 and Figure 5 Let's describe process 600. In the following text, for ease of discussion, we assume that... Figure 2 Process 600 is implemented at device 110-1 in storage system 200. It should be understood that any device 110 in storage system 100 or storage system 200 may be configured to implement process 600 as needed.

[0065] like Figure 6 As shown, at 610, the first device determines whether the remote processing latency exceeds a latency threshold based on latency information. In some embodiments, device 110 may base its determination on, for example, latency information such as, Figure 5 The latency information 550 shown is used to determine whether the remote processing latency exceeds a latency threshold. If the latency information 550 indicates whether the expected processing latency is met, 570 indicates that the expected processing latency is not met, then it is determined that the remote processing latency exceeds the latency threshold. It should be understood that other types of latency information can be used to determine whether the remote processing latency exceeds the latency threshold.

[0066] If it is determined at 610 that the remote processing latency exceeds a latency threshold, then method 600 proceeds to 620. At 620, the first device removes the second device from the candidate device list. For example, device 110-2 used to generate latency information is removed from the candidate device list.

[0067] Alternatively, if the first device determines at 610 that the remote processing latency does not exceed the latency threshold, then the second device will be maintained in the candidate device list. For example, device 110-2 will be maintained in the candidate device list.

[0068] At 630, the first device sets a timer for the second device. For example, device 110-1 sets a timer (e.g., 2000 μs) for device 110-2. It should be understood that the 2000 μs timer length described above is merely illustrative and not limiting. In some embodiments, other timer lengths may be selected.

[0069] At 640, it is determined whether the timer has expired. For example, device 110-1 determines whether the timer for device 110-2 has expired. If it is determined at 640 that the timer has expired, method 600 proceeds to 650. At 650, the first device adds the second device back to the candidate device list. For example, device 110-1 adds device 110-2 back to the candidate device list.

[0070] This method allows for dynamic updating of the candidate device list. If a remote processing experiment on a device in the candidate device list fails to meet expectations, it can be removed from the list. This avoids excessive latency caused by using the device to process the target task.

[0071] Furthermore, by simply judging the latency information, the status of remote devices can be determined without needing to collect the remote processing latency of each remote device separately. Therefore, the embodiments of this disclosure can avoid unnecessary processing work and more easily and conveniently select a second device from the candidate device list to assist the first device in task processing.

[0072] Furthermore, by setting a timer, the device can be added back to the candidate device list after the timer expires. This ensures that the device is added back to the candidate device list after a certain period when it is no longer busy. Consequently, the candidate device list always has a sufficient number of candidate devices to coordinate tasks with the primary device. This approach allows for more flexible coordination of tasks among various devices, thus improving the overall performance of the storage system.

[0073] Figure 7 A schematic block diagram of an example device 700 that can be used to implement embodiments of the present disclosure is shown. For example, such as Figure 1 The storage system 100 shown can be implemented by device 700. For example... Figure 7 As shown, device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 702 or loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. CPU 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0074] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0075] The various processes and handling described above, such as methods 300 and / or 600, can be executed by processing unit 701. For example, in some embodiments, methods 300 and / or 600 can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by CPU 701, one or more actions of methods 300 and / or 600 described above can be performed.

[0076] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0077] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0078] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0079] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0080] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0081] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0082] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0084] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for managing a storage system, comprising: At the first device of the storage system, it is determined whether the load on the first accelerator resource of the first device exceeds a load threshold; If it is determined that the load exceeds the load threshold, a job processing request is sent to a second device in the candidate device list, so that the second device can use its second accelerator resources to process the target job of the first device. The candidate device list indicates devices in the storage system that can be used to assist the first device in processing jobs. Receive latency information from the second device related to the remote processing latency of processing the target job using the resources of the second accelerator; as well as The candidate device list is updated based on the latency information.

2. The method of claim 1, wherein updating the candidate device list comprises: Based on the latency information, determine whether the remote processing latency exceeds the latency threshold; as well as If it is determined that the remote processing latency exceeds the latency threshold, the second device is removed from the candidate device list.

3. The method according to claim 2, further comprising: In response to the removal of the second device from the candidate device list, a timer is set for the second device; as well as In response to the expiration of the timer, the second device is added back to the candidate device list.

4. The method of claim 2, wherein the load threshold is determined based on the delay threshold.

5. The method according to claim 1, wherein the remote processing delay includes the time period between the first moment when the first device sends the request and the second moment when it receives the delay information.

6. The method of claim 1, wherein determining whether the load on the first accelerator resource exceeds the load threshold comprises: Determine the number of jobs that the first device will be processed by the first accelerator resources; as well as If the number exceeds a number threshold, it is determined that the load on the first accelerator resource exceeds the load threshold.

7. The method of claim 1, wherein the first device is connected to the devices in the candidate device list via a high-speed network interface.

8. The method according to claim 1, wherein the first accelerator resource and the second accelerator resource respectively include a Quick Assist Technology (QAT) card.

9. An electronic device, comprising: At least one processor; as well as At least one memory storing computer program instructions, the at least one memory and the computer program instructions being configured, together with the at least one processor, to cause the electronic device to perform actions, the actions including: At the first device of the storage system, it is determined whether the load on the first accelerator resource of the first device exceeds a load threshold; If it is determined that the load exceeds the load threshold, a job processing request is sent to a second device in the candidate device list, so that the second device can use its second accelerator resources to process the target job of the first device. The candidate device list indicates devices in the storage system that can be used to assist the first device in processing jobs. Receive latency information from the second device related to the remote processing latency of processing the target job using the resources of the second accelerator; and The candidate device list is updated based on the latency information.

10. The electronic device of claim 9, wherein updating the candidate device list comprises: Based on the latency information, determine whether the remote processing latency exceeds the latency threshold; as well as If it is determined that the remote processing latency exceeds the latency threshold, the second device is removed from the candidate device list.

11. The electronic device of claim 10, wherein the action further comprises: In response to the removal of the second device from the candidate device list, a timer is set for the second device; as well as In response to the expiration of the timer, the second device is added back to the candidate device list.

12. The electronic device of claim 10, wherein the load threshold is determined based on the delay threshold.

13. The electronic device of claim 9, wherein the remote processing delay includes the time period between the first moment when the first device sends the request and the second moment when it receives the delay information.

14. The electronic device of claim 9, wherein determining whether the load on the first accelerator resource exceeds the load threshold comprises: Determine the number of jobs that the first device will be processed by the first accelerator resources; as well as If the number exceeds a number threshold, it is determined that the load on the first accelerator resource exceeds the load threshold.

15. The electronic device of claim 9, wherein the first device is connected to a device in the candidate device list via a high-speed network interface.

16. The electronic device of claim 9, wherein the first accelerator resource and the second accelerator resource respectively comprise a Quick Assist Technology (QAT) card.

17. A computer program product tangibly stored on a non-volatile computer-readable medium and comprising machine-executable instructions that, when executed, cause a device to perform the method according to any one of claims 1-8.

Images