Resource use method, electronic device and computer program product

By designing service quality strategies for auxiliary resources in the storage system, the problem of low efficiency in the use of auxiliary resources is solved, ensuring that high-priority services obtain more resources, and improving the performance and user experience of backup and recovery services.

CN112685167BActive Publication Date: 2024-12-06EMC IP HLDG CO LLC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN201910989431.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-10-17
Publication Date
2024-12-06
Estimated Expiration
2039-10-17

AI Technical Summary

Technical Problem

Without a Quality of Service (QoS) policy, the auxiliary resource usage efficiency of computing-intensive tasks is low, resulting in high-priority services that may be downgraded, and the request cannot be met when the auxiliary resource load is too large, affecting the performance of backup and recovery services. .

Method used

By designing a quality of service strategy for auxiliary resources (QAT resources) in the storage system, resource allocation is performed from application and operation dimensions, balance resource usage between different applications and operations, ensuring that high-priority applications obtain more resources and in different Adaptively allocate resources between stages to improve overall performance.

Benefits of technology

Improves processing efficiency of high-priority services, improves user experience, and improves the efficiency and performance of the entire storage system through resource balance, supporting traditional backup and new data analysis use cases.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112685167B_ABST
    Figure CN112685167B_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure provide a resource usage method, an electronic device, and a computer program product. The method includes receiving a first request from a first application to process first data using a target resource and a second request from a second application to process second data using the target resource. The method also includes determining a first quantity and a second quantity based on the first request and the second request, the first quantity indicating the amount of data in the first data that will be processed using the target resource, and the second quantity indicating the amount of data in the second data that will be processed using the target resource. The method also includes causing the first application to process a first amount of data in the first data using the target resource, and causing the second application to process a second amount of data in the second data using the target resource. Using the embodiments of the present disclosure, it is possible to balance the use of resources by different applications and different operations of applications, thereby improving the performance of the storage system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of data storage, and more particularly, to a resource use method, an electronic device, and a computer program product. Background Art

[0002] Various storage systems and devices are being used more and more. In some storage systems, in addition to traditional data storage and backup, other types of tasks are added to better utilize the stored data. Such additional tasks (for example, compression) may be computationally intensive. In storage devices such as data domain restorers (DDRs), auxiliary means can be used to speed up such computationally intensive tasks, such as Intel's Quick Assist Technology (QAT).

[0003] In such storage or backup systems, without a Quality of Service (QoS) strategy, high-priority services are likely to be undesirably downgraded. In addition, without good quality of service, the use of auxiliary means will also face more challenges. Summary of the invention

[0004] An embodiment of the present disclosure provides a resource utilization solution.

[0005] In a first aspect of the present disclosure, a resource usage method is provided. The method includes receiving a first request from a first application to process first data using a target resource and a second request from a second application to process second data using the target resource. The method also includes: based on the first request and the second request, determining a first quantity and a second quantity, the first quantity indicating the amount of data in the first data that will be processed using the target resource, and the second quantity indicating the amount of data in the second data that will be processed using the target resource. The method also includes causing the first application to process a first amount of data in the first data using the target resource. The method also includes causing the second application to process a second amount of data in the second data using the target resource.

[0006] In a second aspect of the present disclosure, a resource usage method is provided. The method includes, in response to receiving a request from an application to process data using a target resource, determining a target quantity indicating the amount of data to be processed by the application using the target resource. The method also includes determining a demand for the target resource by a plurality of operations associated with the application. The method also includes allocating the target quantity of data among the plurality of operations based on the demand and the target resource.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the device to perform an action. The action includes receiving a first request from a first application to process first data using a target resource and a second request from a second application to process second data using the target resource. The action also includes: based on the first request and the second request, determining a first quantity and a second quantity, the first quantity indicating the amount of data in the first data that will be processed using the target resource, and the second quantity indicating the amount of data in the second data that will be processed using the target resource. The action also includes causing the first application to process a first amount of data in the first data using the target resource. The action also includes causing the second application to process a second amount of data in the second data using the target resource.

[0008] In a fourth aspect of the present disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, the instructions causing the device to perform an action when executed by the processor. The action includes determining a target quantity in response to receiving a request from an application to process data using a target resource, the target quantity indicating the amount of data to be processed by the application using the target resource. The action also includes determining a demand for the target resource by multiple operations associated with the application. The action also includes allocating the target quantity of data among the multiple operations based on the demand and the target resource.

[0009] In a fifth aspect of the present disclosure, there is provided a computer program product tangibly stored on a computer readable medium and comprising machine executable instructions which, when executed, cause a machine to perform the method according to the first aspect.

[0010] In a sixth aspect of the present disclosure, there is provided a computer program product tangibly stored on a computer readable medium and comprising machine executable instructions which, when executed, cause a machine to perform the method according to the second aspect.

[0011] This Summary is provided to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the exemplary embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present disclosure. In the accompanying drawings:

[0013] Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present disclosure may be implemented;

[0014] Figure 2 A flowchart showing a process of resource utilization according to some embodiments of the present disclosure;

[0015] Figure 3 A flowchart showing a process of resource utilization according to some embodiments of the present disclosure;

[0016] Figure 4A A schematic diagram illustrating the imbalance between different operations is shown;

[0017] Figure 4B shows a schematic diagram illustrating the balance between different operations according to some embodiments of the present disclosure;

[0018] Figure 5 A flowchart showing a process of resource utilization according to some embodiments of the present disclosure; and

[0019] Figure 6 A block diagram of an example device that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The principles of the present disclosure will be described below with reference to several example embodiments shown in the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that these embodiments are described only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0021] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0022] As mentioned above, in some storage systems, auxiliary resources can be used to accelerate the processing of compute-intensive applications. In current solutions, there is no QoS policy for auxiliary resources such as QAT resources, but QAT resources are allocated according to the requests of each application, or in other words, QAT resources are used to accelerate the requested amount of data. When the load of QAT resources is too large to meet the requested amount, the application will wait until the request is met.

[0023] However, the inventors have recognized that there are several problems in this implementation. In the absence of a QoS policy for auxiliary resources, some important applications such as deduplication applications may be affected when the workload is large. The following is an example of a deduplication system, but it should be understood that similar problems may also exist in other data storage or backup systems.

[0024] In the deduplication system, there are non-critical applications or processes, such as garbage collection (GC) applications and cloud migration applications. These non-critical applications also require auxiliary resources such as QAT to help with calculations such as compression and decompression. In fact, the inventors realize that these non-critical applications may occupy too many auxiliary resources and reduce normal backup business performance. Therefore, in order to provide backup / restore services according to the service level agreement (SLA), it is expected to allocate more QAT resources to high-priority backup business applications.

[0025] As mentioned above, in the field of data storage, it has expanded from traditional data storage or backup to secondary storage. For secondary storage, in addition to backup and recovery, there are more types of use cases, including instant access, instant recovery, and even background big data analysis on secondary storage. In the QoS design for QAT, these newly introduced applications need to be considered. Therefore, for secondary storage, it is necessary to determine the priority of these new applications and provide QAT resources according to their priority.

[0026] In addition, applications in deduplication systems usually include different stages of computationally intensive tasks. Taking ordinary backup as an example, its pipeline includes three stages: Secure Hash Algorithm 1 (SHA 1), compression, and encryption. In theory, the performance of a pipeline depends on the slowest stage. This means that it may be meaningless to use more QAT resources for the fast stage. Therefore, it is necessary to monitor the different stages of the pipeline in a timely manner and balance these stages through adaptive allocation of QAT resources.

[0027] The present disclosure provides a resource utilization solution to at least eliminate one or more of the above defects. Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure may be implemented. Figure 1 A plurality of applications 110-150 are running on the storage system 101 shown (such as a deduplication system). Each of the plurality of applications 110-150 is associated with a plurality of different operations. As shown, application 110 includes operations 111-114, application 120 includes operations 121-123, application 130 includes operations 131-134, application 140 includes operations 141-143, and application 150 includes operations 151-153. In the following description, applications 110-150 may be collectively referred to as application 102. Figure 1The number of applications and the number of operations associated with the applications shown in are exemplary only and are not intended to be limiting.

[0028] The operation here may refer to the different stages included in the pipeline of the corresponding application for processing data. In this article, stage and operation may be used interchangeably. For example, in the case where application 110 is a backup application, operation 111 may be SHA 1, operation 112 may be compression, operation 113 may be encryption, and operation 114 may be input and output IO. As another example, in the case where application 120 is a data recovery application, operation 121 may be IO, operation 122 may be decryption, and operation 123 may be decompression. As another example, in the case where application 130 is a big data analysis application, operation 131 may be IO, operation 132 may be decryption, operation 133 may be decompression, and operation 134 may be analysis or calculation. Other types of applications may also be included in the storage system 101, such as GC applications and migration applications, such as migration applications to the cloud.

[0029] Applications 110-150 may request to use target resources 170 to assist in processing data, such as accelerating the processing of data, particularly accelerating computationally intensive tasks. Target resources 170 herein may be any resource that may be used by an application or process in storage system 101 to assist in processing data, such as a resource for accelerating data processing. Target resources 170 may be resources in software form, hardware form, or a combination of the two. In some embodiments, target resources 170 may be a processor for accelerating data processing, such as a QAT chip, etc.

[0030] The resource usage module 160 including a plurality of submodules 161-166 can coordinate the use of the target resource 170 by at least some of the applications 110-150. Alternatively or additionally, the resource usage module 160 can also coordinate the use of the target resource 170 by various operations or various stages of the pipeline in the application 102. In some embodiments, the resource usage module 160 can also utilize some parameters of the storage system 101, such as the global deduplication rate 180 and the local compression rate 190, as further described below.

[0031] It should be understood that the division of resource usage module 160 into multiple submodules 161-166 that perform different functions is for illustrative purposes only and is not intended to be limiting, and in some embodiments, resource usage module 160 may include more or fewer submodules. Although resource usage module 160 is shown as being within storage system 101, it should be understood that resource usage module 160 may also be implemented independently of storage system 101, in which case resource usage module 160 may communicate with storage system 101 in any appropriate manner to implement the resource usage scheme proposed in the present disclosure. In addition, in this document, allocating more resources to an application or similar expressions may refer to enabling the application to utilize target resources to process a greater amount of data than another application.

[0032] In the present disclosure, it is proposed that the QoS policy for target resources (such as QAT resources) can be designed from two dimensions, that is, the use of target resources can be balanced. From the perspective of each application dimension (also referred to as horizontal dimension in this article) of the storage system 101, the priorities of different applications (e.g., applications 110-150) / customers can be determined, and more resources can be allocated to applications for key businesses in the storage system 101. For example, compared with garbage collection / migration to cloud applications, backup / recovery applications should get more resources.

[0033] On the other hand, from the perspective of different operation dimensions (also referred to as vertical dimensions herein) of the same application, application 102 may typically include multiple computationally intensive stages (also referred to as operations herein) in a pipeline, such as SHA1, compression, and encryption mentioned above. Theoretically, the performance of a pipeline depends on the slowest stage (the so-called "short board"). Therefore, QoS policies may be used for different stages in the pipeline so that the use of target resources can be balanced between different stages. In aspects of the present disclosure, the QoS policies of the above two dimensions may be used separately or in combination.

[0034] In the embodiments of the present disclosure, by balancing between different applications, more auxiliary resources can be allocated to applications with higher priority or more important functions, such as backup / recovery applications, thereby improving the user experience. On the other hand, by balancing between different operations or stages of an application, more auxiliary resources can be used to promote "short board" operations or stages, thereby improving the performance of the entire pipeline. In addition, the resources proposed in the present disclosure can support both traditional backup and secondary storage by considering traditional backup use cases and new data analysis use cases.

[0035] The following will refer to Figure 2-Figure 5 The embodiments of the present disclosure are described in detail. Figure 2 FIG. 2 is a flowchart of a resource usage process 200 according to some embodiments of the present disclosure. Figure 1 2. In such an example, process 200 may be implemented by resource utilization module 160.

[0036] In block 210, the resource usage module 160 receives a first request from a first application to process first data using a target resource 170 and a second request from a second application to process second data using the target resource 170. For example, the resource usage module 160 may obtain requests from the application 110 and the application 120 for the target resource 170 within a period of time. Hereinafter, for the purpose of illustration, the application 110 is taken as an example of the first application, and the application 120 is taken as an example of the second application.

[0037] The first request may include the amount of data that the first application 110 requests to process using the target resource 170, which may also be referred to herein as the total amount of first data. For example, the first application 110 may request to use the target resource 170 to accelerate the processing of 100 data segments. Similarly, the second request may include the amount of data that the second application 120 requests to process using the target resource 170, which may also be referred to herein as the total amount of second data. For example, the second application 120 may request to use the target resource 170 to accelerate the processing of 50 data segments.

[0038] In some embodiments, the resource usage module 160 may also receive requests from more applications for the target resource 170. For example, the resource usage module 160 may also receive requests from one or more of the applications 130-150 for the target resource 170.

[0039] At block 220 , resource usage module 160 determines a first quantity and a second quantity based on the received first and second requests. The first quantity indicates the amount of the first data to be processed using target resource 170 , and the second quantity indicates the amount of the second data to be processed using target resource 170 .

[0040] The first quantity may be less than the total amount of the first data requested by the first application 110. For example, when the total amount of the first data is 100 data segments, the first quantity may be, for example, 80 data segments. Similarly, the second data may also be less than the amount of the second data requested by the second application 120. For example, when the total amount of the second data is 50 data segments, the second quantity may be, for example, 30 data segments. It should be understood that the term "data segment" herein is only an example unit for describing the amount of data, and the embodiments of the present disclosure may be applied when any data amount unit is used.

[0041] In some embodiments, a traffic throttling mechanism may be applied. When the target resource 170 is overloaded and cannot process incoming requests, the resource usage module 160 may balance the utilization of the target resource 170 by different applications. For example, the resource usage module 160 (e.g., the throttling submodule 161) may determine whether the utilization of the target resource 170 exceeds a threshold utilization. If the utilization of the target resource 170 exceeds the threshold utilization, the resource usage module 160 may determine the first quantity and the second quantity based on the first request and the second request. If the utilization of the target resource 170 does not exceed the threshold utilization, the resource usage module 160 may directly allocate the target resource 170 with the amount of data requested by the first application 110, the second application 120, and possibly other applications.

[0042] In some embodiments, the first quantity and the second quantity can be determined based on the priority. For example, the priority of the application 110-150 can be pre-set by the interface submodule 162. As just one example, the application 110-150 can be divided into two priorities: high priority and low priority. Applications such as backup and recovery can be assigned high priority, while applications such as GC and migration can be assigned low priority. The priority of the application 110-150 can be determined, for example, based on SLA. The resource usage module 160 may tend to allocate more resources to applications with high priority. In other words, compared with low priority applications, the resource usage module 160 can enable applications with high priority to utilize the target resource 170 to process more data. If the first application 110 and the second application 120 have the same priority, the same resources can be allocated to them.

[0043] In some embodiments, the amount of data that an individual application can process using the target resource 170 can be determined based on the weight for the application. For example, the resource usage module 160 can obtain a first weight for the first application 110 and a second weight for the second application 120. The resource usage module 160 can determine the total amount of the first data based on the first request, and determine the total amount of the second data based on the second request. Then, the resource usage module 160 can determine the first amount and the second amount based on the first weight, the second weight, the total amount of the first data, and the total amount of the second data.

[0044] As just one example, assume that the total amount of first data requested by the first application 110 is 100 data segments and the total amount of second data requested by the second application 120 is 50 data segments. Based on the first weight and the second weight, the resource usage module 160 may, for example, determine that the first amount is, for example, 80 data segments and the second amount is 30 data segments.

[0045] The first weight, the second weight, and possible weights for other applications may be based on the priorities of the corresponding applications. For example, applications with the same priority may have the same weight. These weights may also be based on the applications themselves. For example, the weights of different applications may be set independently of other applications.

[0046] In some embodiments, the weight of the application 102 can also be adjusted based on the execution status of the application 102, for example, so that the user can flexibly adjust it according to his requirements. The resource usage module 160 can monitor the execution performance of at least one of the first application 110 and the second application 120, and obtain the first weight and the second weight that are updated based on the monitored execution performance. For example, the resource usage module 160 can monitor the execution performance of the first application 110 via the operation performance monitoring submodule 163, that is, the processing status of the first application 110 on the data, such as the storage amount of data, delay, etc.

[0047] In some embodiments, the resource usage module 160 may obtain the performance requirement for the first application 110 in any suitable manner, and compare the monitored execution performance of the first application 110 with the performance requirement. The resource usage module 160 may further adjust the first weight and the second weight according to the comparison result. For example, if the execution performance of the first application 110 cannot meet the performance requirement, the first weight may be increased, and the second weight may be reduced accordingly.

[0048] In some embodiments, the resource usage module 160 may provide the monitored execution performance to the user and receive updates to the first weight and the second weight from the user. For example, the resource usage module 160 may present the monitored execution performance of the first application 110 to the user via the interface submodule 162. The user may determine whether to adjust the first weight, the second weight, and other possible weights, for example, based on the SLA. The resource usage module 160 may receive updates to the weights from the user via the interface submodule 162 and determine the updated weights via the weight allocation submodule 165.

[0049] In the case where the weight of an application is based on the priority of the application, the adjustment of the weight of the application may be an adjustment of the weight associated with the priority. For example, in the case where the application is divided into two levels of high priority and low priority, the weight for the high priority and the weight for the low priority may be adjusted. It should be understood that although the above example embodiment is described with respect to the first application 110, the resource usage module 160 may monitor the execution performance of any application.

[0050] Continue to refer Figure 2At block 230, the resource usage module 160 enables the first application 110 to utilize the target resource 170 to process a first amount of data in the first data. At block 240, the resource usage module 160 enables the second application 120 to utilize the target resource 170 to process a second amount of data in the second data. For example, for the above example where the first amount is determined to be 80 data segments and the second amount is determined to be 30 data segments, the resource usage module 160 can enable the first application 110 to utilize the target resource 170 to speed up the processing of the 80 data segments, and enable the second application 120 to utilize the target resource 170 to speed up the processing of the 30 data segments, via the resource allocation submodule 166.

[0051] In such an embodiment, more auxiliary resources (e.g., QAT resources) can be allocated to applications with higher priority or more important, such as backup / restore applications, through balancing between different applications. In other words, important applications such as backup or restore can use auxiliary resources to speed up the processing of more data. In this way, the efficiency of the storage system and the user experience can be improved.

[0052] As mentioned above, alternatively or additionally, the use of auxiliary resources may also be balanced for different operations or phases of the same application. Figure 3 FIG. 3 is a flowchart showing a process 300 of resource utilization according to some embodiments of the present disclosure. Figure 1 , Figure 4A and Figure 4B 300 is described. In such an example, process 300 may be performed by resource usage module 160. In addition, balancing for different operations may be performed for one or more of applications 110-150, and process 300 is described with respect to first application 110 for illustrative purposes only. In some embodiments, process 300 may be viewed as a specific implementation of block 230 of process 200.

[0053] First, refer to Figure 4A To describe the execution model of a pipeline with multiple stages or operations. Figure 4A A schematic diagram 400 illustrating the imbalance between different operations is shown. Figure 4A In the example of , there are three tasks or pipelines, and each pipeline needs to perform three operations 401, 402, and 403 in sequence. In other words, each pipeline includes three stages, and the costs of these three stages (corresponding required computing resources or latency) are 1, 2, and 9, respectively. For example, in the case where the application is data recovery, operations 401, 402, and 403 may correspond to IO, decryption, and decompression, respectively.

[0054] Each next stage can only start working after the previous stage has completed and provided output, which is usually called the dependency relationship between different stages. Figure 4A The total time 410 of the three tasks is "1+2+9x3=30". Figure 4A It can be seen that there are still many pauses in the pipeline, so there is still room for optimization.

[0055] refer to Figure 3 At block 310, the resource usage module 160 determines the requirements of the plurality of operations associated with the first application 110 on the target resource 170. For example, the resource usage module 160 may determine the requirements of the plurality of operations 111-114 on the target resource 170 via the operation monitoring submodule 164. The requirements of each of the plurality of operations 111-114 on the target resource 170 may be determined separately. These operations may be a plurality of operations that need to be performed sequentially, such as Figure 4A In some embodiments, the demand for target resources 170 of an operation or a phase may be determined based on the time cost or the total resource cost required for the operation or the phase.

[0056] In some embodiments, the demand for the target resource 170 by an individual operation can be determined based on other metrics. For example, the resource usage module 160 can determine the total amount of data processed by the individual operation and the throughput of the individual operation for an individual operation in multiple operations 111-114, and determine the demand for the individual operation based on the determined total amount and throughput. The determination of the total amount of data processed by the individual operation can be based on the specific type of the operation. In the presence of compression or decompression, special consideration needs to be given to relevant global and local parameters. For example, in the case where the application 110 is a backup application, when determining the demand for the target resource 170 by, for example, operations 112, 113, and 114, the global deduplication rate 180 and / or the local compression rate 190 need to be considered to determine the amount of data processed by the corresponding operation (due to the presence of deduplication and / or compression, the amount of data processed by two adjacent operations may be different).

[0057] At block 320, the resource usage module 160 allocates the first amount of data (determined at block 220) among the plurality of operations 111-114 based on the determined demand and the target resource 170. For example, an allocation ratio may be determined based on the determined demand of each operation for the target resource 170, and the determined first amount of data may be allocated to each of the plurality of operations 111-114 based on the allocation ratio.

[0058] As an example, in the case where the first number is determined to be 80 data segments, the resource use module 160 may determine, based on the requirements of each of the multiple operations 111-114 for the target resource 170, to allocate 10 data segments to the operation 111, allocate 20 data segments to the operation 112, allocate 40 data segments to the operation 113, and allocate the remaining 10 data segments to the operation 114. In other words, during the data processing, the operation 111 may use the target resource 170 to speed up the processing of the data of the 10 data segments, the operation 112 may use the target resource 170 to speed up the processing of the data of the 20 data segments, the operation 113 may use the target resource 170 to speed up the processing of the data of the 40 data segments, and the operation 114 may use the target resource 170 to speed up the processing of the data of the 10 data segments.

[0059] The following is a specific example to illustrate how to balance different operations or stages by allocating between different operations or stages. In this example, the application is a deduplication application, and the target resource is a QAT resource. First, the "short board" stage or operation of a pipeline is determined via the operation monitoring submodule 164, and the "short board" stage or operation will cost a large computing cost or time. Secondly, specific to deduplication applications, due to the operations of global deduplication and local compression, the amount of data processed by each stage or operation is different. Therefore, it is necessary to determine the global deduplication rate and the local compression rate, and estimate the workload of each stage. Together with the statistical information of the throughput of each stage, the demand for QAT resources in each stage can be obtained. For example, it can be the ratio of QAT resources allocated to each stage. Finally, based on the estimated demand or demand ratio for QAT resources obtained in the previous step, the resource usage module 160 can adaptively adjust the weight of each stage so as to allocate proportional QAT resources to each stage to balance each stage in a pipeline.

[0060] Figure 4B A schematic diagram 450 is shown illustrating the balance between different operations according to some embodiments of the present disclosure. Figure 4B In the example of FIG. 4 , the use of target resource 170 is balanced between operations 410 , 420 , and 430 , as described above with respect to process 300 . Figure 4B In the example above, we can see an improvement in the balancing results. The total cost of the three tasks is the same, i.e. 12x3. Instead of 1+2+9, the cost of a pipeline becomes 4x3. Each operation costs 4. Therefore, when these pipelines are running, no pauses will occur. The total time cost of the three tasks, 460, is "4+4+4x3=20". Therefore, compared to Figure 4A , we can see the benefits gained through the balance between different operations.

[0061] In such an embodiment, by balancing the different operations or stages of the application, more auxiliary resources can be utilized to facilitate the "short board" operations or stages, thereby improving the performance of the entire pipeline.

[0062] The balancing solution between applications and the balancing solution between operations proposed in this article can be used separately or in combination. Figure 5 FIG. 5 is a flowchart showing a process 500 of resource utilization according to some embodiments of the present disclosure. Figure 1 5. In such an example, process 500 may be implemented by resource utilization module 160.

[0063] At block 510, resource usage module 160 determines whether a request is received from an application to process data using target resource 170. If a request is received from an application (eg, application 130), process 500 proceeds to block 520.

[0064] At block 520, resource usage module 160 determines a target quantity indicating the amount of data to be processed by application 130 using target resource 170. In some embodiments, the target quantity may be the total amount of data requested to be processed by application 130. That is, in such embodiments, the balancing scheme between operations is used alone. In some embodiments, the target quantity may be determined in the manner described above with respect to block 220.

[0065] At block 530, resource usage module 160 determines demand on target resources 170 by a plurality of operations 131-134 associated with application 130. The actions performed at block 530 may be performed in the manner described above with respect to block 310.

[0066] At block 540, resource usage module 160 allocates the target amount of data among the plurality of operations 131-134 based on the demand and target resources 170 determined at block 530. The actions performed at block 540 may be performed in the manner described above with respect to block 320.

[0067] The above describes the balance of the use of auxiliary resources such as QAT resources between different applications (horizontal dimension) and between different operations or stages of applications (vertical dimension). In different aspects of the present disclosure, the balance of these two dimensions can be used separately or in combination.

[0068] Figure 6A schematic block diagram of an example device 600 that can be used to implement an embodiment of the present disclosure is shown. As shown, the device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 602 or loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0069] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0070] The processing unit 601 performs the various methods and processes described above, such as any one of the processes 200, 300, and 500. For example, in some embodiments, any one of the processes 200, 300, and 500 may be implemented as a computer software program or a computer program product, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the CPU 601, one or more steps of any one of the processes 200, 300, and 500 described above may be performed. Alternatively, in other embodiments, the CPU 601 may be configured to perform any one of the processes 200, 300, and 500 by any other suitable means (e.g., by means of firmware).

[0071] According to some embodiments of the present disclosure, a computer readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method according to the present disclosure is implemented.

[0072] Those skilled in the art should understand that the various steps of the method disclosed above can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present disclosure is not limited to any specific combination of hardware and software.

[0073] It should be understood that although several devices or sub-devices of the device are mentioned in the detailed description above, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present disclosure, the features and functions of two or more devices described above can be embodied in one device. Conversely, the features and functions of one device described above can be further divided into multiple devices to be embodied.

[0074] The above description is only an optional embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A resource use method, comprising: Receiving a first request from a first application to process first data using a target resource for accelerating data processing and a second request from a second application to process second data using the target resource; Determining a first quantity and a second quantity based on the first request and the second request, wherein the first quantity indicates an amount of data in the first data to be processed using the target resource, and the second quantity indicates an amount of data in the second data to be processed using the target resource, wherein determining the first quantity and the second quantity comprises: Obtaining a first weight for the first application and a second weight for the second application, wherein the first weight and the second weight are determined based on a service level agreement (SLA) and are adjusted based on execution performance of the first application and the second application; determining a first total amount of the first data based on the first request, the first amount being less than the first total amount; determining a second total amount of the second data based on the second request, the second amount being less than the second total amount; and determining the first quantity and the second quantity based on the first weight, the second weight, the first total amount of the first data, and the second total amount of the second data; causing the first application to utilize the target resource to process the first amount of data in the first data; and The second application is caused to utilize the target resource to process the second amount of data in the second data.

2. The method according to claim 1, wherein obtaining the first weight and the second weight comprises: monitoring execution performance of at least one of the first application and the second application; as well as The first weight and the second weight updated based on the execution performance are acquired.

3. The method according to claim 2, wherein obtaining the updated first weight and the second weight comprises: providing the execution performance to the user; as well as An update to the first weight and the second weight is received from the user.

4. The method of claim 1 , wherein determining the first amount and the second amount comprises: Determining whether the utilization of the target resource exceeds a threshold utilization; as well as In response to determining that the utilization exceeds the threshold utilization, the first amount and the second amount are determined based on the first request and the second request.

5. The method of claim 1 , wherein causing the first application to utilize the target resource to process the first amount of data comprises: Determining demands of a plurality of operations associated with the first application on the target resource; as well as The first amount of data is allocated among the plurality of operations based on the demand and the target resource.

6. The method of claim 5, wherein determining the requirement comprises: For an individual operation of the plurality of operations, determining a total amount of data processed by the individual operation and a throughput of the individual operation; as well as Based on the determined total amount and the throughput, the demand for the individual operation is determined.

7. An electronic device comprising: processor; as well as a memory coupled to the processor, the memory having instructions stored therein, the instructions, when executed by the processor, causing the device to perform actions, the actions comprising: Receiving a first request from a first application to process first data using a target resource for accelerating data processing and a second request from a second application to process second data using the target resource; Determining a first quantity and a second quantity based on the first request and the second request, wherein the first quantity indicates an amount of data in the first data to be processed using the target resource, and the second quantity indicates an amount of data in the second data to be processed using the target resource, wherein determining the first quantity and the second quantity comprises: Obtaining a first weight for the first application and a second weight for the second application, wherein the first weight and the second weight are determined based on a service level agreement (SLA) and are adjusted based on execution performance of the first application and the second application; determining a first total amount of the first data based on the first request, the first amount being less than the first total amount; determining a second total amount of the second data based on the second request, the second amount being less than the second total amount; and determining the first quantity and the second quantity based on the first weight, the second weight, the first total amount of the first data, and the second total amount of the second data; causing the first application to utilize the target resource to process the first amount of data in the first data; and The second application is caused to utilize the target resource to process the second amount of data in the second data.

8. The electronic device according to claim 7, wherein acquiring the first weight and the second weight comprises: monitoring execution performance of at least one of the first application and the second application; as well as The first weight and the second weight updated based on the execution performance are acquired.

9. The electronic device according to claim 8, wherein acquiring the updated first weight and the second weight comprises: providing the execution performance to the user; as well as An update to the first weight and the second weight is received from the user.

10. The electronic device of claim 7, wherein determining the first number and the second number comprises: Determining whether the utilization of the target resource exceeds a threshold utilization; as well as In response to determining that the utilization exceeds the threshold utilization, the first amount and the second amount are determined based on the first request and the second request.

11. The electronic device of claim 7, wherein causing the first application to utilize the target resource to process the first amount of data comprises: Determining demands of a plurality of operations associated with the first application on the target resource; as well as The first amount of data is allocated among the plurality of operations based on the demand and the target resource.

12. The electronic device of claim 11, wherein determining the requirement comprises: For an individual operation of the plurality of operations, determining a total amount of data processed by the individual operation and a throughput of the individual operation; as well as Based on the determined total amount and the throughput, the demand for the individual operation is determined.

13. A computer program product tangibly stored on a computer readable medium and comprising machine executable instructions which, when executed, cause a machine to perform the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • System and methods for allocating shared storage resources

    CN102388381A