Resource scheduling method and device, and electronic device

Abstract

Embodiments of the present application relate to a resource scheduling method, device and electronic equipment, the method comprising: periodically acquiring the usage of PMEM resources corresponding to each big data component in all big data components; when receiving a resource scheduling allocation request sent by a first big data component, determining the allocable PMEM resources according to the usage of PMEM resources corresponding to each big data component except the first big data component acquired in the current period; and allocating the allocable PMEM resources to the task queue corresponding to the first big data component for executing the task to be executed. In this way, it can be ensured that each big data component has reasonable resources to support task execution when executing tasks. When some big data components are insufficient in resources, the pre-configured PMEM resources of other big data components can be flexibly scheduled by the resource scheduling system for use by the big data components insufficient in resources, so as to realize reasonable allocation of all PMEM resources.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer technology, and particularly relate to a resource scheduling method and device and electronic equipment. BACKGROUND

[0002] A big data platform integrates many commonly used big data components, such as storage, scheduling, and computing type components of a distributed file system (Hadoop Distributed File System, HDFS), a Hadoop database (HBase, a distributed, column-oriented open source database), a distributed computing engine Spark, a distributed memory database Redis, and the like, involving all aspects of the big data field.

[0003] At present, when performing persistent memory (PMEM) resource allocation for big data components, the following two methods are included:

[0004] The first method is that all PMEM resources in a cluster node can only be occupied by one type of big data component at the same time, and other components cannot use the PMEM resources. Although this method can greatly improve the execution efficiency of the big data component currently executing a task, it cannot guarantee that the tasks of other components are executed, and the PMEM resources are only used for one task, so that a lot of resources are left unused, resulting in resource waste.

[0005] The second method is that the PMEM resources in the cluster node are shared by the tasks of all big data components, and different proportions of PMEM resources are allocated to different types of big data components. This method can cause the PMEM resources of some types of big data components to be insufficient, and the PMEM resources of other types of big data components to be wasted.

[0006] In summary, there is no reasonable PMEM resource scheduling scheme in the prior art to guarantee that all types of big data components can be allocated to reasonable resources as much as possible, and to avoid resource waste. SUMMARY

[0007] The present application provides a resource scheduling method, device and electronic equipment to solve the technical problem that all types of big data components cannot be allocated to reasonable resources as much as possible in the prior art, and resource waste occurs.

[0008] In a first aspect, the present application provides a resource scheduling method, which includes:

[0009] periodically acquire the usage of the PMEM resource corresponding to each big data component in all big data components respectively;

[0010] When the resource scheduling allocation request sent by the first big data component is received, the usage of the PMEM resource corresponding to each big data component except the first big data component is acquired in the current period, and the allocable PMEM resource is determined;

[0011] The allocable PMEM resource is allocated to the task queue corresponding to the first big data component to execute the task to be executed, wherein the first big data component is any one of all big data components.

[0012] In a second aspect, the present application provides a resource scheduling device, which comprises:

[0013] The acquisition module is configured to periodically acquire the usage of the PMEM resource corresponding to each big data component in all big data components respectively, and receive the resource scheduling allocation request sent by each big data component;

[0014] The processing module is configured to, when the resource scheduling allocation request sent by the first big data component is received, determine the allocable PMEM resource according to the usage of the PMEM resource corresponding to each big data component except the first big data component acquired in the current period;

[0015] The allocation module is configured to allocate the allocable PMEM resource to the task queue corresponding to the first big data component to execute the task to be executed, wherein the first big data component is any one of all big data components.

[0016] In a third aspect, an electronic device is provided, which carries a resource scheduling system and comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0017] The memory is configured to store a computer program;

[0018] The processor is configured to execute the program stored on the memory, and implement the steps of the resource scheduling method of any one of the embodiments of the first aspect.

[0019] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to implement the steps of the resource scheduling method of any one of the embodiments of the first aspect.

[0020] Compared with the prior art, the above technical solution provided by the embodiments of the present application has the following advantages:

[0021] The method provided by the embodiment of the application periodically acquires the usage of the PMEM resource corresponding to each big data component in all big data components. After receiving the resource scheduling and distribution request sent by the first big data component, the usage of the PMEM resource corresponding to each big data component except the first big data component is acquired, and the distributable PMEM resource is determined according to the acquired usage. Then, the distributable PMEM resource is distributed to the task queue corresponding to the first big data component, and is used to execute the task to be executed. In this way, the reasonable PMEM resource can be ensured for each big data component to support the execution of the task. At the same time, when the PMEM resource of some big data component is insufficient, the preconfigured PMEM resource of other big data components can be flexibly scheduled by the resource scheduling system, and is used by the big data component with insufficient resource to complete the execution of the task. In this way, the situation that the PMEM resource of the big data component is insufficient can be avoided as much as possible, and the waste of the PMEM resource can be avoided, and the reasonable distribution of all PMEM resources is realized. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A resource scheduling method flow diagram is provided for the embodiment of the application.

[0023] Figure 2 Another resource scheduling method flow diagram is provided for the embodiment of the application.

[0024] Figure 3 Another resource scheduling method flow diagram is provided for the embodiment of the application.

[0025] Figure 4 Another resource scheduling method flow diagram is provided for the embodiment of the application.

[0026] Figure 5 Another resource scheduling method flow diagram is provided for the embodiment of the application.

[0027] Figure 6 A resource scheduling device structure diagram is provided for the embodiment of the application.

[0028] Figure 7 An electronic device structure diagram is provided for the embodiment of the application. DETAILED DESCRIPTION

[0029] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0030] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0031] For the technical problems mentioned in the background art, the embodiments of the present application provide a resource scheduling method, which is specifically described with reference to Figure 1 Figure 1 A flowchart of a resource scheduling method provided by the embodiments of the present application is shown.

[0032] Before introducing the steps of the method, the application scenarios to which the method is applicable are first described. The method can be applied in a server cluster, and each server cluster includes multiple servers (nodes). Each normally working node can carry the same PMEM device and carry the tasks to be executed corresponding to all big data components, such as spark tasks, Hbase tasks, and Redis tasks. Each node is allocated a task queue for executing tasks, and different task queues are pre-configured with different proportions of PMEM resources according to the types of big data components. Of course, when the PMEM device on any node fails, the resource scheduling system can also be informed, and the corresponding big data component is informed to distribute tasks to other nodes for execution.

[0033] Taking an example, assuming that the PMEM resource on each node is 100T, there are 10 nodes in total, and the total resource is 1000T. The types of big data components include Spark, Hbase, and Redis. The 100T resource on the same node is allocated to Spark by 40%, to Hbase by 30%, and to Redis by the remaining 30%. Therefore, the total PMEM resource obtained by Spark is 400T, and the total PMEM resource obtained by the other two big data components is 300T respectively.

[0034] ​The above is the allocation of total resources and PMEM device resources on each node, i.e. the amount of PMEM resources that can be occupied by various types of big data components after pre-configuration. However, this configuration is not fixed, but when there is free PMEM resource of a big data component at a certain moment or time period, and other big data components need more PMEM resources to perform tasks, the resource scheduling and allocation system can be flexibly scheduled and allocated to ensure that most or even all big data components have reasonable PMEM resources to perform tasks, and can also try to avoid waste of PMEM resources.

[0035] In the following, how to realize resource scheduling will be described in detail, please refer to Figure 1 The method is executed by the resource scheduling system, and the method steps specifically include:

[0036] Step 110, periodically acquiring the usage of the PMEM resource corresponding to each big data component in all big data components.

[0037] Step 120, after receiving the resource scheduling and allocation request sent by the first big data component, determining the allocable PMEM resource according to the usage of the PMEM resource corresponding to each big data component except the first big data component which has been acquired in the current period.

[0038] Specifically, the staff can configure a set of device information collection scripts on each node in advance, and set a timing task. In this way, each node can periodically collect the usage of the PMEM resource corresponding to each big data component carried by itself, and write the collection result into the MYSQL table.

[0039] The resource scheduling system can periodically read the usage of the PMEM resource from the MYSQL through the pre-configured collection data reading interface, which can include the usage of the resource. Of course, it can also include the remaining PMEM resource amount of each big data component.

[0040] In an optional example, the period is 30 seconds. Considering that the execution time of each task is usually about half an hour, so 30 seconds is enough to monitor the PMEM resource usage of different big data components.

[0041] If the usage includes the remaining PMEM resource amount of each big data component, then the resource scheduling system can directly determine the allocable PMEM resource according to the remaining PMEM resource amount of each big data component.

[0042] If the usage includes the total capacity of PMEM of each big data component and the resource usage, the resource scheduling system can also determine the allocable PMEM resource of each big data component according to the total capacity and the resource usage.

[0043] Or, if the usage only includes the resource usage of each big data component, the resource scheduling system can be pre-configured with the total capacity of resource of each big data component, or the total number of PMEM devices, the capacity of each PMEM device, and the configuration ratio of each big data component. The allocable PMEM resource of each big data component can also be calculated.

[0044] No matter in which way, the allocable PMEM resource can be determined, thus, the specific way is not limited here.

[0045] After receiving the resource scheduling allocation request sent by the first big data component, the allocable PMEM resource can be determined according to any of the above ways.

[0046] Step 130, the allocable PMEM resource is allocated to the task queue corresponding to the first big data component to execute the task to be executed.

[0047] The first big data component is any one of all big data components.

[0048] Specifically, after obtaining the allocable PMEM resource, the allocable PMEM resource is immediately allocated to the task queue corresponding to the first big data component to execute the corresponding task.

[0049] The resource scheduling method provided by the embodiment of the application periodically acquires the usage of the PMEM resource corresponding to each big data component in all big data components. After receiving the resource scheduling allocation request sent by the first big data component, the allocable PMEM resource is determined according to the usage of the PMEM resource corresponding to each big data component except the first big data component acquired in the current period. Then, the allocable PMEM resource is allocated to the task queue corresponding to the first big data component to execute the task to be executed. In this way, it can be ensured that each big data component has reasonable PMEM resource to support the execution of the task when executing the task. At the same time, when the PMEM resource of some big data component is insufficient, the pre-configured PMEM resource of other big data component can also be flexibly scheduled by the resource scheduling system for the big data component with insufficient resource to use to complete the execution of the task. In this way, it can be ensured that the insufficient PMEM resource of the big data component does not occur, and the waste of PMEM resource can also be avoided, and the reasonable allocation of all PMEM resources is realized.

[0050] Optionally, on the basis of the above embodiment, considering that the allocable PMEM resource is the resource allocated from the task queue corresponding to other big data components, after the allocable PMEM resource is allocated to the task queue corresponding to the first big data component to execute the to-be-executed task, the method further includes the following method steps, which are specifically described with reference to Figure 2 .

[0051] Step 210, when it is determined that the task queue corresponding to the first big data component has executed the to-be-executed task, the allocable PMEM resource is recovered.

[0052] Step 220, the allocable PMEM resource is allocated to the task queue corresponding to the big data component to which the allocable PMEM resource originally belongs.

[0053] The determination of whether the task queue corresponding to the first big data component has executed the to-be-executed task can be obtained in the following ways:

[0054] Firstly, when the task queue corresponding to the first big data component has executed the to-be-executed task, the response information that the task has been executed is automatically reported.

[0055] Secondly, the use of the PMEM resource corresponding to each big data component is periodically fed back to determine whether the task queue corresponding to the first big data component has completed the task (the PMEM resource is automatically released after the task queue corresponding to the big data component completes the task).

[0056] No matter which way is used to determine that the task queue corresponding to the first big data component has completed the task, the PMEM resource previously allocated (lent) to the first big data component will be recovered, and then the resource is allocated (returned) to the task queue corresponding to the big data component to which the resource originally belongs.

[0057] In a specific example, for example, the real-time requirement of the Spark task is high, and the task duration is not long, the task queue allocated for the Spark task is a high-priority task queue, and the corresponding PMEM resource is 40%-50% of the entire PMEM device. The real-time requirement of the HBase query is relatively high, and the HBase query is assigned to a normal-priority queue, and 30%-40% of the entire PMEM device is allocated to the HBase query; the Redis is an in-memory database, and the demand for memory and PMEM resource is continuous, and then the Redis can be allocated to a low-priority queue, and 20%-30% of the entire PMEM device is allocated to the Redis.

[0058] When the HBase finds that the allocated resource is insufficient when performing a task, a resource scheduling allocation request can be sent to the resource scheduling system. After receiving the resource scheduling allocation request, the resource scheduling system determines whether there is allocatable PMEM resource according to the PMEM resource usage of the big data components other than the HBase obtained in the current period.

[0059] When it is determined that there is allocatable PMEM resource, for example, both Spark and Redis have sufficient margins, considering that the Spark task has a higher real-time requirement and the task duration is not long, the PMEM resource occupied by the Spark task is the highest proportion, therefore, the resource usage in the task queue can be first checked, if there is a margin, the PMEM resource of the task queue can be directly called to the HBase task queue.

[0060] Of course, if the PMEM resource corresponding to the Spark component is insufficient to support the PMEM resource required by the HBase component to be allocated, the PMEM resources of multiple components can be superimposed to support the task performed by the HBase component.

[0061] Optionally, the resource scheduling allocation request includes the PMEM resource to be allocated, therefore, whether the allocatable PMEM resource is sufficient can be determined according to the PMEM resource to be allocated.

[0062] Optionally, on the basis of any one of the above embodiments, the method can further include the following cases.

[0063] When it is determined that there is no allocatable PMEM resource according to the PMEM resource usage of each big data component other than the first big data component obtained in the current period, the method can further include the following steps, which are specifically described with reference to the method embodiment shown in Figure 3 .

[0064] Step 310, suspending allocation of PMEM resource to the task queue corresponding to the first big data component.

[0065] Step 320, after obtaining the PMEM resource usage of each big data component in all big data components in the next period, determining the allocatable PMEM resource.

[0066] Step 330, allocating the allocatable PMEM resource to the task queue corresponding to the first big data component.

[0067] Specifically, as Figure 1In the embodiment shown, each cycle is, for example, 30 seconds, so there is no great impact on the execution of the task. Therefore, the PMEM resource can be temporarily allocated to the task queue corresponding to the first big data component, and then, after obtaining the use of the PMEM resource corresponding to each of the big data components in the next cycle, the allocable PMEM resource is determined. The specific acquisition condition is as described above, and will not be described in detail here.

[0068] The allocable PMEM resource is allocated to the task queue corresponding to the first big data component. The execution condition is still as described above, and will not be described in detail here.

[0069] Of course, it is also possible that there is still no allocable resource in the next cycle, so a cycle threshold can be set, that is, as long as it is ensured that the allocable PMEM resource can be obtained within the preset period, the allocable PMEM resource is allocated to the task queue corresponding to the first big data component.

[0070] In another optional embodiment, considering that even after a preset number of cycles, there is still no allocable PMEM resource to schedule to the first big data component, in this case, the method can further include the following implementation, including:

[0071] The second control instruction is used to instruct the first big data component to execute the task to be executed in the task queue in the actual memory corresponding to the first big data component.

[0072] That is, when there is still no allocable PMEM resource within the preset number of cycles, it means that it is currently a peak period of processing tasks, and there is no spare resource that can be called by the first big data component, so the first big data component needs to execute the task to be executed in the task queue in the actual memory corresponding to the first big data component.

[0073] Optionally, in addition to the case where the allocable PMEM resource is sufficient to support the task corresponding to the first big data component and the case where there is no allocable PMEM resource, it is also possible to determine that there is an allocable PMEM resource, but the resource cannot support the task to be executed by the first big data component. In this case, the method can further include the following method steps in addition to the method steps of the above-mentioned embodiments, see Figure 4 As shown, including:

[0074] Step 410, comparing the size of the allocable PMEM resource and the PMEM resource to be allocated.

[0075] Among them, the PMEM resource to be allocated is carried in the resource scheduling request.

[0076] Step 420, when it is determined that the allocable PMEM resource is less than the PMEM resource to be allocated, determining a resource quantity difference between the allocable PMEM resource and the PMEM resource to be allocated.

[0077] Step 430, issuing a first control instruction to the first big data component.

[0078] The first control instruction is used to instruct the first big data component to execute the to-be-executed task corresponding to the resource quantity difference in the actual memory corresponding to the first big data component.

[0079] That is, the allocable resource is allocated to the first big data component, so that the first big data component executes the corresponding task according to the allocable resource, and the remaining task needs to be executed in the actual memory corresponding to the first big data component.

[0080] Specifically, the task to be executed here is actually to store data into the actual memory or into the PMEM, so as to read the stored data for subsequent execution of other operations.

[0081] Further optionally, on the basis of any of the above embodiments, there can be a special case in which, for the allocation of resource scheduling, the following operation can also be performed, which will be described in detail with reference to Figure 5 The figure shows another resource scheduling allocation method flowchart. Figure 5 The figure shows another resource scheduling allocation method flowchart.

[0082] Specifically, on the basis of the above embodiment, before determining the allocable PMEM resource according to the use of the PMEM resource corresponding to each big data component other than the first big data component obtained in the current period, the method further comprises:

[0083] Step 510, determining the type of the first big data component.

[0084] Step 520, when it is determined that the type of the first big data component is a first preset type, no longer determining the allocable PMEM resource for the first big data component; and issuing a third control instruction to the first big data component.

[0085] The third control instruction is used to instruct the first big data component to execute the to-be-executed task in the task queue in the actual memory corresponding to the first big data component. The first preset type here is a big data component whose proportion of occupying the overall resource of the PMEM device is higher than a preset threshold.

[0086] In a specific example, for example, as introduced above, the Spark task has high real-time requirement, and the task duration is not long, and the PMEM resource occupied by the task has already been the highest proportion of the entire PMEM device. Therefore, when the resource required by the Spark task is insufficient, the resource scheduling system will no longer schedule the PMEM resource of other big data components for the Spark task to execute the corresponding task. Instead, a third control instruction is directly issued to instruct the first big data component to execute the task to be executed in the task queue in the actual memory corresponding to the first big data component.

[0087] Further optionally, on the basis of any of the above embodiments, there can be a special case in which, for the allocation of the resource scheduling, the following operation can be performed:

[0088] Before determining the allocatable PMEM resource according to the use of the PMEM resource corresponding to each big data component other than the first big data component obtained in the current period, the method further comprises:

[0089] Determine the type of each big data component, so that when the type of the second big data component is determined to be a second preset type, the allocatable PMEM resource of the second big data component is no longer counted. The second preset type, for example, is a big data component whose proportion of occupying the overall resource of the PMEM device is lower than a second preset threshold, and the task occupying time is higher than a preset time threshold.

[0090] In a specific example, for example, considering that Redis is used as a memory database, the demand for memory and PMEM resource is continuous, so as far as possible, the PMEM resource of Redis is not scheduled to be allocated to other big data components to execute tasks. Therefore, when determining the allocatable PMEM resource according to the use of the PMEM resource corresponding to each big data component other than the first big data component obtained in the current period, this type of big data component is excluded.

[0091] The above is the method for resource scheduling provided by the present application. The following describes other embodiments of the resource scheduling provided by the present application. See the following for details.

[0092] Figure 6 A resource scheduling device structure schematic diagram provided by an embodiment of the present application, the device comprises: an acquisition module 601, a processing module 602, and an allocation module 603.

[0093] The acquisition module 601 is configured to periodically acquire the use of the persistent memory PMEM resource corresponding to each big data component in all big data components, and receive the resource scheduling allocation request sent by each big data component;

[0094] The processing module 602 is configured to determine the allocable PMEM resource according to the usage of the PMEM resource corresponding to each of the big data components except the first big data component, which is acquired in the current period, after receiving the resource scheduling allocation request sent by the first big data component.

[0095] The allocation module 603 is configured to allocate the allocable PMEM resource to the task queue corresponding to the first big data component, so as to execute the to-be-executed task, wherein the first big data component is any one of the big data components.

[0096] Optionally, the apparatus further comprises a recycling module 604.

[0097] The processing module 602 is further configured to determine whether the task queue corresponding to the first big data component has executed the to-be-executed task.

[0098] The recycling module 604 is configured to recycle the allocable PMEM resource when it is determined that the task queue corresponding to the first big data component has executed the to-be-executed task.

[0099] The allocation module 603 is further configured to allocate the allocable PMEM resource to the task queue corresponding to the big data component to which the allocable PMEM resource originally belongs.

[0100] Optionally, the allocation module 603 is further configured to suspend the allocation of the PMEM resource to the task queue corresponding to the first big data component when it is determined that there is no allocable PMEM resource according to the usage of the PMEM resource corresponding to each of the big data components except the first big data component, which is acquired in the current period.

[0101] The processing module 602 is further configured to determine the allocable PMEM resource after the acquisition module 601 acquires the usage of the PMEM resource corresponding to each of the big data components in the next period.

[0102] The allocation module 603 is further configured to allocate the allocable PMEM resource to the task queue corresponding to the first big data component.

[0103] Optionally, the resource scheduling allocation request comprises the to-be-allocated PMEM resource.

[0104] The apparatus further comprises a sending module 605.

[0105] The processing module 602 is further configured to determine the resource quantity difference between the allocable PMEM resource and the to-be-allocated PMEM resource when it is determined that the allocable PMEM resource is less than the to-be-allocated PMEM resource.

[0106] The sending module 605 is further configured to send a first control instruction to the first big data component, where the first control instruction is used to instruct the first big data component to execute the to-be-executed task corresponding to the resource amount difference in the actual memory corresponding to the first big data component.

[0107] Optionally, the sending module 605 is further configured to send a second control instruction to the first big data component when there is still no allocable PMEM resource after the preset period is exceeded, where the second control instruction is used to instruct the first big data component to execute the to-be-executed task in the task queue in the actual memory corresponding to the first big data component.

[0108] Optionally, the processing module 602 is further configured to determine the type of the first big data component, and when it is determined that the type of the first big data component is a first preset type, no allocable PMEM resource is determined for the first big data component.

[0109] The sending module is further configured to send a third control instruction to the first big data component, where the third control instruction is used to instruct the first big data component to execute the to-be-executed task in the task queue in the actual memory corresponding to the first big data component.

[0110] Optionally, the processing module 602 is further configured to determine the type of each big data component, so as to when it is determined that the type of the second big data component is a second preset type, no allocable PMEM resource is counted for the second big data component.

[0111] The functions performed by the components in the resource scheduling apparatus provided by the embodiments of the present application have been described in detail in any of the above method embodiments, and thus will not be described here again.

[0112] The resource scheduling device provided by the embodiment of the present application periodically acquires the usage of the PMEM resource corresponding to each big data component in all big data components. After receiving the resource scheduling allocation request sent by the first big data component, the usage of the PMEM resource corresponding to each big data component except the first big data component is acquired in the current period, and the allocable PMEM resource is determined. Then, the allocable PMEM resource is allocated to the task queue corresponding to the first big data component, so as to execute the task to be executed. In this way, it can be ensured that each big data component has reasonable PMEM resource to support the execution of the task when executing the task. At the same time, when the PMEM resource of some big data component is insufficient, the preconfigured PMEM resource of other big data components can be flexibly scheduled by the resource scheduling system, so as to be used by the big data component with insufficient resource to complete the execution of the task. In this way, it can be ensured that the insufficient PMEM resource of the big data component does not occur, and the waste of the PMEM resource is avoided, and the reasonable allocation of all PMEM resources is realized.

[0113] As shown in Figure 7 The embodiment of the present application provides an electronic device, the electronic device carries the resource scheduling system mentioned in any one of the above embodiments, and includes a processor 111, a communication interface 112, a memory 113 and a communication bus 114. The processor 111, the communication interface 112 and the memory 113 complete mutual communication through the communication bus 114.

[0114] The memory 113 is used to store a computer program.

[0115] In an embodiment of the present application, the processor 111 is used to execute the program stored in the memory 113, and realizes the resource scheduling method provided by any one of the preceding method embodiments, including:

[0116] Periodically acquiring the usage of the persistent memory PMEM resource corresponding to each big data component in all big data components;

[0117] After receiving the resource scheduling allocation request sent by the first big data component, the usage of the PMEM resource corresponding to each big data component except the first big data component is acquired in the current period, and the allocable PMEM resource is determined.

[0118] The allocable PMEM resource is allocated to the task queue corresponding to the first big data component, so as to execute the task to be executed. In this way, it can be ensured that each big data component has reasonable PMEM resource to support the execution of the task when executing the task. At the same time, when the PMEM resource of some big data component is insufficient, the preconfigured PMEM resource of other big data components can be flexibly scheduled by the resource scheduling system, so as to be used by the big data component with insufficient resource to complete the execution of the task. In this way, it can be ensured that the insufficient PMEM resource of the big data component does not occur, and the waste of the PMEM resource is avoided, and the reasonable allocation of all PMEM resources is realized.

[0119] Optionally, the allocable PMEM resource is allocated to the task queue corresponding to the first big data component to execute the to-be-executed task, and further comprising: when it is determined that the task queue corresponding to the first big data component has executed the to-be-executed task, the allocable PMEM resource is recovered and allocated to the task queue corresponding to the big data component to which the allocable PMEM resource originally belongs.

[0120] Optionally, when it is determined that there is no allocable PMEM resource according to the usage of the PMEM resource corresponding to each big data component except the first big data component acquired in the current period, further comprising:

[0121] suspending the allocation of the PMEM resource to the task queue corresponding to the first big data component;

[0122] determining the allocable PMEM resource after acquiring the usage of the PMEM resource corresponding to each big data component in all big data components in the next period;

[0123] allocating the allocable PMEM resource to the task queue corresponding to the first big data component.

[0124] Optionally, the resource scheduling allocation request comprises the to-be-allocated PMEM resource;

[0125] when it is determined that the allocable PMEM resource is less than the to-be-allocated PMEM resource, after the allocable PMEM resource is allocated to the task queue corresponding to the first big data component, further comprising:

[0126] determining the resource quantity difference between the allocable PMEM resource and the to-be-allocated PMEM resource;

[0127] issuing a first control instruction to the first big data component, the first control instruction being used to instruct the first big data component to execute the to-be-executed task corresponding to the resource quantity difference in the actual memory corresponding to the first big data component.

[0128] Optionally, when there is still no allocable PMEM resource after exceeding the preset period, further comprising:

[0129] issuing a second control instruction to the first big data component, the second control instruction being used to instruct the first big data component to execute the to-be-executed task in the task queue in the actual memory corresponding to the first big data component.

[0130] Optionally, before determining the allocable PMEM resource according to the usage of the PMEM resource corresponding to each big data component except the first big data component acquired in the current period, further comprising:

[0131] determining the type of the first big data component;

[0132] When it is determined that the type of the first big data component is the first preset type, the allocable PMEM resource is no longer determined for the first big data component; and a third control instruction is issued to the first big data component, wherein the third control instruction is used to instruct the first big data component to execute the task to be executed in the task queue in the actual memory corresponding to the first big data component.

[0133] Optionally, before the allocable PMEM resource is determined, the method further includes:

[0134] The type of each big data component is determined, so that when it is determined that the type of the second big data component is the second preset type, the allocable PMEM resource of the second big data component is no longer counted.

[0135] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the resource scheduling method provided by any one of the preceding method embodiments.

[0136] It should be noted that, in this document, relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0137] The above merely describes specific embodiments of the present application, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features presented herein.

Claims

1. A resource scheduling method, characterized in that, The method is executed by a resource scheduling system, and the method comprises: Periodically acquiring usage of persistent memory (PMEM) resources corresponding to each of all big data components respectively; After receiving a resource scheduling allocation request sent by a first big data component, determining allocable PMEM resources according to the usage of the PMEM resources corresponding to each of all big data components except the first big data component acquired in a current period; Allocating the allocable PMEM resources to a task queue corresponding to the first big data component for executing a task to be executed, wherein the first big data component is any one of all big data components; The resource scheduling allocation request comprises PMEM resources to be allocated; When it is determined that the allocable PMEM resources are less than the PMEM resources to be allocated, after the allocable PMEM resources are allocated to the task queue corresponding to the first big data component, the method further comprises: Determining a resource quantity difference between the allocable PMEM resources and the PMEM resources to be allocated; Issuing a first control instruction to the first big data component, wherein the first control instruction is used to instruct the first big data component to execute a task to be executed corresponding to the resource quantity difference in actual memory corresponding to the first big data component.

2. The method of claim 1, wherein, After the allocable PMEM resources are allocated to the task queue corresponding to the first big data component for executing the task to be executed, the method further comprises: When it is determined that the task queue corresponding to the first big data component has executed the task to be executed, the allocable PMEM resources are recovered and allocated to a task queue corresponding to a big data component to which the allocable PMEM resources originally belong.

3. The method of claim 1, wherein, When it is determined that there is no allocable PMEM resource according to the usage of the PMEM resources corresponding to each of all big data components except the first big data component acquired in the current period, the method further comprises: Suspending allocation of PMEM resources to the task queue corresponding to the first big data component; After acquiring the usage of the PMEM resources corresponding to each of all big data components in a next period, determining allocable PMEM resources; And allocating the allocable PMEM resources to the task queue corresponding to the first big data component.

4. The method of claim 3, wherein, When there is still no allocable PMEM resource after a preset period is exceeded, the method further comprises: Issuing a second control instruction to the first big data component, wherein the second control instruction is used to instruct the first big data component to execute a task to be executed in the task queue in actual memory corresponding to the first big data component.

5. The method according to any one of claims 1 to 3, characterized in that, Before the allocable PMEM resources are determined according to the usage of the PMEM resources corresponding to each of all big data components except the first big data component acquired in the current period, the method further comprises: Determining a type of the first big data component; When it is determined that the type of the first big data component is a first preset type, no PMEM resource that can be allocated is determined for the first big data component, wherein the first preset type is a big data component whose proportion of occupying the overall PMEM resource is higher than a preset threshold value; and a third control instruction is issued to the first big data component, wherein the third control instruction is used to instruct the first big data component to execute the task to be executed in the actual memory corresponding to the first big data component.

6. The method according to any one of claims 1 to 3, characterized in that, Before the step of determining the PMEM resource that can be allocated according to the use of the PMEM resource corresponding to each big data component except the first big data component in the current period, the method further comprises: determining the type of each big data component, so that when it is determined that the type of a second big data component is a second preset type, no PMEM resource that can be allocated is counted for the second big data component, wherein the second preset type is a big data component whose proportion of occupying the overall PMEM resource is lower than a second preset threshold value and whose task occupying time is higher than a preset time threshold value.

7. A resource scheduling apparatus, characterized by comprising: The device comprises: an acquisition module, configured to periodically acquire the use of the persistent memory (PMEM) resource corresponding to each big data component in all big data components, and receive a resource scheduling and allocation request sent by each big data component; a processing module, configured to, after receiving the resource scheduling and allocation request sent by the first big data component, determine the PMEM resource that can be allocated according to the use of the PMEM resource corresponding to each big data component except the first big data component in the current period; an allocation module, configured to allocate the PMEM resource that can be allocated to the task queue corresponding to the first big data component, so as to execute the task to be executed, wherein the first big data component is any big data component in all big data components; the resource scheduling and allocation request comprises the PMEM resource to be allocated; the processing module is further configured to determine the resource quantity difference between the PMEM resource that can be allocated and the PMEM resource to be allocated; a sending module, configured to issue a first control instruction to the first big data component, wherein the first control instruction is used to instruct the first big data component to execute the task to be executed corresponding to the resource quantity difference in the actual memory corresponding to the first big data component.

8. An electronic device, comprising: The electronic device carries the resource scheduling system, and comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; the memory is used to store a computer program; the processor is used to execute the program stored on the memory, so as to realize the steps of the resource scheduling method in any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the resource scheduling method in any one of claims 1 to 6. The computer program is executed by the processor to realize the steps of the resource scheduling method in any one of claims 1 to 6.

Images