Resource allocation method, resource allocation device, electronic device and storage medium

Abstract

The present application provides a resource allocation method, a resource allocation device, an electronic device and a storage medium, which belongs to the field of artificial intelligence technology. The method includes: obtaining the execution status and original queue of the target task; obtaining the task demand resources and idle resource data of the task whose execution status is blocked; wherein the task demand resources are the resources required to execute the target task; calculating the demand ratio of the target task according to the task demand resources and idle resource data, and obtaining the resource occupancy data; screening and processing the idle queue according to the resource occupancy data, obtaining the target queue, and assigning the target task to the target queue; obtaining the current running status of the target task and the current resource data of the original queue; according to the current running status, the task demand resources and the current resource data, assigning the target task to the original queue or keeping the target queue in the target queue. The present application can improve the balance of resource utilization and the efficiency of task processing.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a resource allocation method, a resource allocation device, an electronic device and a storage medium. Background Art

[0002] At present, when cluster resources are allocated to multiple users, each user often occupies a queue resource. This method often makes it easy for cluster resources to be used unevenly, affecting the efficiency of task processing. Therefore, how to improve the balance of resource utilization and speed up task processing has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a resource allocation method, a resource allocation device, an electronic device and a storage medium, aiming to improve the balance of resource utilization and the processing efficiency of tasks.

[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes a resource allocation method, the method comprising:

[0005] Obtaining the execution status and original queue of the target task; wherein the original queue is the queue where the target task is located;

[0006] Acquire the task demand resources and idle resource data of the task whose execution state is in a blocked state; wherein the task demand resources are the resources required to execute the target task, and the idle resource data are the resource data of the idle queue, and the idle queue is an idle queue;

[0007] Calculate the demand ratio of the target task according to the task demand resources and the idle resource data to obtain resource occupancy data;

[0008] Screening the idle queue according to the resource occupancy data to obtain a target queue;

[0009] Allocating the target task to the target queue according to a preset allocation rule;

[0010] Obtaining the current running status of the target task and the current resource data of the original queue;

[0011] According to the current running state, the task required resources and the current resource data, the target task is allocated to the original queue or the target queue is kept in the target queue.

[0012] In some embodiments, the resource occupancy data includes a core occupancy ratio and a resource occupancy ratio, and the step of calculating the demand ratio of the target task according to the task demand resources and the idle resource data to obtain the resource occupancy data includes:

[0013] Extracting features of the task required resources to obtain core number parameters and memory parameters of the target task;

[0014] Extracting features from the idle resource data to obtain available resource parameters and available core number parameters for each of the idle queues;

[0015] Perform core occupancy calculation according to the core number parameter and the available core number parameter to obtain the core number occupancy ratio;

[0016] Memory occupancy calculation is performed according to the memory parameter and the available resource parameter to obtain the resource occupancy ratio.

[0017] In some embodiments, the step of screening the idle queue according to the resource occupancy data to obtain the target queue includes:

[0018] Comparing the core occupancy ratio with a preset first threshold, and comparing the resource occupancy ratio with a preset second threshold, to obtain a comparison result;

[0019] The idle queue is screened according to the comparison result to obtain the target queue.

[0020] In some embodiments, the step of allocating the target task to the target queue according to a preset allocation rule includes:

[0021] Get the current number of tasks in the target queue;

[0022] The target task is allocated to the target queue with the least number of current tasks.

[0023] In some embodiments, the step of allocating the target task to the target queue according to a preset allocation rule includes:

[0024] Obtaining the resource pool type of the target queue and the task type of the target task;

[0025] Calculating the matching degree between the resource pool type and the task type to obtain a matching value, wherein the matching value is used to indicate the degree of suitability between the queue resources of the target queue and the running resources required by the target task;

[0026] A final queue is screened out from the target queue according to the matching value, and the target task is assigned to the final queue, wherein the final queue is one of the target queues.

[0027] In some embodiments, the step of allocating the target task to the original queue or keeping the target queue in the target queue according to the current running state, the task required resources and the current resource data includes:

[0028] If the current running state is a waiting state, performing resource occupancy calculation according to the current resource data and the task requirement data to obtain a current resource occupancy rate;

[0029] According to the current resource occupancy rate, the target task is allocated to the original queue or the target queue is kept in the target queue.

[0030] In some embodiments, the step of obtaining the required resources and idle resource data of the task whose execution state is in a blocked state includes:

[0031] According to the waiting time of the target task and a preset time threshold, determining the target task whose execution state is in a blocked state, and obtaining the blocked task;

[0032] The idle resource data and the task required resources of the blocked task are obtained by means of a web crawler.

[0033] To achieve the above object, a second aspect of an embodiment of the present application provides a resource allocation device, the device comprising:

[0034] A first acquisition module, used to acquire the execution status and original queue of the target task; wherein the original queue is the queue where the target task is located;

[0035] A data acquisition module, used to acquire the task demand resources and idle resource data of the task whose execution state is in a blocked state; wherein the task demand resources are the resources required to execute the target task, and the idle resource data are the resource data of the idle queue, and the idle queue is an idle queue;

[0036] A calculation module, used to calculate the demand ratio of the target task according to the task demand resources and the idle resource data, and obtain resource occupancy data;

[0037] A screening module, used for screening the idle queues according to the resource occupancy data to obtain a target queue;

[0038] A scheduling module, used to allocate the target task to the target queue according to a preset allocation rule;

[0039] A second acquisition module is used to acquire the current running state of the target task and the current resource data of the original queue;

[0040] A processing module is used to allocate the target task to the original queue or keep the target queue in the target queue according to the current running state, the task required resources and the current resource data.

[0041] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program, when executed by the processor, implements the method described in the first aspect above.

[0042] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect above.

[0043] The resource allocation method, resource allocation device, electronic device and storage medium proposed in the present application can timely identify the queue with blocking and the target task in blocking state by acquiring the execution state and original queue of the target task, and acquiring the task demand resources and idle resource data of the task in blocking state, so as to adjust the queue where the target task is located, shorten the waiting time of the target task as much as possible, and improve the task processing efficiency. Further, the demand proportion of the target task is calculated according to the task demand resources and idle resource data to obtain the resource occupation data; the idle queue is screened and processed according to the resource occupation data to obtain the target queue, and then the target task is allocated to the target queue according to the preset allocation rule, and the position of the target task can be adjusted in time according to the resource conditions of different idle queues, and the target task in blocking state is allocated to the appropriate target queue, and the target task is processed by using the resources of the target queue, so that the resources of each queue can be used more evenly, and the situation of resource idleness and task blocking can be effectively avoided, and the task processing efficiency can be improved. Furthermore, the present application will also obtain the current running status of the target task and the current resource data of the original queue, and assign the target task to the original queue or keep the target queue in the target queue according to the current running status, task required resources and current resource data. It can flexibly adjust the position of the target task according to the current running status of the target task and the current resource data of the original queue, reasonably utilize the resources of the original queue, and further improve the processing efficiency of the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a resource allocation method provided in an embodiment of the present application;

[0045] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0046] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.

[0047] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0048] Figure 5 yes Figure 1 Flow chart of step S105 in FIG.

[0049] Figure 6 yes Figure 1 Another flow chart of step S105 in;

[0050] Figure 7 yes Figure 1 Flow chart of step S107 in FIG.

[0051] Figure 8 is a schematic diagram of the structure of a resource allocation device provided in an embodiment of the present application;

[0052] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0055] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this application are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] First, some nouns involved in this application are analyzed:

[0057] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0058] Cluster: A cluster is a group of independent computers connected by a high-speed network. They form a group and are managed as a single system. When a client interacts with the cluster, the cluster acts like an independent server. Cluster configuration is used to improve availability and scalability.

[0059] Capacity Scheduler: It is a multi-user scheduler that divides resources into queues. Each queue can set a certain percentage of minimum resource guarantees and usage limits. At the same time, each user can also set a certain resource usage limit to prevent resource abuse. When a queue has surplus resources, the remaining resources can be temporarily shared with other queues. The container scheduler allows multiple organizations to share the entire cluster. Each organization is configured with a dedicated queue. Each queue is allocated a portion of the entire cluster resources, and a single queue is used within the queue. The container scheduler has the following main features:

[0060] (1) Capacity guarantee: Administrators can set minimum resource guarantees and resource usage limits for each queue, and all applications submitted to the queue share these resources;

[0061] (2) Flexibility: If there are surplus resources in a queue, they can be temporarily shared with queues that need resources. Once a new application is submitted to the queue, the resources released by other queues will be returned to the queue.

[0062] (3) Multi-tenancy supports multiple users sharing the cluster and multiple applications running simultaneously. To prevent a single application, user, or queue from monopolizing resources in the cluster, administrators can add multiple constraints (such as the number of tasks that a single application can run simultaneously).

[0063] (4) Interface security assurance: Each queue has a strict ACL list to define its access users. Each user can specify which users are allowed to view the running status of their application or control the application. In addition, the administrator can specify queue administrators and cluster system administrators.

[0064] (5) Dynamically update the configuration file. Administrators can dynamically modify various configuration parameters as needed to achieve online cluster management.

[0065] Number of cores: refers to the number of cores of the CPU, such as dual-core, quad-core, octa-core, etc. The higher the number of cores, the more tasks can be performed simultaneously and the higher the efficiency.

[0066] Memory: It is an important component of the computer, also known as internal memory and main memory. It is used to temporarily store the calculation data in the CPU and the data exchanged with external memory such as the hard disk.

[0067] At present, when cluster resources are allocated to multiple users, each user often occupies one queue resource. For example, in the same cluster, multiple queues are set up, and each user occupies one queue resource. When a user needs to run a large number of tasks within a certain period of time, it will cause a large number of queue resources to be occupied, and the waiting time of the task will also be relatively long. At this time, the queue resources of other queues are idle, which often easily leads to unbalanced cluster resource utilization and affects task processing efficiency. Therefore, how to improve the balance of resource utilization and speed up task processing has become a technical problem that needs to be solved urgently.

[0068] Based on this, the embodiments of the present application provide a resource allocation method, a resource allocation device, an electronic device and a storage medium, aiming to improve the balance of resource utilization and the efficiency of task processing.

[0069] The resource allocation method, resource allocation device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the resource allocation method in the embodiments of the present application is described.

[0070] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0071] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0072] The resource allocation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The resource allocation method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the resource allocation method, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] Figure 1 is an optional flowchart of the resource allocation method provided in an embodiment of the present application. Figure 1The method may include but is not limited to steps S101 to S107.

[0075] Step S101, obtaining the execution status and original queue of the target task; wherein the original queue is the queue where the target task is located;

[0076] Step S102, obtaining the task demand resources and idle resource data of the task whose execution state is in the blocked state; wherein the task demand resources are the resources required to execute the target task, and the idle resource data are the resource data of the idle queue, and the idle queue is an idle queue;

[0077] Step S103, calculating the demand ratio of the target task according to the task demand resources and idle resource data to obtain resource occupancy data;

[0078] Step S104, screening the idle queues according to the resource occupancy data to obtain the target queue;

[0079] Step S105, assigning the target task to the target queue according to the preset assignment rule;

[0080] Step S106, obtaining the current running status of the target task and the current resource data of the original queue;

[0081] Step S107 , according to the current running state, the task required resources and the current resource data, the target task is assigned to the original queue or the target queue is kept in the target queue.

[0082] Steps S101 to S107 shown in the embodiment of the present application, by obtaining the execution state and original queue of the target task, and obtaining the task demand resources and idle resource data of the execution state in a blocked state, can timely identify the blocked queue and the target task in a blocked state, so as to adjust the queue where the target task is located, shorten the waiting time of the target task as much as possible, and improve the task processing efficiency. Further, the demand ratio of the target task is calculated according to the task demand resources and idle resource data to obtain resource occupancy data; the idle queue is screened and processed according to the resource occupancy data to obtain the target queue, and then the target task is allocated to the target queue according to the preset allocation rules, and the position of the target task can be adjusted in time according to the resource conditions of different idle queues, and the target task in a blocked state is allocated to the appropriate target queue, and the resources of the target queue are used to process the target task, so that the resources of each queue can be used more evenly, and the situation of idle resources and task blocking can be effectively avoided, and the processing efficiency of the task can be improved. Furthermore, the present application will also obtain the current running status of the target task and the current resource data of the original queue, and assign the target task to the original queue or keep the target queue in the target queue according to the current running status, task required resources and current resource data. It can flexibly adjust the position of the target task according to the current running status of the target task and the current resource data of the original queue, reasonably utilize the resources of the original queue, and further improve the processing efficiency of the task.

[0083] In step S101 of some embodiments, a monitoring script may be set in the cluster, and the monitoring script may be used to monitor the resource status of each queue in the cluster and the execution status of the target task of each queue, wherein the execution status of the target task includes a waiting status, a running status, and a completion status. In order to conveniently assign the target task, it is also necessary to obtain the original queue through the monitoring script, and the original queue is the queue where the target task is located. In addition, when obtaining the execution status of the target task through the monitoring script, the task ID of the target task may also be obtained, and the task ID is a unique identifier of the target task, and the queue where the target task is located and the current running status of the target task may be determined according to the task ID, and the basic information of the target task may be extracted through the task ID, wherein the task ID is generally represented in the form of a string.

[0084] It should be noted that in the embodiment of the present application, the cluster is generally a yarn cluster, and the resource scheduler is generally configured as a Capacity Schedule scheduler (container scheduler). The container scheduler can enable multiple users to share a cluster and set up multiple queues. Each user occupies a queue to run his own tasks, thereby meeting the independence of task processing; at the same time, the container scheduler can also allocate certain elastic resources to each queue, thereby improving the utilization efficiency of cluster resources.

[0085] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S202:

[0086] Step S201, according to the waiting time of the target task and the preset time threshold, determine the target task whose execution state is in a blocked state, and obtain the blocked task;

[0087] Step S202, obtaining idle resource data and task required resources of the blocked task by means of a web crawler.

[0088] In step S201 of some embodiments, the preset time threshold can be set according to actual business needs, and the waiting time is the length of time the target task is in the waiting state. For example, the preset time threshold is 5 minutes. The length of time the target task is in the waiting state is compared with the preset time threshold. If the length of time the target task is in the waiting state exceeds 5 minutes, it indicates that the waiting time of the target task is too long and the task running efficiency is low. Then, it is determined that the target task is in a blocked state in terms of execution status, and the target task is treated as a blocked task.

[0089] In step S202 of some embodiments, by presetting a data source and writing a web crawler to crawl data in a targeted manner on the preset data source, task requirement resources and idle resources are obtained, wherein the task requirement resources are the resources required to execute the target task, the idle resource data are the resource data of the idle queue, and the idle queue is an idle queue. The web crawler can be written in the form of a script or other forms, and the preset data source can be the data platform corresponding to the current cluster or the cluster control center, etc., without limitation.

[0090] It should be noted that the task requirement resources include the number of cores required to execute the target task (i.e., the core number parameter), memory parameters, and the task ID of the target task, etc. The resource data of the idle queue includes the queue information of the idle queue, available resource parameters, and available core number parameters, etc.

[0091] See also Figure 3In some embodiments, the resource occupancy data includes a core occupancy ratio and a resource occupancy ratio. Step S103 may include but is not limited to steps S301 to S304:

[0092] Step S301, extracting features of the task required resources to obtain core number parameters and memory parameters of the target task;

[0093] Step S302, extracting features from the idle resource data to obtain available resource parameters and available core number parameters for each idle queue;

[0094] Step S303, performing core occupancy calculation according to the core number parameter and the available core number parameter to obtain a core occupancy ratio;

[0095] Step S304, calculating memory occupancy according to the memory parameter and the available resource parameter to obtain the resource occupancy ratio.

[0096] In step S301 of some embodiments, according to the data type of the task requirement resource, the core number parameter and memory parameter of the target task are obtained from the task requirement resource. Specifically, the data type includes resource category, attribute category, etc., wherein the resource category mainly includes the memory size and core number required for the target task to run, etc., and the attribute category mainly includes the task ID of the target task, the task type, etc., and the task type includes data upload, file outbound, etc., but is not limited thereto. Through this step, the resource data belonging to the resource category in the task requirement resource can be used to obtain the core number parameter and memory parameter of the target task.

[0097] In step S302 of some embodiments, available resource parameters and available core number parameters of each idle queue are obtained from the idle resource data of the entire cluster through a monitoring script.

[0098] In step S303 of some embodiments, a division operation is performed on the core number parameter and the available core number parameter to calculate the core number occupancy of the target task and obtain the core number occupancy ratio, where core number occupancy ratio=core number parameter / available core number parameter.

[0099] In step S304 of some embodiments, a division operation is performed on the memory parameter and the available resource parameter to calculate the memory occupancy of the target task and obtain a resource occupancy ratio, wherein the resource occupancy ratio=memory parameter / available resource parameter.

[0100] Through the above steps S301 to S304, the resource occupancy of the target task in each idle queue can be conveniently estimated, so as to select a suitable queue from the idle queues to provide the target task with the resources required for running.

[0101] See also Figure 4In some embodiments, step S104 may include but is not limited to steps S401 to S402:

[0102] Step S401, comparing the core occupancy ratio with a preset first threshold, and comparing the resource occupancy ratio with a preset second threshold, to obtain a comparison result;

[0103] Step S402, screening the idle queues according to the comparison result to obtain the target queue.

[0104] In step S401 of some embodiments, the preset first threshold and second threshold can be set according to actual business needs without limitation, for example, the first threshold and the second threshold are both 70%. Then, the core occupancy ratio of the target task in each idle queue is compared to see whether it exceeds 70%, and the resource occupancy ratio exceeds 70%, thereby obtaining a comparison result.

[0105] In step S402 of some embodiments, based on the size relationship between the core occupancy ratio and the preset first threshold and the size relationship between the resource occupancy ratio and the preset second threshold, the comparison results of the embodiments of the present application include a first result, a second result, a third result and a fourth result.

[0106] Specifically, the first result is that the core occupancy ratio of the target queue in the idle queue does not exceed the first threshold, and the resource occupancy ratio does not exceed the second threshold, which indicates that the idle queue can accommodate the normal operation of the target task, and allocating the target task to the idle queue will not cause excessive resource occupancy and no blocking will occur, so the idle queue will be used as the target queue.

[0107] The second result is that the core occupancy ratio of the target queue in the idle queue exceeds the first threshold, but the resource occupancy ratio does not exceed the second threshold, which indicates that allocating the target task to the idle queue may cause excessive resource occupancy and may cause a blocking state. Therefore, it is not suitable to use the idle queue as the target queue.

[0108] The third result is that the core occupancy ratio of the target queue in the idle queue does not exceed the first threshold, but the resource occupancy ratio exceeds the second threshold, which indicates that allocating the target task to the idle queue may cause excessive resource occupancy and may cause a blocking state. Therefore, it is not suitable to use the idle queue as the target queue.

[0109] The fourth result is that the core occupancy ratio of the target queue in the idle queue exceeds the first threshold, and the resource occupancy ratio exceeds the second threshold, indicating that the target queue cannot accommodate the normal operation of the target task. Allocating the target task to the idle queue will cause excessive resource occupancy, and it is not suitable to use the idle queue as the target queue.

[0110] For example, the first threshold and the second threshold are both 70%. The core occupancy ratio of the target task in each idle queue is compared to see whether it exceeds 70%, and whether the resource occupancy ratio exceeds 70%. The idle queue whose core occupancy ratio does not exceed 70% and whose resource occupancy ratio does not exceed 70% is taken as the target queue, wherein the target queue can be used to provide the resources required for the normal operation of the target task.

[0111] See also Figure 5 In some embodiments, step S105 may include but is not limited to steps S501 to S502:

[0112] Step S501, obtaining the current number of tasks in the target queue;

[0113] Step S502: Allocate the target task to the target queue with the least number of current tasks.

[0114] In step S501 of some embodiments, since there may be one or more target queues, in order to better utilize cluster resources and improve task processing efficiency, it is necessary to obtain the current number of tasks in the target queue, where the current number of tasks is the number of tasks existing in the target queue.

[0115] In step S502 of some embodiments, if a target queue has a larger number of current tasks, the waiting time of the target task will be longer if the target task is assigned to the target queue, which affects the processing efficiency of the target task. Therefore, when there are multiple target queues to choose from, it is preferred to assign the target task to the target queue with the least current tasks, so that the target task can be executed faster, thereby improving the processing efficiency of the target task.

[0116] See also Figure 6 In some embodiments, step S105 includes but is not limited to steps S601 to S603:

[0117] Step S601, obtaining the resource pool type of the target queue and the task type of the target task;

[0118] Step S602, calculating the matching degree between the resource pool type and the task type to obtain a matching value, wherein the matching value is used to indicate the degree of suitability between the queue resources of the target queue and the running resources required by the target task;

[0119] Step S603 , a final queue is screened out from the target queue according to the matching value, and the target task is assigned to the final queue, wherein the final queue is one of the queues of the target queue.

[0120] In step S601 of some embodiments, since the target queue may be one or more, in order to better utilize cluster resources and improve task processing efficiency, it is necessary to obtain the resource pool type of the target queue and the task type of the target task, wherein the resource pool type of the target queue can be set according to the needs of different users. Therefore, there will be certain preference differences in the tasks occupied by queue resources of different users for execution. For example, the resource pool types include data transmission, data verification, etc., and the task type of the target task is determined according to the actual needs of the user, wherein the task types include file transfer, file verification, etc.

[0121] In step S602 of some embodiments, the matching degree of the resource pool type and the task type can be calculated according to a preset similarity algorithm (such as a cosine similarity algorithm, etc.) to obtain a matching value. The matching value can be used to characterize the degree of suitability between the queue resources of the target queue and the running resources required for the target task. The size of the matching value can conveniently reflect the degree of adaptability between the resource type that the target task can provide and the task type. The higher the matching value, the more suitable the resources of the target queue are for executing the target task.

[0122] In step S603 of some embodiments, a final queue is screened out from the target queue according to the sizes of the matching values ​​of different target queues, wherein the final queue is a queue in the target queue and the matching value of the final queue is the largest, that is, the final queue is the target queue with the highest degree of suitability between the queue resources and the running resources required by the target task, and the target task is assigned to the final queue, so that the target task can be executed more reasonably, thereby improving the processing effect of the target task.

[0123] Through the above steps S501 to S502, or steps S601 to S603, the position of the target task can be adjusted in time according to the resource conditions of different idle queues, the target task in a blocked state can be allocated to a suitable target queue, and the resources of the target queue can be used to process the target task, so that the resources of each queue can be used more evenly, and resource idleness and task blocking can be effectively avoided, thereby improving the processing efficiency of tasks.

[0124] In step S106 of some embodiments, in order to improve the rationality of resource utilization and task processing efficiency, the current running state of the target task and the current resource data of the original queue can be obtained at each preset time interval, wherein the preset time can be set according to actual business needs, for example, the preset time is 10 minutes. Specifically, the current running state of the target task is monitored by a preset monitoring script every 10 minutes, wherein the current running state of the target task includes waiting, running or completed. While monitoring the current running state of the target task, the preset data source is crawled by a web crawler to obtain the current resource data of the original queue, wherein the current resource data includes the currently available resource parameters. In this way, according to the current running state of the target task and the current resource data of the original queue, it can be considered whether to allocate the target task back to the original queue for operation to improve the rationality of resource occupancy.

[0125] See also Figure 7 In some embodiments, step S107 may also include but is not limited to steps S701 to S702:

[0126] Step S701, if the current running state is the waiting state, the resource occupancy is calculated according to the current resource data and the task requirement data to obtain the current resource occupancy rate;

[0127] Step S702: assign the target task to the original queue or keep the target queue in the target queue according to the current resource occupancy rate.

[0128] In step S701 of some embodiments, in order to improve the rationality of resource utilization and task processing efficiency, it is possible to consider whether to assign the target task back to the original queue for operation based on the current running state of the target task and the current resource data of the original queue. Specifically, if the current running state of the target task is a waiting state, a division operation is performed on the current available resource parameter in the current resource data and the memory parameter in the task requirement data to calculate the occupation of the current resources of the original queue by the target task, and obtain the current resource occupation rate, wherein the current resource occupation rate = memory parameter / current available resource parameter.

[0129] In step S702 of some embodiments, if the current running state of the target task is a waiting state and the current resource occupancy rate is greater than 1, it indicates that the current available resource parameters are less than the memory parameters, and the current resources of the original queue cannot meet the running requirements of the target task. Therefore, the target task is kept in the target queue unchanged and is executed by using the resources of the target queue.

[0130] If there is only one target task, when the current running state of the target task is waiting state and the current resource occupancy rate is less than or equal to 1 (that is, the current available resource parameter is greater than or equal to the memory parameter), it indicates that the current resources of the original queue meet the running requirements of the target task, then the target task is assigned back to the original queue, and the target task is executed through the queue resources of the original queue.

[0131] Furthermore, in order to reduce the workload of scheduling, when there are multiple target tasks, the target tasks whose current resource occupancy rate is less than or equal to 1 are screened out according to the current resource occupancy rate of each target task, and the target task with the largest current resource occupancy rate among the screened target tasks is assigned back to the original queue, and the target task is executed by the queue resources of the original queue, and the remaining target tasks remain unchanged in their respective target queues.

[0132] In addition, if the current running state of the target task is running or completed, it indicates that the target task has been executed in the target queue, and there is no need to consider reallocating the target task back to the original queue.

[0133] In some specific embodiments, in order to reduce the amount of calculation and improve processing efficiency, the memory parameters of the target task whose current running state is the waiting state can be obtained, and the current available resource parameters in the current resource data of the original queue can be obtained, and the size of the memory parameters and the current available resource parameters can be directly compared. Among the target tasks whose memory parameters are less than or equal to the current available resource parameters, the target task whose memory parameter has the smallest difference with the current available resource parameters is selected and reallocated back to the original queue, and the target task is executed by the queue resources of the original queue.

[0134] Through the above steps S701 to S702, the position of the target task can be adjusted more flexibly according to the current running state of the target task and the current resource data of the original queue, so that the resources of the original queue can be reasonably utilized and the task processing efficiency can be further improved.

[0135] The resource allocation method of the embodiment of the present application obtains the execution status and original queue of the target task, and obtains the task demand resources and idle resource data of the task in the blocked state, so that the blocked queue and the target task in the blocked state can be identified in time, so as to adjust the queue where the target task is located, shorten the waiting time of the target task as much as possible, and improve the task processing efficiency. Further, the demand ratio of the target task is calculated according to the task demand resources and idle resource data to obtain resource occupancy data; the idle queue is screened and processed according to the resource occupancy data to obtain the target queue, and then the target task is allocated to the target queue according to the preset allocation rules, and the position of the target task can be adjusted in time according to the resource conditions of different idle queues, and the target task in the blocked state is allocated to the appropriate target queue, and the resources of the target queue are used to process the target task, so that the resources of each queue can be used more evenly, and the situation of idle resources and task blocking can be effectively avoided, and the task processing efficiency can be improved. Further, the present application also obtains the current running state of the target task and the current resource data of the original queue, and according to the current running state, the task demand resources and the current resource data, the target task is allocated to the original queue or the target queue is kept in the target queue, and the position of the target task can be adjusted flexibly according to the current running state of the target task and the current resource data of the original queue, and the resources of the original queue can be reasonably utilized, and the processing efficiency of the task can be further improved. The resource allocation method of the embodiment of the present application can conveniently allocate resources to the blocked tasks by monitoring the usage of cluster resources and the blocking of tasks, improve the utilization efficiency of cluster resources, and also improve the operation efficiency of tasks and reduce the cost of manual intervention.

[0136] See also Figure 8 The embodiment of the present application further provides a resource allocation device, which can implement the above resource allocation method, and the device includes:

[0137] The first acquisition module 801 is used to acquire the execution status and original queue of the target task; wherein the original queue is the queue where the target task is located;

[0138] The data acquisition module 802 is used to acquire the task demand resources and idle resource data of the task whose execution state is in the blocked state; wherein the task demand resources are the resources required to execute the target task, and the idle resource data are the resource data of the idle queue, and the idle queue is an idle queue;

[0139] The calculation module 803 is used to calculate the demand ratio of the target task according to the task demand resources and idle resource data to obtain resource occupancy data;

[0140] A screening module 804 is used to screen the idle queues according to the resource occupancy data to obtain a target queue;

[0141] The scheduling module 805 is used to allocate the target tasks to the target queue according to the preset allocation rules;

[0142] The second acquisition module 806 is used to acquire the current running state of the target task and the current resource data of the original queue;

[0143] The processing module 807 is used to allocate the target task to the original queue or keep the target queue in the target queue according to the current running state, the task required resources and the current resource data.

[0144] In some embodiments, the data acquisition module 802 includes:

[0145] A determination unit, configured to determine a target task in a blocked execution state according to a waiting time of the target task and a preset time threshold, and obtain a blocked task;

[0146] The crawling unit is used to obtain idle resource data and task requirement resources of blocked tasks by means of a web crawler.

[0147] In some embodiments, the resource occupancy data includes a core occupancy ratio and a resource occupancy ratio, and the calculation module 803 includes:

[0148] The first extraction unit is used to extract features of the task required resources to obtain core number parameters and memory parameters of the target task;

[0149] The second extraction unit is used to extract features from the idle resource data to obtain available resource parameters and available core number parameters of each idle queue;

[0150] A first ratio calculation unit, configured to calculate the number of cores occupied according to the number of cores parameter and the number of available cores parameter to obtain the number of cores occupied ratio;

[0151] The second ratio calculation unit is used to perform memory occupancy calculation according to the memory parameter and the available resource parameter to obtain the resource occupancy ratio.

[0152] In some embodiments, the screening module 804 includes:

[0153] A comparison unit, used to compare the core occupancy ratio with a preset first threshold, and to compare the resource occupancy ratio with a preset second threshold, to obtain a comparison result;

[0154] The screening unit is used to screen the idle queues according to the comparison result to obtain the target queue.

[0155] In some embodiments, the scheduling module 805 includes:

[0156] The task quantity acquisition unit is used to obtain the current task quantity of the target queue;

[0157] The first allocation unit is used to allocate the target task to the target queue with the least number of current tasks.

[0158] In some other embodiments, the scheduling module 805 includes:

[0159] A type acquisition unit, used to acquire the resource pool type of the target queue and the task type of the target task;

[0160] A matching calculation unit, used to calculate the matching degree between the resource pool type and the task type to obtain a matching value, wherein the matching value is used to represent the degree of suitability between the queue resources of the target queue and the running resources required by the target task;

[0161] The second allocation unit is used to filter out a final queue from the target queue according to the matching value, and allocate the target task to the final queue, wherein the final queue is one of the target queues.

[0162] In some embodiments, the processing module 807 includes:

[0163] A resource calculation unit, for calculating resource occupancy based on current resource data and task requirement data to obtain current resource occupancy rate if the current running state is a waiting state;

[0164] The control unit is used to allocate the target task to the original queue or keep the target queue in the target queue according to the current resource occupancy rate.

[0165] The specific implementation of the resource allocation device is substantially the same as the specific implementation of the resource allocation method described above, and will not be described in detail herein.

[0166] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above resource allocation method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0167] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0168] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0169] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the resource allocation method of the embodiment of this application;

[0170] Input / output interface 903, used to implement information input and output;

[0171] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0172] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0173] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0174] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned resource allocation method.

[0175] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0176] The resource allocation method, resource allocation device, electronic device and storage medium provided by the embodiment of the present application obtain the execution state and original queue of the target task, and obtain the task demand resources and idle resource data of the execution state in a blocked state, so that the blocked queue and the target task in a blocked state can be identified in time, so as to adjust the queue where the target task is located, shorten the waiting time of the target task as much as possible, and improve the task processing efficiency. Further, the demand ratio of the target task is calculated according to the task demand resources and idle resource data to obtain resource occupancy data; the idle queue is screened and processed according to the resource occupancy data to obtain the target queue, and then the target task is allocated to the target queue according to the preset allocation rules, and the position of the target task can be adjusted in time according to the resource conditions of different idle queues, and the target task in a blocked state is allocated to the appropriate target queue, and the resources of the target queue are used to process the target task, so that the resources of each queue can be used more evenly, and the situation of idle resources and task blocking can be effectively avoided, and the processing efficiency of the task can be improved. Further, the present application also obtains the current running state of the target task and the current resource data of the original queue, and according to the current running state, the task demand resources and the current resource data, the target task is allocated to the original queue or the target queue is kept in the target queue, and the position of the target task can be adjusted flexibly according to the current running state of the target task and the current resource data of the original queue, and the resources of the original queue can be reasonably utilized, and the processing efficiency of the task can be further improved. The resource allocation method of the embodiment of the present application can conveniently allocate resources to the blocked tasks by monitoring the usage of cluster resources and the blocking of tasks, improve the utilization efficiency of cluster resources, and also improve the operation efficiency of tasks and reduce the cost of manual intervention.

[0177] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0178] It can be understood by those skilled in the art that Figure 1-7 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0179] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0181] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0182] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0183] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0184] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0186] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0187] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A resource allocation method, characterized in that: The method comprises: Obtaining the execution status and original queue of the target task; wherein the original queue is the queue where the target task is located; Acquire the task demand resources and idle resource data of the task whose execution state is in a blocked state; wherein the task demand resources are the resources required to execute the target task, and the idle resource data are the resource data of the idle queue, and the idle queue is an idle queue; Calculate the demand ratio of the target task according to the task demand resources and the idle resource data to obtain resource occupancy data; Screening the idle queue according to the resource occupancy data to obtain a target queue; Allocating the target task to the target queue according to a preset allocation rule; Obtaining the current running status of the target task and the current resource data of the original queue; Allocate the target task to the original queue or keep the target queue in the target queue according to the current running state, the task required resources and the current resource data; The resource occupancy data includes a core occupancy ratio and a resource occupancy ratio. The resource occupancy data is obtained by calculating the demand ratio of the target task according to the task demand resources and the idle resource data, including: Extracting features of the task required resources to obtain core number parameters and memory parameters of the target task; Extracting features from the idle resource data to obtain available resource parameters and available core number parameters for each of the idle queues; Perform core occupancy calculation according to the core number parameter and the available core number parameter to obtain the core number occupancy ratio; Memory occupancy calculation is performed according to the memory parameter and the available resource parameter to obtain the resource occupancy ratio.

2. The resource allocation method according to claim 1, characterized in that: The step of screening the idle queue according to the resource occupancy data to obtain the target queue includes: Comparing the core occupancy ratio with a preset first threshold, and comparing the resource occupancy ratio with a preset second threshold, to obtain a comparison result; The idle queue is screened according to the comparison result to obtain the target queue.

3. The resource allocation method according to claim 1, characterized in that: The step of allocating the target task to the target queue according to a preset allocation rule comprises: Get the current number of tasks in the target queue; The target task is allocated to the target queue with the least number of current tasks.

4. The resource allocation method according to claim 1, characterized in that: The step of allocating the target task to the target queue according to a preset allocation rule comprises: Obtaining the resource pool type of the target queue and the task type of the target task; Calculating the matching degree between the resource pool type and the task type to obtain a matching value, wherein the matching value is used to characterize the degree of suitability between the queue resources of the target queue and the running resources required by the target task; A final queue is screened out from the target queue according to the matching value, and the target task is assigned to the final queue, wherein the final queue is one of the target queues.

5. The resource allocation method according to claim 1, characterized in that: The step of allocating the target task to the original queue or keeping the target queue in the target queue according to the current running state, the task required resources and the current resource data comprises: If the current running state is a waiting state, then performing resource occupancy calculation according to the current resource data and the task required resources to obtain the current resource occupancy rate; According to the current resource occupancy rate, the target task is allocated to the original queue or the target queue is kept in the target queue.

6. The resource allocation method according to any one of claims 1 to 5, characterized in that: The step of obtaining the task required resources and idle resource data in the blocked execution state includes: According to the waiting time of the target task and a preset time threshold, determining the target task whose execution state is in a blocked state, and obtaining the blocked task; The idle resource data and the task required resources of the blocked task are obtained by means of a web crawler.

7. A resource allocation device, characterized in that: The device comprises: A first acquisition module, used to acquire the execution status and original queue of the target task; wherein the original queue is the queue where the target task is located; A data acquisition module, used to acquire the task demand resources and idle resource data of the task whose execution state is in a blocked state; wherein the task demand resources are the resources required to execute the target task, and the idle resource data are the resource data of the idle queue, and the idle queue is an idle queue; A calculation module, used to calculate the demand ratio of the target task according to the task demand resources and the idle resource data, and obtain resource occupancy data; A screening module, used for screening the idle queues according to the resource occupancy data to obtain a target queue; A scheduling module, used to allocate the target task to the target queue according to a preset allocation rule; A second acquisition module is used to acquire the current running state of the target task and the current resource data of the original queue; A processing module, configured to allocate the target task to the original queue or keep the target queue in the target queue according to the current running state, the task required resources and the current resource data; The resource occupancy data includes a core occupancy ratio and a resource occupancy ratio. The resource occupancy data is obtained by calculating the demand ratio of the target task according to the task demand resources and the idle resource data, including: Extracting features of the task required resources to obtain core number parameters and memory parameters of the target task; Extracting features from the idle resource data to obtain available resource parameters and available core number parameters for each of the idle queues; Perform core occupancy calculation according to the core number parameter and the available core number parameter to obtain the core number occupancy ratio; Memory occupancy calculation is performed according to the memory parameter and the available resource parameter to obtain the resource occupancy ratio.

8. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the resource allocation method as described in any one of claims 1 to 6 are realized.

9. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the resource allocation method according to any one of claims 1 to 6.

Images