Resource scheduling method and device

Abstract

The invention provides a resource scheduling method and device, and relates to the technical field of resource management. In the application, after the service execution request for the target service is received, a plurality of resource scheduling strategies can be set for the target service according to the first resource demand of the target service carried in the service execution request, so that fixed resources are not reserved for the target service any more, and the resource utilization rate of the target service is improved. Instead, a plurality of resource scheduling strategies are pertinently set according to the resource demand of the target service, so that the flexibility of resource scheduling is improved; moreover, the target resource scheduling strategy meeting the service execution duration requirement set for the target service can be screened out from the plurality of resource scheduling strategies, so that the overall utilization rate of the resources is relatively high. Therefore, by adopting the mode, the overall utilization rate of the resources can be ensured to be relatively high under the condition of flexibly scheduling the resources for the target service.

Description

Technical Field

[0001] The present application relates to the technical field of resource management, and in particular to a resource scheduling method and device. Background Art

[0002] With the rapid development of the Internet, the application of big data has become more and more extensive. Due to the limited nature of resources (such as network resources), it is necessary to reasonably allocate or schedule the limited number of resources to ensure that the business (such as a job) can proceed smoothly.

[0003] At present, resource scheduling strategies that reserve fixed resources for different services or resource-aware load balancing strategies are usually adopted to achieve resource scheduling for services. Among them, the aforementioned resource-aware load balancing strategy is to schedule resources that meet business needs based on resource occupancy.

[0004] However, using the above-mentioned resource scheduling method, since the resources reserved for the business are fixed, it will be impossible to flexibly adjust the resource scheduling according to the operation of the business; and simply scheduling resources to meet business needs may result in low overall resource utilization, and thus cannot guarantee a high overall resource utilization. Summary of the invention

[0005] The embodiments of the present application provide a resource scheduling method and device, which can ensure a high overall resource utilization rate while flexibly scheduling resources for services.

[0006] In a first aspect, an embodiment of the present application provides a resource scheduling method, the method comprising:

[0007] Receive a service execution request initiated by a target object for a target service, and obtain a first resource requirement of the target service from the service execution request; the first resource requirement includes: at least one resource required to complete the target service, and a resource requirement amount corresponding to the at least one resource;

[0008] A plurality of resource scheduling strategies are set for the first resource requirement based on the schedulable resource set; wherein each resource scheduling strategy is used to indicate at least one resource allocated to the target service and a resource allocation amount corresponding to the at least one resource;

[0009] Filter out a target resource scheduling strategy that meets the business execution time requirement set for the target business from multiple resource scheduling strategies;

[0010] The scheduling resource combination corresponding to the target resource scheduling strategy is used as the scheduling resource combination of the target business.

[0011] In an optional embodiment, a target resource scheduling strategy that meets the service execution duration requirement set for the target service is screened out from multiple resource scheduling strategies, including:

[0012] Acquire historical business execution information associated with the target business; the historical business execution information includes: sub-business execution information corresponding to multiple target sub-businesses of the target business, and business stage information corresponding to multiple business stages of the target business, wherein each business stage corresponds to at least one target sub-business;

[0013] Based on historical business execution information, determine the expected execution time of the target business under multiple resource scheduling strategies;

[0014] Based on the obtained estimated execution times of multiple services, a target resource scheduling strategy that meets the requirements of the execution time of the services is selected from multiple resource scheduling strategies.

[0015] In an optional embodiment, before acquiring the historical service execution information associated with the target service, the method further includes:

[0016] Obtaining log information of a service execution device; the log information includes historical service execution information of the target service;

[0017] Based on the event type identifiers of the multiple services, multiple historical service execution information is parsed from the log information, and the multiple service execution information is saved in a preset service execution information query table; wherein each event type identifier is used to indicate a service execution event of the corresponding service.

[0018] In an optional embodiment, based on historical service execution information, determining the service execution estimated duration of the target service under multiple resource scheduling strategies respectively includes:

[0019] Obtaining a first duration and a second duration based on historical business execution information; wherein the first duration is an average startup duration or a maximum startup duration of a target sub-business corresponding to the target business, and the second duration is an average cleanup duration of a target sub-business corresponding to the target business;

[0020] For multiple resource scheduling policies, perform the following operations respectively:

[0021] Based on the estimated operation duration, the first duration and the second duration of the target business under the first resource scheduling strategy, the estimated service execution duration of the target business under the first resource scheduling strategy is obtained; wherein the first resource scheduling strategy is any one of multiple resource scheduling strategies.

[0022] In an optional embodiment, if the at least one resource indicated by the first resource scheduling policy includes a kernel and a memory, the kernel is used to execute at least one target sub-service of the target service;

[0023] Based on the estimated operation duration of the target service under the first resource scheduling strategy, the first duration, and the second duration, the estimated service execution duration of the target service under the first resource scheduling strategy is obtained, including:

[0024] The estimated duration of the business execution is determined based on the number of cores, the total amount of memory, the first duration, the second duration, and the estimated running duration.

[0025] In an optional embodiment, if the execution concurrency of the target service is equal to the number of cores, the first duration is the average startup duration;

[0026] Based on the number of cores, the total amount of memory, the first duration, the second duration, and the estimated running duration, determine the estimated duration of business execution, including:

[0027] Add the first duration and the second duration to obtain a third duration;

[0028] Determine the estimated duration of business execution based on execution concurrency, total memory, third duration, and estimated running duration.

[0029] In an optional embodiment, if the execution concurrency of the target service is less than the number of cores, the first duration is the maximum startup duration;

[0030] Based on the number of cores, the total amount of memory, the first duration, the second duration, and the estimated running duration, determine the estimated duration of business execution, including:

[0031] Subtract the first duration from the second duration to obtain a fourth duration;

[0032] Based on the execution concurrency, the total memory, the fourth duration, the estimated running duration, and the fifth duration obtained based on the historical business execution information, the estimated business execution duration is determined; wherein the fifth duration is the maximum cleanup duration of the target sub-business corresponding to the target business.

[0033] In an optional embodiment, after taking the scheduling resource combination corresponding to the target resource scheduling policy as the scheduling resource combination of the target service, the method further includes:

[0034] Send a push message of the target resource scheduling strategy to the target object; the push message is used to instruct the target object to select the target resource scheduling strategy;

[0035] If the target object does not select the target resource scheduling policy within the set time range, the resource scheduling policy of the target business is adjusted to the target resource scheduling policy.

[0036] In an optional embodiment, after taking the scheduling resource combination corresponding to the target resource scheduling policy as the scheduling resource combination of the target service, the method further includes:

[0037] Receiving a second resource demand of a target object for a target business;

[0038] If multiple resource scheduling strategies do not meet the second resource requirement, a resource configuration warning message is sent to the target object; the resource configuration warning message is used to remind the target object to adjust the second resource requirement;

[0039] And / or, sending a resource release message to the target object; the resource release message is used to instruct the target object to stop the service that does not run according to the resource scheduling policy, and release the resources occupied by the service.

[0040] In a second aspect, an embodiment of the present application further provides a resource scheduling device, the device comprising:

[0041] The information transceiver module is used to receive a service execution request initiated by a target object for a target service, and obtain a first resource requirement of the target service from the service execution request; the first resource requirement includes: at least one resource required to complete the target service, and the resource requirement amount corresponding to the at least one resource;

[0042] A policy setting module, configured to set a plurality of resource scheduling policies for a first resource requirement based on a set of schedulable resources; each resource scheduling policy is configured to indicate at least one resource allocated to a target service and a resource allocation amount corresponding to each of the at least one resource;

[0043] A policy screening module is used to screen out a target resource scheduling policy that meets the service execution time requirement set for the target service from multiple resource scheduling policies;

[0044] The resource confirmation module is used to use the scheduling resource combination corresponding to the target resource scheduling strategy as the scheduling resource combination of the target business.

[0045] In an optional embodiment, when a target resource scheduling strategy that meets the service execution duration requirement set for a target service is screened out from multiple resource scheduling strategies, the strategy screening module is specifically used to:

[0046] Acquire historical business execution information associated with the target business; the historical business execution information includes: sub-business execution information corresponding to multiple target sub-businesses of the target business, and business stage information corresponding to multiple business stages of the target business, wherein each business stage corresponds to at least one target sub-business;

[0047] Based on historical business execution information, determine the expected execution time of the target business under multiple resource scheduling strategies;

[0048] Based on the obtained estimated execution times of multiple services, a target resource scheduling strategy that meets the requirements of the execution time of the services is selected from multiple resource scheduling strategies.

[0049] In an optional embodiment, before acquiring the historical service execution information associated with the target service, the policy screening module is further used to:

[0050] Obtaining log information of a service execution device; the log information includes historical service execution information of the target service;

[0051] Based on the event type identifiers of the multiple services, multiple historical service execution information is parsed from the log information, and the multiple service execution information is saved in a preset service execution information query table; wherein each event type identifier is used to indicate a service execution event of the corresponding service.

[0052] In an optional embodiment, when determining the estimated service execution duration of the target service under multiple resource scheduling strategies based on the historical service execution information, the strategy screening module is specifically used to:

[0053] Obtaining a first duration and a second duration based on historical business execution information; wherein the first duration is an average startup duration or a maximum startup duration of a target sub-business corresponding to the target business, and the second duration is an average cleanup duration of a target sub-business corresponding to the target business;

[0054] For multiple resource scheduling policies, perform the following operations respectively:

[0055] Based on the estimated operation duration, the first duration and the second duration of the target business under the first resource scheduling strategy, the estimated service execution duration of the target business under the first resource scheduling strategy is obtained; wherein the first resource scheduling strategy is any one of multiple resource scheduling strategies.

[0056] In an optional embodiment, if the at least one resource indicated by the first resource scheduling policy includes a kernel and a memory, the kernel is used to execute at least one target sub-service of the target service;

[0057] When obtaining the estimated service execution duration of the target service under the first resource scheduling strategy based on the estimated operation duration of the target service under the first resource scheduling strategy, the first duration, and the second duration, the strategy screening module is specifically used to:

[0058] The estimated duration of the business execution is determined based on the number of cores, the total amount of memory, the first duration, the second duration, and the estimated running duration.

[0059] In an optional embodiment, if the execution concurrency of the target service is equal to the number of cores, the first duration is the average startup duration;

[0060] When determining the estimated duration of business execution based on the number of cores, the total amount of memory, the first duration, the second duration, and the estimated running duration, the policy screening module is specifically used to:

[0061] Add the first duration and the second duration to obtain a third duration;

[0062] Determine the estimated duration of business execution based on execution concurrency, total memory, third duration, and estimated running duration.

[0063] In an optional embodiment, if the execution concurrency of the target service is less than the number of cores, the first duration is the maximum startup duration;

[0064] When determining the estimated duration of business execution based on the number of cores, the total amount of memory, the first duration, the second duration, and the estimated running duration, the policy screening module is specifically used to:

[0065] Subtract the first duration from the second duration to obtain a fourth duration;

[0066] Based on the execution concurrency, the total memory, the fourth duration, the estimated running duration, and the fifth duration obtained based on the historical business execution information, the estimated business execution duration is determined; wherein the fifth duration is the maximum cleanup duration of the target sub-business corresponding to the target business.

[0067] In an optional embodiment, after the scheduling resource combination corresponding to the target resource scheduling policy is used as the scheduling resource combination of the target service, the information transceiver module is further used to:

[0068] Send a push message of the target resource scheduling strategy to the target object; the push message is used to instruct the target object to select the target resource scheduling strategy;

[0069] If the target object does not select the target resource scheduling policy within the set time range, the resource scheduling policy of the target business is adjusted to the target resource scheduling policy.

[0070] In an optional embodiment, after the scheduling resource combination corresponding to the target resource scheduling policy is used as the scheduling resource combination of the target service, the information transceiver module is further used to:

[0071] Receiving a second resource demand of a target object for a target business;

[0072] If multiple resource scheduling strategies do not meet the second resource requirement, a resource configuration warning message is sent to the target object; the resource configuration warning message is used to remind the target object to adjust the second resource requirement;

[0073] And / or, sending a resource release message to the target object; the resource release message is used to instruct the target object to stop the service that does not run according to the resource scheduling policy, and release the resources occupied by the service.

[0074] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0075] Processor; and

[0076] Memory for storing programs,

[0077] The program includes instructions, and when the instructions are executed by a processor, the processor executes the resource scheduling method as described in the first aspect.

[0078] In a fourth aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the resource scheduling method as described in the first aspect.

[0079] In a fifth aspect, the present application provides a computer program product, which, when called by a computer, enables the computer to execute the steps of the resource scheduling method as described in the first aspect.

[0080] The beneficial effects of this application are as follows:

[0081] In the resource scheduling method provided in the embodiment of the present application, after receiving the service execution request of the target service, multiple resource scheduling strategies can be set for the target service according to the first resource demand of the target service carried in the service execution request. In this way, fixed resources are no longer reserved for the target service, but multiple resource scheduling strategies are set in a targeted manner according to the resource demand of the target service, thereby improving the flexibility of resource scheduling; and after obtaining the aforementioned multiple resource scheduling strategies, the target resource scheduling strategy that meets the service execution time requirement set for the target service can be screened out from the multiple resource scheduling strategies, thereby ensuring a high overall resource utilization rate. It can be seen that by adopting this method, a high overall resource utilization rate can be ensured while flexibly scheduling resources for the service.

[0082] In addition, other features and advantages of the present application will be described in the subsequent description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described here are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0084] Figure 1 A schematic diagram of an optional system architecture applicable to the embodiments of the present application;

[0085] Figure 2 A schematic diagram of an implementation flow of a resource scheduling method provided in an embodiment of the present application;

[0086] Figure 3 A logical schematic diagram of a resource scheduling strategy for determining a target business provided in an embodiment of the present application;

[0087] Figure 4 A schematic diagram of a method for screening a target resource scheduling strategy provided in an embodiment of the present application;

[0088] Figure 5 A schematic diagram of the composition structure of a log collection and monitoring visualization module provided in an embodiment of the present application;

[0089] Figure 6 A schematic diagram of the working principle of a log collector provided in an embodiment of the present application;

[0090] Figure 7 A schematic diagram of the workflow of a log parser provided in an embodiment of the present application;

[0091] Figure 8 A schematic diagram of a workflow of a resource allocation prediction model provided in an embodiment of the present application;

[0092] Fig. 9 A method based on the embodiment of the present application is provided Figure 2 Schematic diagram of the specific scene;

[0093] Fig.10 A schematic diagram of the structure of a resource scheduling device provided in an embodiment of the present application;

[0094] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0096] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0097] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0098] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0099] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0100] The following is a brief introduction to the design concept of the embodiment of the present application:

[0101] At present, big data processing systems generally face problems such as uneven resource allocation, high response delay, and low resource utilization. Traditional resource scheduling strategies are mostly based on static configuration files or simple load balancing algorithms, which often cannot accurately adapt to complex and changing workload characteristics. For example, although the distributed file system HDFS and distributed resource management YARN of the distributed system Hadoop provide a basic distributed storage and computing framework, its resource management mechanism is relatively extensive and it is difficult to respond quickly and accurately to the ever-changing task requirements.

[0102] The resource scheduling strategy based on the static configuration file is a resource scheduling strategy that reserves fixed resources for different services, or can be understood as policy-based resource reservation and preemption. In this way, resource reservation rules are preset for different types of tasks or user groups, and resource preemption is allowed according to priority when resources are tight, ensuring the smooth execution of high-priority tasks. This method balances resource fairness and efficiency through sophisticated policy configuration.

[0103] The above resource scheduling strategy based on simple load balancing is also a resource-aware load balancing strategy. In this way, not only the current resource occupancy is considered, but also the priority of the task, the resource demand characteristics, and the actual performance differences of the server are considered. Through comprehensive evaluation, the task is intelligently assigned to the node that is most suitable for its execution, avoiding the resource waste or slow task response problems that may be caused by the "one-size-fits-all" allocation method.

[0104] However, using the above-mentioned resource scheduling method, since the resources reserved for the business are fixed, it will be impossible to flexibly adjust the resource scheduling according to the operation of the business; and simply scheduling resources to meet business needs may result in low overall resource utilization, and thus cannot guarantee a high overall resource utilization.

[0105] In view of this, in order to solve or improve the above problems, the embodiment of the present application proposes a resource scheduling method, which may specifically include: first, receiving a business execution request initiated by a target object for a target business, and obtaining a first resource demand of the target business from the business execution request; the first resource demand includes: at least one resource required to complete the target business, and the resource demand corresponding to at least one resource; then, setting multiple resource scheduling strategies for the first resource demand based on a set of schedulable resources; wherein each resource scheduling strategy is used to indicate at least one resource allocated to the target business, and the resource allocation amount corresponding to at least one resource; furthermore, selecting a target resource scheduling strategy that meets the business execution time requirements set for the target business from multiple resource scheduling strategies; finally, using the scheduling resource combination corresponding to the target resource scheduling strategy as the scheduling resource combination of the target business. In this way, not only can the resource scheduling strategy be flexibly set according to the resource requirements of the target business, but also the target resource scheduling strategy that meets the business execution time requirements can be selected from multiple resource scheduling strategies, thereby ensuring the overall utilization of resources.

[0106] In particular, the preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0107] See also Figure 1 As shown, it is a schematic diagram of a system architecture applicable to an embodiment of the present application, and the system architecture may include: a terminal device 101, a server 102, and a resource cluster 103. The server 102 may schedule one or more resources (such as nodes) from the resource cluster 103 for the execution of services, and the terminal device 101 and the server 102 may exchange information through a communication network, wherein the communication mode adopted by the communication network may include: a wireless communication mode and a wired communication mode.

[0108] Exemplarily, the terminal device 101 can access the network through cellular mobile communication technology to communicate with the server 102. The cellular mobile communication technology, for example, includes the fifth generation mobile communication (5th generation mobile networks, 5G) technology or the next generation mobile communication technology.

[0109] Optionally, the terminal device 101 may access the network through a short-range wireless communication method to communicate with the server 102. The short-range wireless communication method, for example, includes wireless fidelity (Wi-Fi) technology.

[0110] The embodiment of the present application does not impose any restriction on the number of communication devices involved in the above system architecture. For example, the above system architecture may include more terminal devices, or more servers, or other network devices. Figure 1 As shown, only the terminal device 101, the server 102 and the resource cluster 103 are described as examples, and the above communication devices and their respective functions are briefly introduced below.

[0111] The terminal device 101 is a device that can provide voice and / or data connectivity to users, and can be a device that supports wired and / or wireless connection.

[0112] Exemplarily, terminal device 101 may include, but is not limited to: mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminal devices in industrial control, wireless terminal devices in unmanned driving, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, or wireless terminal devices in smart homes, etc.

[0113] In addition, a related client may be installed on the terminal device 101, and the client may be software, for example, an application (APP), a browser, a short video software, etc., or a web page, a mini-program, etc.; it should be noted that the terminal device 101 in the embodiment of the present application may enable the above-mentioned client related to resource scheduling to send a business execution request or resource requirement initiated by the target object for the target business to the server 102, so as to subsequently generate or set a resource scheduling strategy for the target business, thereby completing the resource scheduling for the target business.

[0114] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence (AI) platforms.

[0115] It is worth mentioning that in an embodiment of the present application, the server 102 can receive a business execution request initiated by a target object for a target business, and obtain a first resource requirement for the target business from the business execution request; set multiple resource scheduling strategies for the first resource requirement based on a schedulable resource set (i.e., a resource set that can be scheduled in the resource cluster 103); select a target resource scheduling strategy that meets the business execution time requirement set for the target business from multiple resource scheduling strategies; and use the scheduling resource combination corresponding to the target resource scheduling strategy as the scheduling resource combination for the target business.

[0116] The resource cluster 103 includes multiple resource nodes (not in Figure 1 As shown in FIG. 1 , each resource node has a certain service processing capability, wherein the larger the resource amount of the resource node, the stronger the service processing capability. Taking the resource amount as the size of the memory as an example, if the memory of resource node A is greater than the memory of resource node B, it can be determined that the service processing capability of resource node A is stronger than that of resource node B. It should be noted that the type of resource is not specifically limited in the embodiments of the present application.

[0117] It should also be noted that in the embodiment of the present application, the multiple resource nodes included in the resource cluster 103 are all used for processing corresponding services. Specifically, after receiving a service execution request for a certain service from the terminal device 101, the server 102 can dispatch some resources (or resource nodes) from the resource cluster 103 to execute the aforementioned certain service.

[0118] The resource scheduling method provided by the exemplary embodiment of the present application is described below in combination with the above-mentioned system architecture and with reference to the accompanying drawings. It should be noted that the above-mentioned system architecture is only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect.

[0119] See also Figure 2 As shown, it is a schematic diagram of an implementation process of a resource scheduling method provided in an embodiment of the present application. The execution subject takes a server as an example. The specific implementation process of the method is as follows:

[0120] S201: Receive a service execution request initiated by a target object for a target service, and obtain a first resource requirement of the target service from the service execution request.

[0121] The first resource requirement may include: at least one resource required to complete the target business, and the resource requirement corresponding to the at least one resource. For example, the at least one resource may include: a central processing unit (CPU) and a memory, then the first resource requirement may include the CPU requirement and the memory requirement. The CPU requirement may also be referred to as the kernel requirement.

[0122] It should be noted that the at least one resource mentioned above may also include an input / output (I / O) interface of a disk, etc., which is not specifically limited in the embodiment of the present application.

[0123] S202: Setting a plurality of resource scheduling strategies for a first resource requirement based on a set of schedulable resources.

[0124] The above-mentioned schedulable resource set may be a resource set that has not been scheduled, and each resource scheduling strategy may be used to indicate at least one resource allocated to the target service and the resource allocation amount corresponding to the at least one resource.

[0125] Taking the above-mentioned schedulable resource set as 10 cores and the memory of each core being any memory value between 5GB and 40GB as an example, the server can traverse the number of cores from 10 to 1, and the amount of memory (or memory size) of a single core from 40GB to 5GB, thereby obtaining a series of multiple resource scheduling strategies that meet the above-mentioned first resource requirement.

[0126] See also Figure 3As shown, the server can filter out a set of schedulable resources that can be scheduled for target business execution from the aforementioned multiple resources based on the resource usage corresponding to the multiple resources in the resource cluster, thereby setting all possible resource scheduling strategies for the first resource demand of the target business, for example, 5 resource scheduling strategies.

[0127] S203: Filter out a target resource scheduling strategy that meets the service execution duration requirement set for the target service from the multiple resource scheduling strategies.

[0128] Optionally, the target resource scheduling strategy that meets the business execution time requirements can be a resource scheduling strategy in which the estimated business execution time is less than or equal to the historical maximum completion time of the target business among multiple resource scheduling strategies, and the estimated business execution time is the largest. The aforementioned estimated business execution time is also the estimated time required to complete the target business using the corresponding resource scheduling strategy. In this way, on the basis of meeting the business completion deadline requirements (i.e., the historical maximum completion time), resource waste is reduced as much as possible, thereby maximizing resource utilization, that is, improving the overall resource utilization rate.

[0129] Taking 5 resource scheduling strategies (resource scheduling strategy 1 to resource scheduling strategy 5) as an example, and the estimated business execution times corresponding to the above 5 resource scheduling strategies are: 1min58s, 2min27s, 1min32s, 2min54s, 2min13s, respectively. If it is assumed that the historical maximum completion time of the target business is 2min30s, the server can ensure that the target resource scheduling strategy that meets the above business execution time requirements is resource scheduling strategy 2.

[0130] In order to improve the flexibility and adaptability of resource scheduling policy adjustment, the server can predict the expected service execution duration of the target service under the above-mentioned multiple resource scheduling policies based on the historical service execution information of the target service, and then select the target resource scheduling policy that meets the service execution duration requirements from the above-mentioned multiple resource scheduling policies. Figure 4 As shown, it is a schematic diagram of a method for screening a target resource scheduling strategy provided in an embodiment of the present application, which is as follows:

[0131] S401: Acquire historical business execution information associated with the target business.

[0132] The above-mentioned historical business execution information, that is, the business execution information of the target business within the set historical time range, may include: sub-business execution information corresponding to multiple target sub-businesses of the target business, and business stage information corresponding to multiple business stages of the target business, wherein each business stage corresponds to at least one target sub-business.

[0133] Taking a big data computing engine (such as Spark) as an example, the target business can be a job, so the target sub-business can be a task, and the business stage can be a stage. Among them, a job is the business submitted to Spark for processing, a stage refers to how many stages each job processing process is divided into, and a task refers to how many tasks each job processing process is divided into. A task is the smallest unit of job operation.

[0134] In order to improve the efficiency of obtaining historical business execution information, before executing step S401, the server can also parse multiple historical business execution information from the log information after obtaining the log information of the business execution device based on the event type identifiers of each of the multiple businesses, and save the multiple business execution information in a preset business execution information query table, so that the historical business execution information of the target business can be quickly obtained from the business execution information query table later.

[0135] Each event type identifier can be used to indicate a business execution event of the corresponding business. In other words, the event type identifier can be used to indicate one or more execution information of any one of the business, sub-business or business stage, so each business event type identifier can include multiple ones, and the embodiment of the present application has a good definition of this. For example, the event type identifier can indicate execution information such as job start, stage start, task completion, etc.

[0136] In addition, the above-mentioned business execution device is the core component for completing the business, which is responsible for receiving the business execution request of the target object (i.e., the user), parsing the business execution request, and scheduling resources and executing the business, etc. It can be various big data processing platforms or big data computing engines, etc., and the embodiments of this application do not specifically limit this. For example, the above-mentioned business execution device can be a big data computing engine such as Apache Hadoop Map Reduce, Apache Spark, and Apache Flink.

[0137] In an optional implementation, the server can record the log information of the service execution device through the log collection and monitoring visualization model. Figure 5 As shown in the figure, the aforementioned log collection and monitoring visualization module is mainly composed of a log collector, a log parser, and a job execution process monitoring module, which is used to realize the comprehensive automatic collection, deep collection, and real-time monitoring of the operation logs of the big data computing engine (such as Spark). Figure 6 As shown, the log collector is responsible for collecting and / or storing various information inside the big data computing engine, mainly including two types of information: task execution information and cluster resource information.

[0138] In terms of task execution information, the log collector can capture detailed information during the execution of jobs, stages, and tasks. At the job level, the life cycle of the job can be recorded, including the start and end time, and any intermediate state changes. At the stage level, the execution of the stage can be monitored, and the start and end time of each stage, as well as the status of the tasks within the stage, can be recorded. At the task level, the execution time, data read and write volume, shuffle information, and the status of the task (such as success, failure, cancellation, etc.) can be collected for each task.

[0139] On the other hand, in terms of cluster resource information, the log collector can monitor the CPU usage of each task, including CPU usage rate and CPU usage time; it can also monitor the memory usage of each task, especially the memory usage during task execution, such as peak memory usage, memory overflow, etc. The log collector can also count the number of running tasks to understand the number of tasks running at the same time. In addition, the log collector can also record the number of tasks queued so that the efficiency of task scheduling can be determined later.

[0140] By collecting this information, the log collector provides a basis for evaluating the execution efficiency of the job, resource usage, and possible bottlenecks such as data locality, network communication impact, hardware resource utilization, and task scheduling efficiency.

[0141] The log parser can process and analyze the log information collected by the log collector. Figure 7 As shown, in an optional implementation, the server can perform the following method steps through a log parser:

[0142] S701: Open the log information.

[0143] Specifically, when executing step S701, the log parser enables access to the log information (or log file) to accurately read the content therein.

[0144] S702: Read the event type identifier.

[0145] The event type identifier may also be referred to as a monitoring identifier, and of course there may be other names, which are not limited in the embodiments of the present application. The log parser reads the event type identifiers in the log information, which indicate what type of event the log information records, for example, information about work or a stage, or information about a task. Optionally, the aforementioned event type identifier may be a specific identifier.

[0146] S703: Time-classify the log information according to the event type identifier.

[0147] Specifically, when executing step S703, after reading the event type identifier, the log parser can classify the log information according to the event type identifier to distinguish business execution events (or execution information) occurring in different time periods, for example, job start, stage start, task completion and other execution information.

[0148] S704: Parse the execution information corresponding to different stages of the job / stage / task.

[0149] For each type, the log parser parses and understands the specific events represented by the event type identifier, such as job submission, stage division, task execution, etc. For each level, key parameters can be deeply analyzed, including but not limited to the job's unique identifier, name, submission and completion time, status, resource consumption overview; stage ID, associated jobs, number of tasks, Shuffle data flow details, execution time and resource usage statistics; as well as the task's specific ID, belonging stage, execution time, status, execution node, data processing volume, Shuffle operation time and memory usage, etc.

[0150] S705: Save the multiple business execution information in a preset business execution information query table.

[0151] The preset business execution information query table may be a Hive table in the data warehouse of Hadoop. Therefore, after the log parser obtains the execution information through parsing, it may store the execution information in the Hive table for subsequent query and analysis.

[0152] S706: Is it completely saved? If yes, go to step S707; if no, go to step S702.

[0153] S707: Send the preset business execution information query table to the job execution process monitoring module.

[0154] The above-mentioned job execution process monitoring module can continuously track the execution progress and resource consumption of the job, and record key performance indicators such as job execution time, CPU usage, memory usage, disk I / O, etc. These data will be used as input for the resource allocation prediction model used for the subsequent selection of target resource scheduling strategies.

[0155] Exemplarily, the above-mentioned job execution process monitoring module can be a monitoring tool composed of Prometheus+Grafana, which is used for real-time display and alarm. Based on the above method, the log parser not only collects and stores log information, but also works with the monitoring tool to provide richer data analysis and visualization services.

[0156] S402: Determine the estimated execution duration of the target business under multiple resource scheduling strategies based on historical business execution information.

[0157] In an optional implementation, when executing step S402, after obtaining the historical business execution information of the target business, the server can obtain the first duration and the second duration of the target business based on the historical business execution information. The first duration can be the average startup duration (which can be recorded as: avg_Startup) or the maximum startup duration (which can be recorded as: max_Startup) of the target sub-business corresponding to the target business, and the second duration can be the average cleanup duration (which can be recorded as: avg_Cleanup) of the target sub-business corresponding to the target business. Optionally, the server can also obtain the maximum cleanup duration (which can be recorded as: max_Cleanup) and the average completion duration (which can be recorded as: T of the target sub-business corresponding to the target business) from the historical business execution information. ave ) and the historical maximum completion time (which can be recorded as: T max ), etc., which is not limited in the embodiments of the present application.

[0158] Further, for any one of the above-mentioned multiple resource scheduling strategies, i.e., the first resource scheduling strategy, the server can obtain the estimated service execution duration of the target service under the first resource scheduling strategy based on the estimated operation duration, the first duration, and the second duration of the target service under the first resource scheduling strategy. The aforementioned estimated operation duration can also be understood as the actual processing duration of the target service, i.e., excluding the duration of service startup and cleanup.

[0159] For example, the calculation method of the estimated execution time of the above service can be specifically expressed as follows:

[0160] T temp =T i +T1+T2

[0161] Among them, T temp Indicates the expected execution time of the target business, T i Indicates the estimated running time of the target service, T1 indicates the first running time, and T2 indicates the second running time.

[0162] Optionally, if at least one resource indicated by the first resource scheduling strategy includes a kernel and a memory, wherein the kernel is used to execute at least one target sub-business corresponding to the target business, then the server can determine the estimated business execution duration according to the number of kernels, the total memory amount of the memory, the first duration, the second duration, and the estimated running duration. Exemplarily, the estimated business execution duration of the target business can be obtained in the following two ways:

[0163] Method A: If the execution concurrency of the target business is equal to the number of cores, the server can determine that the first duration is the average startup duration, and then sum the first duration and the second duration to obtain the third duration, and then determine the estimated execution duration of the business based on the execution concurrency, total memory, third duration and estimated running duration.

[0164] It should be noted that the above-mentioned number of cores is also the maximum execution concurrency of the target business. Exemplarily, if the resources indicated by the above-mentioned first resource scheduling strategy include 8 cores, then the maximum execution concurrency of the target business is 8, that is, 8 cores work in parallel at the same time. Therefore, the execution concurrency of the target business in method A is equal to the number of cores, which can also be understood as the execution concurrency of the target business is the maximum execution concurrency under the first resource scheduling strategy. It should also be noted that the execution concurrency of the target business can also be understood as the number of target sub-businesses into which the target business is divided.

[0165] For example, for the first resource scheduling strategy, the calculation method of the estimated service execution duration corresponding to method A can be specifically expressed as follows:

[0166]

[0167] Among them, T temp is the estimated execution time of the target business under mode A, n is the number of cores, m is the total memory, T i It is the estimated operation duration of the target business, T1 is the first duration, T2 is the second duration, and T1+T2 is the third duration.

[0168] Based on the above method A, the server can decompose the target business into n target subtasks. Since each target subtask requires a certain startup time and cleanup time, when these target subtasks are assigned to memory parameters with a total memory of m for execution, if the execution concurrency remains unchanged (that is, the execution concurrency of the target business is equal to the number of cores), it means that each memory parameter will process n / m target subtasks at the same time. In this way, because each memory parameter will process n / m subtasks at the same time, each subtask of the sub-memory unit needs to be started and cleaned up once. The average startup time T1 and the average cleanup time T2 will be added together, and then multiplied by the number of subtasks (n / m) to get the total startup and cleanup time of all subtasks. Finally, add the actual execution time T of the target business i , we can get the total estimated execution time T temp .

[0169] Method B: If the execution concurrency of the target business is less than the number of cores, the server can determine the first duration as the maximum startup duration, and then obtain the fourth duration by subtracting the first duration from the second duration, and then determine the estimated execution duration of the business based on the execution concurrency, the total memory, the fourth duration and the estimated operation duration, and the fifth duration obtained based on the historical business execution information. The fifth duration is the maximum cleanup duration of the target sub-business corresponding to the target business.

[0170] For example, for the first resource scheduling strategy, the calculation method of the estimated service execution duration corresponding to method B can be specifically expressed as follows:

[0171]

[0172] Among them, T temp is the estimated execution time of the target business under mode B, nl is the execution concurrency, l∈(0,n) and is an integer, T1 is the first time duration, T2 is the second time duration, T1-T2 is the fourth time duration, T i is the estimated running time of the target business, and T5 is the fifth time, that is, the maximum cleanup time max_Cleanup.

[0173] Based on the above method B, when the execution concurrency is reduced, since each memory (parameter) only processes one target subtask, the startup time of a single target subtask can be obtained by subtracting the average cleanup time (i.e., the second duration T2) from the maximum startup time (i.e., the first duration T1). Among them, the startup time of a single target subtask (i.e., T1-T2) multiplied by the number of each target subtask (n-1 / m) is the business startup time on each memory (i.e., unit memory). In addition, since there is only one target subtask running in each memory, there is no need to consider the cleanup time of other target subtasks, and it only needs to be cleaned once, so just add this time's resource cleanup time, for example, the fifth duration T5. Finally, add the actual execution time of the current task T i , we can get the total estimated execution time T temp .

[0174] S403: Based on the obtained multiple service execution estimated durations, select a target resource scheduling strategy that meets the service execution duration requirement from multiple resource scheduling strategies.

[0175] For example, after calculating the service execution estimated durations corresponding to the multiple resource scheduling strategies obtained by the above method A or method B, the target resource scheduling strategy that meets the service execution duration requirements can be selected from the above multiple resource scheduling strategies. temp Less than or equal to the historical maximum completion time T of the target businessmax The resource scheduling strategy of is used as the candidate resource scheduling strategy. If there are multiple candidate resource scheduling strategies, the candidate resource scheduling strategy corresponding to the maximum estimated service execution time can be used as the target resource scheduling strategy.

[0176] Through the above steps, two possible optimal resource allocation schemes (i.e., target resource scheduling strategies) will eventually be found: resource allocation scheme 1 and resource allocation scheme 2. Both schemes are based on meeting the job deadline requirements (i.e., business execution duration requirements) and minimize resource waste to maximize resource utilization. Resource allocation scheme 1 is the optimal scheduling resource combination found without reducing execution concurrency, while resource allocation scheme 2 is the optimal scheduling resource combination found after reducing execution concurrency. In this way, it can adapt to different scenarios. For example, in some cases, reducing execution concurrency may help improve overall efficiency, while in other cases, maintaining high execution concurrency may be more beneficial. In general, by simulating different resource allocation situations and predicting the expected execution duration of business, users can be guided to make the best resource allocation decisions, which can save resource costs to the greatest extent while meeting job deadline requirements.

[0177] Optionally, the above-mentioned method A and the above-mentioned method B can be combined, that is, the server can also determine at least one candidate resource scheduling strategy from multiple resource scheduling strategies according to the above-mentioned method A, and then filter out the target resource scheduling strategy from the above-mentioned at least one candidate resource scheduling strategy according to the above-mentioned method B.

[0178] The specific method of determining at least one candidate resource scheduling strategy from multiple resource scheduling strategies may be the same as the specific method of selecting the target resource scheduling strategy from the aforementioned at least one candidate resource scheduling strategy. For example, both are comparing and judging the expected duration of business execution with the historical maximum completion duration.

[0179] S204: Using the scheduling resource combination corresponding to the target resource scheduling policy as the scheduling resource combination of the target service.

[0180] Among them, the above-mentioned target resource scheduling strategy can also be called the optimal resource allocation plan. Of course, there may be other names, and the embodiments of the present application do not make specific limitations on this.

[0181] In an optional implementation, after executing step S204, the server may also send a push message of the target resource scheduling strategy to the target object. If the target object does not select the target resource scheduling strategy according to the push message within a set time range (e.g., 3 days, etc.), the current resource scheduling strategy of the target business may be directly adjusted to the target resource scheduling strategy. The aforementioned push message is used to instruct the target object to select the target resource scheduling strategy. On the contrary, if the target object selects the target resource scheduling strategy according to the push message within the set time range, the target resource scheduling strategy may be directly adapted.

[0182] Exemplarily, the server can perform the above method steps through a resource allocation prediction model, which provides users with an accurate and efficient resource planning tool by deeply analyzing historical operation data (i.e., historical business execution information), real-time monitoring of cluster resource status (i.e., schedulable resource set), and understanding the characteristics of specific loads. It can predict the completion time of each target sub-business, target stage, and even the entire target business based on the historical performance of the target business, combined with factors such as input data scale and task characteristics, thereby helping users to reasonably estimate and schedule the most appropriate computing resources (such as the number of CPU cores) while meeting the application deadline requirements (i.e., business execution time requirements).

[0183] In this way, not only can resource utilization be significantly improved, resource waste and operating costs can be reduced, but also business can be completed on time, avoiding business execution delays or failures caused by improper resource allocation. In addition, the resource allocation prediction model also has the ability to cope with dynamic changes in cluster resources (i.e., a set of schedulable resources). For example, when a resource node fails, it can still provide a reliable business execution time estimate (i.e., the expected duration of business execution) to ensure system stability and reliability.

[0184] Therefore, the above resource allocation prediction model is a prediction model that generates the best resource allocation solution (i.e., target resource scheduling strategy) based on job history information (i.e., historical business execution information) and cluster resource (i.e., schedulable resource set) information. Figure 8 As shown in the figure, the training and resource allocation optimization process of the resource allocation prediction model is as follows:

[0185] S801a: Obtaining job history information.

[0186] S801b: Obtain cluster resource information.

[0187] It should be noted that there is no order between S801a and S801b. For example, S801a may occur before S801b, or after S801b, or may occur at the same time. This embodiment of the present application is not limited to this.

[0188] Based on the above steps S801a and S801b, the resource allocation prediction model can extract relevant information from the recorded job history information, such as business execution time, resource usage, etc. At the same time, the resource status of the current cluster (i.e., the schedulable resource set) can be continuously monitored, such as CPU usage, memory usage, etc.

[0189] S802: Inputting the job history information and cluster resource information into a resource allocation prediction model to obtain an optimal resource allocation solution.

[0190] Exemplarily, the resource allocation prediction model mentioned above can be a resource allocation prediction model based on a linear regression model. First, key features can be extracted from massive historical logs, such as the operating parameters of the job, resource consumption, etc., and combined with the real-time resource usage metrics provided by the Ganglia monitoring system. Subsequently, these feature data are used to train a resource allocation prediction model, which can predict the expected execution time of the job based on input features (such as data size, number of tasks, etc.). The accuracy of the prediction is ensured by continuously optimizing the model parameters.

[0191] When a new job is input into the trained resource allocation prediction model, the completion time of each task, stage and even the entire job can be predicted based on the input data scale, task characteristics, etc. of the job, and corresponding resource allocation suggestions can be given.

[0192] S803: Generate a parameter script adapted to the big data computing engine.

[0193] Specifically, when executing S803, the server may generate a parameter script according to the predicted optimal resource allocation solution. The parameter script includes a recommended resource allocation strategy for subsequent reference by the user or for direct application to job execution.

[0194] S804: Push the best resource allocation solution.

[0195] In this way, the prediction results are pushed to users, so that subsequent users can adjust the job parameters of the big data computing engine according to the suggestions to improve execution efficiency.

[0196] S805: If the suggestion is not adopted within the specified time, the task controller may automatically check and consider terminating the inefficient task.

[0197] S806: If the suggestion is adopted within the specified time, the user can click one-click adaptation.

[0198] The adoption of the above suggestions means that the user confirms the optimization, and by clicking the one-click adaptation function, the recommended resource allocation plan can be quickly applied to the actual operation to achieve automated optimization.

[0199] Through the method steps described in S801 to S806 above, the resource allocation prediction model can help users better manage resources, reduce operating costs, improve job execution efficiency, and ensure system stability.

[0200] In an optional implementation, after executing step S204, the server may also receive a second resource requirement of the target object for the target business; if multiple resource scheduling strategies do not meet the second resource requirement, a resource configuration warning message is sent to the target object; and / or a resource release message is sent to the target object. The resource configuration warning message may be used to remind the target object to adjust the second resource requirement; the resource release message may be used to instruct the target object to stop the business that is not running according to the resource scheduling strategy, and to release the resources occupied by the business.

[0201] The fact that none of the above-mentioned multiple resource scheduling strategies meet the second resource requirement can be understood as the resource amounts corresponding to the scheduling resource combinations corresponding to the above-mentioned multiple resource scheduling strategies are all smaller than the resource requirements corresponding to the second resource requirement.

[0202] Therefore, see Fig. 9 As shown in FIG. 1 , after the resource allocation prediction model is optimized, for the same task (e.g., target business) resubmitted by the user, the server (e.g., task control system) automatically starts a set of sophisticated monitoring and optimization processes: First, the server captures the task submission request through the user interface (UI) of the big data computing engine, and parses the execution parameters of the task to obtain the resource configuration details. Subsequently, the server uses the resource allocation threshold set by the optimized resource allocation prediction model to strictly compare the resource requirements (i.e., the second resource requirements) of the current task request. If it is detected that the resource requirements exceed the optimization threshold, indicating that there is a potential problem of resource over-configuration, the server can trigger an early warning to notify the user and provide an option to directly execute the task termination command (e.g., "Yarn application-kill appid") to terminate the execution of non-optimized tasks to avoid invalid resource occupation. After obtaining feedback based on this, the user can choose to adjust the task resource configuration recommended by the resource allocation prediction model and resubmit the task, thereby ensuring efficient operation of the task while achieving efficient utilization and management of cluster resources. Otherwise, the task continues to execute.

[0203] It can be seen that the log-driven intelligent resource scheduling mechanism improved by the embodiment of the present application has revolutionized the traditional static resource allocation method. First, by integrating advanced log analysis technology and machine learning algorithms, it is possible to dynamically monitor the running status of big data jobs and instantly analyze task requirements and resource utilization efficiency. Secondly, a self-optimizing resource allocation prediction model is introduced to automatically adjust resource thresholds according to real-time job performance (such as a set of schedulable resources) and historical log data (such as historical business execution information) to prevent resource redundancy and task bottlenecks. In addition, early warning and automatic intervention functions are also implemented, allowing users to respond to resource over-allocation problems in real time, effectively improving system response speed and overall computing performance.

[0204] In summary, in the resource scheduling method provided in the embodiment of the present application, after receiving the business execution request of the target business, multiple resource scheduling strategies can be set for the target business according to the first resource demand of the target business carried in the business execution request. In this way, fixed resources are no longer reserved for the target business, but multiple resource scheduling strategies are set according to the resource demand of the target business, thereby improving the flexibility of resource scheduling; and, after obtaining the aforementioned multiple resource scheduling strategies, the target resource scheduling strategy that meets the business execution time requirements set for the target business can also be screened out from the multiple resource scheduling strategies, thereby ensuring that the overall utilization of resources is high. It can be seen that this method can intelligently monitor the capabilities of all running tasks and analyze the task execution status and resource consumption in real time, predict potential resource bottlenecks, and automatically execute efficient and flexible resource allocation decisions, thereby ensuring that the overall utilization of resources is high while flexibly scheduling resources for the business.

[0205] In addition, due to the integration of deep log analysis technology and machine learning-driven resource prediction models, intelligent resource allocation and real-time performance optimization during job execution can be achieved. By actively identifying and correcting over-allocation of resources, not only is resource waste reduced, but also the dual improvement of task execution efficiency and cluster resource utilization is achieved. Moreover, it can also improve the current big data resource allocation or management technology, such as slow static configuration response, insufficient prediction accuracy and adaptability, imbalance between optimization granularity and efficiency, lack of log analysis and feedback mechanism, insufficient cross-layer resource coordination, limited system robustness and elasticity, low user participation and transparency, incomplete cost-effectiveness considerations, and weak data security protection.

[0206] Further, based on the same technical concept, the embodiment of the present application provides a resource scheduling device, which is used to implement the above method flow of the embodiment of the present application. Fig.10As shown, the resource scheduling device 1000 includes: an information transceiver module 1001, a policy setting module 1002, a policy screening module 1003 and a resource determination module 1004, wherein:

[0207] The information transceiver module 1001 is used to receive a service execution request initiated by a target object for a target service, and obtain a first resource requirement of the target service from the service execution request; the first resource requirement includes: at least one resource required to complete the target service, and the resource requirement amount corresponding to the at least one resource;

[0208] A policy setting module 1002 is used to set a plurality of resource scheduling policies for a first resource requirement based on a set of schedulable resources; each resource scheduling policy is used to indicate at least one resource allocated to a target service and a resource allocation amount corresponding to each of the at least one resource;

[0209] A policy screening module 1003 is used to screen out a target resource scheduling policy that meets the service execution time requirement set for the target service from multiple resource scheduling policies;

[0210] The resource confirmation module 1004 is used to use the scheduling resource combination corresponding to the target resource scheduling policy as the scheduling resource combination of the target service.

[0211] In an optional embodiment, when a target resource scheduling strategy that meets the service execution duration requirement set for a target service is screened out from multiple resource scheduling strategies, the strategy screening module 1003 is specifically used to:

[0212] Acquire historical business execution information associated with the target business; the historical business execution information includes: sub-business execution information corresponding to multiple target sub-businesses of the target business, and business stage information corresponding to multiple business stages of the target business, wherein each business stage corresponds to at least one target sub-business;

[0213] Based on historical business execution information, determine the expected execution time of the target business under multiple resource scheduling strategies;

[0214] Based on the obtained estimated execution times of multiple services, a target resource scheduling strategy that meets the requirements of the execution time of the services is selected from multiple resource scheduling strategies.

[0215] In an optional embodiment, before acquiring the historical service execution information associated with the target service, the policy screening module 1003 is further used to:

[0216] Obtaining log information of a service execution device; the log information includes historical service execution information of the target service;

[0217] Based on the event type identifiers of the multiple services, multiple historical service execution information is parsed from the log information, and the multiple service execution information is saved in a preset service execution information query table; wherein each event type identifier is used to indicate a service execution event of the corresponding service.

[0218] In an optional embodiment, when determining the estimated service execution duration of the target service under multiple resource scheduling strategies based on the historical service execution information, the strategy screening module 1003 is specifically used to:

[0219] Obtaining a first duration and a second duration based on historical business execution information; wherein the first duration is an average startup duration or a maximum startup duration of a target sub-business corresponding to the target business, and the second duration is an average cleanup duration of a target sub-business corresponding to the target business;

[0220] For multiple resource scheduling policies, perform the following operations respectively:

[0221] Based on the estimated operation duration, the first duration and the second duration of the target business under the first resource scheduling strategy, the estimated service execution duration of the target business under the first resource scheduling strategy is obtained; wherein the first resource scheduling strategy is any one of multiple resource scheduling strategies.

[0222] In an optional embodiment, if the at least one resource indicated by the first resource scheduling policy includes a kernel and a memory, the kernel is used to execute at least one target sub-service of the target service;

[0223] When obtaining the estimated service execution duration of the target service under the first resource scheduling strategy based on the estimated operation duration of the target service under the first resource scheduling strategy, the first duration, and the second duration, the strategy screening module 1003 is specifically used to:

[0224] The estimated duration of the business execution is determined based on the number of cores, the total amount of memory, the first duration, the second duration, and the estimated running duration.

[0225] In an optional embodiment, if the execution concurrency of the target service is equal to the number of cores, the first duration is the average startup duration;

[0226] When determining the estimated service execution duration based on the number of cores, the total amount of memory, the first duration, the second duration, and the estimated running duration, the policy screening module 1003 is specifically used to:

[0227] Add the first duration and the second duration to obtain a third duration;

[0228] Determine the estimated duration of business execution based on execution concurrency, total memory, third duration, and estimated running duration.

[0229] In an optional embodiment, if the execution concurrency of the target service is less than the number of cores, the first duration is the maximum startup duration;

[0230] When determining the estimated service execution duration based on the number of cores, the total amount of memory, the first duration, the second duration, and the estimated running duration, the policy screening module 1003 is specifically used to:

[0231] Subtract the first duration from the second duration to obtain a fourth duration;

[0232] Based on the execution concurrency, the total memory, the fourth duration, the estimated running duration, and the fifth duration obtained based on the historical business execution information, the estimated business execution duration is determined; wherein the fifth duration is the maximum cleanup duration of the target sub-business corresponding to the target business.

[0233] In an optional embodiment, after the scheduling resource combination corresponding to the target resource scheduling policy is used as the scheduling resource combination of the target service, the information transceiver module 1001 is further used to:

[0234] Send a push message of the target resource scheduling strategy to the target object; the push message is used to instruct the target object to select the target resource scheduling strategy;

[0235] If the target object does not select the target resource scheduling policy within the set time range, the resource scheduling policy of the target business is adjusted to the target resource scheduling policy.

[0236] In an optional embodiment, after the scheduling resource combination corresponding to the target resource scheduling policy is used as the scheduling resource combination of the target service, the information transceiver module 1001 is further used to:

[0237] Receiving a second resource demand of a target object for a target business;

[0238] If multiple resource scheduling strategies do not meet the second resource requirement, a resource configuration warning message is sent to the target object; the resource configuration warning message is used to remind the target object to adjust the second resource requirement;

[0239] And / or, sending a resource release message to the target object; the resource release message is used to instruct the target object to stop the service that does not run according to the resource scheduling policy, and release the resources occupied by the service.

[0240] Based on the description of the above method embodiment and device embodiment, the exemplary embodiment of the present invention further provides an electronic device, including: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method according to the embodiment of the present invention when executed by the at least one processor.

[0241] An embodiment of the present application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute a method according to an embodiment of the present application.

[0242] An embodiment of the present application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute a method according to an embodiment of the present application.

[0243] See also Fig.11 As shown, the structured block diagram of the electronic device 1100 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent the computer device of various forms of digital electronics, such as, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described herein and / or required.

[0244] like Fig.11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An I / O interface 1105 is also connected to the bus 1104.

[0245] Multiple components in the electronic device 1100 are connected to the I / O interface 1105, including: an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 can be any type of device that can input information to the electronic device 1100, and the input unit 1106 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 1107 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1108 can include but is not limited to a disk, an optical disk. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a worldwide interoperability for microwave access (WiMax) device, a cellular communication device, and / or the like.

[0246] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a CPU, a graphics processing unit (GPU), various AI computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned resource scheduling method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. In some embodiments, the computing unit 1101 may be configured to perform the above-mentioned resource scheduling method by any other appropriate means (e.g., by means of firmware).

[0247] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0248] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0249] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0250] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tub (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0251] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0252] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0253] Furthermore, it should be understood that what is disclosed above is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope covered by the present application.

Claims

1. A resource scheduling method, characterized in that: include: Receiving a service execution request initiated by a target object for a target service, and acquiring a first resource requirement of the target service from the service execution request; The first resource requirement includes: at least one resource required to complete the target business, and the resource demand corresponding to the at least one resource; Based on the schedulable resource set, a plurality of resource scheduling strategies are set for the first resource demand; wherein each resource scheduling strategy is used to indicate at least one resource allocated for the target service, and a resource allocation amount corresponding to each of the at least one resource; Filtering out a target resource scheduling strategy that meets the service execution duration requirement set for the target service from the multiple resource scheduling strategies; The scheduling resource combination corresponding to the target resource scheduling strategy is used as the scheduling resource combination of the target service.

2. The method according to claim 1, characterized in that The step of selecting, from the multiple resource scheduling strategies, a target resource scheduling strategy that meets the service execution duration requirement set for the target service comprises: Acquire historical business execution information associated with the target business; the historical business execution information includes: sub-business execution information corresponding to multiple target sub-businesses of the target business, and business stage information corresponding to multiple business stages of the target business, wherein each business stage corresponds to at least one target sub-business; Based on the historical service execution information, determining the estimated service execution duration of the target service under the multiple resource scheduling strategies respectively; Based on the obtained multiple service execution estimated durations, the target resource scheduling strategy that meets the service execution duration requirement is screened out from the multiple resource scheduling strategies.

3. The method according to claim 2, characterized in that Before acquiring the historical service execution information associated with the target service, the method further includes: Obtaining log information of a service execution device; the log information includes historical service execution information of the target service; Based on the event type identifiers of the respective multiple services, multiple historical service execution information is parsed from the log information, and the multiple service execution information is saved in a preset service execution information query table; wherein each event type identifier is used to indicate a service execution event of the corresponding service.

4. The method according to claim 2, characterized in that The determining, based on the historical service execution information, the estimated service execution durations of the target service under the multiple resource scheduling strategies respectively includes: Obtaining a first duration and a second duration based on the historical business execution information; wherein the first duration is the average startup duration or the maximum startup duration of the target sub-business corresponding to the target business, and the second duration is the average cleanup duration of the target sub-business corresponding to the target business; For the multiple resource scheduling strategies, perform the following operations respectively: Based on the estimated operation duration of the target business under the first resource scheduling strategy, the first duration and the second duration, determine the estimated service execution duration of the target business under the first resource scheduling strategy; wherein the first resource scheduling strategy is any one of the multiple resource scheduling strategies.

5. The method according to claim 4, characterized in that If the at least one resource indicated by the first resource scheduling policy includes a kernel and a memory, the kernel is used to execute at least one target sub-service of the target service; Then, determining the estimated service execution duration of the target service under the first resource scheduling strategy based on the estimated operation duration of the target service under the first resource scheduling strategy, the first duration, and the second duration includes: The estimated execution time of the service is determined based on the number of cores of the core, the total memory amount of the memory, the first time, the second time, and the estimated operation time.

6. The method according to claim 5, characterized in that If the execution concurrency of the target service is equal to the number of cores, the first duration is the average startup duration; The determining the estimated execution time of the service based on the number of cores of the core, the total memory amount of the memory, the first time, the second time, and the estimated operation time includes: Summing the first duration and the second duration to obtain a third duration; The estimated execution duration of the business is determined based on the execution concurrency, the total memory amount, the third duration and the estimated operation duration.

7. The method according to claim 5, characterized in that If the execution concurrency of the target service is less than the number of cores, the first duration is the maximum startup duration; The determining the estimated execution time of the service based on the number of cores of the core, the total memory amount of the memory, the first time, the second time, and the estimated operation time includes: Subtract the first duration from the second duration to obtain a fourth duration; Based on the execution concurrency, the total memory, the fourth duration, the estimated running duration, and the fifth duration obtained based on the historical business execution information, the estimated business execution duration is determined; wherein the fifth duration is the maximum cleanup duration of the target sub-business corresponding to the target business.

8. The method according to any one of claims 1 to 7, characterized in that After taking the scheduling resource combination corresponding to the target resource scheduling strategy as the scheduling resource combination of the target service, the method further includes: Sending a push message of the target resource scheduling strategy to the target object; the push message is used to instruct the target object to select the target resource scheduling strategy; If the target object does not select the target resource scheduling policy within the set time range, the resource scheduling policy of the target service is adjusted to the target resource scheduling policy.

9. The method according to any one of claims 1 to 7, characterized in that After taking the scheduling resource combination corresponding to the target resource scheduling strategy as the scheduling resource combination of the target service, the method further includes: receiving a second resource demand of the target object for the target business; If none of the multiple resource scheduling strategies meet the second resource requirement, a resource configuration warning message is sent to the target object; the resource configuration warning message is used to remind the target object to adjust the second resource requirement; And / or, sending a resource release message to the target object; the resource release message is used to instruct the target object to stop the business that is not running according to the resource scheduling policy, and release the resources occupied by the business.

10. A resource scheduling device, characterized in that: include: An information transceiver module, used to receive a service execution request initiated by a target object for a target service, and obtain a first resource requirement of the target service from the service execution request; The first resource requirement includes: at least one resource required to complete the target business, and the resource demand corresponding to the at least one resource; A policy setting module, configured to set a plurality of resource scheduling policies for the first resource demand based on a set of schedulable resources; wherein each resource scheduling policy is used to indicate at least one resource allocated for the target service and a resource allocation amount corresponding to each of the at least one resource; A policy screening module, used to screen out a target resource scheduling policy that meets the service execution duration requirement set for the target service from the multiple resource scheduling policies; The resource confirmation module is used to use the scheduling resource combination corresponding to the target resource scheduling policy as the scheduling resource combination of the target service.

Images