Task scheduling method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a task scheduling method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of computers. The task scheduling method comprises the following steps: receiving a task instruction; based on the parameters of the task instruction, judging the task type of a target task corresponding to the task instruction; the task types comprise a first task type and a second task type, and the priority of the first task type is higher than that of the second task type; when it is judged that the task type of the target task is the first task type, the task instruction is submitted to a corresponding first cluster, and the first cluster is used for executing the target task of the first task type. The task priority is determined based on the parameters of the task instruction, and the cluster is scheduled to execute the corresponding target task based on the priority of the target task, so that intelligent scheduling of the task is realized, resource management can be optimized, and system performance and system stability can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a task scheduling method, device, electronic equipment and computer-readable storage medium. Background Art

[0002] In today's society where data is growing explosively, how to efficiently process massive amounts of data has become the key to improving the competitiveness of all industries. As one of the core technologies to meet the above challenges, the distributed query engine has significantly improved the efficiency and scalability of data queries through parallel processing and distributed architecture.

[0003] Related technologies use the above-mentioned distributed query engine to process big data. However, as the amount of data increases and the complexity of queries increases, the use of the above-mentioned distributed query engine may cause problems such as insufficient performance, poor stability, and insufficient resource management. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure propose a task scheduling method, device, electronic device and computer-readable storage medium to solve the problems of insufficient performance, poor stability and insufficient resource management that may occur when using a distributed query engine in related technologies.

[0005] According to a first aspect of the present disclosure, a task scheduling method is provided, including: receiving a task instruction; judging a task type of a target task corresponding to the task instruction based on parameters of the task instruction; the task types include a first task type and a second task type, and the priority of the first task type is higher than the priority of the second task type; when it is judged that the task type of the target task is the first task type, submitting the task instruction to a corresponding first cluster, and the first cluster is used to execute the target task of the first task type.

[0006] According to a second aspect of the present disclosure, a task scheduling device is provided, including: a receiving module for receiving a task instruction; a judging module for judging, based on parameters of the task instruction, a task type of a target task corresponding to the task instruction; the task types include a first task type and a second task type, and the priority of the first task type is higher than the priority of the second task type; a scheduling module for submitting the task instruction to a corresponding first cluster when it is judged that the task type of the target task is the first task type, and the first cluster is used to execute the target task of the first task type.

[0007] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to perform the above-mentioned task scheduling method.

[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned task scheduling method.

[0009] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0010] In the task scheduling method provided in the example implementation of the present disclosure, a task instruction is received; based on the parameters of the task instruction, the task type of the target task corresponding to the task instruction is determined; the task type includes a first task type and a second task type, and the priority of the first task type is higher than the priority of the second task type; when it is determined that the task type of the target task is the first task type, the task instruction is submitted to the corresponding first cluster, and the first cluster is used to execute the target task of the first task type. The embodiment of the present disclosure determines the task type based on the parameters of the task instruction, and when it is determined that the target task corresponding to the task instruction belongs to the first task type with a higher priority, the task instruction is submitted to the first cluster, thereby realizing intelligent scheduling of the target task and optimizing system resource management, thereby improving the performance and stability of the system; in addition, since the first cluster is used to execute the target task of the first task type, by submitting the target task with a higher priority to the first cluster, resource snatching between high-priority tasks and low-priority tasks can be avoided, thereby improving resource utilization and system efficiency.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0013] Figure 1 A flowchart of a task scheduling method according to an exemplary embodiment of the present disclosure is shown;

[0014] Figure 2 A flowchart of another task scheduling method according to an exemplary embodiment of the present disclosure is shown;

[0015] Figure 3 A schematic block diagram of a task scheduling device according to an exemplary embodiment of the present disclosure is shown;

[0016] Figure 4A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0019] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] Related technologies use the above-mentioned distributed query engine to process big data. However, with the increase in data volume and query complexity, the existing system may have the following problems when processing the core production tasks of the service-level agreement (SLA):

[0023] (1) Insufficient performance: Existing distributed query engines often have performance bottlenecks when processing complex queries. SLA core production tasks usually require the system to return results in a relatively short period of time. However, in high-concurrency query scenarios, the response time of existing systems increases significantly, making it difficult to meet real-time requirements. Especially in big data environments, due to the large amount of data, complex data formats, and long query execution time, the system is unable to respond in a timely manner, affecting business continuity and user experience.

[0024] (2) Poor stability: Existing systems are prone to resource competition and system crashes under high load. For example, node failures, network delays, data loss, and other issues can have a serious impact on system stability. SLA core production tasks require the system to have high stability and reliability, but existing systems often lack effective fault detection and recovery mechanisms when faced with sudden high loads or abnormal situations, resulting in task interruptions or data loss, and unable to ensure business continuity.

[0025] (3) Inadequate resource management: Existing systems have deficiencies in resource management and are unable to effectively isolate and schedule tasks of different priorities. In a big data processing environment, the priorities and resource requirements of tasks vary. The competition for resources between high-priority tasks and low-priority tasks will reduce the overall efficiency of the system. Existing systems usually adopt simple resource allocation strategies that cannot be dynamically adjusted according to the importance and urgency of tasks, resulting in low resource utilization and failure to fully utilize system performance.

[0026] In order to solve the above problems existing in the related art, the embodiments of the present disclosure provide a task scheduling method, device, electronic device and computer-readable storage medium. The following describes the scheme of the present disclosure with reference to the accompanying drawings for illustration.

[0027] The present disclosure first provides a task scheduling method. Figure 1 As shown, the task scheduling method specifically includes the following steps:

[0028] S110: receiving task instructions;

[0029] S120: Based on the parameters of the task instruction, determine the task type of the target task corresponding to the task instruction; the task type includes a first task type and a second task type, and the priority of the first task type is higher than the priority of the second task type;

[0030] S130: When it is determined that the task type of the target task is the first task type, the task instruction is submitted to the corresponding first cluster, and the first cluster is used to execute the target task of the first task type.

[0031] In the task scheduling method provided in the embodiment of the present disclosure, the task type is determined based on the parameters of the task instruction. When it is determined that the target task corresponding to the task instruction belongs to the first task type with a higher priority, the task instruction is submitted to the first cluster, thereby realizing intelligent scheduling of the target task and optimizing system resource management, thereby improving the performance and stability of the system. In addition, since the first cluster is used to execute the target task of the first task type, by submitting the target task with a higher priority to the first cluster, resource competition between high-priority tasks and low-priority tasks can be avoided, thereby improving resource utilization and system efficiency.

[0032] Next, in another embodiment, the above steps are described in more detail.

[0033] In step S110 , a task instruction is received.

[0034] The task scheduling method provided in the embodiment of the present disclosure is used to optimize the distributed query engine. Exemplarily, the above-mentioned distributed query engine can be a distributed SQL query engine (Presto), which is an open source online analytical processing (OLAP) query engine for high-speed query of large-scale distributed data sets. Presto itself does not store data, but can access multiple data sources and support cascade queries across data sources. In addition, Presto is based on memory operations, which is very fast and has high real-time performance.

[0035] In the disclosed embodiment, the task instruction may be a data query instruction submitted by a business party, wherein the business party is a user or user group having a data query requirement.

[0036] In the disclosed embodiment, the task instruction can be input by the business party through an interactive window, and the task instruction can be submitted to the corresponding cluster based on the subsequent analysis of the task instruction, so that the selected cluster executes the task corresponding to the task instruction. Exemplarily, the interactive window can be an interactive window between the business party and the distributed query engine client, specifically, it can be an interactive window between the business party and the Presto client.

[0037] In the disclosed embodiment, the business party can manually input SQL query instructions through the Presto client interactive window, or input SQL query instructions in a file in the form of Presto-f. In addition, the Java program can be directly called in the form of Presto-jdbc to directly connect to the cluster.

[0038] In step S120, based on the parameters of the task instruction, the task type of the target task corresponding to the task instruction is determined; the task type includes a first task type and a second task type, and the priority of the first task type is higher than the priority of the second task type.

[0039] In the disclosed embodiment, the parameters of the task instruction include a machine identifier and a user group identifier.

[0040] In the embodiment of the present disclosure, the above machine identification is used to indicate the machine that submitted the task instruction. Exemplarily, based on the machine identification, the type of the machine that submitted the task instruction can be determined. For example, the machine that submitted the task instruction can be a production machine or a test machine.

[0041] In the embodiment of the present disclosure, the production machine is a machine used only for executing production tasks, and the test machine is a machine used only for executing test tasks. Moreover, the priority of production tasks is usually higher than that of test tasks. For example, the SLA core production tasks belong to production tasks.

[0042] In the embodiment of the present disclosure, the user group identity identifier is used to indicate the current identity of the user group that submits the task instruction, wherein the user group belongs to the business party.

[0043] In an embodiment of the present disclosure, the identity of the user group includes a first identity and a second identity, and different identities are used to submit task instructions of different task types. Among them, the task types may include a first task type and a second task type, and the priority of the first task type is higher than the priority of the second task. Exemplarily, the user group can submit a task instruction of the first task type through the first identity, and submit a task instruction of the second task type through the second identity. That is, the first identity of the user group is used to submit a task instruction with a higher priority, and the second identity of the user group is used to submit a task instruction with a lower priority.

[0044] In the embodiment of the present disclosure, the first task type may be a generation task type, and the second task type may be other tasks except the generation task. For example, the second task type may be a test task type.

[0045] In the disclosed embodiment, the target task is a task corresponding to the task instruction. For example, when the task type is the first task type, the target task may be a production task, which is submitted by the user group as the first identity through the production machine; when the task type is the second task type, the target task may be a test task, which is submitted by the user group as the second identity through the test.

[0046] In the disclosed embodiment, the above-mentioned task type of the target task corresponding to the task instruction is determined based on the parameters of the task instruction. Exemplarily, it can be implemented as follows: determine the machine type of the machine that submits the task instruction based on the machine identifier; determine the current identity of the user group based on the user group identity identifier; when the machine type is the target machine type and the current identity of the user group is the first identity, determine that the task type of the target task corresponding to the task instruction is the first task type. Specifically, the above-mentioned target machine type can be a production machine, and the above-mentioned first task type can be a production task. Then the above-mentioned process can be implemented as follows: when it is determined that the machine type corresponding to the task instruction is a production machine, and the identity of the user group that submits the task instruction is the first identity, determine that the target task corresponding to the task instruction is a production task.

[0047] In the disclosed embodiment, before determining the task type of the target task corresponding to the task instruction based on the parameters of the task instruction, it is also possible to determine whether the target task corresponding to the current task instruction is connected to the task isolation mode by the following steps: turning on the task isolation mode; determining whether the user group is connected to the task isolation mode. Specifically, the process can be implemented as follows: in response to the operation of turning on the task isolation mode, turning on the task isolation mode; determining whether the user group corresponding to the task instruction is connected to the task isolation mode; when it is determined that the user group is connected to the task isolation mode, determining whether the switch of the task isolation mode is closed; when it is determined that the switch of the task isolation mode is in a closed state, determining whether the user group identity is connected to the task isolation mode, if it is determined that the user group identity is connected to the task isolation mode, then determining that the target task corresponding to the current task instruction is in the task isolation mode, that is, the target task can be executed in the task isolation mode.

[0048] In the disclosed embodiment, the task isolation mode refers to a mode for isolating and executing target tasks of different types. Exemplarily, in this task isolation mode, the execution of a target task of a first task type can be isolated from the execution of a target task of a second task type. Specifically, the isolation of the execution process of target tasks of different types can be achieved by executing target tasks of corresponding types through different clusters, for example, executing production tasks through a production cluster and executing other tasks including test tasks through a test cluster.

[0049] In the embodiments of the present disclosure, a user management interface may also be provided, and controls for managing the above-mentioned task isolation mode may be set on the user management interface. The above-mentioned response to the operation of starting the task isolation mode, to start the task isolation mode, may be implemented as follows: in response to a triggering operation such as a click or long press by the business party (user group) on the above-mentioned task isolation control, to start the task isolation mode.

[0050] In the disclosed embodiment, when determining whether the task isolation mode is turned on, it is possible to further determine whether the user group is connected to the task isolation mode to verify that the current user group has the authority to turn on the task isolation mode; when it is verified that the current user group has the authority to turn on the task mode, and it is determined that the current user group has been connected to the task isolation mode, it is also necessary to determine whether the switch controlling the task isolation mode is closed. If it is closed, continue to determine whether the current identity of the user group is connected to the task isolation mode. For example, if the current identity of the user group is the first identity, determine whether the first identity of the user group is connected to the task isolation mode. If so, enter the task isolation mode.

[0051] In the disclosed embodiment, by enabling the task isolation mode, resource contention between high-priority tasks and low-priority tasks can be avoided, ensuring that high-priority tasks can be executed smoothly. For example, under normal circumstances, the priority of the above-mentioned production tasks is usually higher than other tasks, so the production tasks can be executed separately through the production cluster to ensure that the resources of the high-priority tasks are met.

[0052] In step S130 , when it is determined that the task type of the target task is the first task type, the task instruction is submitted to the corresponding first cluster, and the first cluster is used to execute the target task of the first task type.

[0053] In the embodiment of the present disclosure, the first cluster is a cluster that executes a target task of the first task type. Exemplarily, the first cluster may be a cluster composed of multiple electronic devices such as servers.

[0054] In the embodiment of the present disclosure, the first task type may be a production task type, and the first cluster may be a production cluster, which is used to execute the production task.

[0055] In the disclosed embodiment, in addition to the production cluster, other clusters such as a research cluster and a backtracking cluster may also be included. Among them, the above-mentioned production cluster and backtracking cluster belong to the first cluster, and the production cluster is used to perform routine tasks, and the backtracking cluster is used to perform backtracking tasks. Exemplarily, the backtracking task is a production task that needs to be performed multiple times in a short period of time, and the routine task is a production task that is performed periodically once, such as once a day. The above-mentioned research cluster is a cluster other than the first cluster, which is used to perform test tasks or research tasks.

[0056] In the embodiment of the present disclosure, when it is determined that the task type of the target task is the first task type, the task instruction is submitted to the corresponding first cluster. Exemplarily, it can be implemented as follows: when it is determined that the task type of the target task is the first task type, it is determined whether the target has been successfully registered; if it is determined that the target instruction has been successfully registered, the target task is submitted to the first cluster; if it is determined that the target instruction has not been successfully registered, the target task is submitted to the first cluster based on a pre-set whitelist. Specifically, the submission of the target task to the first cluster based on a pre-set whitelist can be implemented as follows: it is determined whether the user group that submits the task instruction is on the whitelist; wherein the whitelist is a pre-set list of user groups; when it is determined that the user group is on the whitelist, the target task is submitted to the first cluster.

[0057] In the disclosed embodiment, the above-mentioned task name can be managed by a registration center, which is a platform for managing task names in a system that executes the above-mentioned task scheduling method. The above-mentioned determination of whether the task name of the target task has been successfully registered can be implemented as follows: determining whether the target task has been registered in the registration center. If the target task has been registered in the registration center, it is determined that the target task has been successfully registered.

[0058] In the disclosed embodiment, the whitelist is a pre-set user group list. Tasks submitted by user groups in the whitelist can be submitted to the first cluster for processing even if the task names are not registered in the registration center.

[0059] In the disclosed embodiment, the production cluster is provided with an SLA queue and a non-SLA queue, wherein the SLA queue is a task queue composed of SLA core production tasks in the production tasks, and the non-SLA queue is a task queue composed of other production tasks in the production tasks.

[0060] In the embodiment of the present disclosure, the first cluster includes a core sub-cluster and a non-core sub-cluster, the target task executed by the core sub-cluster has a higher priority than the non-core sub-cluster, and the core sub-cluster and the non-core sub-cluster are physically isolated. Exemplarily, the physical isolation can be achieved by hardware isolation.

[0061] In the embodiment of the present disclosure, the core sub-cluster can be used to execute the SLA core production tasks, that is, the tasks in the SLA queue. The non-core sub-cluster can be used to execute other production tasks, that is, the tasks in the non-SLA queue. Exemplarily, the performance of the core sub-cluster is better than that of the non-core sub-cluster.

[0062] In the embodiment of the present disclosure, the task instruction is submitted to the corresponding first cluster. Exemplarily, it can be implemented as follows: determine whether the target task is a preset task; determine whether the target task is a preset task; submit the target task to the corresponding sub-cluster of the first cluster based on the judgment result. Specifically, when it is determined that the target task is a preset task, the target task is submitted to the core sub-cluster; when it is determined that the target task is not a preset task, the target task is submitted to the non-core sub-cluster. Specifically, the preset task can be an SLA core production task. After the first cluster receives the submitted target task, when it is determined that the target task is an SLA core production task, the SLA core production task is submitted to the SLA queue. When the core sub-cluster has computing resources, the SLA core production task is read from the SLA queue according to the queue order and submitted to the core sub-cluster so that the SLA core production task is executed through the core sub-cluster; when the target task is determined to be other production tasks, the production task is submitted to the non-SLA queue. When the non-core sub-cluster has computing resources, the production task is read from the non-SLA queue according to the queue order and submitted to the non-core sub-cluster so that the SLA core production task is executed through the non-core sub-cluster.

[0063] In the disclosed embodiment, by physically isolating different task queues in a production cluster, refined management of resources is achieved, resource contention between high-priority tasks and low-priority tasks is avoided, and resource utilization and system efficiency are improved.

[0064] In the task scheduling method provided in the embodiment of the present disclosure, the task type is determined based on the parameters of the task instruction. When it is determined that the target task corresponding to the task instruction belongs to the first task type with a higher priority, the task instruction is submitted to the first cluster, thereby realizing intelligent scheduling of the target task and optimizing system resource management, thereby improving the performance and stability of the system. In addition, since the first cluster is used to execute the target task of the first task type, by submitting the target task with a higher priority to the first cluster, resource competition between high-priority tasks and low-priority tasks can be avoided, thereby improving resource utilization and system efficiency.

[0065] In another embodiment, the above task scheduling method is described in detail, such as Figure 2 As shown, the task scheduling method includes the following steps:

[0066] S201: Receive task instructions.

[0067] In the disclosed embodiment, the above may be a Presto data query instruction submitted by the business party. The business party is a user or user group with data query requirements. The business party can manually input SQL query instructions through the Presto client interactive window, or input SQL query instructions in a file in the form of Presto-f. In addition, the Java program can be directly called in the form of Presto-jdbc to directly connect to the cluster.

[0068] S202: Determine whether the task isolation mode is enabled.

[0069] In the disclosed embodiment, the above is a mode for isolating and executing different types of target tasks. Exemplarily, in this task isolation mode, different clusters can be used to execute tasks of different priorities. Specifically, the above clusters may include a production cluster, a backtracking cluster, and a research cluster.

[0070] S203: Determine whether the user group is in task isolation mode.

[0071] In the embodiment of the present disclosure, the user group (rcc) is the service party that initiates the task instruction. After the task isolation mode is turned on, it is necessary to further determine whether the user group is connected to the task isolation mode.

[0072] S204: Determine whether the switch of the task isolation mode is turned on.

[0073] In the embodiment of the present disclosure, after determining that the user group accesses the task isolation mode, it is necessary to further determine whether the switch of the task isolation mode is turned on. If it is turned on, continue to step S205; if not, continue to step S207.

[0074] S205: Determine whether the current identity of the user group is in task isolation mode.

[0075] In the disclosed embodiment, the identity of the user group includes a first identity and a second identity, and different identities are used to submit task instructions of different task types. Exemplarily, the first identity of the user group is used to submit a task instruction with a higher priority, and the second identity of the user group is used to submit a task instruction with a lower priority. Specifically, the first identity of the user group is used to submit a production task, and the second identity of the user group is used to submit other tasks. If yes, continue to execute S206, otherwise, continue to execute S207.

[0076] S206: Determine the task priority based on the parameters of the task instruction.

[0077] In the disclosed embodiment, the parameters of the above-mentioned task instruction include rcc (identity of the user group) and ip (address of the machine that submitted the task instruction). The specific implementation of the above-mentioned task priority determination based on the parameters of the task instruction can be as follows: if the parameters of the task instruction are production rcc and production ip, the current task is determined to be a first-level task (corresponding to the above-mentioned first task), and step S207 is continued; if the parameters of the task instruction are development rcc and development ip or other parameters, step S210 is continued. Among them, the production ip is the address of the machine used for production, and the development ip is the address of the machine used for development.

[0078] S207: Determine whether the task name corresponding to the task instruction is registered.

[0079] In the disclosed embodiment, if it is determined that the task name has been successfully registered and the corresponding SQL parameters have also been registered normally, the current task is determined to be a first-level task, the priority of a first-level task is higher than other tasks, and S209 is continued to be executed; otherwise, S208 is continued to be executed.

[0080] S208: Determine whether the user group belongs to the whitelist.

[0081] In the disclosed embodiment, if it is determined that the user group belongs to the whitelist, the corresponding task is determined as a first-level task, and S209 is continued to be executed; otherwise, S210 is continued to be executed.

[0082] S209: Determine whether the first-level task is a routine task or a backtracking task.

[0083] In the embodiment of the present disclosure, if it is determined that the first-level task is a routine task, then S211 is continued to be executed; if it is determined that the first-level task is a backtracking task, then S212 is continued to be executed.

[0084] S210: Submit the task to the survey cluster.

[0085] S211: Submit the task to the production cluster.

[0086] In the disclosed embodiment, the production cluster includes an SLA queue and a non-SLA queue. The SLA core production tasks are scheduled to the SLA queue, and other tasks except the SLA core production tasks are scheduled to the non-SLA queue, and the tasks in the SLA queue and the non-SLA queue are executed by physical isolation.

[0087] S212: Submit the task to the backtracking cluster.

[0088] In the disclosed embodiment, when the number of first-level tasks is large and the equipment resources of the production cluster are insufficient to process the current tasks, some equipment can be dispatched from the research cluster to process the first-level tasks. Exemplarily, the process can be implemented as follows: obtaining the equipment utilization rate of the production cluster; when the equipment utilization rate exceeds a preset threshold, dispatching some equipment from the research cluster, and the equipment retrieved from the research cluster is used to execute the first-level tasks.

[0089] pass Figure 2 The task scheduling method shown can achieve the following functions:

[0090] (1) Improving system performance: By optimizing the execution efficiency of the Presto query engine and combining it with an intelligent scheduling mechanism, the system's performance in high-concurrency query scenarios is improved, ensuring that SLA core production tasks can be completed within the specified time.

[0091] (2) Enhance system stability: By dynamically monitoring the system status and adjusting resource allocation and task scheduling in a timely manner, the risk of resource competition and system crash is reduced, thereby improving the overall stability of the system.

[0092] (3) Optimizing resource management: By physically isolating different task queues in the production cluster, refined resource management is achieved, avoiding resource competition between high-priority tasks and low-priority tasks, and improving resource utilization and system efficiency.

[0093] (4) Intelligent scheduling: Through task analysis and classification, resource monitoring and prediction, and scheduling strategy formulation and execution, SLA tasks are intelligently scheduled to fixed production clusters, ensuring that task priorities and resource requirements are met, thereby improving the system's service quality.

[0094] The above mainly introduces the solution provided by the embodiment of the present invention from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0095] The embodiment of the present invention can divide the task scheduling device into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0096] The present disclosure provides a task scheduling device. Figure 3 As shown, the task scheduling device 300 may include a receiving module 310, a judging module 320 and a scheduling module 330, wherein:

[0097] The receiving module 310 may be used to receive a task instruction;

[0098] The judgment module 320 may be used to judge the task type of the target task corresponding to the task instruction based on the parameters of the task instruction; the task type includes a first task type and a second task type, and the priority of the first task type is higher than the priority of the second task type;

[0099] The scheduling module 330 may be used to submit the task instruction to the corresponding first cluster when it is determined that the task type of the target task is the first task type, and the first cluster is used to execute the target task of the first task type.

[0100] In the disclosed embodiment, the parameters of the task instruction include a machine identifier and a user group identifier. The machine identifier is used to indicate the machine that submits the task instruction. The user group identifier is used to indicate the current identity of the user group that submits the task instruction. The identity of the user group includes a first identity and a second identity. The first identity is used to submit a target task of a first task type, and the second identity is used to submit a target task of a second task type.

[0101] In the disclosed embodiment, the task scheduling device may further include an enabling module, which is specifically used to: enable the task isolation mode in response to an operation to enable the task isolation mode; determine whether the user group corresponding to the task instruction is connected to the task isolation mode; when it is determined that the user group is connected to the task isolation mode, determine whether the switch of the task isolation mode is closed; when it is determined that the switch of the task isolation mode is in a closed state, determine whether the user group identity is connected to the task isolation mode.

[0102] In the embodiment of the present disclosure, the above-mentioned judgment module is specifically used to: determine the machine type of the machine that submits the task instruction based on the machine identification; determine the current identity of the user group based on the user group identity identification; when the machine type is the target machine type and the current identity of the user group is the first identity, judge that the task type of the target task corresponding to the task instruction is the first task type.

[0103] In the embodiment of the present disclosure, the above-mentioned scheduling module is specifically used for: when it is determined that the task type of the target task is the first task type, determining whether the task name of the target task has been successfully registered; if it is determined that the task name of the target instruction has been successfully registered, submitting the target task to the first cluster; if it is determined that the task name corresponding to the target instruction has not been successfully registered, determining whether the user group submitting the task instruction is on the whitelist; wherein the whitelist is a pre-set list of user groups; when it is determined that the user group is on the whitelist, submitting the target task to the first cluster.

[0104] In the embodiment of the present disclosure, the first cluster includes a core sub-cluster and a non-core sub-cluster. The priority of the target task executed by the core sub-cluster is higher than that of the non-core sub-cluster, and the core sub-cluster and the non-core sub-cluster are physically isolated.

[0105] In the embodiment of the present disclosure, the above-mentioned scheduling module is specifically used to: determine whether the target task is a preset task; when it is determined that the target task is a preset task, submit the target task to the core sub-cluster; when it is determined that the target task is not a preset task, submit the target task to the non-core sub-cluster.

[0106] In addition, the specific implementation details of the above-mentioned task scheduling device have been described in detail in the corresponding position of the task scheduling method, so they will not be repeated here.

[0107] Figure 4 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 4 , which shows a structural schematic diagram of an electronic device 400 suitable for implementing the embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0108] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403 to implement the task scheduling method of the embodiment described in the present disclosure. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0109] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0110] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the task scheduling method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0111] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0112] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0113] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0114] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0115] Receive task instructions;

[0116] Based on the parameters of the task instruction, determine the task type of the target task corresponding to the task instruction; the task type includes a first task type and a second task type, and the priority of the first task type is higher than the priority of the second task type;

[0117] When it is determined that the task type of the target task is the first task type, the task instruction is submitted to the corresponding first cluster, and the first cluster is used to execute the target task of the first task type.

[0118] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0119] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0120] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0122] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0123] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0125] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0126] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A task scheduling method, characterized in that: The method comprises: Receive task instructions; Based on the parameters of the task instruction, determining the task type of the target task corresponding to the task instruction; the task type includes a first task type and a second task type, and the priority of the first task type is higher than the priority of the second task type; When it is determined that the task type of the target task is the first task type, the task instruction is submitted to a corresponding first cluster, and the first cluster is used to execute the target task of the first task type.

2. The task scheduling method according to claim 1, characterized in that: The parameters of the task instruction include a machine identifier and a user group identifier. The machine identifier is used to indicate the machine that submits the task instruction. The user group identifier is used to indicate the current identity of the user group that submits the task instruction. The identity of the user group includes a first identity and a second identity. The first identity is used to submit the target task of the first task type, and the second identity is used to submit the target task of the second task type.

3. The task scheduling method according to claim 2, characterized in that: Before determining the task type of the target task corresponding to the task instruction based on the parameters of the task instruction, the method further includes: Turning on the task isolation mode; Determine whether the user group is connected to the task isolation mode.

4. The task scheduling method according to claim 3, characterized in that: The determining, based on the parameters of the task instruction, the task type of the target task corresponding to the task instruction includes: Determining the machine type of the machine that submitted the task instruction based on the machine identification; Determining a current identity of the user group based on the user group identity identifier; When the machine type is the target machine type and the current identity of the user group is the first identity, it is determined that the task type of the target task corresponding to the task instruction is the first task type.

5. The task scheduling method according to claim 1, characterized in that: When it is determined that the task type of the target task is the first task type, submitting the task instruction to the corresponding first cluster includes: When it is determined that the task type of the target task is the first task type, determining whether the target task has been successfully registered; If it is determined that the target instruction has been successfully registered, submitting the target task to the first cluster; If it is determined that the target instruction is not successfully registered, the target task is submitted to the first cluster based on a preset whitelist.

6. The task scheduling method according to claim 1, characterized in that: The first cluster includes a core sub-cluster and a non-core sub-cluster. The priority of the target task executed by the core sub-cluster is higher than that of the non-core sub-cluster. The core sub-cluster and the non-core sub-cluster are physically isolated.

7. The task scheduling method according to claim 6, characterized in that: Submitting the task instruction to the corresponding first cluster includes: Determine whether the target task is a preset task; The target task is submitted to a corresponding subcluster of the first cluster based on the judgment result; the subcluster is the core subcluster or the non-core subcluster.

8. A task scheduling device, characterized in that: The device comprises: A receiving module, used for receiving task instructions; A judgment module, used for judging the task type of the target task corresponding to the task instruction based on the parameters of the task instruction; the task type includes a first task type and a second task type, and the priority of the first task type is higher than the priority of the second task type; The scheduling module is used for submitting the task instruction to the corresponding first cluster when it is determined that the task type of the target task is the first task type, and the first cluster is used for executing the target task of the first task type.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images