Task processing method, task scheduling method, computing device, and computer storage medium

Abstract

Embodiments of the present disclosure provide a task processing method, a task scheduling method, a computing device, and a computer storage medium. The task processing method comprises: receiving a task processing request; in response to the task processing request, creating a container group on the basis of the resources required by a task; detecting whether the residual resources of a container management system meet the operation resource requirements of the container group; and if it is determined that the residual resources of the container management system meet the operation resource requirements of the container group, sending the container group to a scheduling queue of the container management system, so as to schedule the container group to a node of the container management system and process the task processing request. The technical solution provided by the embodiments of the present disclosure can avoid the processing failure of the task processing request, and improve the task processing efficiency.

Description

Task processing, task scheduling method, computing device and computer storage medium

[0001] This disclosure claims priority to the Chinese patent application filed with the China Patent Office on September 7, 2023, with application number 202311153677.4 and application name “Task processing, task scheduling method, computing device and computer storage medium”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a task processing and task scheduling method, a computing device, and a computer storage medium. Background Art

[0003] Database systems, such as data warehouses, are used to integrate, store, and analyze large amounts of structured and unstructured data. To improve database system availability and enable elastic scaling based on load, they can be deployed on container management systems, such as Kubernetes, to store the data they need to manage.

[0004] In related technologies, after receiving a task processing request from a user, the database system typically submits the task processing request to the container management system. The container management system then determines whether its currently available resources meet the requirements for processing the task processing request. If so, a container group is created to handle the task processing request and placed in a scheduling queue, awaiting scheduling to a node. If the currently available resources do not meet the requirements for processing the task processing request, the container management system intercepts the task processing request, preventing the container group from being created and causing the task processing to fail.

[0005] Public content

[0006] The embodiments of the present disclosure provide a task processing and task scheduling method, apparatus, computing device, and computer storage medium.

[0007] In a first aspect, an embodiment of the present disclosure provides a task processing method, including:

[0008] Receive task processing request;

[0009] In response to the task processing request, creating a container group based on resources required by the task;

[0010] Detecting whether the remaining resources of the container management system meet the operating resource requirements of the container group;

[0011] When it is determined that the remaining resources of the container management system meet the running resource requirements of the container group, the container group is sent to the scheduling queue of the container management system so that the container group is scheduled to a node of the container management system to run and process the task processing request.

[0012] In a second aspect, an embodiment of the present disclosure provides a task scheduling method, including:

[0013] receiving, using a scheduling queue, a container group, the container group being created in response to a task processing request and based on resources required by the task, and being sent to the scheduling queue upon detecting that remaining resources of the container management system satisfy the running resource requirements of the container group;

[0014] The container group is scheduled to a target node, so that the container group runs on the target node and processes the task processing request.

[0015] In a third aspect, an embodiment of the present disclosure provides a task processing device, including:

[0016] Request receiving module, used to receive task processing requests;

[0017] a container group creation module, configured to create a container group based on resources required by the task in response to the task processing request;

[0018] A detection module, configured to detect whether the remaining resources of the container management system meet the operating resource requirements of the container group;

[0019] The container group sending module is configured to send the container group to a scheduling queue of the container management system when it is determined that the remaining resources of the container management system meet the operating resource requirements of the container group, so as to schedule the container group to a node of the container management system and process the task processing request.

[0020] In a fourth aspect, an embodiment of the present disclosure provides a task scheduling device, including:

[0021] a container group receiving module, configured to receive a container group using a scheduling queue, the container group being created in response to a task processing request based on resources required by the task, and sending the container group to the scheduling queue upon detecting that remaining resources of the container management system meet the running resource requirements of the container group;

[0022] The container group scheduling module is used to schedule the container group to a target node, so that the container group runs on the target node and processes the task processing request.

[0023] In a fifth aspect, an embodiment of the present disclosure provides a computing device, including a processing component and a storage component;

[0024] The storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the task processing method as described in the first aspect, or to implement the task scheduling method as described in the second aspect.

[0025] In a sixth aspect, an embodiment of the present disclosure provides a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements the task processing method as described in the first aspect, or implements the task scheduling method as described in the second aspect.

[0026] In a seventh aspect, an embodiment of the present disclosure provides a computer program product, comprising a computer program, which, when executed by a computer, implements the task processing method as described in the first aspect, or implements the task scheduling method as described in the second aspect.

[0027] An embodiment of the present disclosure provides a task processing method, which adopts the following technical solutions: receiving a task processing request; creating a container group based on resources required for the task in response to the task processing request; detecting whether the remaining resources of the container management system meet the operating resource requirements of the container group; and sending the container group to a scheduling queue of the container management system when it is determined that the remaining resources of the container management system meet the operating resource requirements of the container group, so as to schedule the container group to a node of the container management system and process the task processing request. A queuing queue is constructed outside the container management system, and the task processing request is first queued in the queuing queue. When it is detected that the remaining resources of the container management system meet the resources required to process the task processing request, the container group used for the task processing request is sent to the container management system, and the container management system schedules the container group, thereby avoiding processing failure of the task processing request and improving task processing efficiency.

[0028] These and other aspects of the present disclosure will become more apparent from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] FIG1 schematically shows a general architecture diagram of the container management system Kubernetes;

[0031] FIG2 schematically shows a flow chart of a task processing method provided by an embodiment of the present disclosure;

[0032] FIG3 schematically shows a diagram of a task processing method provided by an embodiment of the present disclosure;

[0033] FIG4 schematically shows a flow chart of a task scheduling method provided by an embodiment of the present disclosure;

[0034] FIG5 schematically shows a block diagram of a task processing device provided by an embodiment of the present disclosure;

[0035] FIG6 schematically shows a block diagram of a task scheduling device provided by an embodiment of the present disclosure;

[0036] FIG7 schematically shows a block diagram of a computing device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present disclosure.

[0038] In some of the processes described in the specification and claims of the present disclosure and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit "first" and "second" to be different types.

[0039] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0040] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0041] FIG1 schematically shows a general architecture diagram of the container management system Kubernetes.

[0042] As shown in Figure 1, Kubernetes divides the nodes in the cluster into a master node (Master) and at least one worker node (Node). The Master and Node can be physical machines or virtual machines in a cloud environment. The cloud environment is specifically a computing cluster that includes at least one cloud computing device (such as a central server). In some implementations, the Master and Node can also be physical machines or virtual machines in an edge environment. The edge environment is specifically a computing cluster that includes at least one edge computing device (such as an edge server). In other implementations, the Master and Node can also be end devices or virtual machines on end devices. It should be noted that the Master and Node can be physical machines or virtual machines in the same environment, or they can be physical machines or virtual machines in different environments. For example, the Master can be a physical machine in a cloud environment, and the Node can be a physical machine in an edge environment.

[0043] The Master runs a set of processes related to cluster management, such as the kube-apiserver, kube-controller-manager, kube-scheduler, and etcd. The kube-apiserver relies on etcd for data storage. These processes enable the Master to manage resources, pod scheduling, elastic scaling, security controls, and error correction for the entire cluster.

[0044] Nodes run application processes, providing them with the resources they need to run. Application processes run in pod containers. Nodes also run service processes, such as the kubelet and network proxy (kube-proxy). Nodes use these service processes to create, start, restart, terminate, and load balance pods.

[0045] Figure 1 illustrates a Kubernetes cluster consisting of one master and one node. In some implementations, a Kubernetes cluster can include multiple nodes, allowing applications to be deployed across multiple nodes in a distributed manner. Furthermore, a Kubernetes cluster can also include multiple masters. If one master fails, another master can be activated, ensuring high availability.

[0046] Database systems, such as data warehouses, are used to integrate, store, and analyze large amounts of structured and unstructured data. To improve database system availability and enable elastic scaling based on load, they can be deployed on container management systems, such as Kubernetes, to store the data they need to manage.

[0047] In related technologies, after receiving a task processing request from a user, the database system typically submits the task processing request to the container management system. The container management system then determines whether its currently available resources meet the requirements for processing the task processing request. If so, a container group is created to handle the task processing request and placed in a scheduling queue, awaiting scheduling to a node. If the currently available resources do not meet the requirements for processing the task processing request, the container management system intercepts the task processing request, preventing the container group from being created and causing the task processing to fail.

[0048] In order to solve the technical problems existing in the related art, the embodiment of the present disclosure provides a task processing method, which adopts the following methods: receiving a task processing request; creating a container group based on the resources required for the task in response to the task processing request; detecting whether the remaining resources of the container management system meet the operating resource requirements of the container group; and sending the container group to the scheduling queue of the container management system when it is determined that the remaining resources of the container management system meet the operating resource requirements of the container group, so as to schedule the container group to a node of the container management system and process the task processing request. A queuing queue is constructed outside the container management system, and the task processing request is first queued in the queuing queue. When it is detected that the remaining resources of the container management system meet the resources required to process the task processing request, the container group used for the task processing request is sent to the container management system, and the container management system schedules the container group, thereby avoiding the processing failure of the task processing request and improving the task processing efficiency.

[0049] FIG2 schematically shows a flow chart of a task processing method provided by an embodiment of the present disclosure. As shown in FIG2 , the task processing method may include the following steps:

[0050] 201, receiving a task processing request;

[0051] 202, in response to the task processing request, creating a container group based on resources required by the task;

[0052] 203, checking whether the remaining resources of the container management system meet the operating resource requirements of the container group;

[0053] 204 , when it is determined that the remaining resources of the container management system meet the running resource requirements of the container group, the container group is sent to the scheduling queue of the container management system so that the container group is scheduled to a node of the container management system and the task processing request is processed.

[0054] The task processing method provided by the embodiment of the present disclosure can be executed by a database system.

[0055] According to an embodiment of the present disclosure, a task processing request may include, for example, a big data (BigData) analysis and processing task, an artificial intelligence processing task, and the like.

[0056] According to an embodiment of the present disclosure, a task processing request may carry a resource requirement, which may refer to the amount of resources consumed to process the task processing request. Resources may include, for example, computing resources, storage resources, and the like.

[0057] According to an embodiment of the present disclosure, after obtaining a task processing request, the database system may obtain the resource requirements carried in the task processing request from the task processing request, and create a container group according to the resource requirements.

[0058] According to an embodiment of the present disclosure, a container group has an operating resource requirement. The operating resource requirement may be the amount of resources that the container management system needs to provide when the container group is running. The operating resource requirement may match the resource requirement. For example, the operating resource requirement may be greater than or equal to the resource requirement.

[0059] According to an embodiment of the present disclosure, after a container group is created, the container group may be first placed in a queue of a database system, and then the current remaining amount of resources of the container management system may be obtained.

[0060] According to an embodiment of the present disclosure, when it is determined that the current remaining resources of the container management system are greater than the resource operation requirements of the container group, it indicates that the remaining resources of the container management system can support the operation of the container group. At this time, the container group can be dequeued, that is, the container group can be sent to the scheduling queue of the container management system.

[0061] According to an embodiment of the present disclosure, after a container group is sent to a scheduling queue of a container management system, the container management system can execute its scheduling policy and schedule the container group to a working node in the container management system. The working node provides resources for the container group so that the container group can run to process task processing requests.

[0062] According to an embodiment of the present disclosure, after creating a container group based on the resources required by the task, the task processing method further includes:

[0063] The generated container group is placed in the queue of the database system.

[0064] According to an embodiment of the present disclosure, detecting whether the remaining resources of the container management system meet the operating resource requirements of the container group can be specifically implemented as follows:

[0065] Check the queue to see if there is a waiting container group currently waiting to be scheduled, and the creation time of the waiting container group is earlier than that of the container group.

[0066] If there is a waiting container group currently waiting for scheduling, determine the first scheduling priority of the waiting container group and the second scheduling priority of the container group;

[0067] When the second scheduling priority is higher than the first scheduling priority, it is detected whether the remaining resources of the container management system meet the running resource requirements of the container group.

[0068] According to an embodiment of the present disclosure, there may be multiple container groups in a queue, and each container group may correspond to a different task processing request.

[0069] According to an embodiment of the present disclosure, each container group can be configured with a different priority. A container group with a high priority can be allocated resources first, dequeued first, and sent to a scheduling queue of a container management system for scheduling and operation.

[0070] According to an embodiment of the present disclosure, the priority of a container group may be determined by the priority of a task processing request.

[0071] According to an embodiment of the present disclosure, if the first scheduling priority is higher than the second scheduling priority, running resources can be allocated to the waiting container group first; if the first scheduling priority is equal to the second scheduling priority, running resources can be allocated to the waiting container group first according to the first-in-first-out rule.

[0072] According to an embodiment of the present disclosure, the task processing method further includes:

[0073] A first resource that meets the running resource requirement is allocated to the container group, so that after the container group is scheduled to a node of the container management system, the container group is run using the first resource.

[0074] According to an embodiment of the present disclosure, allocating a first resource that meets the running resource requirement to a container group can be achieved by marking the first resource as belonging to the container group in the container management system without temporarily performing actual resource configuration. After the container group is scheduled to a node, the container group is run using the first resource marked as belonging to the container.

[0075] According to an embodiment of the present disclosure, the task processing method further includes:

[0076] determining a second resource remaining in the container management system;

[0077] dividing the second resource into a third resource and a fourth resource according to the first scheduling priority and the second scheduling priority respectively;

[0078] The second resource is allocated to the waiting container group, and the fourth resource is allocated to the container group.

[0079] According to an embodiment of the present disclosure, to prevent low-priority waiting container groups from waiting too long, a fair allocation strategy can be adopted during resource allocation. Specifically, the remaining secondary resources of the container management system are evenly distributed to each container group, using their respective priorities as weights.

[0080] According to an embodiment of the present disclosure, when dividing the second resource, the first scheduling priority and the second scheduling priority may be first converted into weights, and then the weights may be accumulated to obtain a weight sum.

[0081] After the total weight is calculated, the resource ratio that each container group should obtain can be obtained based on the ratio of the weight to the total weight. Therefore, the resource amount obtained by each container group can be calculated based on the resource ratio.

[0082] According to embodiments of the present disclosure, using priority as a weight can evenly distribute the remaining resource quota in the container management system. This ensures that high-priority container groups receive more resources, while ensuring that low-priority container groups are not completely ignored, thus preventing low-priority container groups from waiting for too long.

[0083] According to an embodiment of the present disclosure, in response to a task processing request, creating a container group may be specifically implemented as follows:

[0084] In response to the task processing request, multiple container groups are created.

[0085] When it is determined that the remaining resources of the container management system meet the running resource requirements of the container group, sending the container group to the scheduling queue of the container management system includes:

[0086] When it is determined that remaining resources of the container management system meet the running resource requirements of at least one container group among the multiple container groups, the at least one container group is sent to a scheduling queue of the container management system.

[0087] According to the embodiments of the present disclosure, when processing tasks related to big data, artificial intelligence, etc., the task complexity is often high, and multiple container groups need to be created to process the highly complex tasks.

[0088] According to an embodiment of the present disclosure, when a task processing request requires multiple container groups to process, the resource demand is usually high. At this time, in order to meet the operating resource requirements of all container groups, the multiple container groups corresponding to the task processing request need to queue for a long time in the queue, waiting for resource allocation, which will result in a long waiting time for task processing.

[0089] In the disclosed embodiments, when the remaining resources of the container management system meet the running resource requirements of any container group, the container group that meets the requirements can be dequeued first, allowing it to run on the scheduled node and process its portion of the processing tasks. This reduces resource waiting time, fully utilizes limited resources for task processing, and avoids resource waiting deadlock when processing tasks across multiple container groups.

[0090] According to an embodiment of the present disclosure, after creating a container group, the task processing method further includes:

[0091] Configure the pause scheduling label for the container group.

[0092] When it is determined that the remaining resources of the container management system meet the operating resource requirements of the container group, the method further includes:

[0093] Remove the pause scheduling flag configured for the container group.

[0094] The dispatch queues that send container groups to the container management system include:

[0095] In response to the pause scheduling tag being removed, the container group is sent to a scheduling queue of the container management system.

[0096] According to an embodiment of the present disclosure, after a generated container group is placed in a queue, a pause scheduling tag can be configured on the container group. When a container group is marked as paused, the container group can be placed in a cache and temporarily not participate in further scheduling.

[0097] According to an embodiment of the present disclosure, configuring a pause scheduling tag for the container group may be specifically implemented as follows:

[0098] When it is detected that the remaining resources of the container management system do not meet the running resource requirements of the container group, a pause scheduling tag is configured for the container group.

[0099] According to an embodiment of the present disclosure, a container group may be configured with a pause scheduling flag when the remaining resources of the container management system are less than the running resource requirements of the container group. However, the pause scheduling flag may also be configured when a high-priority container group with a higher priority than the container group exists in the queue.

[0100] According to an embodiment of the present disclosure, the pause scheduling tag may be removed when it is determined that the remaining resources of the container management system are greater than or equal to the running resource requirements of the container group.

[0101] FIG3 schematically shows a diagram of a task processing method provided by an embodiment of the present disclosure.

[0102] As shown in FIG3 , the task processing method provided by the embodiment of the present disclosure can be logically divided into four layers, namely, the task management layer, the operation layer, the task scheduling layer, and the container group scheduling layer.

[0103] When a user submits a task processing request in the database system console, the task management layer receives the task processing request, obtains the metadata information in the task processing request, and then converts the task into a Kubernetes Custom Resource (CR) based on the metadata and submits it to the API server. Task metadata may include information such as the task name, description, type, and resource requirements.

[0104] The task scheduling layer can detect the API-Server. When it detects the creation of a CR in the API-Server, it can create a specified number of container groups based on the metadata information contained in the CR.

[0105] The task scheduling layer can first use the scheduler to create a task group. A task group can include multiple container groups, and each task group corresponds to a task processing request. After detecting the creation of a container group, it can first configure a pause scheduling tag for the container group and send the container group to the queue.

[0106] The queuing queue first assigns the container group to the task group to which it belongs. The queuing queue then distributes the remaining resource quota of the container management system to the task groups in the queue according to a specific algorithm. There are two resource allocation algorithms: the first is a priority scheduling algorithm, which considers the priority of the task group when allocating resources. Task groups with higher priorities receive the requested resources first. When priorities are consistent, resources are allocated on a first-come, first-served basis. The second is a weighted fair scheduling algorithm, which uses each task group's priority as a weight to evenly distribute the remaining resources of the management system. After receiving resources, the task group further allocates them to the container groups within it. When a container group within a task group meets its operating resource requirements, it can be dequeued and the pause scheduling label on the pod is removed. When the resources obtained by a task group cannot meet the needs of its own container group, the container group is queued within the task group.

[0107] Container groups with the pause scheduling label can be placed in the cache first and do not participate in the scheduling policy. When the pause scheduling label is removed, the container group can be sent to the scheduling queue of the container management system to execute the scheduling logic of the container management system.

[0108] The task processing method provided by the disclosed embodiments expands upon the container management system's scheduler, encapsulating a queue layer in front of the existing scheduler. Only container groups scheduled through the queue can enter the container management system's scheduler for final scheduling. This solution can integrate with the container management system's scheduler, supporting task-level queuing and job-priority-aware resource allocation, improving resource allocation and task processing efficiency.

[0109] FIG4 schematically shows a flow chart of a task scheduling method provided by an embodiment of the present disclosure. As shown in FIG4 , the task scheduling method may specifically include the following steps:

[0110] 401 , receiving a container group using a scheduling queue, the container group being created in response to a task processing request and being sent to the scheduling queue upon detecting that remaining resources of the container management system meet the operating resource requirements of the container group;

[0111] 402. Schedule the container group to the target node so that the container group runs on the target node.

[0112] FIG5 schematically shows a block diagram of a task processing device provided by an embodiment of the present disclosure. As shown in FIG5 , the task processing device 500 may include:

[0113] Request receiving module 501, used for receiving task processing request;

[0114] A container group creation module 502 is configured to create a container group based on resources required by a task in response to a task processing request;

[0115] Detection module 503, used to detect whether the remaining resources of the container management system meet the operating resource requirements of the container group;

[0116] The container group sending module 504 is configured to send the container group to the scheduling queue of the container management system when it is determined that the remaining resources of the container management system meet the running resource requirements of the container group, so that the container group is scheduled to a node of the container management system and the task processing request is processed.

[0117] According to an embodiment of the present disclosure, the task processing device 500 further includes:

[0118] The queue writing module is used to put the created container group into the queue of the database system.

[0119] According to an embodiment of the present disclosure, the detection module includes:

[0120] The search submodule is used to find out whether there is a waiting container group waiting to be scheduled in the queue, and the creation time of the waiting container group is earlier than that of the container group;

[0121] A first determining submodule is configured to determine a first scheduling priority of the waiting container group and a second scheduling priority of the container group if there is currently a waiting container group waiting for scheduling;

[0122] The detection submodule is configured to detect whether the remaining resources of the container management system meet the operating resource requirements of the container group when the second scheduling priority is higher than the first scheduling priority.

[0123] According to an embodiment of the present disclosure, the task processing device 500 further includes:

[0124] The configuration module is configured to allocate a first resource that meets the running resource requirement to the container group, so that after the container group is scheduled to a node of the container management system, the container group is run using the first resource.

[0125] According to an embodiment of the present disclosure, the task processing device 500 further includes:

[0126] A first determining module, configured to determine remaining second resources of the container management system;

[0127] a partitioning module, configured to partition the second resource into a third resource and a fourth resource according to the first scheduling priority and the second scheduling priority;

[0128] The resource allocation module is configured to allocate the second resource to the waiting container group and allocate the fourth resource to the container group.

[0129] According to an embodiment of the present disclosure, the container group creation module 502 includes:

[0130] A container group creation submodule is used to create multiple container groups in response to task processing requests;

[0131] According to an embodiment of the present disclosure, the container group sending module 504 includes:

[0132] The first container sending submodule is configured to send the at least one container group to a scheduling queue of the container management system when it is determined that remaining resources of the container management system meet the operating resource requirements of at least one container group among the multiple container groups.

[0133] According to an embodiment of the present disclosure, the task processing device 500 further includes:

[0134] The label configuration module is used to configure the pause scheduling label for the container group;

[0135] According to an embodiment of the present disclosure, the task processing device 500 further includes:

[0136] The label removal module is used to remove the pause scheduling label configured for the container group;

[0137] According to an embodiment of the present disclosure, the container group sending module 504 includes:

[0138] The second container sending submodule is configured to send the container group to a scheduling queue of the container management system in response to the removal of the pause scheduling tag.

[0139] According to an embodiment of the present disclosure, the tag configuration module includes:

[0140] The label configuration unit is configured to configure a pause scheduling label for the container group when detecting that the remaining resources of the container management system do not meet the running resource requirements of the container group.

[0141] The task scheduling device of FIG5 can execute the task processing method of the embodiment shown in FIG1, and its implementation principle and technical effect are not described in detail here. The specific manner in which each module and unit of the task processing device in the above embodiment performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0142] FIG6 schematically shows a block diagram of a task scheduling device provided by an embodiment of the present disclosure. As shown in FIG6 , the task scheduling device 600 may include:

[0143] The container group receiving module 601 is configured to receive a container group using a scheduling queue. The container group is created in response to a task processing request based on the resources required by the task and is sent to the scheduling queue upon detecting that the remaining resources of the container management system meet the running resource requirements of the container group.

[0144] The container group scheduling module 602 is configured to schedule the container group to a target node, so that the container group runs on the target node and processes the task processing request.

[0145] The task scheduling device of FIG6 can execute the task scheduling method of the embodiment shown in FIG4, and its implementation principle and technical effect are not described in detail. The specific manner in which each module and unit performs operations in the task scheduling device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

[0146] In one possible design, the task processing apparatus and task scheduling apparatus provided by the embodiments of the present disclosure may be implemented as a computing device. As shown in FIG7 , the computing device may include a storage component 701 and a processing component 702 .

[0147] The storage component 701 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 702 to implement the task processing method and task scheduling method provided by the embodiments of the present disclosure.

[0148] Of course, a computing device may also include other components, such as input / output interfaces and communication components. The input / output interface provides an interface between the processing component and peripheral interface modules, such as output devices and input devices. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0149] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0150] When the computing device is a physical device, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device.

[0151] The embodiments of the present disclosure further provide a computer-readable storage medium storing a computer program. When the computer program is executed by a computer, the task processing method and the task scheduling method provided by the embodiments of the present disclosure can be implemented.

[0152] The embodiments of the present disclosure further provide a computer program product, including a computer program. When the computer program is executed by a computer, the task processing method and the task scheduling method provided by the embodiments of the present disclosure can be implemented.

[0153] The processing components in the above embodiments may include one or more processors to execute computer instructions to complete all or part of the steps in the above methods. Of course, the processing components may also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0154] The memory component is configured to store various types of data to support operations in the device. The memory component can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0155] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0157] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

1. A task processing method, characterized in that: Applied to a database system, the method comprises: Receive task processing request; In response to the task processing request, creating a container group based on resources required by the task; Detecting whether the remaining resources of the container management system meet the operating resource requirements of the container group; When it is determined that the remaining resources of the container management system meet the running resource requirements of the container group, the container group is sent to the scheduling queue of the container management system so as to schedule the container group to a node of the container management system and process the task processing request.

2. The method according to claim 1, characterized in that The method further comprises: Putting the created container group into a queue of the database system; The detecting whether the remaining resources of the container management system meet the operating resource requirements of the container group includes: Find out whether there is a waiting container group currently waiting for scheduling in the queue, and the creation time of the waiting container group is earlier than that of the container group; If there is currently a waiting container group waiting for scheduling, determine a first scheduling priority of the waiting container group and a second scheduling priority of the container group; When the second scheduling priority is higher than the first scheduling priority, it is detected whether the remaining resources of the container management system meet the running resource requirements of the container group.

3. The method according to claim 2, characterized in that The method further comprises: A first resource that meets the running resource requirement is allocated to the container group, so that after the container group is scheduled to a node of a container management system, the container group is run using the first resource.

4. The method according to claim 2, characterized in that: The method further comprises: determining a second resource remaining in the container management system; According to the first scheduling priority and the second scheduling priority, respectively, the second resource is divided into a third resource and a fourth resource; The second resource is allocated to the waiting container group, and the fourth resource is allocated to the container group.

5. The method according to any one of claims 1 to 4, characterized in that: In response to the task processing request, creating a container group includes: In response to the task processing request, creating a plurality of container groups; When it is determined that the remaining resources of the container management system meet the running resource requirements of the container group, sending the container group to the scheduling queue of the container management system includes: When it is determined that the remaining resources of the container management system meet the running resource requirement of at least one container group among the multiple container groups, the at least one container group is sent to the scheduling queue of the container management system.

6. The method according to any one of claims 1 to 5, characterized in that: After the container group is created, the method further includes: configuring a pause scheduling tag for the container group; In the case where it is determined that the remaining resources of the container management system meet the operating resource requirements of the container group, the method further includes: removing the pause scheduling tag configured for the container group; The step of sending the container group to the scheduling queue of the container management system includes: In response to the pause scheduling tag being removed, sending the container group to a scheduling queue of the container management system.

7. The method according to claim 6, characterized in that Configuring a pause scheduling tag for the container group includes: When it is detected that the remaining resources of the container management system do not meet the running resource requirements of the container group, a pause scheduling tag is configured for the container group.

8. A task scheduling method, characterized in that: include: Receiving a container group using a scheduling queue, the container group is created in response to a task processing request based on resources required by the task, and is sent to the scheduling queue when it is detected that remaining resources of the container management system meet the operating resource requirements of the container group; The container group is scheduled to a target node, so that the container group runs on the target node and processes the task processing request.

9. A computing device, characterized in that including a processing component and a storage component; The storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the task processing method as described in any one of claims 1 to 7, or to implement the task scheduling method as described in claim 8.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, the task processing method according to any one of claims 1 to 7 is implemented, or the task scheduling method according to claim 8 is implemented.

11. A computer program product, characterized in that It comprises a computer program, which, when executed by a computer, implements the task processing method according to any one of claims 1 to 7, or implements the task scheduling method according to claim 8.

Images