Task processing method and device, storage medium and electronic device

Abstract

This invention discloses a task processing method, apparatus, storage medium, and electronic device. The method includes: upon receiving a target task to be processed in a big data cluster, obtaining the available resources of the big data cluster; if the available resources are insufficient to process the target task, creating a target container instance within a container cluster to assist the big data cluster in processing the target task by running the target container instance; and after the target task is processed, controlling the container cluster to delete the created target container instance. This invention solves the technical problem of resource waste when resources from big data clusters and container clusters are used interchangeably.

Description

Technical Field

[0001] This invention relates to the field of clusters, and more specifically, to a task processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] In existing technologies, when a big data cluster receives a task to be processed, if the resources in the cluster are insufficient to process the task, the task will remain in a waiting state until the remaining resources in the cluster are sufficient to process the task.

[0003] To address this issue, existing technologies have proposed a solution that combines big data clusters and container clusters, where a portion of the resources of the container cluster are provided to the big data cluster.

[0004] However, the amount of available resources in a big data cluster changes in real time. If the resources of a container cluster are provided to the big data cluster, the resources of the big data cluster itself may already be sufficient, thus resulting in a waste of resources. Summary of the Invention

[0005] This invention provides a task processing method, apparatus, storage medium, and electronic device to at least solve the technical problem of resource waste when big data clusters and container clusters share resources.

[0006] According to one aspect of the present invention, a task processing method is provided, comprising: when a big data cluster receives a target task to be processed, obtaining the available resources of the big data cluster; when the available resources are insufficient to process the target task, creating a target container instance in a container cluster to assist the big data cluster in processing the target task by running the target container instance; and after the target task is processed, controlling the container cluster to delete the created target container instance.

[0007] According to another aspect of the present invention, a task processing apparatus is provided, comprising: an acquisition module, configured to acquire the available resources of the big data cluster when the big data cluster receives a target task to be processed; a creation module, configured to create a target container instance in a container cluster when the available resources are insufficient to process the target task, so as to assist the big data cluster in processing the target task by running the target container instance; and a first control module, configured to control the container cluster to delete the created target container instance after the target task is processed.

[0008] As an optional example, the above-mentioned acquisition module includes: an acquisition unit, configured to acquire a task tag of the first task when the big data cluster receives a first task to be processed; and a determination unit, configured to determine the first task as the target task when the task tag is a first tag, wherein the first tag is used to indicate that the first task is allowed to be processed by the container cluster.

[0009] As an optional example, the creation module includes a sending unit, which sends the required resources of the target task to the container cluster, and the container cluster creates the target container instance based on the required resources, wherein the target container instance has a built-in big data cluster computing process.

[0010] As an optional example, when creating the target container instance, the container cluster determines the upper and lower limits of resources that the target container instance can use based on the required resource amount; and creates the target container instance according to the upper and lower limits of resources.

[0011] As an optional example, the above apparatus further includes: a second control module, configured to control the container cluster to reduce the resource limit of the target container instance to release resources for processing the second task when the container cluster receives a second task to be processed and the available resources of the container cluster are insufficient to process the second task, wherein the second task is a task with a higher priority than the target task.

[0012] As an optional example, the above apparatus further includes: an insertion module, used to insert a third-party service process into the target container instance when the target container instance is created, wherein the third-party service process is used to control the distribution, interaction and reporting of the target task between the target container instance and the third-party server.

[0013] As an optional example, the creation module includes: an allocation unit, configured to allocate each subtask of the target task to the target container instance for processing by the target container instance; or allocate a portion of the subtasks of the target task to the target container instance for processing by the target container instance, and allocate the remaining subtasks to the big data cluster for processing by the big data cluster.

[0014] According to another aspect of the present invention, a storage medium is also provided, wherein a computer program is stored in the storage medium, and the computer program is executed by a processor to perform the above-described task processing method.

[0015] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the task processing method described above through the computer program.

[0016] In this embodiment of the invention, when a big data cluster receives a target task to be processed, it acquires the available resources of the big data cluster; if the available resources are insufficient to process the target task, it creates a target container instance within a container cluster to assist the big data cluster in processing the target task; and after the target task is completed, it controls the container cluster to delete the created target container instance. Because in this method, if the big data cluster's available resources are insufficient when processing the target task, the container cluster creates a target container instance to help the big data cluster process the target task, thereby achieving the goal of sharing resources between the big data cluster and the container cluster without wasting resources, and thus solving the technical problem of resource waste when sharing resources between the big data cluster and the container cluster. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0018] Figure 1 This is a flowchart of an optional task processing method according to an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram illustrating a big data computing cluster borrowing resources from a container cluster in an optional task processing method according to an embodiment of the present invention.

[0020] Figure 3 This is a diagram of the architecture of YARN, an optional task processing method according to an embodiment of the present invention;

[0021] Figure 4 This is a system architecture diagram of an optional task processing method according to an embodiment of the present invention;

[0022] Figure 5 This is a system flowchart of an optional task processing method according to an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of the structure of an optional task processing device according to an embodiment of the present invention;

[0024] Figure 7 This is a schematic diagram of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] According to a first aspect of the present invention, a task processing method is provided, optionally, as follows: Figure 1 As shown, the above method includes:

[0028] S102, when the big data cluster receives the target task to be processed, obtain the available resources of the big data cluster;

[0029] S104. When the available resources are insufficient to handle the target task, create a target container instance within the container cluster to assist the big data cluster in handling the target task by running the target container instance.

[0030] S106, after the target task is completed, control the container cluster to delete the created target container instance.

[0031] Optionally, this embodiment can be applied to the process of scaling up a big data cluster (which can be used for computing and can also be called a big data computing cluster) using a container cluster. The scaling process is dynamic and changes with the amount of available resources in the big data cluster.

[0032] The method described in this embodiment can be applied to servers providing container resource elastic scaling services (hereinafter referred to as third-party services). The servers of third-party services can coordinate task processing between container clusters and big data clusters.

[0033] The resources in the aforementioned big data clusters and container clusters can include cluster CPU resources, storage resources, etc. Storage resources can include memory resources and disk resources.

[0034] In this embodiment, for a big data cluster, the amount of available resources in the big data cluster varies with the number of tasks being processed. When the big data cluster receives a target task to be processed, the target task's status is divided into a preparation stage, a waiting stage, and a processing stage. When the target task is in the waiting stage, it means that the amount of available resources in the big data cluster is insufficient to process the target task. In this case, the container cluster is needed to assist in processing the target task. (Another scenario is when a user creates a task and specifies that the task will be processed by the big data cluster. In this case, even if the available resources of the big data cluster are insufficient, the container cluster will not assist in processing the target task. This part will be detailed below.)

[0035] In this embodiment, a third-party service can monitor the target task and available resources of the big data cluster. If the available resources are insufficient and the target task is in a waiting state (padding state), a message is sent to the container cluster, which then creates a target container instance. After the container cluster creates the target container instance, it can use the cluster's resources to assist in processing the target task. After processing the target task, the target container instance can be deleted.

[0036] In the above method, if the available resources of the big data cluster are insufficient when processing the target task, the container cluster will create a target container instance to help the big data cluster process the target task, thus achieving the goal of sharing resources between the big data cluster and the container cluster. Furthermore, since the creation of the target container instance is real-time, occurring when the big data cluster's resources are insufficient, and it is deleted after use, no resources are wasted.

[0037] As an optional example, the big data cluster receiving a target task to be processed includes: if the big data cluster receives a first task to be processed, obtaining the task label of the first task; if the task label is a first tag, determining the first task as the target task, wherein the first tag is used to indicate that the first task is allowed to be processed by the container cluster.

[0038] In this embodiment, when a user creates a first task to be executed on a big data cluster, a task label can be assigned to the first task. If the task label is the first label, it means that if the available resources of the big data cluster are insufficient, the task can be processed by the container cluster. If the task label of the first task is the second label, it means that even if the available resources of the big data cluster are insufficient, the first task is not allowed to be processed by the container cluster.

[0039] In this embodiment, the first task with the first label is designated as the target task. The third-party server can determine when the available resources of the big data cluster are insufficient, and the container cluster will create a target container instance to assist in executing the target task. If the first task is labeled with the second label, then the first task will be processed by the big data cluster when it is idle.

[0040] As an optional example, when the available resources are insufficient to handle the target task, creating a target container instance within the container cluster includes: sending the required resources for the target task to the container cluster, and having the container cluster create the target container instance based on the required resources, wherein the target container instance has a built-in big data cluster computing process.

[0041] Optionally, in this embodiment, if the big data cluster receives the target task but the available resources are insufficient, a target container instance needs to be created in the container cluster. When creating the target container instance, the resource requirements of the target task can be determined. Then, the target container cluster is created according to the resource requirements of the target task. In this embodiment, the built-in big data cluster computing process can report the resource usage of the target task.

[0042] As an optional example, sending the required resources for the target task to the container cluster, and having the container cluster create a target container instance based on the required resources, includes: when creating the target container instance, the container cluster determines the upper and lower limits of resources that the target container instance can use based on the required resources; and creating the target container instance according to the upper and lower limits of resources.

[0043] In this embodiment, when the big data cluster receives the target task and the available resources are insufficient, when creating a target container instance in the container cluster, the amount of container cluster resources to be occupied by the target container instance can be determined based on the aforementioned required resource amount. For example, a target container instance may occupy a certain number of central processing unit (CPU) cores, including the maximum and minimum number of cores required. It may also occupy a certain amount of memory, including the maximum and minimum memory requirements.

[0044] When the target container instance uses the resources of the container cluster to process the target task, the actual amount of resources used by the target container instance can be returned by the big data cluster computing process. For example, if 3 cores are actually used, it falls between the maximum and minimum number of cores used; if 20MB of memory is actually used, it falls between the maximum and minimum memory used. The actual amount of resources used cannot exceed the upper and lower limits of the target container instance's resources.

[0045] As an optional example, the above method further includes: when the container cluster receives a second task to be processed, and when the available resources of the container cluster are insufficient to process the second task, controlling the container cluster to reduce the resource limit of the target container instance to release resources to process the second task, wherein the second task is a task with a higher priority than the target task.

[0046] In this embodiment, when the big data cluster receives a target task and the available resources are insufficient, after creating a target container instance in the container cluster, the target container instance will consume resources of the container cluster to process the target task. If the container cluster receives other higher-priority second tasks in addition to the target task, the higher-priority second tasks need to be processed first. In this case, the upper limit of the resources that the target container instance can use can be lowered. For example, if it originally occupied a maximum of 4 CPU cores and a maximum of 10GB of memory, it can be adjusted to occupy a maximum of 2 CPU cores and a maximum of 4GB of memory.

[0047] You can reduce the maximum resource size of the target container instance to the minimum resource size, or a value greater than the minimum resource size but less than the maximum resource size. The freed-up resources can then be used by the container cluster to handle higher-priority tasks, such as backend service tasks, application service tasks, database tasks, and gateway service tasks.

[0048] As an optional example, the above method also includes: when creating a target container instance, placing a third-party service process in the target container instance, wherein the third-party service process is used to control the distribution, interaction and reporting of target tasks between the target container instance and the third-party server.

[0049] In this embodiment, when creating the target container instance, in addition to the aforementioned big data cluster computing process, a third-party service process can also be included. The third-party service process is a client process of a third-party server located within the target container instance. The third-party service process controls the processing of tasks within the target container instance according to the instructions of the third-party server, including issuing, interacting, and reporting tasks.

[0050] As an alternative example, assisting the big data cluster in processing target tasks by running a target container instance includes: assigning each subtask of the target task to the target container instance for processing; or assigning a portion of the subtasks of the target task to the target container instance for processing, and assigning the remaining subtasks to the big data cluster for processing.

[0051] Optionally, in this embodiment, after the target container instance is created, the target task can be handled by the target container instance in the container cluster. The target task can be broken down into multiple subtasks by the big data cluster. If the big data cluster has available resources (but insufficient to handle the target task), it can handle a portion of the subtasks, leaving the remaining subtasks to be handled by the target container instance in the container cluster. Of course, all subtasks of the target task can be handled by the target container instance in the container cluster. Regardless of whether the subtasks are handled by the big data cluster, if there are multiple target container instances, the subtasks assigned to each target container instance can be handled by one or more subtasks by each target container instance.

[0052] Figure 2 This is a schematic diagram illustrating how the big data computing cluster borrows resources from the container cluster in this embodiment. Figure 2 In this implementation, the big data cluster can be a Hadoop cluster. When the remaining resources of the big data cluster are insufficient, the Kubernetes container cluster uses idle resources to create container instances (pods) for the big data cluster to use. Each pod contains a built-in big data cluster computing process (yarn-nodemanager), which actively reports heartbeats to the big data computing cluster to increase its computing resources. This allows the big data cluster to reuse container resources, improving utilization (in this embodiment, a very large-scale big data cluster can be maintained; container resources can be used to supplement computing resources when needed). After the big data cluster task is completed, the container resources can be quickly released (pod deletion), returning the resources to other services within the container cluster.

[0053] Figure 3 This is the architecture diagram of YARN in this embodiment. The YARN process is as follows: 1. The client submits a task to the application manager (ResourceManager). 2. The application manager passes the task to the node manager (NodeManager). The node manager, acting as the master node, uses the MapReduce (MR) application instance (MRAppMstr) to divide the task into multiple subtasks. 3. The division result is reported to the application manager. 4. The resource information to be used is reported to the resource scheduler (ResourceScheduler). 5. The division result is reported to the master node (NodeManager), and the task is...

[0054] The task is handed over to the slave node NodeManager. The master node NodeManager's MapTask monitors the execution process of task 5, and the slave node NodeManager (one NodeManager is the master and one is the slave)...

[0055] The node executes its Map Task and Reduce Task. 7. Return the execution results. 8. Report the results. Yarn is a big data task scheduler and resource manager. A yarn-slave (standardly called a node-manager) created in a container can supplement the scheduler's management after being registered with the machine.

[0056] A well-organized resource pool allows clients submitting tasks to utilize the resources provided by the container cluster. Figure 4 This is a system architecture diagram for this embodiment. The third-party server on the management side, BingoServer (containing...)

[0057] The system manages and uses resources in a hybrid manner (e.g., elastic scaling of container resources). A third-party client is created within the container cluster's pods for task scheduling. The monitoring module is Promethus, used to monitor task execution. NM stands for NodeManager, and VM stands for Virtual Machine. The controller is...

[0058] kubelete.

[0059] 5 Figure 5 This is the system flowchart for this embodiment. After a user submits a task to the big data cluster, if the big data cluster's resources are insufficient, the container cluster will create a pod, register the pod's resources with the big data cluster, and allow the big data cluster to use it. Once use is complete, the container resources are returned. The container pod contains a yarn-nodemanager, whose main capability is to provide distributed task resource management.

[0060] In this embodiment, the reporting mechanism between him and the master is modified (to report the actual pod resource size used). This allows the master to be aware of the expansion of container resources. At the same time, cgroup technology is used to limit the resource size used by yarn-nodemanager. If the container cluster has other more urgent tasks, the pod resources of the container cluster used by the big data cluster can be partially or completely relinquished.

[0061] It should be noted that, for the sake of simplicity, all the aforementioned method embodiments are referred to as...

[0062] The description refers to a series of actions, but those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or in the same order.

[0063] This is done in a timely manner. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0064] According to another aspect of the embodiments of this application, a task processing apparatus is also provided, such as... Figure 6 As shown, it includes:

[0065] The acquisition module 602 is used to acquire the available resources of the big data cluster when the big data cluster receives the target task to be processed.

[0066] Create module 604 to create a target container instance within a container cluster when the available resources are insufficient to handle the target task, so as to assist the big data cluster in processing the target task by running the target container instance.

[0067] The first control module 606 is used to control the container cluster to delete the target container instance created after the target task is completed.

[0068] Optionally, this embodiment can be applied to the process of scaling up a big data cluster (which can be used for computing and can also be called a big data computing cluster) using a container cluster. The scaling process is dynamic and changes with the amount of available resources in the big data cluster.

[0069] The apparatus in this embodiment can be applied to the server of the container resource elastic scaling service (hereinafter referred to as the third-party service) 5. The server of the third-party service can coordinate task processing between the container cluster and the big data cluster.

[0070] The resources in the aforementioned big data clusters and container clusters can include cluster CPU resources, storage resources, etc. Storage resources can include memory resources and disk resources.

[0071] In this embodiment, for a big data cluster, the available resources within the cluster vary with the number of tasks being processed. When the big data cluster receives a target task to be processed, the task's status is divided into three stages: preparation, waiting, and processing. When the target task is in the waiting stage, it indicates that the available resources in the big data cluster are insufficient to process it. In this case, a container cluster is needed to assist in processing the target task. (Another scenario is when the user creates a task and specifies the task...)

[0072] The task will be handled by the big data cluster. Even if the available resources of the big data cluster are insufficient, the container cluster will not assist in handling the target task. This will be detailed below.

[0073] In this embodiment, a third-party service can monitor the target task and available resources of the big data cluster. If the available resources are insufficient and the target task is in a waiting state (padding state), a message is sent to the container cluster, which then creates a target container instance. After the container cluster creates the target container instance, it can use the cluster's resources to assist in processing the target task. After processing the target task, the target container instance can be deleted.

[0074] In the above method, if the available resources of the big data cluster are insufficient when processing the target task, the container cluster will create a target container instance to help the big data cluster process the target task, thus achieving the goal of sharing resources between the big data cluster and the container cluster. Furthermore, since the creation of the target container instance is real-time, occurring when the big data cluster's resources are insufficient, and it is deleted after use, no resources are wasted.

[0075] As an optional example, the above-mentioned acquisition module includes: an acquisition unit, used to acquire the task label of the first task when the big data cluster receives the first task to be processed; and a determination unit, used to determine the first task as the target task when the task label is a first label, wherein the first label is used to indicate that the first task is allowed to be processed by the container cluster.

[0076] In this embodiment, when a user creates a first task to be executed on a big data cluster, a task label can be assigned to the first task. If the task label is the first label, it means that if the available resources of the big data cluster are insufficient, the task can be processed by the container cluster. If the task label of the first task is the second label, it means that even if the available resources of the big data cluster are insufficient, the first task is not allowed to be processed by the container cluster.

[0077] In this embodiment, the first task with the first label is designated as the target task. The third-party server can determine when the available resources of the big data cluster are insufficient, and the container cluster will create a target container instance to assist in executing the target task. If the first task is labeled with the second label, then the first task will be processed by the big data cluster when it is idle.

[0078] As an optional example, the above creation module includes: a sending unit, used to send the required resources of the target task to the container cluster, and the container cluster creates a target container instance based on the required resources, wherein the target container instance has a built-in big data cluster computing process.

[0079] Optionally, in this embodiment, if the big data cluster receives the target task but the available resources are insufficient, a target container instance needs to be created in the container cluster. When creating the target container instance, the resource requirements of the target task can be determined. Then, the target container cluster is created according to the resource requirements of the target task. In this embodiment, the built-in big data cluster computing process can report the resource usage of the target task.

[0080] As an optional example, when creating a target container instance, the container cluster described above determines the upper and lower limits of resources that the target container instance can use based on the required amount of resources; and creates the target container instance according to the upper and lower limits of resources.

[0081] In this embodiment, when the big data cluster receives the target task and the available resources are insufficient, when creating a target container instance in the container cluster, the amount of container cluster resources to be occupied by the target container instance can be determined based on the aforementioned required resource amount. For example, a target container instance may occupy a certain number of central processing unit (CPU) cores, including the maximum and minimum number of cores required. It may also occupy a certain amount of memory, including the maximum and minimum memory requirements.

[0082] When the target container instance uses the resources of the container cluster to process the target task, the actual amount of resources used by the target container instance can be returned by the big data cluster computing process. For example, if 3 cores are actually used, it falls between the maximum and minimum number of cores used; if 20MB of memory is actually used, it falls between the maximum and minimum memory used. The actual amount of resources used cannot exceed the upper and lower limits of the target container instance's resources.

[0083] As an optional example, the above apparatus further includes: a second control module, configured to control the container cluster to reduce the resource limit of the target container instance to release resources for processing the second task when the container cluster receives a second task to be processed and the available resources of the container cluster are insufficient to process the second task, wherein the second task is a task with a higher priority than the target task.

[0084] In this embodiment, when the big data cluster receives a target task and the available resources are insufficient, after creating a target container instance in the container cluster, the target container instance will consume resources of the container cluster to process the target task. If the container cluster receives other higher-priority second tasks in addition to the target task, the higher-priority second tasks need to be processed first. In this case, the upper limit of the resources that the target container instance can use can be lowered. For example, if it originally occupied a maximum of 4 CPU cores and a maximum of 10GB of memory, it can be adjusted to occupy a maximum of 2 CPU cores and a maximum of 4GB of memory.

[0085] You can reduce the maximum resource size of the target container instance to the minimum resource size, or a value greater than the minimum resource size but less than the maximum resource size. The freed-up resources can then be used by the container cluster to handle higher-priority tasks, such as backend service tasks, application service tasks, database tasks, and gateway service tasks.

[0086] As an optional example, the above apparatus further includes: an insertion module, used to insert a third-party service process into the target container instance when the target container instance is created, wherein the third-party service process is used to control the distribution, interaction and reporting of target tasks between the target container instance and the third-party server.

[0087] In this embodiment, when creating the target container instance, in addition to the aforementioned big data cluster computing process, a third-party service process can also be included. The third-party service process is a client process of a third-party server located within the target container instance. The third-party service process controls the processing of tasks within the target container instance according to the instructions of the third-party server, including issuing, interacting, and reporting tasks.

[0088] As an optional example, the above creation module includes: an allocation unit, used to allocate each subtask of the target task to the target container instance, which then processes the target task; or to allocate a portion of the subtasks of the target task to the target container instance, which then processes them, and allocate the remaining subtasks to the big data cluster, which then processes them.

[0089] Optionally, in this embodiment, after the target container instance is created, the target task can be handled by the target container instance in the container cluster. The target task can be broken down into multiple subtasks by the big data cluster. If the big data cluster has available resources (but insufficient to handle the target task), it can handle a portion of the subtasks, leaving the remaining subtasks to be handled by the target container instance in the container cluster. Of course, all subtasks of the target task can be handled by the target container instance in the container cluster. Regardless of whether the subtasks are handled by the big data cluster, if there are multiple target container instances, the subtasks assigned to each target container instance can be handled by one or more subtasks by each target container instance.

[0090] For other examples of this embodiment, please refer to the examples above, which will not be repeated here.

[0091] Figure 7 This is a schematic diagram of an optional electronic device according to an embodiment of this application, such as... Figure 7As shown, it includes a processor 702, a communication interface 704, a memory 706, and a communication bus 708. The processor 702, communication interface 704, and memory 706 communicate with each other via the communication bus 708.

[0092] Memory 706 is used to store computer programs;

[0093] When processor 702 executes a computer program stored in memory 706, it performs the following steps:

[0094] When the big data cluster receives a target task to be processed, obtain the available resources of the big data cluster;

[0095] When the available resources are insufficient to handle the target task, create a target container instance within the container cluster to assist the big data cluster in handling the target task by running the target container instance.

[0096] After the target task is completed, control the container cluster to delete the target container instance that was created.

[0097] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0098] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0099] As an example, the memory 706 described above may include, but is not limited to, the acquisition module 602, the creation module 604, and the first control module 606 from the task processing device described above. Furthermore, it may include, but is not limited to, other module units from the task processing device described above, which will not be elaborated upon in this example.

[0100] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0101] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0102] Those skilled in the art will understand that Figure 7 The structure shown is for illustrative purposes only. The device that implements the above task processing method can be a terminal device, such as a smartphone (e.g., an Android phone, an iOS phone), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal devices. Figure 7 This does not limit the structure of the aforementioned electronic devices. For example, the electronic device may also include components that are more... Figure 7 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 7 The different configurations shown.

[0103] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0104] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is executed by a processor to perform the steps in the above-described task processing method.

[0105] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0106] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0107] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0108] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A task processing method, characterized in that, include: When the big data cluster receives a target task to be processed, the available resources of the big data cluster are obtained; If the available resources are insufficient to handle the target task, a target container instance is created within the container cluster to assist the big data cluster in handling the target task by running the target container instance. After the target task is completed, control the container cluster to delete the target container instance that was created; Where the available resources are insufficient to process the target task, creating a target container instance within the container cluster includes: sending the required resources of the target task to the container cluster, and having the container cluster create the target container instance based on the required resources, wherein the target container instance has a built-in big data cluster computing process. The step of sending the required resources for the target task to the container cluster, and having the container cluster create the target container instance based on the required resources, includes: when creating the target container instance, the container cluster determines the upper and lower limits of resources that the target container instance can use based on the required resources; and creates the target container instance according to the upper and lower limits of resources. The method further includes: when the container cluster receives a second task to be processed, and when the available resources of the container cluster are insufficient to process the second task, controlling the container cluster to reduce the resource limit of the target container instance to release resources to process the second task, wherein the second task is a task with a higher priority than the target task.

2. The method according to claim 1, characterized in that, The big data cluster receives the following target tasks to be processed: When the big data cluster receives a first task to be processed, the task tag of the first task is obtained; When the task label is the first label, the first task is identified as the target task, wherein the first label is used to indicate that the first task is allowed to be processed by the container cluster.

3. The method according to claim 1, characterized in that, The method further includes: When creating the target container instance, a third-party service process is placed in the target container instance. The third-party service process is used to control the distribution, interaction, and reporting of the target task between the target container instance and the third-party server.

4. The method according to any one of claims 1 to 2, characterized in that, The step of assisting the big data cluster in processing the target task by running the target container instance includes: Assign each subtask of the target task to the target container instance, and have the target container instance process the target task; or A portion of the subtasks in the target task are assigned to the target container instance for processing, while the remaining subtasks are assigned to the big data cluster for processing.

5. A task processing device, characterized in that, include: The acquisition module is used to acquire the available resources of the big data cluster when the big data cluster receives the target task to be processed. A creation module is used to create a target container instance within a container cluster when the available resources are insufficient to process the target task, so as to assist the big data cluster in processing the target task by running the target container instance; The control module is used to control the container cluster to delete the created target container instance after the target task is processed; Where the available resources are insufficient to process the target task, creating a target container instance within the container cluster includes: sending the required resources of the target task to the container cluster, and having the container cluster create the target container instance based on the required resources, wherein the target container instance has a built-in big data cluster computing process. The step of sending the required resources for the target task to the container cluster, and having the container cluster create the target container instance based on the required resources, includes: when creating the target container instance, the container cluster determines the upper and lower limits of resources that the target container instance can use based on the required resources; and creates the target container instance according to the upper and lower limits of resources. The device further includes: when the container cluster receives a second task to be processed, and when the available resources of the container cluster are insufficient to process the second task, controlling the container cluster to reduce the resource limit of the target container instance to release resources to process the second task, wherein the second task is a task with a higher priority than the target task.

6. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the method described in any one of claims 1 to 4.

7. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 4 through the computer program.

Images