Task processing method and device, storage medium and electronic device

Abstract

The application provides a task processing method and device, a storage medium and an electronic device, and the method comprises the following steps: receiving a target task submitted by a task submission device, and extracting parameter setting content from the target task based on a submission task type corresponding to the target task; calculating a target resource quantity required by the target task based on the parameter setting content; determining a residual resource quantity of each running device cluster in at least one running device cluster, and selecting a matching running device cluster from the at least one running device cluster based on the target resource quantity and the residual resource quantity of each running device cluster, wherein the residual resource quantity of the matching running device cluster is greater than or equal to the target resource quantity; and sending the target task to the matching running device cluster, so that the matching running device cluster executes the target task. The application can realize high availability guarantee for the target task through the matching running device cluster, thereby realizing efficient task processing.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a task processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] Currently, big data processing engines are crucial tools for the informatization and intelligent development of modern society. They can be used to process and analyze massive amounts of data, providing more efficient and intelligent services for applications across various fields. However, the task processing methods in the big data field are diverse, and while related technologies can utilize at least one cluster to execute tasks, situations such as cluster resource overload make it difficult to dynamically adjust scheduling based on cluster status, resulting in the inability to meet the requirements for high availability assurance. Therefore, determining the appropriate running device cluster for the target task to achieve high availability assurance and thus efficient task processing has become a research hotspot. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a task processing method, apparatus, storage medium, and electronic device to solve the problems that when cluster resources are fully loaded, related technologies are unable to dynamically adjust scheduling according to the cluster status, resulting in the inability to meet the requirements for high availability of tasks and thus low task processing efficiency. In other words, embodiments of the present invention can determine the matching running device cluster for the target task, so as to achieve high availability guarantee for the target task by matching the running device cluster, thereby achieving efficient task processing and effectively improving the task processing efficiency of the target task.

[0004] According to one aspect of the present invention, a task processing method is provided, the method being applied to a task scheduling device, the method comprising:

[0005] The system receives a target task submitted by a task submission device and extracts parameter settings from the target task based on the submission task type corresponding to the target task. The parameter settings include parameter information for each of the M parameters under the submission task type, where M is a non-negative integer.

[0006] Based on the parameter settings, calculate the amount of target resources required for the target task;

[0007] Determine the remaining resource quantity of each running device cluster in at least one running device cluster, and select a matching running device cluster from the at least one running device cluster based on the target resource quantity and the remaining resource quantity of each running device cluster, wherein the remaining resource quantity of the matching running device cluster is greater than or equal to the target resource quantity.

[0008] The target task is sent to the matching running device cluster so that the matching running device cluster executes the target task.

[0009] According to another aspect of the present invention, another task processing method is provided, the method being applied to a task submission device, the method comprising:

[0010] Obtain a task submission instruction, which carries the task type and parameter data to be transmitted;

[0011] According to the submitted task type, the transmitted parameter data is identified to obtain the parameter information of each parameter among the M parameters under the submitted task type, where M is a non-negative integer;

[0012] According to the submitted task type, the parameter information of each parameter is concatenated to obtain the target task, which can be recognized by the corresponding running device cluster.

[0013] The target task is sent to the task scheduling device so that the task scheduling device can schedule the target task.

[0014] According to another aspect of the present invention, a task processing apparatus is provided, the apparatus operating on a task scheduling device, the apparatus comprising:

[0015] The first input unit is used to receive the target task submitted by the task submission device.

[0016] The first processing unit is configured to extract parameter setting content from the target task based on the submission task type corresponding to the target task; wherein, the parameter setting content includes parameter information of each of the M parameters under the submission task type, where M is a non-negative integer;

[0017] The first processing unit is further configured to calculate the amount of target resources required for the target task based on the parameter settings.

[0018] The first processing unit is further configured to determine the remaining resource quantity of each running device cluster in at least one running device cluster, and select a matching running device cluster from the at least one running device cluster based on the target resource quantity and the remaining resource quantity of each running device cluster, wherein the remaining resource quantity of the matching running device cluster is greater than or equal to the target resource quantity.

[0019] The first output unit is used to send the target task to the matching running device cluster so that the matching running device cluster executes the target task.

[0020] According to another aspect of the present invention, another task processing apparatus is provided, the apparatus operating on a task submission device, the apparatus comprising:

[0021] The second input unit is used to obtain a task submission instruction, which carries the task type and transmission parameter data.

[0022] The second processing unit is used to identify parameters in the transmitted parameter data according to the submitted task type, and obtain parameter information of each parameter among M parameters under the submitted task type, where M is a non-negative integer;

[0023] The second processing unit is further configured to concatenate the parameter information of each parameter according to the submitted task type to obtain a target task, wherein the target task can be identified by the corresponding running device cluster.

[0024] The second output unit is used to send the target task to the task scheduling device so that the task scheduling device can schedule the target task.

[0025] According to another aspect of the present invention, an electronic device is provided, the electronic device including a processor and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the methods mentioned above.

[0026] According to another aspect of the present invention, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods mentioned above.

[0027] In this embodiment of the invention, after the task scheduling device receives a target task submitted by the task submission device, it extracts parameter settings from the target task based on the corresponding task type, and calculates the required target resources based on these parameter settings. The parameter settings include the parameter information of each of the M parameters under the task type. Then, it determines the remaining resources of each running device cluster in at least one running device cluster, and selects a matching running device cluster from the at least one cluster based on the target resource quantity and the remaining resources of each running device cluster. The remaining resources of the matching running device cluster are greater than or equal to the target resource quantity. Based on this, the target task can be sent to the matching running device cluster to execute the target task. High availability of the target task can be ensured through the matching running device cluster, thereby achieving efficient task processing and effectively improving the task processing efficiency. Furthermore, before submitting a target task, the task submission device can obtain a task submission instruction, which carries the task type and transmission parameter data. Then, according to the task type, it can identify the parameters in the transmission parameter data to obtain the parameter information of each of the M parameters under the task type. The parameter information of each parameter is then concatenated according to the task type to obtain the target task. The target task can be identified by the corresponding running device cluster, and thus sent to the task scheduling device for scheduling. Based on this, the complexity of the underlying big data architecture can be shielded from users (i.e., objects), so that users do not need to involve a lot of underlying technologies when submitting tasks, thereby conveniently realizing task submission and accelerating task processing efficiency. Attached Figure Description

[0028] Further details, features, and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0029] Figure 1 A flowchart illustrating a task processing method according to an exemplary embodiment of the present invention is shown;

[0030] Figure 2 A flowchart illustrating another task processing method according to an exemplary embodiment of the present invention is shown;

[0031] Figure 3 A flowchart illustrating yet another task processing method according to an exemplary embodiment of the present invention is shown;

[0032] Figure 4 A schematic diagram illustrating an embodiment of obtaining a task submission instruction according to an exemplary embodiment of the present invention is shown;

[0033] Figure 5A schematic diagram illustrating an example of acquiring a target task according to an exemplary embodiment of the present invention is shown;

[0034] Figure 6 A schematic diagram illustrating client-server interaction according to an exemplary embodiment of the present invention is shown.

[0035] Figure 7 A flowchart illustrating another task processing method according to an exemplary embodiment of the present invention is shown;

[0036] Figure 8a A schematic block diagram of a task processing apparatus according to an exemplary embodiment of the present invention is shown;

[0037] Figure 8b A schematic block diagram of another task processing apparatus according to an exemplary embodiment of the present invention is shown;

[0038] Figure 9 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0039] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0040] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0041] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0042] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0043] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0044] In this embodiment of the invention, a task processing system is provided, which may include, but is not limited to, a task submission device, a task scheduling device, and a cluster of multiple running devices, etc.; the invention does not limit the scope of the invention. The task submission device, the task scheduling device, and the devices in the cluster of running devices can all be electronic devices; optionally, the task submission device and the task scheduling device can be the same electronic device, that is, the task submission device and the task scheduling device can be different modules within the same electronic device; or, the task submission device and the task scheduling device can also be different electronic devices, etc.; the invention does not limit the scope of the invention.

[0045] The electronic device can be a terminal or a server; the terminal mentioned here can include, but is not limited to, smartphones, tablets, laptops, desktop computers, smartwatches, smart voice interaction devices, smart home appliances, etc. The server mentioned here can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0046] Based on the above description, embodiments of the present invention propose a task processing method, which can be executed by the aforementioned electronic device (i.e., task scheduling device); such as Figure 1 As shown, the task processing method may include the following steps S101-S104:

[0047] S101, receive the target task submitted by the task submission device, and extract the parameter setting content from the target task based on the submission task type corresponding to the target task; wherein, the parameter setting content includes the parameter information of each parameter among the M parameters under the submission task type, where M is a non-negative integer.

[0048] It should be noted that task processing methods in the big data field are diverse. This means that the big data engines executing tasks can include, but are not limited to: Hadoop (a distributed computing framework designed to solve computation and storage problems in large-scale data processing), Spark (a distributed computing framework designed to provide fast, flexible, and scalable data processing and high-performance computing capabilities), Hive (a distributed query language designed to provide fast query and analysis capabilities for distributed datasets), and Presto (a distributed query language designed to provide high-performance, scalable query capabilities and data interaction functions), etc.; this invention does not limit this. Based on this, the above-mentioned task submission type can be any of the following types: Hive, Spark, Presto, MR (MapReduce, a programming framework including Map tasks (responsible for decomposing complex tasks into several simple tasks for processing) and Reduce tasks (responsible for summarizing the results of the Map phase)), Hadoop, and HiveSQL to PrestoSQL conversion, etc.; this invention does not limit this.

[0049] Optionally, the type of task to be submitted can be represented by a type name (such as "Hadoop" to represent the Hadoop type), or by a type number (such as number "1" to represent the Hadoop type), etc.; this invention does not limit this.

[0050] For example, when submitting a task of type Hadoop or Hive, the M parameters under the task type may include, but are not limited to: mapreduce.job.running.map.limit (a parameter controlling runtime parallelism, i.e., the maximum number of Map tasks that can run concurrently), mapreduce.map.memory.mb (the upper limit of resources that Map tasks can use), mapreduce.map.java.opts (JVM parameters for Map tasks), mapreduce.reduce.memory.mb (the upper limit of resources that Reduce tasks can use), mapreduce.reduce.java.opts (JVM parameters for Reduce tasks), yarn.app.mapreduce.am.resource.mb (when running on yarn (Yet Another Resource Negotiator, a type of resource coordinator), which is used to allocate memory for AM (Application Master, responsible for scheduling and coordinating each specific application)). When submitting a task of type Spark, the M parameters may include, but are not limited to: executor-memory (a memory parameter indicating the memory size corresponding to an executor), executor-cores (CPU (Central Processing Time)), etc. Unit / Processor (Central Processing Unit) core parameters, used to indicate the number of CPU cores corresponding to an executor, num-executors (number of executors) and other parameters, are not limited in this invention.

[0051] Optionally, the task scheduling device may include a configuration parameter parsing layer. In this case, the task scheduling device can extract parameter settings from the target task through the configuration parameter parsing layer. Optionally, the configuration parameter parsing layer can identify parameter settings for different task submission types using regular expressions or other methods. Furthermore, the configuration parameter parsing layer can also perform operations such as formatting and annotation (e.g., unit conversion) on the identified parameters (i.e., each parameter under the task submission type) to obtain parameter information for each parameter.

[0052] S102, based on the parameter settings, calculate the amount of target resources required for the target task.

[0053] The target resource quantity can include the amount of memory (i.e., memory size) and the number of CPU cores required by the target task. In other words, the task scheduling device can determine the amount of memory and CPU cores required by the target task based on the parameter settings to calculate the target resource quantity. For example, taking a Spark task as the submitted task type (i.e., the target task is a Spark task), the task scheduling device can calculate the target resource quantity based on the parameters of executor-memory (memory parameter), executor-cores (CPU core parameter), and num-executors (number of executors parameter). Specifically, when the executor-memory parameter is 5GB, the executor-cores parameter is 2, and the num-executors parameter is 20, the target task requires 100GB of memory (i.e., 5 * 20) and 40 CPU cores (i.e., 2 * 20). That is, the target task requires at least 100GB of memory and 40 CPU cores. GB is a unit of memory storage capacity, and 1GB equals 1024MB (another unit of storage).

[0054] In this embodiment of the invention, the task scheduling device can further determine the target parameter judgment rule and determine whether the parameter setting content meets the target parameter judgment rule. The target parameter judgment rule is used to indicate the parameter range defined by each of the N limited parameters, so that the parameter information of each limited parameter in the parameter setting content is within the corresponding parameter range, thus satisfying the target parameter judgment rule. N is a positive integer. The N limited parameters may include Q parameters from M parameters, where Q is a non-negative integer and Q is less than or equal to M; that is, the M parameters may include Q limited parameters, thereby determining whether the parameter information of each of the Q limited parameters is within the corresponding parameter range. Optionally, the target parameter judgment rule can also be used to indicate the suggested value, unit, and task type of each limited parameter, etc., which is not limited in this invention. Optionally, the parameter range of a limited parameter can be represented by a maximum value (i.e., a parameter threshold), that is, the parameter range of a limited parameter can refer to a parameter range that does not exceed the maximum value. Optionally, the parameter threshold of any limited parameter can be set according to experience or according to actual needs, which is not limited in this invention. It should be noted that the target parameter judgment rules can be set based on experience or based on actual needs, and this invention does not limit them; that is, the N limiting parameters and the parameter range of each limiting parameter can be set based on experience or based on actual needs, and this invention does not limit them.

[0055] For example, let's illustrate this by showing that the range of a single limiting parameter is represented by its maximum value. Assume there are N limiting parameters, including `yarn.app.mapreduce.am.resource.mb` (task type: Hadoop / Hive), `mapreduce.job.running.map.limit` (task type: Hadoop / Hive), and `executor-memory` (task type: Spark). The maximum values ​​of the parameters are 5120MB, 2000, and 30GB, respectively. Assuming the submitted task type is Hadoop, and the Q limiting parameters among the M parameters include the `yarn.app.mapreduce.am.resource.mb` and `mapreduce.job.running.map.limit` parameters, the task scheduling device can determine whether the parameter information of `yarn.app.mapreduce.am.resource.mb` exceeds 5120MB and whether the parameter information of `mapreduce.job.running.map.limit` exceeds 2000, thus determining whether the parameter information of each of the Q limiting parameters is within its corresponding range. Based on this, if the parameter information of `yarn.app.mapreduce.am.resource.mb` is less than or equal to 5120MB, and the parameter information of `mapreduce.job.running.map.limit` is less than or equal to 2000, then it can be determined that the parameter information of each of the Q limiting parameters is within its corresponding range, thus confirming that the parameter settings meet the target parameter judgment rules.

[0056] Accordingly, if the parameter settings meet the target parameter judgment rules, the task scheduling device can calculate the target resource quantity required by the target task based on the aforementioned parameter settings. If the parameter settings do not meet the target parameter judgment rules, an error message can be returned to the task submission device to provide an error message to the user, allowing the user to modify the corresponding parameter information. Optionally, the error message can be a text message, a numerical message, etc.; this invention does not limit this. It should be understood that when the parameter settings do not meet the target parameter judgment rules, the submission of the target task can be interrupted, thereby interrupting the execution of the target task.

[0057] Optionally, if the parameter settings do not meet the target parameter judgment rules, the task scheduling device can also correct the parameters whose parameter information is not within the corresponding parameter range among the Q limited parameters based on the suggested values ​​of each limited parameter, so that the corrected parameter information is equal to the corresponding suggested value; or, it can also correct the parameters whose parameter information is not within the corresponding parameter range among the Q limited parameters based on the maximum value of each limited parameter, and so on.

[0058] Optionally, the task scheduling device may also include a task value verification layer, such as... Figure 2 As shown; in this case, the task scheduling device can use the task value verification layer to determine whether the parameter settings meet the target parameter judgment rules, etc. Optionally, the limited parameters can also refer to the parameters that must be submitted at startup. That is, all limited parameters under the submitted task type included in the N limited parameters must be parameters from the M parameters, i.e., the Q limited parameters must include all limited parameters under the submitted task type from the N limited parameters.

[0059] In summary, the embodiments of the present invention can have the capabilities of task optimization and parameter tuning, thereby enabling accurate and reliable task processing and data analysis. This can effectively avoid problems such as low target task running efficiency, poor data processing quality, and even cluster failure caused by an excessive number of target resources.

[0060] S103, determine the remaining resource quantity of each running device cluster in at least one running device cluster, and select a matching running device cluster from at least one running device cluster based on the target resource quantity and the remaining resource quantity of each running device cluster, wherein the remaining resource quantity of the matching running device cluster is greater than or equal to the target resource quantity.

[0061] In this embodiment of the invention, the task scheduling device can first determine at least one running device cluster from multiple running device clusters; wherein, the multiple running device clusters may include at least one Presto cluster and at least one Yarn cluster; the Presto cluster may include a Presto big data engine to execute Presto type tasks (i.e., Presto tasks); the Yarn cluster may include, but is not limited to, a Hive big data engine, a Spark big data engine, and a Hadoop big data engine, that is, the Yarn cluster can be used to execute Hive tasks, Spark tasks, and Hadoop tasks, etc.

[0062] Furthermore, when the submitted task type is Presto, at least one running device cluster may include at least one Presto cluster; in this case, the task scheduling device can directly obtain the remaining resources of each running device cluster from the at least one running device cluster. Correspondingly, when the submitted task type is not Presto, at least one running device cluster may include at least one Yarn cluster; in this case, the task scheduling device can invoke the Yarn platform to obtain the remaining resources of each running device cluster from the at least one running device cluster.

[0063] Optionally, the task scheduling device may first obtain the remaining resources of each of the multiple running device clusters; then, when determining the remaining resources of each running device cluster in at least one running device cluster, it may select the remaining resources of each running device cluster in at least one running device cluster from the remaining resources of each running device cluster in the multiple running device clusters based on the type of task submitted.

[0064] Specifically, when selecting a matching running device cluster from at least one running device cluster based on the target resource quantity and the remaining resource quantity of each running device cluster, the task scheduling device can select the first cluster with the largest remaining resource quantity from the at least one running device cluster. If the remaining resource quantity of the first cluster is greater than or equal to the target resource quantity, then the first cluster is selected as the matching running device cluster. The remaining resource quantity of the first cluster is greater than the remaining resource quantity of any other cluster in the at least one running device cluster. Alternatively, the task queuing quantity of the first cluster can be determined. If the remaining resource quantity of the first cluster is greater than or equal to the target resource quantity, then the first cluster is selected as the matching running device cluster. If the number of tasks in the first cluster is less than or equal to the target resource quantity, and the number of tasks in the queue is less than or equal to the preset queuing threshold, then the first cluster is selected as the matching running device cluster. Alternatively, at least one second cluster with a number of tasks in the queue less than or equal to the preset queuing threshold can be selected from at least one running device cluster. Based on the remaining resource quantity of each running device cluster, a third cluster with the largest remaining resource quantity can be selected from at least one second cluster. If the remaining resource quantity of the third cluster is greater than or equal to the target resource quantity, then the third cluster is selected as the matching running device cluster. The remaining resource quantity of the third cluster is greater than the remaining resource quantity of any cluster other than the third cluster in at least one second cluster.

[0065] It should be understood that when the first cluster meets the cluster matching conditions, it can be used as the matching running device cluster. In this case, the cluster matching condition means that the remaining resource quantity is greater than or equal to the target resource quantity. That is, when the remaining resource quantity of the first cluster is greater than or equal to the target resource quantity, the first cluster meets the cluster matching condition. Alternatively, it can mean that the remaining resource quantity is greater than or equal to the target resource quantity, and the number of tasks in the queue is less than or equal to a preset queuing threshold. In other words, when the remaining resource quantity of the first cluster is greater than or equal to the target resource quantity, and the number of tasks in the queue is less than or equal to the preset queuing threshold, the first cluster meets the cluster matching condition. Similarly, when the third cluster meets the cluster matching conditions, it can be used as the matching running device cluster. In this case, the cluster matching condition can mean that the remaining resource quantity is greater than or equal to the target resource quantity, and the number of tasks in the queue is less than or equal to a preset queuing threshold. That is, when the remaining resource quantity of the third cluster is greater than or equal to the target resource quantity, and the number of tasks in the queue is less than or equal to the preset queuing threshold, the third cluster meets the cluster matching condition.

[0066] Optionally, when selecting the first cluster with the largest remaining resources from at least one running device cluster, the remaining resources of each running device cluster can be sorted from smallest to largest, and the first cluster with the largest remaining resources can be selected from at least one running device cluster based on the sorting result; or, the remaining resources of each running device cluster can be sorted from largest to smallest, and the first cluster with the largest remaining resources can be selected from at least one running device cluster based on the sorting result, and so on.

[0067] Specifically, when at least one running device cluster is at least one Presto cluster, the task scheduling device can directly obtain the task queue count of each running device cluster from each Presto cluster; when at least one running device cluster is at least one Yarn cluster, it can call the Yarn platform to obtain the task queue count of each running device cluster. Optionally, the above-mentioned preset queuing threshold can be 5, 6, etc., and this invention does not limit it.

[0068] Furthermore, if the fourth cluster in at least one operating device cluster does not meet the cluster matching conditions, the task scheduling device can store the target task in the scheduling layer queue, where the fourth cluster is either the first cluster or the third cluster; wherein, the fourth cluster does not meet the cluster matching conditions, meaning that the remaining resources of the fourth cluster are less than the target resource quantity, and / or the number of tasks queued in the fourth cluster is greater than the preset queuing threshold. Specifically, when the fourth cluster is the first cluster, and the cluster matching condition is that the remaining resource quantity is greater than or equal to the target resource quantity, the fourth cluster does not meet the cluster matching condition if the remaining resource quantity of the fourth cluster is less than the target resource quantity. When the fourth cluster is the first cluster or the third cluster, and the cluster matching condition is that the remaining resource quantity is greater than or equal to the target resource quantity, and the number of tasks in the queue is less than or equal to the preset queuing threshold, the fourth cluster does not meet the cluster matching condition if the remaining resource quantity of the fourth cluster is less than the target resource quantity, and / or the number of tasks in the queue of the fourth cluster is greater than the preset queuing threshold. In other words, when the remaining resource quantity of the fourth cluster is less than the target resource quantity, or the number of tasks in the queue of the fourth cluster is greater than the preset queuing threshold, or the remaining resource quantity of the fourth cluster is less than the target resource quantity and the number of tasks in the queue of the fourth cluster is greater than the preset queuing threshold, the fourth cluster does not meet the cluster matching condition.

[0069] Based on this, when a fifth cluster satisfying the cluster matching conditions is detected in at least one running device cluster, the task scheduling device can use the fifth cluster as the matching running device cluster. Optionally, if the fourth cluster is the first cluster, then the fifth cluster can be the running device cluster with the largest remaining resource amount among at least one running device cluster at the current system time. That is, the fifth cluster with the largest remaining resource amount can be selected from at least one running device cluster based on the remaining resource amount of each running device cluster at the current system time. If the fourth cluster is the third cluster, then at least one sixth cluster with a task queuing number less than or equal to a preset queuing threshold can be selected from at least one running device cluster based on the task queuing number of each running device cluster at the current system time, and the fifth cluster with the largest remaining resource amount can be selected from at least one sixth cluster based on the remaining resource amount of each running device cluster at the current system time. Accordingly, if the fourth cluster meets the cluster matching condition (i.e., its remaining resources are greater than or equal to the target resource quantity), then the fifth cluster meets the cluster matching condition (i.e., its remaining resources are greater than or equal to the target resource quantity). Similarly, if the fourth cluster meets the cluster matching condition (i.e., its remaining resources are greater than or equal to the target resource quantity, and its task queue count is less than or equal to a preset queuing threshold), then the fifth cluster meets the cluster matching condition (i.e., its remaining resources are greater than or equal to the target resource quantity, and its task queue count is less than or equal to a preset queuing threshold). Therefore, this embodiment of the invention uses the same determination method and the same cluster matching condition regardless of when it determines the matching running device cluster. That is, the determination method for the fifth cluster is the same as that for the fourth cluster, and the cluster matching conditions for determining whether the fifth and fourth clusters can be used as matching running device clusters are the same.

[0070] S104, the target task is sent to the matching running device cluster so that the matching running device cluster can execute the target task.

[0071] In this embodiment of the invention, when the fourth cluster in at least one running device cluster does not meet the cluster matching conditions, the target task is stored in the scheduling layer queue. Then, after detecting the fifth cluster that meets the cluster matching conditions and determining the matching running device cluster, the target task can be sent to the matching running device cluster. In other words, the task scheduling device can temporarily store the target task in the scheduling layer queue, wait for the cluster to have resources and for the task queuing situation to meet expectations, and then submit the target task to the matching running device cluster.

[0072] Furthermore, when the task scheduling device detects a failure in the matching running device cluster and the target task has not been completed, it can select a switching running device cluster from at least one running device cluster based on the target resource quantity and the remaining resources of each running device cluster. The remaining resources of the switching running device cluster are greater than or equal to the target resource quantity, and the switching running device cluster is a running device cluster other than the matching running device cluster from at least one running device cluster. Then, the target task can be sent to the switching running device cluster so that the switching running device cluster can execute the target task, thereby switching the hit cluster configuration. This achieves the switching of the target task's hit cluster from the matching running device cluster to the switching running device cluster. Figure 2 As shown. It should be understood that the switching of the running device cluster meets the above-mentioned cluster matching conditions, and the method for determining the switching running device cluster is the same as the method for determining the matching running device cluster. Therefore, this embodiment of the invention can determine the status and progress of the target task, and when the matching running device cluster fails, it can dynamically adjust scheduling, load balance, and automatically schedule the target task to a cluster with normal capabilities based on the cluster status, thus meeting the requirements for high availability assurance and failover capability of the target task.

[0073] In summary, the task scheduling device can verify the rationality of the parameters set in the submitted task (i.e., the parameter settings mentioned above), including resource parameters such as memory and CPU cores. Simultaneously, it can intelligently schedule tasks to different clusters based on factors such as the remaining resources and load conditions of the cluster, thereby maximizing resource utilization.

[0074] In this embodiment of the invention, after the task scheduling device receives a target task submitted by the task submission device, it extracts parameter settings from the target task based on the corresponding task type, and calculates the required target resources based on these parameter settings. The parameter settings include the parameter information of each of the M parameters under the task type. Then, it determines the remaining resources of each running device cluster in at least one running device cluster, and selects a matching running device cluster from the at least one cluster based on the target resource quantity and the remaining resources of each running device cluster. The remaining resources of the matching running device cluster are greater than or equal to the target resource quantity. Based on this, the target task can be sent to the matching running device cluster to execute the target task. High availability of the target task can be ensured through the matching running device cluster, thereby achieving efficient task processing and effectively improving the task processing efficiency.

[0075] Based on the above description, this embodiment of the invention also proposes another task processing method; correspondingly, this task processing method can be executed by the aforementioned electronic device (i.e., task submission device); please refer to... Figure 3 The task processing method may include the following steps S301-S304:

[0076] S301, Obtain the task submission instruction. The task submission instruction carries the task type and parameter data to be passed.

[0077] Optionally, the task submission device may include a unified access layer module for task submission (also known as the unified access layer for task submission). In this case, the task submission device can call the unified access layer module to obtain the task submission instruction. The aforementioned transmitted parameter data may refer to a command that includes parameter information for each parameter under the task type being submitted.

[0078] In one implementation, the task submission command can be submitted manually. Optionally, the unified access layer module for task submission can include a task submission portal. The task submission device can provide users with an open task submission portal to shield the diversity of underlying big data engines and the differences in usage methods. In this case, users can directly use the zy_client-t tool (i.e., the task submission tool) to submit tasks, where -t is followed by the big data engine to be used (i.e., the task type), including but not limited to -t hive for Hive tasks (i.e., Hive type tasks), -t spark for Spark tasks (i.e., Spark type tasks), -t hadoop for Hadoop tasks (i.e., Hadoop type tasks), and -t presto for Presto tasks (i.e., Presto type tasks). The zy_client tool (i.e., the zy_client-t tool) can also support user-defined development to meet business needs.

[0079] Based on this, when a user enters a task submission instruction (i.e., task submission command) into the task submission portal using the parameter zy_client-ttaskType, the task submission device can obtain the task submission instruction. In other words, the user-input task submission instruction can be used as the obtained task submission instruction. The taskType parameter is the task type parameter, used to indicate the type of task to be submitted.

[0080] In another implementation, the submission mode of the aforementioned task submission instruction can be a platform submission mode, which supports SQL (Structured Query Language) type tasks. Based on this, when obtaining a task submission instruction, the task submission device can acquire the target structured query data and the task type, determine the complexity calculation rules, and calculate the complexity of the target structured query data based on these rules. Then, based on the complexity of the target structured query data, resource requirement assessment data can be determined, and based on the task type and resource requirement assessment data, parameter setting content can be generated. The parameter setting content includes parameter information for each of the M parameters under the task type, where M is a non-negative integer. Based on this, the parameter setting content can be encapsulated based on the task type to obtain the task submission instruction.

[0081] Optionally, the task submission device may include a submission platform (also known as an SQL-based task submission platform). The task submission device can obtain the target structured query data and the task type through the submission platform. Correspondingly, users can develop SQL for different business scenarios through the SQL editing window of the submission platform. That is, users can input the target structured query data through the SQL editing window, select the big data engine type (i.e., the aforementioned task type) on the submission platform, and then click buttons such as save or execute. The submission platform can then automatically submit the SQL task, enabling the task submission device to obtain the target structured query data and the task type. The submission platform primarily supports Hive, Spark, Presto, and HiveSQL to PrestoSQL conversion (i.e., Hive to Presto conversion), etc. Figure 4 As shown; where Hive to Presto type can be the result of SQL conversion of Hive type, such as Corel SQL conversion.

[0082] It should be noted that the above complexity calculation rules can be used to indicate that the complexity of the target structured query data is determined by the number of target keywords. Based on this, when calculating the complexity of the target structured query data based on the complexity calculation rules, the task submission device can determine the number of parameters for each statistical parameter in at least one statistical parameter, and use the number of parameters for each statistical parameter to calculate the number of target keywords. Then, based on the number of target keywords and the complexity calculation rules, the complexity of the target structured query data is calculated. Optionally, the task submission device may include an SQL complexity calculation module. In this case, the task submission device can call the SQL complexity calculation module to calculate the complexity of the target structured query data. Optionally, if the number of target keywords exceeds a preset keyword threshold, it may cause parsing to consume a lot of time and the job to be in a queue for a long time. In this case, the task submission device can output optimization suggestions through the submission platform to suggest that the user optimize the SQL and control the number of SQL keywords (i.e., the number of target keywords). Otherwise, execution will not continue, i.e., the task will not be submitted. Optionally, the preset keyword threshold can be 20, 23, etc., and this invention does not limit it.

[0083] The statistical parameter may include, but is not limited to: JOIN function parameters (functions for concatenating any number of strings), GROUPBY function parameters (an aggregation function), ORDERBY function parameters (a sorting function), DISTINCT function parameters (a function for returning uniquely distinct values), window functions, INSERT function parameters (insertion functions), UPDATE function parameters (update functions), and DELETE function parameters (deletion functions), etc.; this invention does not limit this. Optionally, the number of target keywords may be equal to the number of JOINs + the number of GROUPBYs + the number of ORDERBYs + the number of DISTINCTs + the number of window functions + MAX(MAX(number of INSERTs, UPDATEs, and DELETEs), 1).

[0084] Optionally, the above complexity calculation rules can be set based on experience or actual needs, and this invention does not limit this. For example, suppose the complexity calculation rules are as follows: when the number of keywords is less than or equal to 3, the complexity is the first complexity; when the number of keywords is less than or equal to 6 and greater than or equal to 4, the complexity is the second complexity; when the number of keywords is less than or equal to 19 and greater than or equal to 7, the complexity is the third complexity; when the number of keywords is greater than or equal to 20, the complexity is the fourth complexity. In this case, assuming the number of target keywords is 12, the complexity of the target structured query data can be the third complexity. The first, second, third, and fourth complexities increase sequentially, and this invention does not limit the specific values ​​of the first, second, third, and fourth complexities; for example, the first, second, third, and fourth complexities can be 1, 1.5, 2, and 4, respectively.

[0085] Optionally, the submission platform in the task submission device may also include a resource application assessment module (also known as a cluster resource application assessment module, etc.). In this case, the task submission device can determine the resource requirement assessment data through the resource application assessment module, that is, to conduct a cluster resource application assessment on the target task to obtain the resource requirement assessment data. The resource requirement assessment data may include the memory, CPU, and parallelism (i.e., the number of executors) required. In other words, the resource application assessment module can automatically analyze the memory, CPU, and parallelism resources required by the target task.

[0086] Specifically, when determining resource requirement assessment data based on the complexity of the target structured query data, the required assessment memory and the number of assessment CPU cores for a single executor can be determined separately based on the complexity of the target structured query data. This resource requirement assessment data includes assessment memory and the number of assessment CPU cores. Optionally, the resource requirement assessment data may also include the number of assessment executors. In this case, the number of assessment executors can also be determined based on the complexity of the target structured query data. It should be noted that this invention does not limit the specific method for determining the resource requirement assessment data.

[0087] In one implementation, the evaluation memory, the number of evaluation CPU cores, and the number of evaluation executors can each be positively correlated with the complexity. In this case, the complexity corresponds one-to-one with the memory, the number of CPU cores, and the number of executors. For example, when the complexity of the target structured query data is a first complexity (e.g., 1), the evaluation memory, the number of evaluation CPU cores, and the number of evaluation executors can be 5GB, 1, and 10, respectively; when the complexity of the target structured query data is a second complexity (e.g., 1.5), the evaluation memory, the number of evaluation CPU cores, and the number of evaluation executors can be 10GB, 1, and 30, respectively; when the complexity of the target structured query data is a third complexity (e.g., 2), the evaluation memory, the number of evaluation CPU cores, and the number of evaluation executors can be 20GB, 2, and 40, respectively; when the complexity of the target structured query data is a fourth complexity (e.g., 4), the evaluation memory, the number of evaluation CPU cores, and the number of evaluation executors can be 40GB, 5, and 50, respectively, and so on.

[0088] In another implementation, the complexity corresponds one-to-one with the memory range, the CPU core range, and the number of executors, respectively. In this case, the evaluation memory range, the evaluation CPU core range, and the evaluation executor number can be determined based on the complexity of the target structured query data. Then, the evaluation memory can be determined from the evaluation memory range, and the evaluation CPU core number can be determined from the evaluation CPU core range. Optionally, a memory number can be randomly selected from the evaluation memory range as the evaluation memory, and a specified number of cores can be selected from the evaluation CPU core range as the evaluation CPU core number, and so on. For example, when the complexity of the target structured query data is of the first complexity, the estimated memory range can refer to a maximum memory resource allocation of 5GB for a single executor, an estimated CPU core range can refer to a maximum of 1 core for a single executor, and the estimated number of executor cores can be 10; when the complexity of the target structured query data is of the second complexity, the estimated memory range can refer to a maximum memory resource allocation of 10GB for a single executor, an estimated CPU core range can refer to a maximum of 1 core for a single executor, and the estimated number of executor cores can be 30; when the complexity of the target structured query data is of the third complexity, the estimated memory range can refer to a maximum memory resource allocation of 20GB for a single executor, an estimated CPU core range can refer to a maximum of 2 cores for a single executor, and the estimated number of executor cores can be 40; when the complexity of the target structured query data is of the fourth complexity, the estimated memory range can refer to a maximum memory resource allocation of 40GB for a single executor, an estimated CPU core range can refer to a maximum of 5 cores for a single executor, and the estimated number of executor cores can be 50, and so on; the present invention does not limit this.

[0089] Accordingly, the task submission device can generate parameter settings based on the user-selected big data engine type (i.e., the task type) and resource requirement assessment data. For example, assuming the task type is Spark, and the resource requirement assessment data indicates that an executor requires 20G of memory (i.e., the assessed memory mentioned above) and 2 CPU cores (i.e., the assessed number of CPU cores mentioned above), then --executor-memory=20G and --executor-core=2 can be generated to obtain the parameter settings. At this time, the parameter settings include the parameter information of each parameter under the Spark type.

[0090] Optionally, the submission platform may also include a command encapsulation module. In this case, the parameter settings can be encapsulated through this module. Specifically, the target structured query data can be formatted and saved to a local custom file to obtain an SQL file. Then, based on the parameter settings, the submission command is assembled into the form zy_client-t taskType.... to obtain the task submission instruction. After encapsulation, this command is input to the unified task submission access layer for cluster submission. In other words, the task submission instruction can be submitted to the unified task submission access layer to continue submitting tasks.

[0091] Optionally, in other embodiments, the submission platform may also be an electronic device located outside the task submission device, and the present invention does not limit this. Optionally, the submission platform may also be the DCloud platform (a big data task submission platform), or the iQiyi platform (another big data task submission platform), etc., which can provide richer task submission and management functions to meet the needs of different scenarios, and the present invention does not limit this.

[0092] S302, according to the task submission type, perform parameter identification on the transmitted parameter data to obtain the parameter information of each parameter among the M parameters under the task submission type, where M is a non-negative integer.

[0093] In this embodiment of the invention, the task submission device can identify the -t parameter (i.e., the task type parameter) to determine the type of task being submitted. Then, it can identify and break down the parameters according to the characteristics of different big data engines to obtain the parameter information for each parameter under the submitted task type. Examples include the parameter information for the --conf setting in Hive, and the parameter information for --executor-memory, --executor-core, --jars, etc., in Spark, and so on.

[0094] Optionally, the task submission device may include a command recognition layer. In this case, the task submission device can use the command recognition layer to identify the parameters of the transmitted parameter data.

[0095] S303 concatenates the parameter information of each parameter according to the submitted task type to obtain the target task, which can be recognized by the corresponding running device cluster.

[0096] In this embodiment of the invention, the parameter information of each parameter can be concatenated into a command that the cluster can recognize, according to the characteristics of different big data engines and task submission requirements (such as the required memory). For example, a Hadoop command could be hadoopdf-put / home / test.orc hdfs: / / user / baihua / test, a Hive command could be beeline-conf mapred.job.name=testtask-f test.sql, a Spark command could be spark-submit--master yarn--class com.zy.task.TestTask test.jar input output, a Presto command could be presto-sql-e "select a,b from testdb.testtable limit 1;", and so on; this invention does not limit this.

[0097] Furthermore, these commands can be concatenated into a submission execution command executable by the task submission device. This submission execution command can include the target task, thereby enabling the target task to be sent to the task scheduling device when the submission execution command is executed. For example, taking a Hadoop-type submission task as an example, assuming the target task is hadoopdf-put / home / test.orc hdfs: / / user / baihua / test, that is, assuming the command used to implement the target task is hadoopdf-put / home / test.orc hdfs: / / user / baihua / test, then the submission execution command could be zy_client-thadoopdf-put / home / test.orc hdfs: / / user / baihua / test.

[0098] Optionally, the task submission device may include a command splicing layer. In this case, the task submission device can perform splicing operations through the command splicing layer, such as... Figure 5As shown in the diagram. The client can refer to the task submission device, while the server can include task scheduling devices, server interfaces, offline big data clusters (i.e., the aforementioned cluster of multiple running devices), etc. The server can be used to handle resource scheduling, allocation, task execution, and other functions. It should be understood that the client and server can jointly provide intelligent services. Optionally, when receiving tasks, the cluster in the server can receive them through a server interface (such as HiveServer2).

[0099] In summary, the task submission device can identify user-submitted task submission instructions or encapsulated task submission instructions, and compile and assemble them into target tasks that are compatible with the syntax specifications of the corresponding big data engine.

[0100] S304, the target task is sent to the task scheduling device so that the task scheduling device can schedule the target task.

[0101] In this embodiment of the invention, the task submission device can also obtain tracing indication data of the task submission instruction. The tracing indication data includes at least one of the following: a submission object identifier and a submission device identifier. The tracing indication data is used to trace the target task. The tracing indication data can also be sent to the task scheduling device so that the task scheduling device stores the tracing indication data. In this case, the task scheduling device can trace the target task based on the tracing indication data. The submission object identifier can be used to indicate the submitter of the target task (i.e., the submitter of the task submission instruction), and the submission device identifier can be used to indicate the task submission device of the target task. Optionally, the submission object identifier and the submission device identifier can be character identifiers or numeric identifiers, etc., and the invention does not limit this. For example, the submission device identifier can be the submission device IP (Internet Protocol). Optionally, the tracing indication data can also include a submission department identifier, which is used to indicate the submission department of the target task. The submission department identifier can be a character identifier or a numeric identifier, etc., and the invention does not limit this.

[0102] Optionally, the task submission device may include a data acquisition center (also known as a task submission data acquisition center). The data acquisition center can collect valuable information through the unified access layer for task submission, and then report the target task and traceability indication data to the task scheduling device on the server for storage and analysis. Optionally, big data engines such as Hadoop, Hive, Spark, and Presto can all encapsulate clients (i.e., task submission devices) to listen and report from different dimensions.

[0103] It should be noted that the task submission device (i.e., the client) can obtain the necessary information, including but not limited to the submitter, department, IP address, task type, and command, by listening to the task submission entry point and SQL execution interface of each engine. This can be achieved by embedding code into the engine's execution script. For example, for the Spark engine, this information is collected in the Spark-SQL script; for the Hive engine, it's collected in the Beeline script. Alternatively, code injection can be used to listen to the SQL interface of each big data engine. For instance, all SQL executions in the Spark engine go through the SQLSession interface class, so listening to this interface allows access to all SQL information. Correspondingly, the server can store different data in different storage media. For example, structured data such as the submitter and department can be directly stored in MySQL; semi-structured data such as task execution logs can be stored locally as files, and then managed through a log system for storage, lifecycle eviction, and analysis. For example,... Figure 6 As shown, when the storage time of logs exceeds the preset storage time threshold (such as 15 days or 16 days), they can be stored in a big data engine (such as the Hive engine).

[0104] For example, such as Figure 7 As shown, the task submission device can obtain task submission instructions through the unified access layer for task submission. It can then acquire target tasks and traceability indication data through the data acquisition center within the task submission device, and send these data to the task scheduling device. The task scheduling device can then store the traceability indication data and schedule the target tasks to be executed by a matching cluster of running devices. In this embodiment of the invention, the task scheduling device can also be referred to as the task scheduling center.

[0105] In summary, the client and server work together to complete the entire task information collection, reporting, and processing stages of the entire chain. This stage plays an invaluable role in task tracking, analysis, optimization, and tracing. Optionally, the intelligent service may also include a task decision module, i.e., a task scheduling device, for intelligent scheduling and cluster switching scheduling. It should be noted that, in order to meet the needs of simplified task submission and processing for big data engines in hybrid cloud, private cloud, and public cloud scenarios, this embodiment of the invention proposes a set of intelligent task processing modes applicable to components such as Hive, Spark, Hadoop, and Presto, ultimately achieving comprehensive processing across multiple dimensions, including task submission, information chain construction, optimization, management, and scheduling.

[0106] This invention, upon receiving a task submission instruction, allows the task submission device to identify the parameters in the transmitted parameter data according to the task type. This identifies the parameter information for each of the M parameters under the task type. The task submission instruction carries the task type and transmitted parameter data. The parameter information is then concatenated according to the task type to obtain the target task, which can be recognized by the corresponding running device cluster. Furthermore, the target task can be sent to a task scheduling device for scheduling. This invention shields users from the complexity of the underlying big data architecture, allowing for convenient task submission without involving extensive underlying technologies, thus accelerating task processing efficiency. In other words, it enables one-click submission and processing of tasks from different big data engines, shielding businesses from the diversity of underlying engine operations and the complexity of infrastructure computing power. It enables big data engine task deployment, task collection, task monitoring, task scheduling, task management, and intelligent scheduling, thereby improving the intelligence level of the big data engine and achieving efficient, accurate, and reliable task processing and data analysis, providing more efficient and intelligent services for applications in various fields.

[0107] Based on the description of the relevant embodiments of the above task processing method, this embodiment of the invention also proposes a task processing device, which can be a computer program (including program code) running in an electronic device, and the electronic device here refers to a task scheduling device; such as Figure 8a As shown, the task processing device may include a first input unit 801, a first processing unit 802, and a first output unit 803. The task processing device can perform... Figure 1 The task processing method shown indicates that the task processing device can operate the above-mentioned unit:

[0108] The first input unit 801 is used to receive the target task submitted by the task submission device;

[0109] The first processing unit 802 is used to extract parameter setting content from the target task based on the submission task type corresponding to the target task; wherein, the parameter setting content includes parameter information of each of the M parameters under the submission task type, where M is a non-negative integer;

[0110] The first processing unit 802 is further configured to calculate the amount of target resources required for the target task based on the parameter settings.

[0111] The first processing unit 802 is further configured to determine the remaining resource quantity of each running device cluster in at least one running device cluster, and select a matching running device cluster from the at least one running device cluster based on the target resource quantity and the remaining resource quantity of each running device cluster, wherein the remaining resource quantity of the matching running device cluster is greater than or equal to the target resource quantity.

[0112] The first output unit 803 is used to send the target task to the matching running device cluster so that the matching running device cluster executes the target task.

[0113] In one implementation, when the first processing unit 802 selects a matching running device cluster from the at least one running device cluster based on the target resource quantity and the remaining resource quantity of each running device cluster, it may specifically be used to:

[0114] Based on the remaining resources of each operating device cluster, a first cluster with the largest remaining resources is selected from the at least one operating device cluster. If the remaining resources of the first cluster are greater than or equal to the target resource quantity, then the first cluster is selected as the matching operating device cluster; or...

[0115] Determine the number of tasks queued in the first cluster. If the remaining resources of the first cluster are greater than or equal to the target resource quantity, and the number of tasks queued in the first cluster is less than or equal to a preset queuing threshold, then the first cluster is selected as the matching running device cluster; or...

[0116] From the at least one running device cluster, select at least one second cluster whose task queuing number is less than or equal to the preset queuing threshold, and based on the remaining resources of each running device cluster, select a third cluster with the largest remaining resources from the at least one second cluster. If the remaining resources of the third cluster are greater than or equal to the target resource quantity, then the third cluster is used as the matching running device cluster.

[0117] In another embodiment, the first processing unit 802 can also be used for:

[0118] If the fourth cluster in the at least one running device cluster does not meet the cluster matching condition, the target task is stored in the scheduling layer queue, and the fourth cluster is the first cluster or the third cluster; wherein, the fourth cluster does not meet the cluster matching condition means that the remaining resources of the fourth cluster are less than the target resource quantity, and / or the number of tasks queued in the fourth cluster is greater than the preset queuing threshold.

[0119] When a fifth cluster that meets the cluster matching conditions is detected in the at least one operating device cluster, the fifth cluster is taken as the matching operating device cluster.

[0120] In another embodiment, the first processing unit 802 can also be used for:

[0121] Determine the target parameter judgment rule, and determine whether the parameter setting content satisfies the target parameter judgment rule. The target parameter judgment rule is used to indicate the parameter range defined by each of the N limited parameters, so that the parameter information of each limited parameter in the parameter setting content is within the corresponding parameter range, and the target parameter judgment rule is satisfied. N is a positive integer.

[0122] If the parameter settings satisfy the target parameter judgment rule, then the step of calculating the target resource quantity required for the target task based on the parameter settings is executed.

[0123] The first output unit 803 can also be used for:

[0124] If the parameter settings do not meet the target parameter judgment rules, an error message is returned to the task submission device.

[0125] In another embodiment, the first processing unit 802 can also be used for:

[0126] When a fault is detected in the matching running device cluster and the target task has not been completed, a switching running device cluster is selected from the at least one running device cluster based on the target resource quantity and the remaining resource quantity of each running device cluster. The remaining resource quantity of the switching running device cluster is greater than or equal to the target resource quantity, and the switching running device cluster is a running device cluster other than the matching running device cluster among the at least one running device clusters.

[0127] The first output unit 803 can also be used for:

[0128] The target task is sent to the switching operation device cluster so that the switching operation device cluster executes the target task.

[0129] Based on the description of the relevant embodiments of the above task processing method, this embodiment of the invention also proposes another task processing device. This task processing device can be a computer program (including program code) running on an electronic device, and the electronic device here refers to a task submission device; such as Figure 8b As shown, the task processing device may include a second input unit 804, a second processing unit 805, and a second output unit 806. The task processing device can perform... Figure 3The task processing method shown indicates that the task processing device can operate the above-mentioned unit:

[0130] The second input unit 804 is used to obtain a task submission instruction, which carries the task type and transmission parameter data.

[0131] The second processing unit 805 is used to identify parameters in the transmitted parameter data according to the submitted task type, and obtain parameter information of each parameter among M parameters under the submitted task type, where M is a non-negative integer;

[0132] The second processing unit 805 is further configured to concatenate the parameter information of each parameter according to the submitted task type to obtain a target task, wherein the target task can be identified by the corresponding running device cluster.

[0133] The second output unit 806 is used to send the target task to the task scheduling device so that the task scheduling device can schedule the target task.

[0134] In one implementation, when the second input unit 804 receives the task submission instruction, it may specifically be used for:

[0135] Obtain the target structured query data and the submitted task type;

[0136] Determine the complexity calculation rules, and calculate the complexity of the target structured query data based on the complexity calculation rules;

[0137] Based on the complexity of the target structured query data, resource requirement assessment data is determined, and based on the submitted task type and the resource requirement assessment data, parameter setting content is generated. The parameter setting content includes parameter information of each parameter among the M parameters under the submitted task type.

[0138] Based on the submitted task type, the parameter settings are encapsulated to obtain the task submission instruction.

[0139] In another implementation, the complexity calculation rule is used to indicate that the complexity of the target structured query data is determined by the number of target keywords; when calculating the complexity of the target structured query data based on the complexity calculation rule, the second input unit 804 can specifically be used to:

[0140] Determine the statistical quantity of each statistical parameter in at least one statistical parameter, and use the statistical quantity of each statistical parameter to calculate the number of target keywords;

[0141] The complexity of the target structured query data is calculated based on the number of target keywords and the complexity calculation rules.

[0142] In another embodiment, the second processing unit 805 can also be used for:

[0143] Obtain the traceability indication data of the task submission instruction, wherein the traceability indication data includes at least one of the following: submission object identifier and submission device identifier, and the traceability indication data is used to trace the source of the target task;

[0144] The second output unit 806 can also be used for:

[0145] The source tracing indication data is sent to the task scheduling device so that the task scheduling device stores the source tracing indication data.

[0146] According to one embodiment of the present invention, Figure 1 Each step involved in the method shown can be derived from... Figure 8a The tasks are executed by the individual units within the illustrated task processing apparatus; for example... Figure 1 Step S101 shown can be performed by Figure 8a The first input unit 801 and the first processing unit 802 shown are executed together, and steps S102 and S103 can both be performed by... Figure 8a The first processing unit 802 shown is executed, and step S104 can be performed by... Figure 8a The first output unit 803 shown executes, and so on. Accordingly, Figure 3 Each step involved in the method shown can be derived from... Figure 8b The tasks are executed by the individual units within the illustrated task processing apparatus; for example... Figure 3 Step S301 shown can be performed by Figure 8b The second input unit 804 shown is executed, and steps S302 and S303 can both be performed by... Figure 8b The second processing unit 805 shown is executed, and step S304 can be performed by... Figure 8b The second output unit 806 shown in the figure performs, and so on.

[0147] According to another embodiment of the present invention, Figure 8a and Figure 8bEach unit in the task processing device shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of the present invention. The above units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, any task processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0148] According to another embodiment of the present invention, the following can be performed by running on a general-purpose electronic device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). Figure 1 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 8a The task processing apparatus shown herein, and the task processing method for implementing embodiments of the present invention; and can be executed by running on a general-purpose electronic device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), etc. Figure 3 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 8b The task processing apparatus shown herein, and the task processing method for implementing embodiments of the present invention, are described. The computer program may be recorded on, for example, a computer storage medium, loaded onto the aforementioned electronic device via the computer storage medium, and run therein.

[0149] In this embodiment of the invention, after the task scheduling device receives a target task submitted by the task submission device, it extracts parameter settings from the target task based on the corresponding task type, and calculates the required target resources based on these parameter settings. The parameter settings include the parameter information of each of the M parameters under the task type. Then, it determines the remaining resources of each running device cluster in at least one running device cluster, and selects a matching running device cluster from the at least one cluster based on the target resource quantity and the remaining resources of each running device cluster. The remaining resources of the matching running device cluster are greater than or equal to the target resource quantity. Based on this, the target task can be sent to the matching running device cluster to execute the target task. High availability of the target task can be ensured through the matching running device cluster, thereby achieving efficient task processing and effectively improving the task processing efficiency. Furthermore, before submitting a target task, the task submission device can obtain a task submission instruction, which carries the task type and transmission parameter data. Then, according to the task type, it can identify the parameters in the transmission parameter data to obtain the parameter information of each of the M parameters under the task type. The parameter information of each parameter is then concatenated according to the task type to obtain the target task. The target task can be identified by the corresponding running device cluster, and thus sent to the task scheduling device for scheduling. Based on this, the complexity of the underlying big data architecture can be shielded from users (i.e., objects), so that users do not need to involve a lot of underlying technologies when submitting tasks, thereby conveniently realizing task submission and accelerating task processing efficiency.

[0150] Based on the description of the method and apparatus embodiments above, an exemplary embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method according to an embodiment of the present invention.

[0151] An exemplary embodiment of the present invention also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0152] An exemplary embodiment of the present invention also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of the present invention.

[0153] refer to Figure 9The present invention will now be described in the form of a structural block diagram of an electronic device 900 that can serve as a server or client of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0154] like Figure 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 may also store various programs and data required for the operation of the device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0155] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, output unit 907, storage unit 908, and communication unit 909. Input unit 906 can be any type of device capable of inputting information to electronic device 900. Input unit 906 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 907 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 908 may include, but is not limited to, disk and optical disk. Communication unit 909 allows electronic device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0156] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above. For example, in some embodiments, the task processing methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. In some embodiments, the computing unit 901 can be configured to perform task processing methods by any other suitable means (e.g., by means of firmware).

[0157] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable task processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0158] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0159] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0161] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0162] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0163] Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A task processing method, characterized in that, The method is applied to a task scheduling device, and the method includes: The system receives a target task submitted by a task submission device, and extracts parameter settings from the target task based on the corresponding task type. The target task is obtained by the task submission device after performing complexity calculations, resource assessments, and concatenation according to engine type on the target structured query data. The parameter settings include parameter information for each of the M parameters under the task type. The task type supports any one of the big data engine types: Hive, Spark, Presto, and Hadoop, where M is a non-negative integer. Based on the parameter settings, calculate the amount of target resources required for the target task; Determine the remaining resource quantity of each running device cluster in at least one running device cluster, and select a matching running device cluster from the at least one running device cluster based on the target resource quantity and the remaining resource quantity of each running device cluster, wherein the remaining resource quantity of the matching running device cluster is greater than or equal to the target resource quantity. The target task is sent to the matching running device cluster so that the matching running device cluster executes the target task.

2. The method according to claim 1, characterized in that, The step of selecting a matching operating device cluster from the at least one operating device cluster based on the target resource quantity and the remaining resource quantity of each operating device cluster includes: Based on the remaining resources of each operating device cluster, a first cluster with the largest remaining resources is selected from the at least one operating device cluster. If the remaining resources of the first cluster are greater than or equal to the target resource quantity, then the first cluster is selected as the matching operating device cluster; or... Determine the number of tasks queued in the first cluster. If the remaining resources of the first cluster are greater than or equal to the target resource quantity, and the number of tasks queued in the first cluster is less than or equal to a preset queuing threshold, then the first cluster is selected as the matching running device cluster; or... From the at least one running device cluster, select at least one second cluster whose task queuing number is less than or equal to the preset queuing threshold, and based on the remaining resources of each running device cluster, select a third cluster with the largest remaining resources from the at least one second cluster. If the remaining resources of the third cluster are greater than or equal to the target resource quantity, then the third cluster is used as the matching running device cluster.

3. The method according to claim 2, characterized in that, The method further includes: If the fourth cluster in the at least one running device cluster does not meet the cluster matching condition, the target task is stored in the scheduling layer queue, and the fourth cluster is the first cluster or the third cluster; wherein, the fourth cluster does not meet the cluster matching condition means that the remaining resources of the fourth cluster are less than the target resource quantity, and / or the number of tasks queued in the fourth cluster is greater than the preset queuing threshold. When a fifth cluster that meets the cluster matching conditions is detected in the at least one operating device cluster, the fifth cluster is taken as the matching operating device cluster.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Determine the target parameter judgment rule, and determine whether the parameter setting content satisfies the target parameter judgment rule. The target parameter judgment rule is used to indicate the parameter range defined by each of the N limited parameters, so that the parameter information of each limited parameter in the parameter setting content is within the corresponding parameter range, and the target parameter judgment rule is satisfied. N is a positive integer. If the parameter settings satisfy the target parameter judgment rule, then the step of calculating the target resource quantity required for the target task based on the parameter settings is executed. If the parameter settings do not meet the target parameter judgment rules, an error message is returned to the task submission device.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: When a fault is detected in the matching running device cluster and the target task has not been completed, a switching running device cluster is selected from the at least one running device cluster based on the target resource quantity and the remaining resource quantity of each running device cluster. The remaining resource quantity of the switching running device cluster is greater than or equal to the target resource quantity, and the switching running device cluster is a running device cluster other than the matching running device cluster among the at least one running device clusters. The target task is sent to the switching operation device cluster so that the switching operation device cluster executes the target task.

6. A task processing method, characterized in that, The method is applied to a task submission device, and the method includes: Obtaining a task submission instruction includes: obtaining target structured query data and the task submission type; determining complexity calculation rules and calculating the complexity of the target structured query data based on the complexity calculation rules; determining resource requirement assessment data based on the complexity of the target structured query data, and generating parameter setting content based on the task submission type and the resource requirement assessment data, wherein the parameter setting content includes parameter information of each of the M parameters under the task submission type; encapsulating the parameter setting content based on the task submission type to obtain a task submission instruction; the task submission instruction carries the task submission type and transmitted parameter data, and the task submission type supports any one of the following big data engine types: Hive, Spark, Presto, and Hadoop. According to the submitted task type, the transmitted parameter data is identified to obtain the parameter information of each parameter among the M parameters under the submitted task type, where M is a non-negative integer; According to the submitted task type, the parameter information of each parameter is concatenated to obtain the target task, which can be recognized by the corresponding running device cluster. The target task is sent to the task scheduling device so that the task scheduling device can schedule the target task.

7. The method according to claim 6, characterized in that, The complexity calculation rule is used to indicate that the complexity of the target structured query data is determined by the number of target keywords; the calculation of the complexity of the target structured query data based on the complexity calculation rule includes: Determine the statistical quantity of each statistical parameter in at least one statistical parameter, and use the statistical quantity of each statistical parameter to calculate the number of target keywords; The complexity of the target structured query data is calculated based on the number of target keywords and the complexity calculation rules.

8. The method according to claim 6 or 7, characterized in that, The method further includes: Obtain the traceability indication data of the task submission instruction, wherein the traceability indication data includes at least one of the following: submission object identifier and submission device identifier, and the traceability indication data is used to trace the source of the target task; The source tracing indication data is sent to the task scheduling device so that the task scheduling device stores the source tracing indication data.

9. A task processing device, characterized in that, The device operates on a task scheduling device, and the device includes: The first input unit is used to receive the target task submitted by the task submission device. The first processing unit is used to extract parameter setting content from the target task based on the submission task type corresponding to the target task. The target task is obtained by the task submission device after performing complexity calculation, resource evaluation, and concatenation according to engine type on the target structured query data. The parameter setting content includes parameter information of each of the M parameters under the submission task type. The submission task type supports any one of the big data engine types: Hive, Spark, Presto, and Hadoop, where M is a non-negative integer. The first processing unit is further configured to calculate the amount of target resources required for the target task based on the parameter settings. The first processing unit is further configured to determine the remaining resource quantity of each running device cluster in at least one running device cluster, and select a matching running device cluster from the at least one running device cluster based on the target resource quantity and the remaining resource quantity of each running device cluster, wherein the remaining resource quantity of the matching running device cluster is greater than or equal to the target resource quantity. The first output unit is used to send the target task to the matching running device cluster so that the matching running device cluster executes the target task.

10. A task processing device, characterized in that, The device operates on a task submission device, and the device includes: The second input unit is used to obtain a task submission instruction, including: obtaining target structured query data and the task submission type; determining complexity calculation rules, and calculating the complexity of the target structured query data based on the complexity calculation rules; determining resource requirement assessment data based on the complexity of the target structured query data, and generating parameter setting content based on the task submission type and the resource requirement assessment data, wherein the parameter setting content includes parameter information of each of the M parameters under the task submission type; encapsulating the parameter setting content based on the task submission type to obtain a task submission instruction; the task submission instruction carries the task submission type and transmitted parameter data, and the task submission type supports any one of the big data engine types: Hive, Spark, Presto, and Hadoop. The second processing unit is used to identify parameters in the transmitted parameter data according to the submitted task type, and obtain parameter information of each parameter among M parameters under the submitted task type, where M is a non-negative integer; The second processing unit is further configured to concatenate the parameter information of each parameter according to the submitted task type to obtain a target task, wherein the target task can be identified by the corresponding running device cluster. The second output unit is used to send the target task to the task scheduling device so that the task scheduling device can schedule the target task.

11. An electronic device, characterized in that, include: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-5; or, when executed by the processor, cause the processor to perform the method according to any one of claims 6-8.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5; or, the computer instructions are used to cause the computer to perform the method according to any one of claims 6-8.

Images