Task processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a task processing method and device, computer equipment, a computer readable storage medium and a computer program product, and the method comprises the steps: determining a first initial task execution parameter of a target task based on task data of the target task; determining an initial queue of a target task indicated by the first initial task execution parameter in a task processing cluster and a queue resource of the initial queue; determining a target task execution parameter of a target task based on a queue resource and the first initial task execution parameter; and distributing the target task to a first queue indicated by the target task execution parameter. The initial task execution parameter of the target task is determined based on the task data of the target task, so that the initial task execution parameter of the target task is determined more reasonably; the task execution parameter of the target task is further optimized, so that the target task can be distributed to a more reasonable queue, a better queue resource allocation scheme is realized for the target task, and the task execution efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a task processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the continuous development of computer communication technology, the widespread application of terminals such as smart phones, tablets and laptops has triggered a rapid growth of data. Big data is increasingly having an important impact on people's work and life, the production and operation of enterprises, the operation of social systems, and the governance capacity of the country. In the field of big data, all walks of life have a wide range of demands for distributed computing and big data processing tasks (Spark tasks). For example, Spark tasks can be used to efficiently process large-scale data sets through in-memory computing and parallel processing technology. Spark tasks can also be used to clean, transform and aggregate data, as well as extract valuable information, so as to better help all walks of life complete production work based on big data. Summary of the invention

[0003] In response to the above technical problems, embodiments of the present application provide a task processing method, apparatus, computer device, computer-readable storage medium, and computer program product.

[0004] In a first aspect, an embodiment of the present application provides a task processing method, including:

[0005] Determining a first initial task execution parameter of the target task based on the task data of the target task;

[0006] Determine an initial queue of the target task indicated by the first initial task execution parameter in a task processing cluster, and queue resources of the initial queue;

[0007] Determining target task execution parameters of the target task based on the queue resources and the first initial task execution parameters;

[0008] The target task is dispatched to a first queue indicated by the target task execution parameter, where the first queue is a queue in the task processing cluster.

[0009] In a second aspect, an embodiment of the present application provides a task processing device, including:

[0010] A first determining unit, configured to determine a first initial task execution parameter of the target task based on the task data of the target task;

[0011] A second determining unit, configured to determine an initial queue of the target task indicated by the first initial task execution parameter in the task processing cluster, and queue resources of the initial queue;

[0012] A third determining unit, configured to determine a target task execution parameter of the target task based on the queue resources and the first initial task execution parameter;

[0013] The processing unit is used to dispatch the target task to a first queue indicated by the target task execution parameter, where the first queue is a queue in the task processing cluster.

[0014] In a third aspect, an embodiment of the present application further provides a computer device, including a memory storing a plurality of instructions; a processor loads instructions from the memory to execute the steps of any task processing method provided in the embodiment of the present application.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the steps of any task processing method provided in an embodiment of the present application.

[0016] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps of any task processing method provided in the embodiment of the present application.

[0017] By adopting the scheme of the embodiment of the present application, the initial task execution parameters of the target task can be determined based on the task data of the target task, which makes the determination of the initial task execution parameters of the target task more reasonable compared to fixed initial task execution parameters; and then combining the queue resources of the initial queue of the target task in the task processing cluster to determine the target task execution parameters, the task execution parameters of the target task can be further optimized, so that the target task can be dispatched to a more reasonable queue, and a better queue resource allocation plan can be implemented for the target task, thereby improving the efficiency of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a schematic diagram of an application environment of the task processing method provided in an embodiment of the present application;

[0020] Figure 2 This is a flowchart of an embodiment of a task processing method provided in an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of an application scenario of the task processing method provided in an embodiment of the present application;

[0022] Figure 4 is a flowchart of another embodiment of the task processing method provided in the embodiments of the present application;

[0023] Figure 5 It is a structural diagram of a task processing system of the task processing method provided in an embodiment of the present application;

[0024] Figure 6 is a flowchart of another embodiment of the task processing method provided in the embodiments of the present application;

[0025] Figure 7 is a structural diagram of a task processing device provided in an embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of the internal structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0028] In one embodiment of the present application, the task processing method can be run on a local terminal device or a server. When the game interaction method is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0029] In order to better understand a task processing method, apparatus, computer device, computer-readable storage medium, and computer program product provided in an embodiment of the present application, an application environment applicable to an embodiment of the present application is first described below.

[0030] See also Figure 1 , Figure 1FIG. 1 is a schematic diagram showing an application environment of a task processing method provided by an embodiment of the present application. As an implementation mode, the task processing method provided by an embodiment of the present application can be applied to the same electronic device. The electronic device can be as follows: Figure 1 The server 110 shown in the figure can be connected to the terminal device 120 through a network. The network is used to provide a medium for a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as a wired communication link, a wireless communication link, etc., which are not limited in the embodiments of the present application. Optionally, in other embodiments, the electronic device can also be a smart phone, a laptop computer, etc.

[0031] It should be understood that Figure 1 The server 110, network and terminal device 120 are merely illustrative. Depending on the implementation requirements, there may be any number of servers, networks and terminal devices. For example, the server 110 may be a physical server or a server cluster composed of multiple servers, and the terminal device 120 may be a mobile phone, a tablet, a desktop computer, a laptop computer, and the like. It is understood that the embodiments of the present application may also allow multiple terminal devices 120 to access the server 110 at the same time.

[0032] In some embodiments, the terminal device 120 can acquire the target task initiated by the user and obtain the target task initiated by the user and the target task data of the target task. Further, the terminal device 120 sends the target task and the target task data of the target task to the server 110 through the network, and when the server 110 receives the target task and the target task data of the target task, these target tasks and the target task data of the target task can be processed by the task processing method described in the embodiment of the present application.

[0033] As another implementation, see Figure 2 , users can create target tasks through the task development system, and the task development system can send the target task and the target task data of the target task to the scheduling recommendation module and the scheduling system respectively, so that the scheduling recommendation module recommends task execution parameters to the target task, and the scheduling system dispatches the target task with the configured task execution parameters to the task processing cluster (i.e. Hadoop), wherein the scheduling recommendation module is connected to the database (i.e. dorisDB) in communication. Then, the scheduling recommendation module can monitor the execution results of the target task, and the task processing cluster can send the task execution results of the target task in the task processing cluster to the scheduling recommendation module, so as to continue to recommend new task execution parameters for the target task through the scheduling recommendation module.

[0034] The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is a terminal device as an example. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0035] Spark tasks can be submitted to the task processing cluster that processes Spark tasks through batch computing, and then the task processing cluster uniformly processes all tasks based on resource management tools. However, when the existing task scheduling system submits Spark tasks to the task processing cluster for task execution, it usually uses the default task execution parameters in the task processing cluster, or uses the task execution parameters manually set for the Spark task. Therefore, during the execution of the Spark task, the task processing cluster needs to adjust the parameters adapted to the Spark task execution according to the actual execution situation. However, with the growth of business volume, more and more businesses need to generate scheduling tasks, and the number of Spark tasks accounts for an increasing proportion in the scheduling system. Due to the continuous increase in the number of scheduling tasks and the dependency level, the existing method of allocating task execution parameters for tasks is not sufficient to meet the needs of rapid parameter configuration of large batches of tasks during execution, and cannot quickly allocate adapted task execution parameters to each task, resulting in low efficiency in task execution.

[0036] For the task processing method of this embodiment, please refer to Figure 3 By adding an intelligent recommendation module (i.e., scheduling recommendation module) to the original task processing flow, before the target Spark task is actually submitted to the big data cluster (i.e., task processing cluster) for execution, the intelligent recommendation module is used to tune the task execution parameters according to the target Spark task itself, and then the real-time resources and task concurrency of the current cluster are combined to perform rapid real-time re-tuning, so as to tend to achieve the optimal resource allocation plan for the Spark task, meet the needs of rapid parameter configuration of large batches of tasks during execution, quickly allocate adaptive task execution parameters to each task, and improve the efficiency of task execution.

[0037] Please refer to Figure 4 The specific process of the task processing method may be as follows: Step 201 to Step 204, wherein:

[0038] Step 201: Determine a first initial task execution parameter of the target task based on task data of the target task.

[0039] Among them, the target task can be a Spark task. Specifically, a Spark task refers to a task executed under a distributed computing framework, and a Spark task can be a distributed computing implemented based on the mapreduce algorithm. A Spark task can be an application written by a user to process data through the interface (API) and functions provided by Spark. These tasks usually use Spark's Resilient Distributed Dataset (RDD) as a data model, and use Spark's computing engine to execute tasks in parallel on multiple computing nodes in a task processing cluster. In addition, Spark tasks can cover a variety of data processing and analysis scenarios. For example, Spark tasks can include data cleaning, transformation and aggregation, statistical analysis, machine learning training, and stream processing. Through Spark tasks, users can use Spark's high performance and scalability to process big data and perform complex data analysis and calculations in a distributed environment.

[0040] In the embodiment of the present application, the task data of the target task may include the SQL (Structured Query Language) syntax of the task, the input / output data volume of the task, the task lineage, and the source data. In the subsequent steps, the target historical tasks that are similar to the target task can be determined from the executed historical tasks (i.e., the executed tasks) through the task data of the target task.

[0041] The task execution parameters may include the queue information of the cluster queue to which the task is assigned, the CPU resources required to occupy the cluster queue, the memory resources, the specified timeout for task execution, the required number of threads, the number of concurrent threads, the shuffle read and write parameters, and the broadcast information. The queue information is used to indicate the target queue to which the target task is assigned when it is executed in the task processing cluster, and the specified timeout for task execution is the time to limit the target task to execute the task.

[0042] In one specific embodiment, see Figure 5 The present application also provides a task processing system, which includes a scheduling recommendation module, a task scheduling module, and a task processing cluster. The scheduling recommendation module is connected to the task scheduling module, and the task scheduling module is connected to the task processing cluster. The task processing cluster includes several cluster queues. Specifically, the target task to be processed and the task data of the target task can be obtained through the task scheduling module.

[0043] In one embodiment, historical task execution parameters matching the target task can be determined from historical records based on the task data of the target task and configured as the first initial task execution parameters of the target task, wherein the historical task execution parameters are used by at least one historical task when it is executed in a task processing cluster.

[0044] Specifically, the historical records may include all executed historical tasks and all corresponding task execution parameters. The present application determines the historical task execution parameters that match the target task from all task execution parameters according to the target task data, and configures them as the initial task execution parameters of the target task.

[0045] For example, see Figure 5 , the scheduling recommendation module can determine the historical task execution parameters that match the target task from the historical records based on the task data of the target task, and send the historical task execution parameters to the task scheduling module so that the task scheduling module configures the historical task execution parameters as the initial task execution parameters of the target task.

[0046] In summary, the embodiments of the present application can configure task execution parameters corresponding to similar historical tasks for the target task, so that the target task can be quickly configured with task execution parameters that are adapted to it, meeting the needs of quickly configuring parameters for large batches of tasks during execution, and quickly allocating adapted task execution parameters to each task, which can effectively improve the efficiency of task parameter configuration.

[0047] Step 202: Determine the initial queue of the target task indicated by the first initial task execution parameter in the task processing cluster and the queue resources of the initial queue.

[0048] Among them, the total resources of the task processing cluster include resources such as CPU resources, memory resources, and the number of dispatchable tasks. The total cluster queue resources of the cluster queue cannot be greater than the maximum resources of the task processing cluster. Furthermore, the cluster queue resources include CPU resources, memory resources, disk resources, and concurrency resources, among which CPU resources are the total available processor resources allocated to the queue, which can be the number of cores or percentage of the specified CPU resources. Memory resources are the total available memory resources allocated to the queue. Disk resources are the total available disk space and bandwidth allocated to the queue, which are used to store and read the input data, output data, or temporary data of the task. Concurrency resources are the maximum number of tasks executed simultaneously, or can be the number of tasks executed within a specific time window.

[0049] In an embodiment of the present application, the initial queue of the target task in the task processing cluster can be determined according to the queue information in the first initial task execution parameter, and the queue resources of the initial queue can be determined. The queue resources of the initial queue are: the available queue resources determined based on the total queue resources configured for the initial queue and the task execution parameters of other tasks executed in the initial queue and before the target task.

[0050] In a specific embodiment, the scheduling recommendation module may determine the initial queue of the target task in the task processing cluster based on the historical task execution data, and determine the queue resources of the initial queue.

[0051] Step 203: Determine a target task execution parameter of the target task based on the queue resources and the first initial task execution parameter.

[0052] Specifically, the queue resources of the initial queue can be compared with the queue resources required by the initial task execution parameters, and the target task execution parameters of the target task are finally configured based on the comparison result. At this time, if the available queue resources do not support the execution of the target task based on the first initial task execution parameters, the target task execution parameters of the target task are finally configured based on the available queue resources and the first initial task execution parameters. Alternatively, if the available queue resources support the execution of the target task based on the first initial task execution parameters, the first initial task execution parameters are configured as the target task execution parameters of the target task finally configured.

[0053] In a specific implementation, the queue resources of the initial queue can be compared with the first initial task execution parameters through the scheduling recommendation module, and the target task execution parameters of the target task final configuration can be determined based on the comparison result, so as to send the target task execution parameters to the task scheduling module so that the task scheduling module configures the target task execution parameters as the target task execution parameters of the target task.

[0054] In summary, when the target task is about to be dispatched to the task processing cluster, the embodiment of the present application can optimize the parameters again in combination with the queue resources of the initial queue indicated by the first initial task execution parameter, thereby achieving a better queue resource allocation scheme for the target task.

[0055] Step 204: dispatch the target task to a first queue indicated by the target task execution parameter, where the first queue is a queue in the task processing cluster.

[0056] In an embodiment of the present application, a corresponding target task instance can be generated based on the target task. When the task instance starts to be dispatched, after it is detected that the expected execution time point of the target task instance has been reached, the target task instance is dispatched to the target queue indicated by the target task execution parameters for task execution.

[0057] In a specific embodiment, the target task may be dispatched to a target cluster queue indicated by the target task execution parameter through a task scheduling module, so that the target task is executed through the target cluster queue when the expected execution time point is reached.

[0058] Based on the above introduction, the task processing method of the present application will be further illustrated by examples below. The specific implementation example of the task processing method is as follows.

[0059] In one embodiment, determining the first initial task execution parameter of the target task based on the task data of the target task includes:

[0060] Matching the task data of the target task with the task data of the executed task;

[0061] If the task data of the first executed task matches the task data of the target task, first initial task execution parameters of the target task are determined based on the task execution parameters of the first executed task.

[0062] In the embodiment of the present application, the target historical task (i.e., the first executed task) that is similar to the target task can be determined from the executed historical tasks (i.e., the executed tasks) through the task data of the target task, thereby determining the first executed task whose task data matches the target task in the executed tasks. For example, some specific statements and operations can be used to determine whether two tasks are similar, and the determination method includes the following methods:

[0063] (1) The basic structures of the SQL queries of the two tasks can be compared to see whether they are similar. If the structures of the two queries are very similar, it may mean that they perform similar data operations, that is, they are similar tasks.

[0064] (2) You can compare whether the column and table names used in the SQL queries of the two tasks are similar. If they reference the same columns or tables, it may mean that they are operating on similar data sets, that is, they are similar tasks.

[0065] (3) The semantic meanings of the two task SQL queries can be compared to see if they are similar. If the purposes and operations of the two task queries are very similar, then they are similar tasks.

[0066] (4) You can compare whether the result sets of the SQL queries of two tasks are similar. You can do this by executing the queries corresponding to the two tasks and comparing the results. If the result sets returned by the queries corresponding to the two tasks are similar, then they are similar tasks.

[0067] For another example, whether two tasks are similar can be determined based on the input / output data volumes of the two tasks, and the input and output data volumes of the two tasks, the units of the input and output data of the tasks, and the input and output data modes of the tasks can be compared to determine whether the two tasks are similar.

[0068] For another example, we can determine whether two tasks are similar based on their task lineage. Task lineage refers to tracking and recording the dependencies and data flows between tasks in the data processing process, which indicates which task's output is the source of a task's data, and which subsequent tasks use the output of the task. Task lineage is to find the connection between related data in the process of data tracing. We can determine the pre-dependencies between tasks by analyzing the input-output relationships between tasks. Specifically, we can compare the input data sources, dependencies, and data flow paths of two tasks to determine whether they are similar.

[0069] Optionally, in order to quickly configure task execution parameters that are adapted to the target task, meet the requirement of quickly configuring parameters when executing a large number of tasks, and quickly assign adapted task execution parameters to each task, the method may include:

[0070] Acquire first task data of the target task in at least one task similarity dimension from the task data of the target task;

[0071] Based on the first task data, determining, from the executed tasks, a first executed task that is similar to the target task in at least one task similarity dimension;

[0072] The historical task execution parameters corresponding to the first executed task are obtained and configured as the first initial task execution parameters of the target task.

[0073] Specifically, the task similarity dimension includes at least one of a task content similarity dimension, a blood relationship dimension, and an inter-task dependency dimension.

[0074] Specifically, task data of at least one dimension of the task content similarity dimension, the blood relationship dimension, and the task dependency dimension of the task data of the target task can be obtained, and based on the task data of at least one dimension, an executed task that is similar to the target task in at least one dimension is determined as the first executed task; then, the historical task execution parameters corresponding to the first executed task are obtained and configured as the initial task execution parameters of the target task.

[0075] For example, task data of a dimension similar to the task content of the task data of the target task may be obtained, that is, the SQL syntax of the target task may be obtained, and then, based on the SQL syntax of the target task, an executed task whose SQL syntax is similar to the SQL syntax of the target task may be determined as the first executed task. Specifically, at least one of the basic structure of the SQL query, the column and table names used in the SQL query, the semantic meaning of the SQL query, and the result set of the SQL query that is similar to the target task may be determined as the first executed task.

[0076] For another example, an executed task similar to the target task may be determined from executed historical tasks as the first executed task based on the input / output data volume, task lineage, and source data of the target task, which will not be elaborated here.

[0077] Optionally, in order to quickly batch process and batch adjust multiple similar target tasks to be executed, save computing resources of computer equipment, and quickly configure task execution parameters of multiple similar tasks, multiple tasks to be processed can be screened, and multiple similar tasks can be uniformly configured with task execution parameters. Specifically, this application can support the simultaneous identification of multiple similar tasks, and the specific scenarios include the following:

[0078] (1) Multiple pending tasks that have no strong dependency on unfinished upstream tasks have their pending tasks instances A, B, and C configured with the same task execution parameters. When the cluster queue resources are sufficient to support the simultaneous execution of task instances A, B, and C, the task processing cluster executes task instances A, B, and C according to the task execution parameters.

[0079] (2) When multiple pending tasks that do not currently have strong dependencies on unfinished upstream tasks have the same configuration, task instances A, B, and C are pending tasks, and when cluster queue resources can only support the simultaneous execution of task instances A and B, or only support the execution of task instance A according to a predetermined plan, then N (total available cluster resources X / optimal resources for a single task Y=N, rounded) tasks are recommended based on the maximum resources of the cluster queue resources, and the remaining resources are recommended for the remaining tasks according to the minimum configuration.

[0080] (3) When there is a to-be-executed first task instance that is strongly dependent on an upstream task that has not been completed, if the cluster queue resources can support the rapid completion of the first task, the resources of the first task are adjusted in real time to enable it to be quickly completed. After completion, the optimal execution resources are quickly allocated to the first task instance, and the first task instance is called up for execution.

[0081] (4) When there is a to-be-executed second task instance that is strongly dependent on the downstream second task, if the cluster queue resources can support the rapid completion of the second task, the resources of the second task instance are adjusted in real time to enable it to be quickly executed and completed. After completion, the optimal execution resources for the second task are quickly allocated, and the second task is called up for execution.

[0082] In one embodiment, in order to determine whether the queue resources of the initial queue are sufficient for the target task to perform task execution based on the first initial task execution parameters, the step of "determining the target task execution parameters of the target task based on the queue resources and the first initial task execution parameters" includes:

[0083] Obtaining task execution parameters of a task to be completed in the initial queue, wherein the task to be completed is a task in the initial queue that is located before the target task;

[0084] Based on the task execution parameters of the task to be completed, determining a first queue resource that the initial queue is allowed to allocate to the target task, where the first queue resource is used to execute the target task;

[0085] A target task execution parameter is determined according to the first queue resource and the first initial task execution parameter.

[0086] In an embodiment of the present application, the initial queue for subsequent task execution can be determined based on the queue information in the first initial task execution parameters, and then the task execution parameters of all other tasks executed in the initial queue and before the target task can be determined, so as to determine the available queue resources that the initial queue can use for the target task when executing the target task based on the task execution parameters of other tasks, wherein the queue resources may include CPU resources, memory resources, number of available threads, number of concurrency and other resources.

[0087] Specifically, the step of “determining the target task execution parameter according to the first queue resource and the first initial task execution parameter” includes:

[0088] Compare the first queue resources with the second queue resources, wherein the second queue resources are queue resources of the initial queue indicated by the first initial task execution parameter and required to execute the target task;

[0089] If the first queue resources are less than the second queue resources, a target task execution parameter configured for the target task is determined based on the first queue resources and the first initial task execution parameter.

[0090] Further, in order to optimize the first initial task execution parameters according to the queue resources available in the initial queue, so as to quickly configure the task execution parameters for the target task again and determine the task execution parameters adapted when the target task is executed, the step of "determining the target task execution parameters configured for the target task based on the first queue resources and the first initial task execution parameters" includes:

[0091] determining a target type of the first queue resources;

[0092] Adjusting the first initial task execution parameter based on the target type to obtain a second initial task execution parameter;

[0093] The second initial task execution parameter is configured as the target task execution parameter of the target task.

[0094] For example, if the first queue resources are less than the second queue resources, that is, the first queue resources do not support the target task to perform task execution based on the first initial task execution parameters, then the target type of the first queue resources that do not support the target task execution in the first initial task execution parameters is determined, and the target type may be a CPU resource type. Then, based on the queue available CPU resources of the CPU resource type in the first queue resources, the amount of CPU resources required to perform the target task of the CPU resource type in the first initial task execution parameters may be reduced, thereby obtaining the second initial task execution parameters.

[0095] For another example, if the first queue resources are less than the second queue resources, then the target type of the first queue resources that do not support the target task execution in the first initial task execution parameters is determined, and the target type may be a memory resource type. Then, based on the queue available memory resources of the memory resource type in the first queue resources, the amount of memory resources required to execute the target task of the memory resource type in the first initial task execution parameters may be reduced, thereby obtaining the second initial task execution parameters.

[0096] Optionally, in order to quickly configure the task execution parameters for the target task again, the method further includes:

[0097] If the first queue resource is less than the second queue resource, obtaining a third queue resource of each queue in the queue set in the task processing cluster, wherein the third queue resource is a queue resource that a queue in the queue set is allowed to allocate to the target task, and the third queue resource is used to execute the target task;

[0098] Determining a second queue from the queue set based on the third queue resources and the task data of the target task;

[0099] Based on the queue information of the second queue, adjusting the parameters of the first initial task execution parameters to obtain second initial task execution parameters;

[0100] The second initial task execution parameter is configured as the target task execution parameter of the target task.

[0101] For example, the third queue resources of each queue in the queue set in the task processing cluster can be obtained, and then based on the third queue resources of each queue and the task data of the target task, the second queue currently available for the target task to perform task execution based on the first initial task execution parameter is determined from the queue set. Then, based on the queue information of the second queue, the queue information in the first initial task execution parameter is adjusted to obtain the second initial task execution parameter, and the second initial task execution parameter is configured as the target task execution parameter of the target task, so that the subsequent target task can perform task execution operations in the second queue based on the second initial task execution parameter.

[0102] In another specific implementation, when the first queue resources are much larger than the second queue resources required by the first initial task execution parameters, the task execution parameters can be reconfigured to optimize the task execution parameters of the target task so that the target task can be executed faster and better. The step of "determining the target task execution parameters according to the first queue resources and the first initial task execution parameters" includes:

[0103] If the queue resources of the target type in the first queue resources are greater than a preset threshold, a target task execution parameter configured for the target task is determined based on the first queue resources and the first initial task execution parameter.

[0104] In one embodiment, the step of “determining the target task execution parameters configured for the target task based on the first queue resources and the first initial task execution parameters” includes:

[0105] Based on the queue resources of the target type in the first queue resources, adjusting the first initial task execution parameter to obtain a second initial task execution parameter;

[0106] The second initial task execution parameter is configured as the target task execution parameter of the target task.

[0107] Specifically, based on the queue resources of the target type in the first queue resources, the queue resources corresponding to the target queue resource information of the target type indicated by the first initial task execution parameter are added to obtain the second initial task execution parameter; then, the second initial task execution parameter is configured as the target task execution parameter of the target task.

[0108] For example, if the queue available CPU resources of the CPU resource type in the first queue resources are greater than the preset threshold corresponding to the CPU resource type, then based on the queue available CPU resources of the first queue resources, the amount of CPU resources required to execute the target task of the CPU resource type in the first initial task execution parameters is increased to obtain the second initial task execution parameters.

[0109] In one embodiment, in order to quickly execute the target task based on the pre-configured first initial task execution parameter, the method further includes:

[0110] If the first queue resources are greater than or equal to the second queue resources, the first initial task execution parameters are configured as the target task execution parameters configured for the target task.

[0111] In an embodiment of the present application, if the available CPU resources, memory resources, disk resources and concurrency resources of the initial queue support the execution of the target task based on the first initial task execution parameters, the first initial task execution parameters are configured as the target task execution parameters of the final configuration of the target task, so that the target task is executed in the initial queue based on the first initial task execution parameters.

[0112] Since the task execution of the target task is affected by many factors such as the current resource occupancy of the task processing cluster, the total amount of task throughput data, the disk read and write efficiency of the data storage node, and the network, on the one hand, the resources of the task processing cluster can be optimized. For example, if the task execution parameters configured for T+1 day are not ideal after the target task is executed, the task execution parameters of the target task can be optimized on T+2 day, so that the optimal task execution parameters can be infinitely close to each day under the condition of excluding other factors, and the scenario where the resources of the task processing cluster itself change (such as increase or decrease of resources) can be adapted. Specifically, the method provided in the embodiment of the present application also includes:

[0113] After the target task is executed, obtaining first task execution effect information corresponding to the target task and second task execution effect information of a first executed task, wherein the first executed task is a task whose task data matches the task data of the target task among the executed tasks;

[0114] comparing the first task execution effect information with the second task execution effect information;

[0115] If the second task execution effect information is better than the first task execution effect information, the task execution parameters of the first executed task are configured as the target task execution parameters of the target task.

[0116] Among them, the task execution effect information includes at least one of the task execution time and the cluster queue resources used for task execution. For example, after the task execution of the target task is completed, the first task execution time corresponding to the target task and the second task execution time of the first executed task are obtained. Then, the first task execution time is compared with the second task execution time. If the second task execution time is shorter than the first task execution time, the historical task execution parameters of the first executed task are configured as the target task execution parameters of the target task. If the second task execution time is longer than the first task execution time, the current task execution parameters of the target task are kept and continue to be configured as the target task execution parameters of the target task.

[0117] The embodiment of the present application can determine the historical task execution parameters that match the target task from the historical records based on the task data of the target task before submitting the target task to the task processing cluster for execution, and configure the historical task execution parameters as the first initial task execution parameters of the target task to perform parameter optimization configuration, thereby meeting the demand for rapid parameter configuration of a large number of tasks during execution, and quickly allocating adapted task execution parameters to each task; then, when the target task is about to be dispatched to the task processing cluster, the cluster queue resources of the initial queue indicated by the first initial task execution parameters are combined to perform parameter optimization configuration again, thereby achieving a better cluster queue resource allocation plan for the target task and improving the efficiency of task execution.

[0118] Optionally, the step of “determining the target task execution parameter of the target task based on the queue resources and the first initial task execution parameter” includes:

[0119] In a current task cycle, target task execution parameters are determined based on the queue resources and the first initial task execution parameters.

[0120] In an embodiment of the present application, the target task is a periodically executed task. In the current task cycle, the target task to be processed in the next task cycle and the task data of the target task can be obtained, and then the target task execution parameters of the target task are determined based on the queue resources and the first initial task execution parameters. Among them, the current task cycle and the next task cycle are continuous periodic time periods. In the current task cycle, the target task to be processed in the next task cycle and the target task data of the target task are obtained, and the process of determining the target task execution parameters of the target task is performed in the current task cycle.

[0121] Optionally, the method further includes:

[0122] After the target task is executed, the task execution effect information corresponding to the target task and the number of configuration times of the task execution parameters configured for the target task are obtained;

[0123] An analysis is performed based on the task execution effect information and the configuration times to generate a target analysis report for the target task.

[0124] For example, the execution results and parameter tuning records of the task instances of the target task can be collected to generate an analysis report to support the decision whether to perform cluster-level tuning on the cluster queue of the task processing cluster, as well as an analysis report on whether the current priority division of the target task is reasonable, and whether the guarantee classification needs to be changed. Among them, the execution result of the target task is the actual number of CPU resources consumed, memory size and other cluster queue resource expenditures, and actual execution time and other parameters under the premise of successful execution of the task instance. The parameter tuning record refers to the number of times the task execution parameter is tuned in the shortest successful execution record of the task within the specified period, the content of the tuning, and other data.

[0125] Specifically, the analysis report that determines whether to support the decision-making of cluster-level tuning of the cluster queues of the task processing cluster can indicate the optimization and execution results of the task throughput of each cluster queue resource in the specified period when the tasks running in the current task scheduling system are roughly unchanged or grow normally on a daily basis. For example, when the execution time and / or throughput of tasks under each cluster queue have not undergone a specified change for a long time (the number can be customized, such as increasing the task throughput by 5% in 10 minutes), the overall arrangement of the current task processing cluster and cluster queue resources can be maintained; if a negative specified change occurs, it means that the task processing cluster and cluster queue resources are insufficient to support the processing of the current task data volume, and expansion or priority adjustment is required. This analysis report is used to remind users that they need to expand or adjust the priority of the task processing cluster or cluster queue.

[0126] Specifically, the analysis report on whether the current priority division of the target task is reasonable can determine whether the current priority division of the target task is reasonable. It can be judged whether it is reasonable by determining the execution efficiency of the weighted queues assigned to the current target task and its upstream dependent tasks, and whether it meets the expected execution completion time. For example, target task A has a strong dependency as follows: UP1->UP2->…A, and the expected guaranteed completion time of target task A is 10 o'clock. In fact, under the current queue, the target task A at the end of the entire link can only be completed at 10:30 at the earliest during the entire report cycle. It can be confirmed that the current guarantee level is too low, and the task link needs to be switched to a cluster queue with a larger resource weight to perform task execution operations.

[0127] Based on the above introduction, the following example will be used to further illustrate the task processing method of this application. For example, please refer to Figure 6 Taking the application of the embodiment of the present application in a task processing system as an example, the task processing system includes a task development module, a scheduling recommendation module, a task scheduling module and a task processing cluster. The target task is a periodically executed task. The specific embodiment of the task processing method is as follows:

[0128] (1) The user can generate a target task (i.e., a Spark task) through the task development module and send the target task to the task scheduling module so that the task data of the target task is generated in the task scheduling system. At the same time, the task development module notifies the scheduling recommendation module to prepare to recommend task execution parameters.

[0129] (2) Through the scheduling recommendation module, based on the target task data of the target task, if a similar task similar to the target task can be identified in the historical decision support data of the local database, the optimal task key configuration information (i.e., task execution parameters) submitted to the task processing cluster by the similar task in the historical record is obtained, and submitted to the task scheduling module, so that the task scheduling module configures the historical task execution parameters as the first initial task execution parameters of the target task, and generates / updates the configuration of the task execution parameters of the target task in the task scheduling module.

[0130] (3) The task scheduling module generates a periodic task instance to be executed on day T+1. Before 0:00 on day T+1, the scheduling recommendation module monitors the queue information of the initial queue in the task processing cluster in real time, and determines whether the queue resources of the initial queue support the task instance of the target task to be executed based on the first initial task execution parameter. Specifically, there are two situations:

[0131] If the available queue resources do not support the execution of the target task based on the first initial task execution parameters, the target task execution parameters of the target task are determined based on the available queue resources and the first initial task execution parameters. Specifically, the task execution parameters that meet the requirements of the rapid execution of the periodic task instance of the target task on T+1 day and the fastest throughput that the task processing cluster can achieve can be calculated in real time, and pushed to the task scheduling system to modify the task execution parameters corresponding to the task instance of the target task on T+1 day in real time.

[0132] If the available queue resources support the execution of the target task based on the first initial task execution parameters, configure the first initial task execution parameters as the target task execution parameters of the final configuration of the target task, and keep the initial task execution parameters as the task execution parameters corresponding to the task instance of the target task on T+1 day.

[0133] (4) The task scheduling system starts dispatching multiple task instances of other tasks at 0:00 on T+1 day, and starts dispatching the task instance of the target task on T+1 day after the expected execution time of the target task is reached.

[0134] (5) The scheduling recommendation module calls the query interface to periodically poll the application information of the periodic task instances of the successfully executed target tasks in YARN, and compares the final task execution time of the target task with the execution time of the target historical tasks to determine whether it is necessary to push the task execution parameters of the target task on T+2. There are two specific cases:

[0135] If the second task execution duration of the target historical task is shorter than the first task execution duration of the target task, the historical task execution parameters of the target historical task are configured as the target task execution parameters of the target task.

[0136] If the second task execution duration of the target historical task is longer than the first task execution duration of the target task, the current task execution parameters of the target task are kept and continue to be configured as the target task execution parameters of the target task.

[0137] (6) Based on the collected execution results and parameter tuning records of multiple periodic task instances of the target task within a specified period, an analysis report can be regularly generated to support the decision-making on whether cluster-level tuning should be performed on the cluster queue of the task processing cluster, as well as an analysis report on whether the current priority division of the target task is reasonable and whether the guarantee classification needs to be changed.

[0138] It should be understood that, although each step in the flowcharts involved in the above-mentioned embodiments is displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0139] Based on the same inventive concept, the embodiment of the present application also provides a task processing device for implementing the task processing method involved above, and a task processing device for implementing the task processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the embodiments of one or more task processing devices and task processing devices provided below can refer to the above limitations on the task processing method and task processing method, and the specific limitations will not be repeated here.

[0140] This embodiment also provides a task processing device, which can be integrated into a terminal device. Figure 7 As shown, the task processing device may include:

[0141] A first determining unit 301 is used to determine a first initial task execution parameter of the target task based on the task data of the target task;

[0142] A second determining unit 302 is used to determine an initial queue of the target task indicated by the first initial task execution parameter in the task processing cluster, and queue resources of the initial queue;

[0143] A third determining unit 303 is used to determine a target task execution parameter of the target task based on the queue resources and the first initial task execution parameter;

[0144] The processing unit 304 is configured to dispatch the target task to a first queue indicated by the target task execution parameter, where the first queue is a queue in the task processing cluster.

[0145] The above-mentioned task processing device can determine the initial task execution parameters of the target task based on the task data of the target task, which makes the determination of the initial task execution parameters of the target task more reasonable compared to the fixed initial task execution parameters; and then determine the target task execution parameters of the target task in combination with the queue resources of the initial queue of the target task in the task processing cluster, which can further optimize the task execution parameters of the target task, so that the target task can be dispatched to a more reasonable queue, thereby realizing a better queue resource allocation plan for the target task and improving the efficiency of task execution.

[0146] In some embodiments, the task processing device further includes:

[0147] A first processing subunit, used for matching the task data of the target task with the task data of the executed task;

[0148] The first processing subunit is further configured to determine first initial task execution parameters of the target task based on the task execution parameters of the first executed task if the task data of the first executed task matches the task data of the target task.

[0149] In some embodiments, the task processing device further includes:

[0150] A first acquisition subunit is used to acquire task execution parameters of a task to be completed in the initial queue, where the task to be completed is a task in the initial queue that is located before the target task;

[0151] A first determining subunit is used to determine, based on the task execution parameters of the task to be completed, a first queue resource that the initial queue is allowed to allocate to the target task, wherein the first queue resource is used to execute the target task;

[0152] The first determining subunit is further configured to determine a target task execution parameter according to the first queue resources and the first initial task execution parameter.

[0153] In some embodiments, the task processing device further includes:

[0154] A second processing subunit is configured to compare the first queue resources with second queue resources, wherein the second queue resources are queue resources of the initial queue indicated by the first initial task execution parameter and required to execute the target task;

[0155] The second determining subunit is configured to determine, if the first queue resources are less than the second queue resources, target task execution parameters configured for the target task based on the first queue resources and the first initial task execution parameters.

[0156] In some embodiments, the task processing device further includes:

[0157] A third determining subunit, configured to determine a target type of the first queue resources;

[0158] A first adjustment subunit, configured to adjust the first initial task execution parameter based on the target type to obtain a second initial task execution parameter;

[0159] The third processing subunit is used to configure the second initial task execution parameter as the target task execution parameter of the target task.

[0160] In some embodiments, the task processing device further includes:

[0161] a second acquisition subunit, configured to acquire, if the first queue resources are less than the second queue resources, a third queue resource of each queue in the queue set in the task processing cluster, wherein the third queue resource is a queue resource that a queue in the queue set is allowed to allocate to the target task, and the third queue resource is used to execute the target task;

[0162] a fourth determining subunit, configured to determine a second queue from the queue set based on the third queue resources and the task data of the target task;

[0163] a fourth processing subunit, configured to adjust the parameters of the first initial task execution parameters based on the queue information of the second queue to obtain second initial task execution parameters;

[0164] The fourth processing sub-unit is further used to configure the second initial task execution parameter as the target task execution parameter of the target task.

[0165] In some embodiments, the task processing device further includes:

[0166] The fifth determining subunit is used to determine the target task execution parameters configured for the target task based on the first queue resources and the first initial task execution parameters if the queue resources of the target type in the first queue resources are greater than a preset threshold.

[0167] In some embodiments, the task processing device further includes:

[0168] A fifth processing subunit, configured to adjust the first initial task execution parameter based on the queue resources of the target type in the first queue resources to obtain a second initial task execution parameter;

[0169] The fifth processing subunit is further used to configure the second initial task execution parameter as the target task execution parameter of the target task.

[0170] In some embodiments, the task processing device further includes:

[0171] The sixth processing subunit is configured to configure the first initial task execution parameter as the target task execution parameter configured for the target task if the first queue resources are greater than or equal to the second queue resources.

[0172] In some embodiments, the task processing device further includes:

[0173] A third acquisition subunit is used to acquire, after the task execution of the target task is completed, first task execution effect information corresponding to the target task and second task execution effect information of the first executed task, wherein the first executed task is a task whose task data in the executed tasks matches the task data of the target task;

[0174] a seventh processing subunit, configured to compare the first task execution effect information with the second task execution effect information;

[0175] The seventh processing sub-unit is further configured to configure the task execution parameters of the first executed task as the target task execution parameters of the target task if the second task execution effect information is better than the first task execution effect information.

[0176] Based on the same inventive concept, the embodiment of the present application also provides a computer device, which may be a server or a terminal device, and includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned task processing method when executing the computer program. Thereby, various functions are realized, for example:

[0177] Determining a first initial task execution parameter of the target task based on the task data of the target task;

[0178] Determine an initial queue of the target task indicated by the first initial task execution parameter in a task processing cluster, and queue resources of the initial queue;

[0179] Determining target task execution parameters of the target task based on the queue resources and the first initial task execution parameters;

[0180] The target task is dispatched to a first queue indicated by the target task execution parameter, where the first queue is a queue in the task processing cluster.

[0181] The above-mentioned computer device can determine the initial task execution parameters of the target task based on the task data of the target task, which makes the determination of the initial task execution parameters of the target task more reasonable compared to the fixed initial task execution parameters; and then determine the target task execution parameters of the target task in combination with the queue resources of the initial queue of the target task in the task processing cluster, which can further optimize the task execution parameters of the target task, so that the target task can be dispatched to a more reasonable queue, thereby realizing a better queue resource allocation plan for the target task and improving the efficiency of task execution.

[0182] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0183] In one embodiment, the computer device is a terminal device, for example, its internal structure diagram can be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a task processing method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0184] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0185] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0186] Since the computer program stored in the computer-readable storage medium can execute any one of the task processing methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the task processing methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0187] Based on the same inventive concept, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.

[0188] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0189] Any reference to the memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Re RAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0190] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0191] In the above-mentioned task processing device, computer-readable storage medium, computer equipment, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described task processing device, computer-readable storage medium, computer program product, computer equipment, and its corresponding units can refer to the description of the task processing method in the above embodiment, and will not be repeated here.

[0192] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The above is a detailed introduction to a task processing method, device, computer equipment, computer-readable storage medium and computer program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A task processing method, characterized in that: include: Determining a first initial task execution parameter of the target task based on the task data of the target task; Determine an initial queue of the target task indicated by the first initial task execution parameter in a task processing cluster, and queue resources of the initial queue; Determining target task execution parameters of the target task based on the queue resources and the first initial task execution parameters; The target task is dispatched to a first queue indicated by the target task execution parameter, where the first queue is a queue in the task processing cluster.

2. The method according to claim 1, characterized in that The step of determining the first initial task execution parameter of the target task based on the task data of the target task includes: Matching the task data of the target task with the task data of the executed task; If the task data of the first executed task matches the task data of the target task, first initial task execution parameters of the target task are determined based on the task execution parameters of the first executed task.

3. The method according to claim 1, characterized in that The determining the target task execution parameter of the target task based on the queue resources and the first initial task execution parameter includes: Obtaining task execution parameters of a task to be completed in the initial queue, wherein the task to be completed is a task in the initial queue that is located before the target task; Based on the task execution parameters of the task to be completed, determining a first queue resource that the initial queue is allowed to allocate to the target task, where the first queue resource is used to execute the target task; A target task execution parameter is determined according to the first queue resource and the first initial task execution parameter.

4. The method according to claim 3, characterized in that The determining the target task execution parameter according to the first queue resource and the first initial task execution parameter includes: Compare the first queue resources with the second queue resources, wherein the second queue resources are queue resources of the initial queue indicated by the first initial task execution parameter and required to execute the target task; If the first queue resources are less than the second queue resources, a target task execution parameter configured for the target task is determined based on the first queue resources and the first initial task execution parameter.

5. The method according to claim 4, characterized in that The determining, based on the first queue resources and the first initial task execution parameters, the target task execution parameters configured for the target task includes: determining a target type of the first queue resources; Adjusting the first initial task execution parameter based on the target type to obtain a second initial task execution parameter; The second initial task execution parameter is configured as the target task execution parameter of the target task.

6. The method according to claim 4, characterized in that The method further comprises: If the first queue resource is less than the second queue resource, obtaining a third queue resource of each queue in the queue set in the task processing cluster, wherein the third queue resource is a queue resource that a queue in the queue set is allowed to allocate to the target task, and the third queue resource is used to execute the target task; Determining a second queue from the queue set based on the third queue resources and the task data of the target task; Based on the queue information of the second queue, adjusting the parameters of the first initial task execution parameters to obtain second initial task execution parameters; The second initial task execution parameter is configured as the target task execution parameter of the target task.

7. The method according to claim 4, characterized in that The determining the target task execution parameter according to the first queue resource and the first initial task execution parameter includes: If the queue resources of the target type in the first queue resources are greater than a preset threshold, a target task execution parameter configured for the target task is determined based on the first queue resources and the first initial task execution parameter.

8. The method according to claim 7, characterized in that The determining, based on the first queue resources and the first initial task execution parameters, the target task execution parameters configured for the target task includes: Based on the queue resources of the target type in the first queue resources, adjusting the first initial task execution parameter to obtain a second initial task execution parameter; The second initial task execution parameter is configured as the target task execution parameter of the target task.

9. The method according to claim 4, characterized in that The method further comprises: If the first queue resources are greater than or equal to the second queue resources, the first initial task execution parameters are configured as the target task execution parameters configured for the target task.

10. The method according to claim 3, characterized in that The method further comprises: After the target task is executed, obtaining first task execution effect information corresponding to the target task and second task execution effect information of a first executed task, wherein the first executed task is a task whose task data matches the task data of the target task among the executed tasks; comparing the first task execution effect information with the second task execution effect information; If the second task execution effect information is better than the first task execution effect information, the task execution parameters of the first executed task are configured as the target task execution parameters of the target task.

11. A task processing device, characterized in that: The device comprises: A first determining unit, configured to determine a first initial task execution parameter of the target task based on the task data of the target task; A second determining unit, configured to determine an initial queue of the target task indicated by the first initial task execution parameter in the task processing cluster, and queue resources of the initial queue; A third determining unit, configured to determine a target task execution parameter of the target task based on the queue resources and the first initial task execution parameter; The processing unit is used to dispatch the target task to a first queue indicated by the target task execution parameter, where the first queue is a queue in the task processing cluster.

12. A computer device, characterized in that: The system comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of the task processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the task processing method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is executed by a processor to perform the steps of the task processing method according to any one of claims 1 to 10.

Images