Task execution method and device, program product and storage medium

Abstract

The embodiment of the invention provides a task execution method and device, a program product and a storage medium, and the method comprises the steps: receiving a target task sent by a task system and executor information, sent by an executor cluster, of a plurality of executors, and executing fragmentation operation on the target task according to the task type, the task function and the executor information of the target task, n subtasks are obtained; configuring a corresponding actuator for each subtask according to the actuator information of the plurality of actuators to obtain task configuration information of each subtask; generating an execution request of the target subtask; sending an execution request to an executor cluster according to the target task configuration information to indicate the executor cluster to respond to the execution request so as to search a target executor from a plurality of executors based on the target task configuration information; and sending an execution instruction to the target executor to instruct the target executor to execute the target subtask. The problem that the task cannot be efficiently executed in the prior art is solved, and the effect of efficiently executing the task is achieved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to a task execution method and device, a program product, and a storage medium. Background Art

[0002] In the current microservice architecture design, almost every microservice has the need for task scheduling. The tasks under the microservice are scattered on the application servers. In high-concurrency scenarios, when scheduling and executing tasks, operation and maintenance personnel are required to check one by one on multiple known server nodes. The workflow definition and management are complex, and there is a problem of inefficient execution of tasks. Summary of the invention

[0003] The embodiments of the present application provide a task execution method and device, a program product, and a storage medium to at least solve the problem that tasks cannot be executed efficiently in the related art.

[0004] According to an embodiment of the present application, a task execution method is provided, which is applied to a task scheduling device, wherein the task scheduling device is connected between a task system and an executor cluster, wherein the executor cluster is deployed in a task service node of the task system, and wherein the executor cluster includes a plurality of executors, including: receiving a target task sent by the task system and executor information of the plurality of executors sent by the executor cluster, and performing a slicing operation on the target task according to the task type, task function and executor information of the target task to obtain N subtasks, wherein N is a natural number greater than 1; configuring a corresponding task for each of the subtasks according to the executor information of the plurality of executors , obtain task configuration information of each of the above subtasks, wherein the above task configuration information includes the above executor information and the task information of the above subtask; generate an execution request for the target subtask, wherein the above target subtask is any subtask of the multiple subtasks, and the above execution request includes the target task configuration information of the above target subtask; send the above execution request to the above executor cluster according to the above target task configuration information to instruct the above executor cluster to respond to the above execution request to search for the target executor from the multiple executors based on the above target task configuration information; send an execution instruction to the above target executor to instruct the above target executor to execute the above target subtask.

[0005] In an exemplary embodiment, a target task sent by the task system and executor information of the plurality of executors sent by the executor cluster are received, and a sharding operation is performed on the target task according to the task type, task function and executor information of the target task to obtain N subtasks, including: determining the data volume of the execution object of the target task according to the task type and the task function, wherein the data volume is used to represent the data scale of the execution object; obtaining sharding parameters based on the data volume and the load information of the executor in the executor information, wherein the load information is used to represent the ability of the executor to carry the data volume; performing the sharding operation on the target task based on the sharding parameters to obtain N subtasks, wherein the sum of the sub-data volumes of the N subtasks is equal to the data volume.

[0006] In an exemplary embodiment, based on the data volume and the load information of the actuators in the actuator information, a sharding parameter is obtained, including: extracting the load data volume of each actuator from the load information, calculating the sum of the load data volumes, and obtaining the total load data volume of the actuator cluster; when the data volume is equal to the total load data volume, determining the first number of the actuators in the actuator cluster as the sharding parameter, wherein the first number is the total number of the actuators; when the data volume is less than the total load data volume, determining the second number of the actuators in the actuator cluster as the sharding parameter according to the load data volume of each actuator, wherein the second number is the number of the actuators whose load data volume is greater than or equal to a preset threshold.

[0007] In an exemplary embodiment, a corresponding executor is configured for each of the subtasks according to executor information of the plurality of executors to obtain task configuration information of each of the subtasks, including: when the number of the N subtasks is less than the total number of the plurality of executors, one executor is configured for each of the subtasks according to the priority information in the executor information to obtain task configuration information of each of the subtasks, wherein the task information of the subtasks includes the sub-execution object of the subtask, identification information of the executor and the priority information; when the number of the subtasks is equal to the total number of the plurality of executors, one executor is configured for each of the subtasks according to the identification information of the executors in the executor information to obtain task configuration information of each of the subtasks, wherein the task information of the subtasks includes the sub-execution object of the subtask, identification information of the sub-execution object and identification information of the executor, and the identification information of the executor includes the identification information of the subtask.

[0008] In an exemplary embodiment, before sending an execution instruction to the target executor to instruct the target executor to execute the target subtask, the method further includes: sending a status request to the target executor, wherein the status request is used to request to obtain the execution status of the target executor through the executor cluster; and in a case where the execution status includes a first field, determining that the target executor is in an idle state, wherein the first field is used to indicate that the execution state is the idle state.

[0009] In an exemplary embodiment, sending an execution instruction to the target executor to instruct the target executor to execute the target subtask includes: when determining that the execution state of the target executor is in an idle state, generating the execution instruction based on the target task configuration information in the execution request, wherein the execution instruction includes the task information of the target subtask, and the task message includes the sub-execution object of the target subtask and the sub-data volume of the sub-execution object; sending the execution instruction to the target executor to instruct the target executor to execute the target subtask based on the sub-execution object and the sub-data volume in the task information of the target subtask.

[0010] According to one embodiment of the present application, a task execution device is also provided, which is applied to a task scheduling device. The task scheduling device is connected between a task system and an executor cluster. The executor cluster is deployed in a task service node of the task system. The executor cluster includes multiple executors, including: a first receiving module, which is used to receive the target task sent by the task system and the executor information of the multiple executors sent by the executor cluster, and perform a slicing operation on the target task according to the task type, task function and executor information of the target task to obtain N subtasks, wherein N is a natural number greater than 1; a first configuration module, which is used to configure a corresponding executor for each of the subtasks according to the executor information of the multiple executors. , obtain the task configuration information of each of the above subtasks, wherein the above task configuration information includes the above executor information and the task information of the above subtask; a first generating module, used to generate an execution request of a target subtask, wherein the above target subtask is any subtask among the multiple subtasks, and the above execution request includes the target task configuration information of the above target subtask; a first indicating module, used to send the above execution request to the above executor cluster according to the above target task configuration information, so as to instruct the above executor cluster to respond to the above execution request, so as to search for the target executor from the multiple executors based on the above target task configuration information; a first sending module, used to send an execution instruction to the above target executor, so as to instruct the above target executor to execute the above target subtask.

[0011] In an exemplary embodiment, the first receiving module includes: a first determining submodule, used to determine the data volume of the execution object of the target task according to the task type and the task function, wherein the data volume is used to represent the data scale of the execution object; a first obtaining submodule, used to obtain sharding parameters based on the data volume and the load information of the executor in the executor information, wherein the load information is used to represent the ability of the executor to carry the data volume; a first executing submodule, used to perform the sharding operation on the target task based on the sharding parameters to obtain N subtasks, wherein the sum of the sub-data volumes of the N subtasks is equal to the data volume.

[0012] In an exemplary embodiment, the first obtaining module includes: a first calculating submodule, used to extract the load data amount of each of the above-mentioned actuators from the above-mentioned load information, calculate the sum of the above-mentioned load data amounts, and obtain the total load data amount of the above-mentioned actuator cluster; a first determining submodule, used to determine the first number of the above-mentioned actuators in the above-mentioned actuator cluster as the above-mentioned sharding parameter when the above-mentioned data amount is equal to the above-mentioned total load data amount, wherein the above-mentioned first number is the total number of the above-mentioned actuators; a second determining submodule, used to determine the second number of the above-mentioned actuators in the above-mentioned actuator cluster as the above-mentioned sharding parameter according to the load data amount of each of the above-mentioned actuators when the above-mentioned data amount is less than the above-mentioned total load data amount, wherein the above-mentioned second number is the number of the above-mentioned actuators whose load data amount is greater than or equal to a preset threshold.

[0013] In an exemplary embodiment, the first configuration module includes: a first configuration submodule, which is used to configure one executor for each of the subtasks according to the priority information in the executor information when the number of the N subtasks is less than the total number of the plurality of executors, so as to obtain task configuration information for each of the subtasks, wherein the task information of the subtasks includes the sub-execution object of the subtask, the identification information of the executor and the priority information; and a second configuration submodule, which is used to configure one executor for each of the subtasks according to the identification information of the executor in the executor information when the number of the subtasks is equal to the total number of the plurality of executors, so as to obtain task configuration information for each of the subtasks, wherein the task information of the subtasks includes the sub-execution object of the subtask, the identification information of the sub-execution object and the identification information of the executor, and the identification information of the executor includes the identification information of the subtask.

[0014] In an exemplary embodiment, the above-mentioned device also includes: a second sending module, used to send an execution instruction to the above-mentioned target executor to instruct the above-mentioned target executor to send a status request to the above-mentioned target executor before executing the above-mentioned target subtask, wherein the above-mentioned status request is used to request to obtain the execution status of the above-mentioned target executor through the above-mentioned executor cluster; a first determination module, used to determine that the above-mentioned target executor is in an idle state when the above-mentioned execution status includes a first field, wherein the above-mentioned first field is used to indicate that the above-mentioned execution state is the above-mentioned idle state.

[0015] In an exemplary embodiment, the first execution module includes: a second generation module, which is used to generate the execution instruction based on the target task configuration information in the execution request when it is determined that the execution state of the target executor is in an idle state, wherein the execution instruction includes the task information of the target subtask, and the task message includes the sub-execution object of the target subtask and the sub-data volume of the sub-execution object; the first execution module is used to send the execution instruction to the target executor to instruct the target executor to execute the target subtask based on the sub-execution object and the sub-data volume in the task information of the target subtask.

[0016] According to another embodiment of the present application, a computer program product is provided, including a computer program, wherein the computer program is configured for a processor to execute the steps of any of the above method embodiments.

[0017] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to enable a processor to execute the steps in any of the above method embodiments.

[0018] According to another embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor is configured to execute the computer program to perform the steps in any one of the method embodiments.

[0019] Through this application, a sharding operation is performed on the target task according to the task type, task function and executor information of the target task to obtain N subtasks, and a corresponding executor is configured for each subtask according to the executor information of multiple executors. An execution request is sent to the executor cluster according to the target task configuration information of the target subtask to instruct the executor cluster to respond to the execution request, to search for the target executor from multiple executors based on the target task configuration information, and to send an execution instruction to the target executor to instruct the target executor to execute the target subtask. Since this application can determine the executor to execute the task in combination with the task type, task function and executor information of the target task, it solves the problem of the inability to efficiently execute tasks in the related art and achieves the effect of efficiently executing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a hardware structure block diagram of a task execution method mobile terminal in an embodiment of the present application;

[0021] Figure 2 is a flowchart of a task execution method according to an embodiment of the present application;

[0022] Figure 3 It is a flowchart of a task execution method of a specific embodiment of the present application;

[0023] Figure 4 It is a structural block diagram of a task execution device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a task execution method mobile terminal in an embodiment of the present application. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a task execution method in an embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] In this embodiment, a task execution method is provided, which is applied to a task scheduling device. The task scheduling device is connected between a task system and an executor cluster. The executor cluster is deployed in a task service node of the task system. The executor cluster includes multiple executors. Figure 2 is a flowchart of a task execution method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0030] Step S202, receiving the target task sent by the task system and the executor information of the plurality of executors sent by the executor cluster, and performing a slicing operation on the target task according to the task type, task function and executor information of the target task to obtain N subtasks, wherein N is a natural number greater than 1;

[0031] Optionally, the task type of the target task refers to the classification of the task, including but not limited to database operation tasks, file processing tasks, and data analysis tasks. The task function refers to the specific function that the task needs to achieve, including but not limited to data cleaning, file merging, and data analysis. The task type and task function determine the execution method of the task and the resources required.

[0032] Optionally, the task system refers to a system for generating and managing tasks, which defines the content, type, priority, etc. of the tasks and sends the tasks to the task scheduling device.

[0033] Optionally, the actuator information refers to data describing the characteristics and status of the actuator, including but not limited to the identification information, port, resource capacity, and load data volume of the actuator.

[0034] Step S204, configuring a corresponding executor for each of the subtasks according to the executor information of the plurality of executors, and obtaining task configuration information of each of the subtasks, wherein the task configuration information includes the executor information and the task information of the subtasks;

[0035] Optionally, the task information of the subtask includes, but is not limited to, identification information of the subtask, task type of the subtask, task function of the subtask, priority information of the subtask, and dependency relationship of the subtask.

[0036] Step S206, generating an execution request of a target subtask, wherein the target subtask is any subtask among the plurality of subtasks, and the execution request includes target task configuration information of the target subtask;

[0037] Step S208, sending the execution request to the executor cluster according to the target task configuration information to instruct the executor cluster to respond to the execution request to search for a target executor from the plurality of executors based on the target task configuration information;

[0038] Optionally, sending an execution request to the executor cluster according to the target task configuration information to instruct the executor cluster to respond to the execution request to search for a target executor from multiple executors based on the target task configuration information, including: instructing the executor cluster to respond to the execution request to search for a target executor from multiple executors based on identification information of the target executor in the target task configuration information.

[0039] Step S210: sending an execution instruction to the target executor to instruct the target executor to execute the target subtask.

[0040] In this embodiment, the execution subject of the above steps can be a terminal, a server, a specific processor set in the terminal or server, or a processor or processing device set relatively independently from the terminal or server, but is not limited thereto.

[0041] Through the above steps, the target task is sliced ​​according to the task type, task function and executor information of the target task to obtain N subtasks, and the corresponding executor is configured for each subtask according to the executor information of multiple executors. An execution request is sent to the executor cluster according to the target task configuration information of the target subtask to instruct the executor cluster to respond to the execution request, to search for the target executor from multiple executors based on the target task configuration information, and to send an execution instruction to the target executor to instruct the target executor to execute the target subtask. Since the present application can determine the executor to execute the task in combination with the task type, task function and executor information of the target task, it solves the problem of inability to efficiently execute tasks in the related art and achieves the effect of efficiently executing tasks.

[0042] In an exemplary embodiment, a target task sent by the task system and executor information of the plurality of executors sent by the executor cluster are received, and a sharding operation is performed on the target task according to the task type, task function and executor information of the target task to obtain N subtasks, including: determining the data volume of the execution object of the target task according to the task type and the task function, wherein the data volume is used to represent the data scale of the execution object; obtaining sharding parameters based on the data volume and the load information of the executor in the executor information, wherein the load information is used to represent the ability of the executor to carry the data volume; performing the sharding operation on the target task based on the sharding parameters to obtain N subtasks, wherein the sum of the sub-data volumes of the N subtasks is equal to the data volume.

[0043] Optionally, for example, the target task is to clean a user data set containing 10 million records. The executor cluster consists of 5 executors, each of which has different processing capabilities and current loads. The executor information includes but is not limited to the identification information, port, resource capacity, and load information of the executor. According to the task type: data processing and task function: clean a user data set containing 10 million records, it is determined that the execution object of the target task is the user data set, and the data volume of the execution object is 10 million records. According to the load and data volume of the executor, it is decided to divide the task into 5 subtasks, each of which processes 2 million records.

[0044] This embodiment calculates the sharding parameters according to the task type and task function of the target task and the executor information of the executor cluster, thereby achieving the purpose of accurately dividing the target task to ensure efficient execution of the task.

[0045] In an exemplary embodiment, based on the data volume and the load information of the actuators in the actuator information, a sharding parameter is obtained, including: extracting the load data volume of each actuator from the load information, calculating the sum of the load data volumes, and obtaining the total load data volume of the actuator cluster; when the data volume is equal to the total load data volume, determining the first number of the actuators in the actuator cluster as the sharding parameter, wherein the first number is the total number of the actuators; when the data volume is less than the total load data volume, determining the second number of the actuators in the actuator cluster as the sharding parameter according to the load data volume of each actuator, wherein the second number is the number of the actuators whose load data volume is greater than a preset threshold.

[0046] Optionally, the load data volume is a specific data volume extracted from the load information, which is used to represent the current processing capacity of each executor. The total load data volume is the sum of the load data volumes of all executors, which is used to represent the overall load of the executor cluster. The second quantity is the minimum number of executors that can execute the target task, that is, the sum of the load data volumes of the minimum number of executors is greater than or equal to the data volume of the target task.

[0047] Optionally, for example, the target task is to process a data analysis task containing 5 million records, including 10 executors, each of which has a different amount of load data. Load information: The load data volume of executors 1 to 10 is 500,000, 600,000, 700,000, 800,000, 900,000, 1 million, 1.1 million, 1.2 million, 1.3 million, and 1.4 million records, respectively. The total load data volume is 9.5 million records. In this case, the data volume (5 million records) is less than the total load data volume (9.5 million records). The preset threshold is 1.1 million records. Second quantity: The number of executors with a load data volume greater than or equal to 1.1 million records is 4 (executors 6 to 10). Divide the target task into 4 subtasks, and each subtask is assigned to an executor with a load data volume greater than or equal to the preset threshold. Each subtask will process approximately more than one million records to ensure efficient execution of the task.

[0048] This embodiment calculates the sharding parameters according to the task type and task function of the target task and the executor information of the executor cluster, thereby achieving the purpose of accurately dividing the target task to ensure efficient execution of the task.

[0049] In an exemplary embodiment, a corresponding executor is configured for each of the subtasks according to executor information of the plurality of executors to obtain task configuration information of each of the subtasks, including: when the number of the N subtasks is less than the total number of the plurality of executors, one executor is configured for each of the subtasks according to the priority information in the executor information to obtain task configuration information of each of the subtasks, wherein the task information of the subtasks includes the sub-execution object of the subtask, identification information of the executor and the priority information; when the number of the subtasks is equal to the total number of the plurality of executors, one executor is configured for each of the subtasks according to the identification information of the executors in the executor information to obtain task configuration information of each of the subtasks, wherein the task information of the subtasks includes the sub-execution object of the subtask, identification information of the sub-execution object and identification information of the executor, and the identification information of the executor includes the identification information of the subtask.

[0050] Optionally, the priority information can be determined based on the amount of data loaded by the executor. The larger the amount of data loaded, the higher the priority of the executor. For example, the target task is to analyze a user data set containing 10 million records. The executor cluster contains 15 executors. The executors are assigned different priorities based on their processing capabilities and current loads. The larger the amount of data loaded, the higher the priority of the executor. Case 1: The number of subtasks is less than the total number of executors. For example, the target task is decomposed into 10 subtasks. The executor with the highest priority will process the subtask with the largest amount of data. The task configuration information of each subtask includes the sub-execution object, the identification information of the executor (for example, executor 1), and the priority information (such as high priority). The data volume of the sub-execution object is part of the data set and is determined based on the amount of data loaded by the executor corresponding to the subtask. Case 2: The number of subtasks is equal to the total number of executors. For example, the target task is decomposed into 15 subtasks. In this case, an executor will be configured for each subtask based on the identification information of the executor. For example, executor 1 will be assigned to subtask 1, and the task configuration information of each subtask includes the sub-execution object, the identification information of the sub-execution object (such as data set part 1) and the identification information of the executor (such as the identification information of executor 1, including the identification information of subtask 1), wherein the data volume of the sub-execution object is part of the data set and is determined based on the load data volume of the executor corresponding to the subtask.

[0051] This embodiment configures a corresponding executor for each subtask according to executor information of multiple executors, configures a suitable executor for each subtask, and generates task configuration information containing all necessary information, thereby achieving the purpose of ensuring that subtasks can be executed correctly and efficiently.

[0052] In an exemplary embodiment, before sending an execution instruction to the target executor to instruct the target executor to execute the target subtask, the method further includes: sending a status request to the target executor, wherein the status request is used to request to obtain the execution status of the target executor through the executor cluster; and in a case where the execution status includes a first field, determining that the target executor is in an idle state, wherein the first field is used to indicate that the execution state is the idle state.

[0053] In an exemplary embodiment, sending an execution instruction to the target executor to instruct the target executor to execute the target subtask includes: when determining that the execution state of the target executor is in an idle state, generating the execution instruction based on the target task configuration information in the execution request, wherein the execution instruction includes the task information of the target subtask, and the task message includes the sub-execution object of the target subtask and the sub-data volume of the sub-execution object; sending the execution instruction to the target executor to instruct the target executor to execute the target subtask based on the sub-execution object and the sub-data volume in the task information of the target subtask.

[0054] Optionally, after receiving the execution status of the target executor sent by the executor cluster, and determining from the execution status that the target executor is in an idle state, sending an execution instruction to the target executor to instruct the target executor to execute the target subtask, receiving the execution log generated by the target subtask executed by the target executor, and monitoring the execution of the target task based on the execution log.

[0055] Optionally, the data volume of the sub-execution object is a part of the data set, which is determined according to the load data volume of the executor corresponding to the subtask. For example, the target task is to analyze a user data set containing 100,000 records, and the subtasks include subtask 1, subtask 2, and subtask 3. The load data volume of the executor corresponding to subtask 1 is 40,000 records, the load data volume of the executor corresponding to subtask 2 is 40,000 records, and the load data volume of the executor corresponding to subtask 3 is 30,000 records. Then, the data volume of the sub-execution object of subtask 1 is 40,000 records, the data volume of the sub-execution object of subtask 2 is 40,000 records, and the data volume of the sub-execution object of subtask 3 is 20,000 records.

[0056] The present invention will be described below in conjunction with specific embodiments:

[0057] Figure 3 This is a flowchart of a task execution method according to a specific embodiment of the present application, comprising the following steps:

[0058] S302, create a target task: in the task system, create a target task and set the basic information of the target task, including task name, task function, task type, and executor routing strategy.

[0059] S304, split the target task: based on the data volume of the target task execution object and the load information of the executor in the executor information, obtain the sharding parameters, perform the sharding operation on the target task based on the sharding parameters, obtain the number of split subtasks, and configure the corresponding executor for each subtask. For example, the target task is to process a data analysis task containing 5 million records, which contains 10 executors, and each executor has a different load data volume. Load information: The load data volume of executors 1 to 10 is 500,000, 600,000, 700,000, 800,000, 900,000, 1 million, 1.1 million, 1.2 million, 1.3 million, and 1.4 million records respectively. The total load data volume is 9.5 million records. In this case, the data volume (5 million records) is less than the total load data volume (9.5 million records). The preset threshold is 1.1 million records. Second quantity: The number of executors whose load data volume is greater than or equal to 1.1 million records is 4 (executors 6 to 10). The target task is divided into 4 subtasks. The number of subtasks is less than the total number of executors. The corresponding subtasks are determined according to the priority information in the executor information. The data volume of the sub-execution object of subtask 1 is 1.4 million records, the data volume of the sub-execution object of subtask 2 is 1.3 million records, the data volume of the sub-execution object of subtask 3 is 1.2 million records, and the data volume of the sub-execution object of subtask 4 is 1.1 million records. The larger the load data volume, the higher the priority of the executor.

[0060] S306, based on the identification information -1 of the target task, send a status request to the target executor -1 to obtain the execution status of the target executor, and when the execution status includes the first field, determine that the target executor is in an idle state. When it is determined that the execution status of the target executor is in an idle state, generate an execution instruction based on the target task configuration information in the execution request, wherein the execution instruction includes the task information of the target subtask, and the task message includes the sub-execution object of the target subtask and the sub-data volume of the sub-execution object. Send an execution instruction to the target executor to instruct the target executor to execute the target subtask based on the sub-execution object and the sub-data volume in the task information of the target subtask.

[0061] S308, receiving an execution log generated by the target subtask executed by the target executor, and monitoring the execution of the target task based on the execution log.

[0062] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0063] In this embodiment, a task execution device is also provided, which is applied to a task scheduling device. The task scheduling device is connected between a task system and an executor cluster. The executor cluster is deployed in a task service node of the task system. The executor cluster includes a plurality of executors. The device is used to implement the above embodiments and preferred implementation modes, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0064] Figure 4 is a structural block diagram of a task execution device according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:

[0065] The first receiving module 402 is used to receive the target task sent by the task system and the executor information of the plurality of executors sent by the executor cluster, and perform a slicing operation on the target task according to the task type, task function and executor information of the target task to obtain N subtasks, where N is a natural number greater than 1;

[0066] A first configuration module 404 is used to configure a corresponding executor for each of the above subtasks according to the executor information of the plurality of executors, and obtain task configuration information of each of the above subtasks, wherein the above task configuration information includes the above executor information and the task information of the above subtasks;

[0067] A first generating module 406 is used to generate an execution request of a target subtask, wherein the target subtask is any subtask among the plurality of subtasks, and the execution request includes target task configuration information of the target subtask;

[0068] A first instruction module 408 is used to send the execution request to the executor cluster according to the target task configuration information to instruct the executor cluster to respond to the execution request to search for a target executor from the plurality of executors based on the target task configuration information;

[0069] The first sending module 410 is used to send an execution instruction to the target executor to instruct the target executor to execute the target subtask.

[0070] In an exemplary embodiment, the first receiving module includes: a first determining submodule, used to determine the data volume of the execution object of the target task according to the task type and the task function, wherein the data volume is used to represent the data scale of the execution object; a first obtaining submodule, used to obtain sharding parameters based on the data volume and the load information of the executor in the executor information, wherein the load information is used to represent the ability of the executor to carry the data volume; a first executing submodule, used to perform the sharding operation on the target task based on the sharding parameters to obtain N subtasks, wherein the sum of the sub-data volumes of the N subtasks is equal to the data volume.

[0071] In an exemplary embodiment, the first obtaining module includes: a first calculating submodule, used to extract the load data amount of each of the above-mentioned actuators from the above-mentioned load information, calculate the sum of the above-mentioned load data amounts, and obtain the total load data amount of the above-mentioned actuator cluster; a first determining submodule, used to determine the first number of the above-mentioned actuators in the above-mentioned actuator cluster as the above-mentioned sharding parameter when the above-mentioned data amount is equal to the above-mentioned total load data amount, wherein the above-mentioned first number is the total number of the above-mentioned actuators; a second determining submodule, used to determine the second number of the above-mentioned actuators in the above-mentioned actuator cluster as the above-mentioned sharding parameter according to the load data amount of each of the above-mentioned actuators when the above-mentioned data amount is less than the above-mentioned total load data amount, wherein the above-mentioned second number is the number of the above-mentioned actuators whose load data amount is greater than or equal to a preset threshold.

[0072] In an exemplary embodiment, the first configuration module includes: a first configuration submodule, which is used to configure one executor for each of the subtasks according to the priority information in the executor information when the number of the N subtasks is less than the total number of the plurality of executors, so as to obtain task configuration information for each of the subtasks, wherein the task information of the subtasks includes the sub-execution object of the subtask, the identification information of the executor and the priority information; and a second configuration submodule, which is used to configure one executor for each of the subtasks according to the identification information of the executor in the executor information when the number of the subtasks is equal to the total number of the plurality of executors, so as to obtain task configuration information for each of the subtasks, wherein the task information of the subtasks includes the sub-execution object of the subtask, the identification information of the sub-execution object and the identification information of the executor, and the identification information of the executor includes the identification information of the subtask.

[0073] In an exemplary embodiment, the above-mentioned device also includes: a second sending module, used to send an execution instruction to the above-mentioned target executor to instruct the above-mentioned target executor to send a status request to the above-mentioned target executor before executing the above-mentioned target subtask, wherein the above-mentioned status request is used to request to obtain the execution status of the above-mentioned target executor through the above-mentioned executor cluster; a first determination module, used to determine that the above-mentioned target executor is in an idle state when the above-mentioned execution status includes a first field, wherein the above-mentioned first field is used to indicate that the above-mentioned execution state is the above-mentioned idle state.

[0074] In an exemplary embodiment, the first execution module includes: a second generation module, which is used to generate the execution instruction based on the target task configuration information in the execution request when it is determined that the execution state of the target executor is in an idle state, wherein the execution instruction includes the task information of the target subtask, and the task message includes the sub-execution object of the target subtask and the sub-data volume of the sub-execution object; the first execution module is used to send the execution instruction to the target executor to instruct the target executor to execute the target subtask based on the sub-execution object and the sub-data volume in the task information of the target subtask.

[0075] An embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0076] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0077] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0078] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0079] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0080] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0081] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0082] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A task execution method, characterized in that: Applied to a task scheduling device, the task scheduling device is connected between a task system and an executor cluster, the executor cluster is deployed in a task service node of the task system, and the executor cluster includes a plurality of executors, including: Receiving the target task sent by the task system and the executor information of the plurality of executors sent by the executor cluster, and performing a slicing operation on the target task according to the task type, task function and the executor information of the target task to obtain N subtasks, wherein N is a natural number greater than 1; According to the executor information of the plurality of executors, a corresponding executor is configured for each of the subtasks to obtain task configuration information of each of the subtasks, wherein the task configuration information includes the executor information and the task information of the subtask; Generate an execution request for a target subtask, wherein the target subtask is any subtask among the multiple subtasks, and the execution request includes target task configuration information of the target subtask; Sending the execution request to the executor cluster according to the target task configuration information to instruct the executor cluster to respond to the execution request to search for a target executor from the plurality of executors based on the target task configuration information; An execution instruction is sent to the target executor to instruct the target executor to execute the target subtask.

2. The method according to claim 1, characterized in that Receiving the target task sent by the task system and the executor information of the plurality of executors sent by the executor cluster, and performing a slicing operation on the target task according to the task type, task function and the executor information of the target task to obtain N subtasks, including: Determining the data volume of the execution object of the target task according to the task type and the task function, wherein the data volume is used to represent the data scale of the execution object; Obtaining a sharding parameter based on the data volume and the load information of the executor in the executor information, wherein the load information is used to indicate the ability of the executor to carry the data volume; The slicing operation is performed on the target task based on the slicing parameters to obtain N subtasks, wherein the sum of the sub-data volumes of the N subtasks is equal to the data volume.

3. The method according to claim 2, characterized in that Based on the data volume and the load information of the executor in the executor information, a sharding parameter is obtained, including: Extracting the load data amount of each of the actuators from the load information, calculating the sum of the load data amounts, and obtaining the total load data amount of the actuator cluster; In a case where the data volume is equal to the total load data volume, determining a first number of the executors in the executor cluster as the sharding parameter, wherein the first number is the total number of the executors; In the case where the data volume is less than the total load data volume, the second number of the executors in the executor cluster is determined as the sharding parameter according to the load data volume of each executor, wherein the second number is the number of the executors whose load data volume is greater than or equal to a preset threshold.

4. The method according to claim 1, characterized in that According to the executor information of the plurality of executors, a corresponding executor is configured for each of the subtasks, and task configuration information of each of the subtasks is obtained, including: In the case where the number of the N subtasks is less than the total number of the plurality of executors, one executor is configured for each subtask according to the priority information in the executor information, to obtain task configuration information for each subtask, wherein the task information of the subtask includes the sub-execution object of the subtask, the identification information of the executor and the priority information; When the number of the subtasks is equal to the total number of the multiple executors, one executor is configured for each subtask according to the identification information of the executor in the executor information to obtain task configuration information for each subtask, wherein the task information of the subtask includes the sub-execution object of the subtask, the identification information of the sub-execution object and the identification information of the executor, and the identification information of the executor includes the identification information of the subtask.

5. The method according to claim 1, characterized in that Before sending an execution instruction to the target executor to instruct the target executor to execute the target subtask, the method further includes: Sending a status request to the target executor, wherein the status request is used to request obtaining the execution status of the target executor through the executor cluster; In a case where the execution state includes a first field, it is determined that the target executor is in an idle state, wherein the first field is used to indicate that the execution state is the idle state.

6. The method according to claim 1, characterized in that Sending an execution instruction to the target executor to instruct the target executor to execute the target subtask includes: In the case of determining that the execution state of the target executor is in an idle state, generating the execution instruction based on the target task configuration information in the execution request, wherein the execution instruction includes the task information of the target subtask, and the task message includes the sub-execution object of the target subtask and the sub-data volume of the sub-execution object; The execution instruction is sent to the target executor to instruct the target executor to execute the target subtask based on the sub-execution object and the sub-data volume in the task information of the target subtask.

7. A device for executing a task, characterized in that: Applied to a task scheduling device, the task scheduling device is connected between a task system and an executor cluster, the executor cluster is deployed in a task service node of the task system, and the executor cluster includes a plurality of executors, including: A first receiving module is used to receive the target task sent by the task system and the executor information of the plurality of executors sent by the executor cluster, and perform a slicing operation on the target task according to the task type, task function and the executor information of the target task to obtain N subtasks, wherein N is a natural number greater than 1; A first configuration module, configured to configure a corresponding executor for each of the subtasks according to the executor information of the plurality of executors, and obtain task configuration information of each of the subtasks, wherein the task configuration information includes the executor information and the task information of the subtask; A first generating module, configured to generate an execution request of a target subtask, wherein the target subtask is any subtask among the plurality of subtasks, and the execution request includes target task configuration information of the target subtask; A first instruction module, used for sending the execution request to the executor cluster according to the target task configuration information, so as to instruct the executor cluster to respond to the execution request, so as to search for a target executor from the plurality of executors based on the target task configuration information; The first sending module is used to send an execution instruction to the target executor to instruct the target executor to execute the target subtask.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 6 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 6 are implemented.

Images