Task execution method and device based on task framework, equipment and storage medium
By separating the scheduling logic and execution logic of tasks in the distributed task scheduling framework, and using the separate design of scheduler, task provisioner and executor, the problem of inefficient task execution in the existing technology is solved, and more efficient resource utilization and system stability are achieved.
Patent Information
- Application Number
- CN202411761792.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-04-25
AI Technical Summary
In the existing distributed task scheduling framework Elastic-Job-Lite, the scheduling logic and execution logic of the task are executed in the same process or thread, resulting in excessive resource usage and inefficient task execution.
A task execution method based on the task framework is designed to separate the scheduling logic and execution logic of tasks through the scheduler, task provisioner and executor. The scheduler determines the target task and sends its execution packet to the task provisioner, which creates the target process and places the executor to execute the task.
By separating the scheduling logic and execution logic, resource utilization is optimized, resource waste is avoided, and task execution efficiency and system scalability, maintainability, reliability and stability are improved.
Smart Images
Figure CN119883539A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of distributed timed task frameworks, and in particular to a task execution method and device based on a task framework, an electronic device, and a storage medium. Background Art
[0002] In the related art, the Elastic-Job-Lite framework (a distributed task scheduling framework) is used to provide coordination services for distributed tasks. However, the Elastic-Job-Lite framework is based on the program of the deployed job framework, which triggers and schedules tasks at corresponding time points. The scheduling logic and execution logic of the task are executed in the same process or thread. If the scheduling logic occupies a large amount of resources, the execution logic of the task cannot fully utilize the resources, resulting in low execution efficiency of the task.
[0003] Therefore, how to separate the scheduling logic and execution logic of a task, and thereby improve the execution efficiency of the task, is a problem that needs to be solved urgently in this field. Summary of the invention
[0004] The embodiment of the present application provides a task execution method based on a task framework to solve the problem of low task execution efficiency caused by the inability to separate the scheduling logic and the execution logic of the task.
[0005] Correspondingly, an embodiment of the present application also provides a task execution device based on a task framework, an electronic device and a storage medium to ensure the implementation and application of the above method.
[0006] In order to solve the above problems, an embodiment of the present application discloses a task execution method based on a task framework, wherein the task framework includes a scheduler, a task dispatcher and an executor, and the scheduler includes a scheduling logic for triggering a task. The method includes:
[0007] Determine the currently triggered target task through the scheduling logic; the target task has a corresponding execution package, and the execution package corresponding to the target task includes the execution logic corresponding to the target task;
[0008] Sending the execution package corresponding to the target task to the task dispatcher;
[0009] Control the task dispatcher to create a target process corresponding to the target task according to the execution package corresponding to the target task;
[0010] The executor is placed in the target process to execute the target task according to the execution logic corresponding to the target task.
[0011] Optionally, the task framework further includes an updater and a registration center, the scheduler and the task dispatcher communicate through the registration center, and the sending of the execution package corresponding to the target task to the task dispatcher includes:
[0012] Uploading the execution package corresponding to the target task to the updater;
[0013] Sending the execution package corresponding to the target task in the updater to the registration center;
[0014] Comparing whether the execution package corresponding to the target task in the registration center is consistent with the initial execution package corresponding to the target task in the task dispatcher;
[0015] If the execution package corresponding to the target task in the registration center is inconsistent with the initial execution package corresponding to the target task in the task dispatcher, the execution package corresponding to the target task in the updater is sent to the task dispatcher.
[0016] Optionally, the scheduler is in communication with a database, the database includes trigger conditions corresponding to a plurality of tasks, and the determining of the currently triggered target task by the scheduling logic includes:
[0017] Scan the trigger conditions corresponding to the multiple tasks respectively through the scheduling logic, and determine whether there is a task that meets the trigger condition;
[0018] If there is a task that satisfies the trigger condition, the task that satisfies the trigger condition is determined as the target task.
[0019] Optionally, the executor includes a plurality of hosts for executing tasks, the database also includes execution information corresponding to the plurality of tasks, and before controlling the task dispatcher to create a target process corresponding to the target task according to the execution package corresponding to the target task, the method further includes:
[0020] Controlling the scheduler to obtain execution information corresponding to the target task;
[0021] Control the scheduler to create a task node corresponding to the target task in the registration center according to the execution information corresponding to the target task;
[0022] The scheduler is controlled to determine a target host in the executor for executing the target task according to execution information corresponding to the target task, and host information corresponding to the target host is stored in a task node corresponding to the target task.
[0023] Optionally, after placing the executor in the target process to execute the target task according to the execution logic corresponding to the target task, the method further includes:
[0024] The task dispatcher monitors the startup status of the target host within a preset time, and uploads the startup status to the task node corresponding to the target task;
[0025] Monitor the task node corresponding to the target task to determine whether the target host is successfully started within a preset time;
[0026] If the target host fails to start successfully within the preset time, the scheduler determines a backup host other than the target host in the executor;
[0027] The standby host is controlled to execute the target task in the target process according to the execution logic corresponding to the target task.
[0028] Optionally, after placing the executor in the target process to execute the target task according to the execution logic corresponding to the target task, the method further includes:
[0029] Determining, in the executor, job information corresponding to the target task;
[0030] The executor is controlled to store the job information corresponding to the target task in the task node corresponding to the target task.
[0031] Optionally, after controlling the executor to store the job information corresponding to the target task in the task node corresponding to the target task, the method further includes:
[0032] If the scheduler monitors that the task node corresponding to the target task is updated, the job information corresponding to the target task is obtained through the scheduler, so that the scheduler stores the job information corresponding to the target task in the database.
[0033] The embodiment of the present application further discloses a task execution device based on a task framework, wherein the task framework includes a scheduler, a task dispatcher and an executor, wherein the scheduler includes a scheduling logic for triggering a task, and the device includes:
[0034] A target task determination module, used to determine the currently triggered target task through the scheduling logic; the target task has a corresponding execution package, and the execution package corresponding to the target task includes the execution logic corresponding to the target task;
[0035] An execution package sending module, used for sending the execution package corresponding to the target task to the task dispatcher;
[0036] A target process creation module, used to control the task dispatcher to create a target process corresponding to the target task according to the execution package corresponding to the target task;
[0037] The target task execution module is used to place the executor in the target process and execute the target task according to the execution logic corresponding to the target task.
[0038] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes a task execution method based on a task framework as described in any one of the embodiments of the present application.
[0039] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the task execution method based on the task framework as described in any one of the embodiments of the present application.
[0040] Compared with the prior art, the embodiments of the present application have the following advantages:
[0041] In an embodiment of the present application, the currently triggered target task is determined by the scheduling logic; the target task has a corresponding execution package, and the execution package corresponding to the target task includes the execution logic corresponding to the target task; the execution package corresponding to the target task is sent to the task dispatcher; the task dispatcher is controlled to create a target process corresponding to the target task according to the execution package corresponding to the target task; the executor is placed in the target process to execute the target task according to the execution logic corresponding to the target task. The embodiment of the present application determines the currently triggered target task through dynamic scheduling logic, and sends its corresponding execution package to the task dispatcher, thereby realizing the flexibility and efficiency improvement of task execution. The execution logic contained in the execution package makes the task execution more modular, easy to maintain and expand. The task dispatcher creates the target process according to the execution package, optimizes resource utilization, avoids resource waste, and improves resource utilization. The executor executes tasks according to the execution logic in the target process, enhances the scalability and maintainability of the system, and improves the reliability and stability of the system through process isolation and execution logic control. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a flowchart of the steps of an embodiment of a task execution method based on a task framework of the present application;
[0043] Figure 2 It is a flowchart of a task execution of this application;
[0044] Figure 3It is a structural block diagram of an embodiment of a task execution device based on a task framework of the present application;
[0045] Figure 4 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0046] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0047] Reference Figure 1 , is a flowchart of a task execution method embodiment based on a task framework of the present application, wherein the task framework includes a scheduler, a task dispatcher and an executor, and the scheduler includes a scheduling logic for triggering a task, including the following steps:
[0048] Step 101: determine the currently triggered target task through the scheduling logic; the target task has a corresponding execution package, and the execution package corresponding to the target task includes the execution logic corresponding to the target task;
[0049] The task framework in the embodiment of the present application is a distributed scheduled task framework (crawl-job), which is mainly composed of five parts: a scheduler (scheduler), a task dispatcher (worker-builder), an executor (worker), a registration center (zookeeper) and a jar package updater (jar-updater).
[0050] Zookeeper is a distributed service management framework based on the observer design pattern. It is responsible for managing the data that needs attention and then accepts the registration of observers. Once the status of this data changes, Zookeeper will be responsible for notifying the observers registered with Zookeeper to respond accordingly.
[0051] Scheduled tasks are short tasks that run at a specified time period. They are used in many scenarios, such as synchronizing third-party data at dawn, cleaning up junk data at dawn, sending weekly reports, etc. Distributed scheduled tasks are a way to manage scheduled tasks by integrating scattered and unreliable scheduled tasks into a unified platform and implementing cluster management scheduling and distributed deployment.
[0052] In the embodiment of the present application, the task scheduler and the scheduler do not need to communicate through the network. As long as the task scheduler can access zookeeper and the scheduler can access zookeeper, the scheduler and the task scheduler do not need to communicate through the network.
[0053] In the related art, the framework closest to the embodiment of the present application is elastic-job-lite (a distributed task scheduling framework, open sourced by Dangdang.com and secondary developed based on Quartz). The purpose of elastic-job-lite is to provide a lightweight decentralized solution, using the form of a jar package to provide the lightest distributed task coordination service. The only thing it relies on during runtime is zookeeper. The solution has the following characteristics:
[0054] 1. Distributed scheduling and coordination, decentralization;
[0055] 2. Elastic expansion and reduction of capacity;
[0056] 3. Failover;
[0057] 4. Missed execution information re-trigger;
[0058] 5. Job shard consistency ensures that the same shard has only one execution instance in a distributed environment.
[0059] Elastic-job-lite will store the job configuration of the scheduled task in a JSON format on a zookeeper node. The registration center is only used for job registration and monitoring information storage. This solution is not friendly to one-time tasks. The task is only executed once, but all task metadata will be stored on the node. At the same time, the terminated nodes will not be deleted, which will lead to the existence of a large number of nodes and affect the performance of zookeeper. In addition, the decentralization of elastic-job-lite does not have a job scheduling center node, but is based on the program that deploys the job framework to trigger the scheduling at the corresponding time point. Resource scheduling and business execution are not decoupled, which will occupy business machine resources.
[0060] Scheduling logic refers to the logical module or algorithm in the scheduler that is responsible for task scheduling. The scheduling logic determines which task should be triggered for execution based on factors such as the current system status, task priority, and time conditions.
[0061] Therefore, in the embodiment of the present application, the scheduled task that is triggered to execute among multiple scheduled tasks is determined by scheduling logic as the target task. The target task can be a periodic task with a long time period, a long time consumption, and is insensitive to scheduling time delay.
[0062] The execution package corresponding to the target task is a collection of all the information required for task execution. This execution package is usually a data structure or file that contains the specific execution logic and related configuration of the task. The code or instructions contained in the execution package define the specific execution steps and operations of the target task. The execution logic is the core part of the correct execution of the task.
[0063] Step 102: Send the execution package corresponding to the target task to the task dispatcher;
[0064] The task dispatcher in the embodiment of the present application is a module responsible for managing and allocating task execution resources. The task dispatcher is responsible for receiving execution packages and executing tasks according to the information in the execution packages.
[0065] Specifically, when the target task is a scheduled task that is currently triggered for execution, the execution package corresponding to the target task will be sent to the task dispatcher so that the task dispatcher can perform subsequent task execution operations according to the information in the execution package.
[0066] Step 103: Control the task dispatcher to create a target process corresponding to the target task according to the execution package corresponding to the target task;
[0067] In the embodiment of the present application, the execution of the target task still needs to be performed by the task dispatcher, that is, after receiving the execution package, the task dispatcher executes the task according to the information in the execution package, and when starting the job, the task dispatcher does not directly execute the job in the current process, but creates a new process for each job. Specifically, the task dispatcher creates a target process based on the information in the execution package. This target process is a process specifically used to execute the target task, ensuring that the task is executed in an independent environment to avoid interference with other tasks.
[0068] In an operating system, a process is an instance of a program that is executed and has independent memory space and system resources. Each process is independent and does not interfere with each other.
[0069] Therefore, when the task dispatcher starts a task, it starts it in the form of a process, so that the resources and data between tasks are completely isolated. When an exception occurs in one task, it will not affect other tasks.
[0070] Specifically, each task runs in an independent process, so the system resources (such as memory, CPU, file handles, etc.) between multiple tasks are completely isolated, and the resource usage of one task will not affect other tasks. In addition, since each task runs in an independent process, the data between multiple tasks is also completely isolated. The data of one task will not be accessed or modified by other tasks, ensuring data security and consistency.
[0071] If an exception occurs during the execution of a task (such as crash, deadlock, memory leak, etc.), since the task is running in an independent process, these exceptions will not affect other tasks. This isolation mechanism improves the stability of the system. Even if a problem occurs in a task, it will not cause the entire system to crash or affect the normal execution of other tasks.
[0072] Step 104: placing the executor in the target process to execute the target task according to the execution logic corresponding to the target task.
[0073] The executor in the embodiment of the present application is a module or component responsible for actually executing the task. The aforementioned target process is specifically used to execute the target task. Placing the executor in the target process can ensure that the executor can run in the environment of the target process.
[0074] The executor performs the specific operations of the target task according to the execution logic in the execution package. The execution logic defines the execution steps and operations of the task to ensure that the task can be completed in the predetermined manner.
[0075] Placing the executor in the process to execute the target task, that is, the executor executes the target task in the form of a process, avoiding resource conflicts between different tasks. The resources of each task (such as memory, CPU, file handles, etc.) are independent and will not affect each other.
[0076] When the target process is completed (that is, the target task is executed), the target process will be destroyed. This operation will not have any impact on the execution of other tasks.
[0077] The embodiment of the present application determines the currently triggered target task through dynamic scheduling logic, and sends its corresponding execution package to the task dispatcher, thereby achieving flexibility and efficiency improvement of task execution. The execution logic contained in the execution package makes task execution more modular, easy to maintain and expand. The task dispatcher creates a target process according to the execution package, optimizes resource utilization, avoids resource waste, and improves resource utilization. The executor executes tasks in the target process according to the execution logic, enhances the scalability and maintainability of the system, and improves the reliability and stability of the system through process isolation and execution logic control.
[0078] In one embodiment of the present application, the task framework further includes an updater and a registration center, the scheduler and the task dispatcher communicate through the registration center, and the step 102 of sending the execution package corresponding to the target task to the task dispatcher includes:
[0079] Uploading the execution package corresponding to the target task to the updater;
[0080] Sending the execution package corresponding to the target task in the updater to the registration center;
[0081] Comparing whether the execution package corresponding to the target task in the registration center is consistent with the initial execution package corresponding to the target task in the task dispatcher;
[0082] If the execution package corresponding to the target task in the registration center is inconsistent with the initial execution package corresponding to the target task in the task dispatcher, the execution package corresponding to the target task in the updater is sent to the task dispatcher.
[0083] In the embodiment of the present application, when a task needs to be modified, an updater is used to update the execution package, thereby realizing the hot update function.
[0084] Specifically, the updater in the embodiment of the present application is responsible for managing the update and version control of the task execution package, the registration center is used to store and manage the metadata of the task execution package, and the scheduler and the task dispatcher communicate through the registration center.
[0085] When the execution package of the target task is updated, the execution package corresponding to the target task will be uploaded to the updater, and then the updater will send the received execution package to the registration center for storage.
[0086] By relying on the registration center to record the execution package information, the execution package can be automatically distributed to the task dispatcher through the updater, reducing manual operation and maintenance.
[0087] Since the task dispatcher needs to create a process based on the execution package, it needs to determine whether the execution package is updated. At this time, it is necessary to compare the execution package in the registration center with the initial execution package in the task dispatcher to check whether the two are consistent. If the execution package in the registration center is inconsistent with the initial execution package in the task dispatcher, it means that the execution package in the task dispatcher is a version before the update, so the updated execution package needs to be sent to the task dispatcher.
[0088] Similarly, if the scheduler and task dispatcher detect that there is an update to the execution package of this task execution, they will restart themselves to apply the latest code to ensure that system components can apply the latest code in a timely manner and avoid using outdated code.
[0089] Specifically, the updater sends the received execution package to the task dispatcher for storage, so that a process can be created based on the updated execution package when subsequent tasks are executed.
[0090] In an embodiment of the present application, without restarting the system or task scheduler, hot update of task code and resources is achieved by uploading the execution package to the updater and updating it to the task scheduler. Hot update can improve the flexibility and response speed of the system and reduce system downtime caused by updates.
[0091] In one embodiment of the present application, the scheduler is in communication with a database, the database includes trigger conditions corresponding to a plurality of tasks, and the step 101, determining the currently triggered target task through the scheduling logic, includes:
[0092] Scan the trigger conditions corresponding to the multiple tasks respectively through the scheduling logic, and determine whether there is a task that meets the trigger condition;
[0093] If there is a task that satisfies the trigger condition, the task that satisfies the trigger condition is determined as the target task.
[0094] In the embodiment of the present application, the task framework is connected to the database through the scheduler for communication, and the database includes relevant information corresponding to the task, for example, the triggering conditions of the task.
[0095] Since the scheduler and the database can communicate, the scheduler will use the scheduling logic to scan the trigger conditions corresponding to multiple tasks in the database to determine whether there is a task whose trigger conditions meet the current conditions, such as system status conditions are met, time conditions are met, etc.
[0096] The scheduling logic will add the scanned trigger conditions to a timer queue in the scheduler to ensure that the trigger conditions can be efficiently managed and scheduled in the memory. This timer queue in the memory is implemented by WheelTimer (a timer) to ensure that the scheduled tasks can be efficiently managed and executed, reducing resource consumption. WheelTimer is an efficient timer implementation method, usually used to handle a large number of scheduled tasks.
[0097] If there are tasks that meet the triggering conditions, the scheduling logic determines these tasks as target tasks, that is, the tasks that need to be executed currently.
[0098] Assume that there are three tasks in the system: Task A, Task B, and Task C. Each task has a corresponding trigger condition, which is stored in the database. The scheduler establishes a communication connection with the database, and the scheduling logic scans the trigger conditions of Task A, Task B, and Task C to determine whether there is a task that meets the current system status or time conditions. Assuming that the current time meets the trigger condition of Task A, the scheduling logic determines Task A as the target task, that is, the task that needs to be executed currently.
[0099] Exemplarily, the target task may be to implement a CrawlJob class (a class for crawler tasks).
[0100] The task framework in the embodiment of the present application communicates with the database, and the scheduler can dynamically obtain the triggering conditions of the task and determine the target task according to the current system status or time conditions, thereby improving the flexibility and response speed of task scheduling; moreover, the triggering conditions of the task are stored in the database, which is convenient for centralized management and maintenance. Developers can directly modify the triggering conditions of the task through the database without modifying the scheduling logic code, which is convenient for independent development, testing and maintenance, and improves the maintainability and scalability of the system; in addition, the scheduling logic determines the target task according to the triggering conditions in the database, ensuring that the task can be correctly triggered in the predetermined manner, reducing the possibility of human error and improving the reliability of the system.
[0101] In one embodiment of the present application, the executor includes a plurality of hosts for executing tasks, the database also includes execution information corresponding to the plurality of tasks, and before the step 103 of controlling the task dispatcher to create a target process corresponding to the target task according to the execution package corresponding to the target task, the method further includes:
[0102] Controlling the scheduler to obtain execution information corresponding to the target task;
[0103] Control the scheduler to create a task node corresponding to the target task in the registration center according to the execution information corresponding to the target task;
[0104] The scheduler is controlled to determine a target host in the executor for executing the target task according to execution information corresponding to the target task, and host information corresponding to the target host is stored in a task node corresponding to the target task.
[0105] Before the task dispatcher in the embodiment of the present application creates a target process, it is necessary to determine a target host for executing the target task.
[0106] The executor includes multiple hosts, which can be distributed on different physical or virtual servers to execute different tasks. Each task has corresponding execution information, which defines on which host the task is executed and the specific configuration of the execution.
[0107] The scheduler communicates with the database to obtain the execution information corresponding to the target task. Then, based on the execution information of the target task, the scheduler creates a task node in the registration center. This task node contains the execution information of the target task for use by the task dispatcher. The scheduler also determines the target host in the executor for executing the target task based on the execution information corresponding to the target task.
[0108] The task dispatcher will continue to monitor the registration center. When the task dispatcher detects that there is a newly created task node on the registration center, it will create a target process.
[0109] The task framework in the embodiment of the present application interacts with the registration center, and the scheduler can dynamically obtain the execution information of the task and determine the target host for executing the task based on this information, thereby improving the flexibility and response speed of task allocation; the execution information is stored in the database, which is convenient for centralized management and maintenance. Developers can directly modify the execution information of the task through the database without modifying the scheduling logic code, which is convenient for independent development, testing and maintenance, and improves the maintainability and scalability of the system; the scheduler determines the target host based on the execution information in the database, ensuring that the task can be executed in the correct environment in the predetermined manner, reducing the possibility of human error and improving the reliability of the system.
[0110] In one embodiment of the present application, after the step 104 of placing the executor in the target process to execute the target task according to the execution logic corresponding to the target task, the method further includes:
[0111] The task dispatcher monitors the startup status of the target host within a preset time, and uploads the startup status to the task node corresponding to the target task;
[0112] Monitor the task node corresponding to the target task to determine whether the target host is successfully started within a preset time;
[0113] If the target host fails to start successfully within the preset time, the scheduler determines a backup host other than the target host in the executor;
[0114] The standby host is controlled to execute the target task in the target process according to the execution logic corresponding to the target task.
[0115] After the target task is executed, the task dispatcher needs to monitor the startup status of the target host and switch to the backup host to continue the task if the target host fails to start.
[0116] Specifically, the task scheduler monitors the startup status of the target host within the preset time, and uploads the startup status of the target host to the task node corresponding to the target task in the registration center; the scheduler also determines whether the target host is successfully started within the preset time based on the startup status in the task node; if the target host fails to start successfully within the preset time, the task scheduler determines a backup host for executing the target task in the executor; the task scheduler controls the backup host to execute the target task in the target process according to the execution logic corresponding to the target task.
[0117] Furthermore, if the task dispatcher monitors the startup of the target task, it will upload the successful startup status to the task node of the registration center. The subsequent scheduler will obtain the startup status of the target host from the task node after a period of time, such as 60 seconds, after the executor starts the task by monitoring the task node. If the target host fails to start, the startup status of the target host will not exist in the task node. Therefore, if the scheduler does not monitor the startup status of the target host in the task node after a certain period of time, it will be considered that the currently specified target host is unavailable, and a backup host will be determined, and the task will be transferred to the backup host for execution.
[0118] Specifically, the standby host is placed in the target process to continue executing the target task, thereby achieving the re-execution of the task.
[0119] The embodiment of the present application monitors the startup status of the target host and switches to the backup host when the startup fails. The system can ensure the high availability of the task and avoid the failure of task execution due to a single point failure. The startup status of the host is centrally managed by the registration center, which facilitates centralized monitoring and maintenance and improves the maintainability and scalability of the system. By switching the backup host, it is ensured that the task can be executed continuously in a predetermined manner, reducing task interruptions caused by host failures and improving the reliability of the system.
[0120] In one embodiment of the present application, after the step 104 of placing the executor in the target process to execute the target task according to the execution logic corresponding to the target task, the method further includes:
[0121] Determining, in the executor, job information corresponding to the target task;
[0122] The executor is controlled to store the job information corresponding to the target task in the task node corresponding to the target task.
[0123] After the task is executed, whether the task is executed successfully or failed, the job information corresponding to the target task will be stored in the task node corresponding to the target task. The job information includes the execution results, status, logs, etc. of the task, which are used for subsequent monitoring, analysis and reporting.
[0124] By storing the job information in the task node in the registration center, the embodiment of the present application enables the system to centrally manage and maintain the execution information of the task, which is convenient for subsequent monitoring, analysis and reporting; and the job information is stored in the task node to ensure that the execution process and results of the task are traceable, which is convenient for subsequent auditing and troubleshooting, and improves the reliability and transparency of the system.
[0125] In one embodiment of the present application, after controlling the executor to store the job information corresponding to the target task in the task node corresponding to the target task, the method further includes:
[0126] If the scheduler monitors that the task node corresponding to the target task is updated, the job information corresponding to the target task is obtained through the scheduler, so that the scheduler stores the job information corresponding to the target task in the database.
[0127] The scheduler in the embodiment of the present application will continuously monitor the information in the task node. If the scheduler monitors that the task node corresponding to the target task contains execution information corresponding to the target task (task node update), the task information corresponding to the target task will be persistently stored in the database.
[0128] By monitoring the updates of task nodes, the scheduler in the embodiment of the present application can obtain the job information of the target task in real time and synchronize it to the database to ensure the real-time and consistency of the job information; by persistently storing the job information in the database, the system can better monitor the execution status of the task, analyze the execution results of the task, and promptly discover and solve problems, thereby improving the monitoring and analysis capabilities of the system.
[0129] The embodiment of the present application determines the currently triggered target task through dynamic scheduling logic, and sends its corresponding execution package to the task dispatcher, thereby achieving flexibility and efficiency improvement of task execution. The execution logic contained in the execution package makes task execution more modular, easy to maintain and expand. The task dispatcher creates a target process according to the execution package, optimizes resource utilization, avoids resource waste, and improves resource utilization. The executor executes tasks in the target process according to the execution logic, enhances the scalability and maintainability of the system, and improves the reliability and stability of the system through process isolation and execution logic control.
[0130] Reference Figure 2 , is a flowchart of a task execution of this application, which specifically includes the following steps:
[0131] Step 1: The scheduler and task dispatcher start and monitor the registration center;
[0132] The scheduler obtains task update information, task node status changes, etc. by listening to the registration center, and then schedules and allocates tasks based on this information. The task dispatcher obtains task update information by listening to the registration center, and starts a process for executing tasks based on the information obtained from the listening, thereby ensuring that task scheduling and execution instructions can be obtained in real time.
[0133] Step 2.1, the program that communicates with the task framework will upload the execution package to the updater;
[0134] By storing the latest execution package in the updater, the task framework can include the latest execution package.
[0135] Step 2.2, the updater updates the execution package of the registration center;
[0136] By updating the execution package in the registry through the updater, you can ensure that the execution package information in the registry is up to date.
[0137] Step 2.3, the task dispatcher compares its own execution package information with that of the registration center;
[0138] The task dispatcher can confirm whether there is a new execution package that needs to be updated or replaced by comparing the execution package information.
[0139] Step 2.4: If the task dispatcher confirms that a new execution package needs to be updated or replaced, the execution package in the updater is updated or replaced;
[0140] By updating or replacing the execution package, you can ensure that the execution package can be updated in time, avoiding the use of the execution package before the update when executing the task.
[0141] Step 3: The scheduler periodically scans the task data (scheduling conditions, execution information, etc.) in the database;
[0142] The scheduler scans the database regularly to ensure that the scheduler can discover new tasks in a timely manner.
[0143] Step 4.1: When the scheduler detects that a new task needs to be executed, it will create a task node corresponding to the task on the registration center;
[0144] By creating task nodes in the registry, you can ensure that tasks can be correctly identified and managed in the task framework.
[0145] Step 4.2: When the task dispatcher monitors the task node, it considers that the task needs to be executed;
[0146] The existence of task nodes determines the actual execution time of tasks, which ensures that tasks can be identified and prepared for execution in a timely manner.
[0147] Step 4.3: The task dispatcher starts a process and places the executor in the process to execute the task;
[0148] Placing the executor in the process to execute tasks can ensure that the task resources between multiple tasks are isolated from each other and do not affect each other.
[0149] Step 4.4, the task dispatcher will update the startup status of the executor to the task node of the registration center;
[0150] By managing the startup status of the executor through the registration center, it can be ensured that the scheduler confirms whether the host needs to be changed to execute the task based on the startup status.
[0151] Step 4.5: After the executor completes the task, it will also update the job information corresponding to the task to the task node of the registration center;
[0152] By storing job information through task nodes, it can be ensured that the job information of the task can be recorded.
[0153] Step 4.6: When the scheduler detects that there is job information corresponding to the task in the task node of the registration center, it will obtain the job information and record it in the database;
[0154] Ensure that the execution results of the task can be persistently stored so that there is a basis for subsequent data tracking.
[0155] The embodiment of the present application determines the currently triggered target task through dynamic scheduling logic, and sends its corresponding execution package to the task dispatcher, thereby achieving flexibility and efficiency improvement of task execution. The execution logic contained in the execution package makes task execution more modular, easy to maintain and expand. The task dispatcher creates a target process according to the execution package, optimizes resource utilization, avoids resource waste, and improves resource utilization. The executor executes tasks in the target process according to the execution logic, enhances the scalability and maintainability of the system, and improves the reliability and stability of the system through process isolation and execution logic control.
[0156] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.
[0157] On the basis of the above embodiments, this embodiment further provides a task execution device based on a task framework, which is applied to electronic devices such as terminal devices and servers.
[0158] Reference Figure 3 , shows a structural block diagram of an embodiment of a task execution device based on a task framework of the present application, wherein the task framework includes a scheduler, a task dispatcher and an executor, and the scheduler includes a scheduling logic for triggering tasks, which may specifically include the following modules:
[0159] The target task determination module 301 is used to determine the currently triggered target task through the scheduling logic; the target task has a corresponding execution package, and the execution package corresponding to the target task includes the execution logic corresponding to the target task;
[0160] An execution package sending module 302 is used to send the execution package corresponding to the target task to the task dispatcher;
[0161] A target process creation module 303, used to control the task dispatcher to create a target process corresponding to the target task according to the execution package corresponding to the target task;
[0162] The target task execution module 304 is used to place the executor in the target process and execute the target task according to the execution logic corresponding to the target task.
[0163] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.
[0164] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).
[0165] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 4 An exemplary apparatus 400 that may be used to implement various embodiments described in this application is schematically illustrated.
[0166] For one embodiment, Figure 4 An exemplary apparatus 400 is shown having one or more processors 402, a control module (chip set) 404 coupled to at least one of the processor(s) 402, a memory 406 coupled to the control module 404, a non-volatile memory (NVM) / storage device 408 coupled to the control module 404, one or more input / output devices 410 coupled to the control module 404, and a network interface 412 coupled to the control module 404.
[0167] The processor 402 may include one or more single-core or multi-core processors, and the processor 402 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 400 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.
[0168] In some embodiments, the apparatus 400 may include one or more computer-readable media (e.g., memory 406 or NVM / storage device 408) having instructions 414 and one or more processors 402 combined with the one or more computer-readable media and configured to execute the instructions 414 to implement a module to perform the actions described in the present disclosure.
[0169] For one embodiment, control module 404 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 402 and / or any suitable device or component in communication with control module 404 .
[0170] The control module 404 may include a memory controller module to provide an interface to the memory 406. The memory controller module may be a hardware module, a software module, and / or a firmware module.
[0171] The memory 406 may be used, for example, to load and store data and / or instructions 414 for the device 400. For one embodiment, the memory 406 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 406 may include a double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).
[0172] For one embodiment, control module 404 may include one or more input / output controllers to provide interfaces to NVM / storage device 408 and input / output device(s) 410 .
[0173] For example, NVM / storage 408 may be used to store data and / or instructions 414. NVM / storage 408 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).
[0174] NVM / storage device 408 may include storage resources that are physically part of the device on which apparatus 400 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 408 may be accessed via input / output device(s) 410 over a network.
[0175] (One or more) input / output devices 410 may provide an interface for the apparatus 400 to communicate with any other appropriate device, and the input / output device 410 may include a communication component, an audio component, a sensor component, etc. The network interface 412 may provide an interface for the apparatus 400 to communicate through one or more networks, and the apparatus 400 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.
[0176] For one embodiment, at least one of the processor(s) 402 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 404. For one embodiment, at least one of the processor(s) 402 may be packaged together with the logic of one or more controllers of the control module 404 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 402 may be integrated on the same die with the logic of one or more controllers of the control module 404. For one embodiment, at least one of the processor(s) 402 may be integrated on the same die with the logic of one or more controllers of the control module 404 to form a system-on-chip (SoC).
[0177] In various embodiments, the device 400 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 400 may have more or fewer components and / or different architectures. For example, in some embodiments, the device 400 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.
[0178] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.
[0179] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0180] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0181] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable task framework-based task execution terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable task framework-based task execution terminal device generate instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0182] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable task execution terminal device based on the task framework to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0183] These computer program instructions can also be loaded onto a computer or other programmable task execution terminal device based on the task framework, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0184] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0185] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0186] The above is a detailed introduction to a task execution method and device based on a task framework, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A task execution method based on a task framework, characterized in that: The task framework includes a scheduler, a task dispatcher and an executor, the scheduler includes a scheduling logic for triggering a task, and the method includes: Determine the currently triggered target task through the scheduling logic; the target task has a corresponding execution package, and the execution package corresponding to the target task includes the execution logic corresponding to the target task; Sending the execution package corresponding to the target task to the task dispatcher; Control the task dispatcher to create a target process corresponding to the target task according to the execution package corresponding to the target task; The executor is placed in the target process to execute the target task according to the execution logic corresponding to the target task.
2. The method according to claim 1, characterized in that The task framework further includes an updater and a registration center, the scheduler and the task dispatcher communicate through the registration center, and the sending of the execution package corresponding to the target task to the task dispatcher includes: Uploading the execution package corresponding to the target task to the updater; Sending the execution package corresponding to the target task in the updater to the registration center; Comparing whether the execution package corresponding to the target task in the registration center is consistent with the initial execution package corresponding to the target task in the task dispatcher; If the execution package corresponding to the target task in the registration center is inconsistent with the initial execution package corresponding to the target task in the task dispatcher, the execution package corresponding to the target task in the updater is sent to the task dispatcher.
3. The method according to claim 2, characterized in that The scheduler is in communication with a database, the database includes trigger conditions corresponding to a plurality of tasks, and the currently triggered target task is determined by the scheduling logic, including: Scan the trigger conditions corresponding to the multiple tasks respectively through the scheduling logic, and determine whether there is a task that meets the trigger condition; If there is a task that satisfies the trigger condition, the task that satisfies the trigger condition is determined as the target task.
4. The method according to claim 3, characterized in that The executor includes a plurality of hosts for executing tasks, the database also includes execution information corresponding to the plurality of tasks, and before controlling the task dispatcher to create a target process corresponding to the target task according to the execution package corresponding to the target task, the method further includes: Controlling the scheduler to obtain execution information corresponding to the target task; Control the scheduler to create a task node corresponding to the target task in the registration center according to the execution information corresponding to the target task; The scheduler is controlled to determine a target host in the executor for executing the target task according to execution information corresponding to the target task, and the host information corresponding to the target host is stored in a task node corresponding to the target task.
5. The method according to claim 4, characterized in that After placing the executor in the target process and executing the target task according to the execution logic corresponding to the target task, the method further includes: The task dispatcher monitors the startup status of the target host within a preset time, and uploads the startup status to the task node corresponding to the target task; Monitor the task node corresponding to the target task to determine whether the target host is successfully started within a preset time; If the target host fails to start successfully within the preset time, the scheduler determines a backup host other than the target host in the executor; The standby host is controlled to execute the target task in the target process according to the execution logic corresponding to the target task.
6. The method according to claim 4, characterized in that After placing the executor in the target process and executing the target task according to the execution logic corresponding to the target task, the method further includes: Determining, in the executor, job information corresponding to the target task; The executor is controlled to store the job information corresponding to the target task in the task node corresponding to the target task.
7. The method according to claim 6, characterized in that After controlling the executor to store the job information corresponding to the target task in the task node corresponding to the target task, the method further includes: If the scheduler monitors that the task node corresponding to the target task is updated, the job information corresponding to the target task is obtained through the scheduler, so that the scheduler stores the job information corresponding to the target task in the database.
8. A task execution device based on a task framework, characterized in that: The task framework includes a scheduler, a task dispatcher and an executor, the scheduler includes a scheduling logic for triggering tasks, and the device includes: A target task determination module, used to determine the currently triggered target task through the scheduling logic; the target task has a corresponding execution package, and the execution package corresponding to the target task includes the execution logic corresponding to the target task; An execution package sending module, used for sending the execution package corresponding to the target task to the task dispatcher; A target process creation module, used to control the task dispatcher to create a target process corresponding to the target task according to the execution package corresponding to the target task; The target task execution module is used to place the executor in the target process and execute the target task according to the execution logic corresponding to the target task.
9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, enables the processor to execute the task execution method based on the task framework as described in any one of claims 1-7.
10. One or more machine-readable media having executable codes stored thereon, which, when executed, enable a processor to execute the task execution method based on a task framework as claimed in any one of claims 1 to 7.