Task scheduling methods, devices, electronic equipment and storage media

Abstract

This disclosure provides a task scheduling method, apparatus, electronic device, and storage medium, relating to the field of data processing technology, and particularly to the fields of intelligent search, web search, and information retrieval. The specific implementation includes: responding to a received selection instruction, selecting a target template from multiple predetermined templates according to the selection instruction; responding to received task information, creating a target task based on the task information and the target template; and scheduling the target task.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more particularly to the fields of intelligent search, web search, and information retrieval. More specifically, this disclosure provides a task scheduling method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] Currently, a range of architectures, such as search systems and recommendation systems, can be equipped with scheduled tasks to handle tasks such as data access, data processing, model training, monitoring, and maintenance. Summary of the Invention

[0003] This disclosure provides a task scheduling method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to one aspect of this disclosure, a task scheduling method is provided, comprising: in response to receiving a selection instruction, selecting a target template from a plurality of predetermined templates according to the selection instruction; in response to receiving task information, creating a target task according to the task information and the target template; and scheduling the target task.

[0005] According to another aspect of this disclosure, a task scheduling apparatus is provided, including a selection module, a creation module, and a scheduling module. The selection module is used to select a target template from a plurality of predetermined templates in response to receiving a selection instruction; the creation module is used to create a target task based on the task information and the target template in response to receiving task information; and the scheduling module is used to schedule the target task.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods provided in this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods provided in this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods provided in this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of the task scheduling method and apparatus according to embodiments of this disclosure;

[0012] Figure 2 This is a schematic flowchart of a task scheduling method according to an embodiment of the present disclosure;

[0013] Figure 3 This is a schematic flowchart of a task scheduling method according to another embodiment of the present disclosure;

[0014] Figure 4 This is a schematic flowchart of a task scheduling method according to another embodiment of the present disclosure;

[0015] Figure 5 This is a schematic system architecture diagram of a task scheduling method according to an embodiment of the present disclosure;

[0016] Figure 6 This is a schematic structural block diagram of a task scheduling apparatus according to embodiments of the present disclosure; and

[0017] Figure 7 This is a structural block diagram of an electronic device used to implement the task scheduling method of the embodiments of this disclosure. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of the task scheduling method and apparatus according to embodiments of this disclosure.

[0020] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0021] like Figure 1As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0022] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0023] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests and feed the processing results back to the terminal devices.

[0024] It should be noted that the task scheduling method provided in this embodiment can generally be executed by server 105. Correspondingly, the task scheduling device provided in this embodiment can generally be located in server 105. The task scheduling method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the task scheduling device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0025] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0026] Figure 2 This is a schematic flowchart of a task scheduling method according to an embodiment of the present disclosure.

[0027] like Figure 2 As shown, the task scheduling method 200 may include operations S210 to S230.

[0028] In operation S210, in response to receiving a selection instruction, a target template is selected from multiple predefined templates according to the selection instruction.

[0029] For example, multiple pre-defined templates can be created in advance according to actual needs, and each pre-defined template can have a corresponding template identifier.

[0030] For example, the business platform layer can provide a human-computer interaction interface, and it can also provide multiple pre-defined templates. Users can make selections on the human-computer interaction interface according to their actual needs. These selections will trigger selection instructions, allowing users to choose the target template based on the template identifiers included in the selection instructions.

[0031] In operation S220, in response to receiving task information, a target task is created based on the task information and the target template.

[0032] For example, users can also perform configuration operations at the business platform layer according to actual needs to configure task-related information, such as task parameters and code. This information can indicate the task content, execution cycle, and start time. Configuration operations will trigger configuration commands, from which the user-configured task information can be obtained.

[0033] For example, task templates and task information can be assembled into template tasks. Tasks may include data export, data import, or issuing coupons for specific products on an e-commerce platform. Tasks may also be scheduled tasks. This embodiment of the disclosure does not limit the content of the tasks.

[0034] Taking data export as an example, the target task can include multiple steps. Steps such as connecting to the database, searching for data and then performing protocol conversion and data verification can be encapsulated into a task template. Other steps required to execute the task can be configured as task information, or the data source address and the target storage address of the exported data can be configured as task information. Therefore, by assembling the task template and task information, the target task can be obtained.

[0035] In operation S230, the target task is scheduled.

[0036] For example, after creating a target task, the task can be executed at a predetermined time and a predetermined number of times, thereby achieving task scheduling.

[0037] In related technologies, multiple tasks can be developed across multiple machines. However, as the number of scheduled task scripts increases, the problems become increasingly apparent. On the one hand, with multiple scripts scattered across multiple machines, omissions are prone to occur during service handover. On the other hand, the stability requirements of the tasks are difficult to guarantee, especially in the event of a single machine failure, where tasks running on the failed machine will all malfunction. Therefore, tasks on multiple machines can be migrated to a business task system to manage multiple tasks. However, in the actual task migration process, it is necessary to develop separate scripts for each task, resulting in high costs for migrating tasks to the business task system.

[0038] According to the technical solution provided in the embodiments of this disclosure, in the process of migrating tasks from other machines to the business task system, pre-defined templates and configuration information can be used to migrate tasks. Since the pre-defined templates can be reused, it is relatively easy to migrate scheduled tasks from other machines to the business task system, thereby reducing the resource and labor costs of task migration and improving the efficiency of task migration.

[0039] In practical applications, this disclosure is applicable to scheduled tasks such as data access, data processing, and training in large-scale and complex scenarios, such as search and recommendation.

[0040] It should be noted that in practical applications, pre-defined templates and task information can be created and configured according to actual needs. This disclosure does not limit the pre-defined templates and task information.

[0041] In one example, the target template includes a task template, which contains code related to the task. The task template is suitable for creating commonly used tasks, where the execution flow is relatively fixed and generalized, and the task parameters can be customized according to the actual needs of the task. Therefore, task templates can be created based on task-related code, eliminating the need to write the task code when using the task template.

[0042] In practical applications, designers can abstract and design a general operator based on multiple similar tasks. This operator can be encapsulated as a task template, which can be loaded and executed uniformly by a general framework and executor. This allows for low-cost configuration of highly procedural tasks and stable task execution.

[0043] Accordingly, when a task template includes task code, the task information may include code parameters. It should be understood that a task template may pre-encapsulate some code for executing that type of task, but the parameters in the code are currently empty; these empty parameters can be configured as task information.

[0044] For example, a task might involve exporting a certain type of data from MySQL or another database, and then performing database creation processing on the exported data. This task could include multiple steps and involve hundreds of lines of code. In practice, without a task template, writing all the code for the task would be complex. Using a task template allows you to pre-encapsulate some steps of the task, such as connecting to the database, performing protocol conversion after data retrieval, and data validation. These encapsulated steps do not require writing code. Therefore, when a task needs to be executed, the user can select a pre-created task template and provide the MySQL (Relational Database Management System) SQL statement, UGI parameters (user account, password, address), and the target storage address to which the data needs to be exported, based on the actual requirements of the task. The task can then be created by assembling the task code with the task information. It should be understood that when multiple tasks reuse the same task template, the task information for each task will differ, and therefore, the tasks themselves will also differ.

[0045] In another example, the target template includes a business environment template, which is suitable for scenarios where business needs flexible development, such as service-type tasks with high business flexibility, or distributed script-type tasks.

[0046] Accordingly, when the target template includes a business environment template, the task information can include code related to the task. In this case, the user can configure the task code, thereby supporting the development of custom code for the business.

[0047] It's important to note that business requirements are diverse, and these requirements need to be mapped to different underlying environments. While the differences between different businesses may be significant, the underlying environments for multiple tasks within the same business line are largely the same and highly similar. Business environment templates can be created based on actual business needs. These templates are reusable, and scripts for multiple target tasks can use the same template. In practical applications, the code for the target task can be overlaid and assembled with the selected business environment template.

[0048] For example, target tasks of custom code classes can be created based on business environment templates, architecture execution frameworks, and configured task code, thereby enabling businesses to migrate from the original physical machines to the business task system at a lower cost.

[0049] For example, businesses can customize business environment templates, and the same template can be reused in multiple tasks. Business environment templates can include environments with Nginx and Redis, basic environments with PHP, and environments with basic product search components. Business environment templates can be selected according to actual needs, and the basic environment supports customization.

[0050] An architecture execution framework is a framework provided for business operations, which can be reused by different business operations without modification. Business operations can execute target tasks in parallel within this framework.

[0051] Task code is the script to be executed. It can be executed in a specific environment and supports common scripting languages ​​such as Python, Shell, and PHP. Multiple tasks can each have their own code; alternatively, multiple tasks can correspond to the same code, and the parameters of the code for different tasks can be different.

[0052] Figure 3 This is a schematic flowchart of a task scheduling method according to another embodiment of the present disclosure.

[0053] like Figure 3 As shown, the task scheduling method 300 may include operations S310 to S320, and may also include operations S331 to S333.

[0054] In some embodiments, method 300 may be performed if the target template includes a task template and the task template includes task-related code. It should be understood that method 300 may also be performed in other cases, such as if the target template includes a business environment template and the task information may include task-related code.

[0055] For example, operations S310 and S320 can refer to operations S210 and S220 described above, and will not be repeated in this embodiment.

[0056] In operation S331, in response to detecting that the current time is the trigger time of the target task, the target task is added to the message queue.

[0057] In operation S332, the target task is retrieved from the message queue using at least one of multiple worker nodes.

[0058] In operation S333, the target task is executed using at least one working node.

[0059] For example, the above operations can be achieved by configuring the database, task manager, task triggers, message queues, and worker nodes.

[0060] The configuration database can store information related to tasks, such as the task content, the task execution cycle, the time when the task was last executed, and the current running status of the task. The running status can include triggered, processing, completed, etc.

[0061] Task Manager can retrieve data from the configuration database and then determine whether the current time is the trigger time for the target task. If it is determined that the task needs to be triggered at the current time, Task Manager can send a trigger command to the task trigger.

[0062] After receiving a trigger, the task trigger performs initialization data loading. This loading process may include sending the target task to a message queue, which can be a Kafka queue. The task trigger can also split the target task into multiple subtasks.

[0063] The message queue can receive target tasks from upstream task triggers and then distribute them to at least one downstream worker node. Furthermore, when a target task is split into multiple subtasks, the message queue can send full information about each subtask to the worker nodes.

[0064] Worker nodes can receive target tasks from message queues. Multiple worker nodes can process their respective target tasks or subtasks individually to execute the target tasks.

[0065] Furthermore, in practical applications, downstream worker nodes execute target tasks by consuming messages from the upstream message queue. If a worker node fails, the message queue can distribute the target task to other worker nodes through a rebalance mechanism.

[0066] In some embodiments, the method 300 described above may further include operation S334, which updates and records the execution status of the target task in real time.

[0067] For example, task triggers can be used to add target tasks to a message queue and register them in the task manager. Registration can include updating the target task's execution status to "triggered." Another example is that when a worker node receives a target task or a subtask derived from a target task, it can synchronize its status with the task manager, thus updating the target task's status to "processing." Yet another example is that when a worker node has completed executing a target task, it can synchronize its status with the task manager, thus updating the target task's status to "completed."

[0068] For example, the Task Manager can maintain status information for target tasks submitted by task triggers and worker nodes. The Task Manager can also record this status information in a configuration database.

[0069] In addition, an observation interface can be provided to obtain the status information of the target task. In practical applications, the status information of the target task can be used to trace the cause of failure. For example, if a target task is scheduled to execute every 5 minutes but fails to execute due to timeout, the status information can be used to determine whether the failure was due to unsuccessful triggering or a failure that occurred during execution and caused it to stop.

[0070] In some embodiments, the operation of adding the target task to the message queue described above may include the following operations: in response to detecting that the target task meets a predetermined splitting condition, splitting the target task into multiple subtasks, and then adding the multiple subtasks to the message queue. Accordingly, the operation of executing the target task using at least one worker node described above may include the following operations: executing multiple subtasks in parallel using at least two worker nodes.

[0071] For example, the predetermined splitting conditions include at least one of the following: the amount of data in the target task is greater than or equal to a first threshold, and the amount of computation in the target task is greater than or equal to a second threshold.

[0072] For example, taking data export tasks as an example, for large-scale data export tasks, the data can be partitioned, with each partition corresponding to a subtask. The target task can be split into 10 subtasks, and multiple subtasks can be executed in parallel by 5 or other numbers of downstream worker nodes, thereby improving the processing efficiency of the target task.

[0073] Figure 4 This is a schematic flowchart of a task scheduling method according to another embodiment of the present disclosure.

[0074] like Figure 4 As shown, the task scheduling method 400 may include operations S410 to S420, and may also include operations S431 to S437.

[0075] In some embodiments, method 400 may be performed when the target template includes a business environment template and the task information may include task-related code. It should be understood that method 400 may also be performed in other cases, such as when the target template includes a task template and the task template includes task-related code.

[0076] For example, operations S410 and S420 can refer to operations S210 and S220 described above, and will not be repeated in this embodiment.

[0077] In operation S431, based on the scheduling interval of the target task and the expected duration required for scheduling the container, it is determined whether the container meets the predetermined pre-scheduling conditions. If it is determined that the container meets the predetermined pre-scheduling conditions, operation S432 can be executed. If it is determined that the container does not meet the predetermined pre-scheduling conditions, operation S435 can be executed.

[0078] For example, scheduling a container can mean starting the container, initializing the container, etc.

[0079] For example, a configuration server can store task-related information for the scheduler to read and trigger scheduling. The information stored by the configuration server can include information configured by the user at the business platform layer, as well as information calculated based on the actual situation of the task. This information may include the task's execution command, execution cycle, current execution status, actual duration required for container scheduling, the next time to schedule the container, and the next time to execute the target task.

[0080] For example, a container can be a Docker container.

[0081] For example, the expected duration can be determined by obtaining at least one actual duration required to schedule the container from historical data of a predetermined time period, and then determining the expected duration based on at least one actual duration and a predetermined quantile value.

[0082] For example, the scheduled time period can be the past 1 day, 7 days, 30 days, etc. The container has been scheduled multiple times within the past scheduled time period, and the actual duration required to schedule the container can be recorded each time it is scheduled. The scheduled quantile value can be the 75th quantile value, the 90th quantile value, etc.

[0083] Taking a scenario where a container was scheduled 100 times in the past 7 days, and the 90th percentile corresponds to an actual duration of 5 minutes, this means that in the past 7 days, the actual duration required for 90 container schedulings was less than or equal to 5 minutes. This indicates that the probability of scheduling a container for less than or equal to 5 minutes is 90%. Therefore, by pre-setting a percentile, we can ensure that the container is scheduled with a high probability within the expected duration.

[0084] For example, the scheduler can read the information stored in the configuration server, then compare the expected duration required to schedule the container with the task execution interval, and determine whether pre-scheduling is needed based on the comparison result.

[0085] For example, pre-scheduling can mean starting a container before the target task is executed, so that the target task can be executed directly after the container starts, without wasting time waiting for the container to start.

[0086] For example, the pre-scheduling conditions include: the scheduling interval duration is greater than the product of the expected duration and the predetermined coefficient, the predetermined coefficient is greater than or equal to 1, and preferably the predetermined coefficient is greater than or equal to 1.5.

[0087] Taking a container scheduling process with an expected duration of 10 minutes and a pre-scheduling coefficient of 1 as an example, if the scheduling interval is 1 hour, pre-scheduling can be performed. If the scheduling interval is 1 minute, pre-scheduling is not required.

[0088] Taking an expected duration of 10 minutes and a scheduling interval of 13 minutes as an example, if the pre-scheduling coefficient is equal to 1, pre-scheduling can be performed; if the pre-scheduling coefficient is equal to 1.5, pre-scheduling can be omitted. It can be seen that when the pre-scheduling coefficient is adjusted to be greater than 1, such as greater than 1.5 or even greater than 2, frequent container startups and additional resource consumption can be avoided.

[0089] In operation S432, the container is pre-scheduled before the target execution time based on the expected duration and the target execution time of the target task.

[0090] For example, containers can be pre-scheduled at a predetermined time, where the time difference between the predetermined time and the target execution time of the objective task equals the expected duration. While this method requires some time to process container scheduling, pre-scheduling allows containers to directly execute the objective task after scheduling is complete, eliminating the need to wait for scheduling to finish before executing the objective task, thus improving execution efficiency. Furthermore, scheduling containers at a predetermined time avoids the waste of resources caused by containers starting up but not executing tasks, preventing them from waiting excessively long after startup.

[0091] For example, a scheduler can be used to generate scheduling instructions, which can then be used to pre-schedule containers.

[0092] In operation S433, the target task is executed using pre-scheduled containers.

[0093] For example, a PaaS (Platform as a Service) management layer can receive pre-scheduling instructions from the scheduler, then create containers based on these instructions, initialize the business environment, run a scheduled task framework, and the framework launches service code to execute the target task. In practical applications, the PaaS management layer can use Kubernetes (K8S). Containers provide features such as resource isolation, space isolation, and independent deployment; a single container can execute multiple target tasks simultaneously.

[0094] In operation S434, once it is determined that the container has completed the target task, the container is released.

[0095] For example, because predetermined pre-scheduling conditions are met, after a container completes its target task, it needs to wait for a period of time before executing the next target task, and this waiting period is longer than the expected time required to schedule the container. Therefore, the container can be released, thus avoiding the container remaining in a running state without executing its target task and consuming additional resources, thereby saving resources.

[0096] In operation S435, the container is scheduled according to the target execution time of the target task.

[0097] For example, containers can be scheduled at the target execution time of the target task. Alternatively, containers can be pre-scheduled during the initial scheduling process, similar to operation S432.

[0098] In operation S436, the target task is executed using the scheduled container.

[0099] In operation S437, if it is determined that the container has completed the target task, the scheduled container is retained.

[0100] For example, if the expected time for container scheduling is 10 minutes, and the pre-scheduling coefficient is 1, the scheduling interval is 1 minute. It can be seen that if the container is released after scheduling, the next time the target task is executed, it will need to wait for the container to complete scheduling, causing the target task to time out. Therefore, it is possible to retain the scheduled container, thereby improving the execution efficiency of the target task and also enhancing stability.

[0101] In some embodiments, the above method may further include the following operation: migrating abnormal containers.

[0102] For example, in response to detecting that a container is in a faulty state, a new container is scheduled, and the execution status of the target task is obtained. Then, the target task is continued to be executed using the new container based on the execution status.

[0103] For example, if a container detects a fault in itself through its own runtime logic, it can report the fault to the scheduler. The scheduler can release the faulty container and schedule a new container to continue executing the target task performed by the faulty container.

[0104] Furthermore, in practical applications, a remote distributed file system can be used to synchronize the execution status of the target task. For example, during the execution of the target task, the execution status of the target task can be updated in real time in a local file, while data synchronization occurs between the local file and the remote distributed file system. When a container fails, the target task can continue running based on its execution status using the synchronized data.

[0105] In related technologies, a single-machine, single-instance approach can be used to execute the target task. However, if the machine crashes, it needs to be restarted before the target task can continue to execute, resulting in low stability. Compared to these related technologies, the embodiments disclosed in this disclosure can migrate the target task to other containers for execution after a container failure, thereby improving stability.

[0106] Figure 5 This is a schematic system architecture diagram of a task scheduling method according to an embodiment of the present disclosure.

[0107] like Figure 5 As shown, the business task system includes a business platform layer 510, a business logic layer 520, and a basic service layer 530.

[0108] The business platform layer 510 can provide a human-computer interaction interface and multiple pre-defined templates, such as multiple task templates and multiple business environment templates. Users can select the required task template or business environment template in the business platform layer 510 according to their actual needs. Users can also configure task information in the business platform layer 510, such as configuring code parameters or task code.

[0109] The business logic layer 520 can include general template tasks and custom code tasks. For service tasks with fixed processes, the processing flow of general template tasks can be used. For other tasks not covered by general template tasks, the processing flow of custom code tasks can be used.

[0110] The basic service layer 530 is used to provide basic services for the target task. For example, for data import and data export tasks, the basic service layer 530 can include various databases, such as MongoDB (a database based on distributed file storage), HDFS (Hadoop Distributed File System), MySQL, and Elasticsearch.

[0111] In practical applications, when a user selects a task template and configures code parameters at the task platform layer, it indicates that the target task belongs to the general template class. The task template and code parameters can be assembled into the target task. General template class tasks can be processed using configuration databases, task managers, task triggers, message queues, and worker nodes.

[0112] For example, a configuration database can store task-related information. The task manager can retrieve data from the configuration database and then determine whether the current time is the trigger time for the target task. If it is determined that the current time requires triggering, the task manager can send a trigger command to the task trigger. The task trigger sends the target task to a message queue, and the worker node retrieves the target task from the message queue and executes it.

[0113] In practical applications, when a user selects a business environment template at the task platform layer and configures task-related code, it indicates that the target task belongs to the custom code class. The business environment template, the architecture execution framework, and the configured task code can be assembled into the target task. Custom code class tasks can be processed using configuration servers, schedulers, PaaS management layers, and containers.

[0114] For example, a configuration server can store task-related information, which the scheduler can read to trigger scheduling. The trigger reads information from the configuration server and compares the scheduling interval of the target task with the expected duration required to schedule the container to determine whether the container meets the predetermined pre-scheduling conditions.

[0115] If predetermined pre-scheduling conditions are met, the scheduler pre-schedules containers at a predetermined time. For example, the scheduler can send a pre-scheduling instruction to the PaaS management layer, which then creates containers based on the instruction. These containers execute the target task. After executing the target task, the containers are released. The containers are pre-scheduled again for the next execution of the target task.

[0116] If the pre-scheduled conditions are not met, the scheduler schedules the container. For example, the scheduler can send a pre-scheduling instruction to the PaaS management layer, which then creates a container based on the instruction. The container executes the target task. After the container executes the target task, the scheduled container is reserved.

[0117] By adopting the aforementioned business task system, efficient business creation and migration can be ensured. In practical applications, for scenarios involving complex business tasks, the cloud-native scheduled task system can divide tasks into general template tasks and custom code tasks based on different usage scenarios. During the migration process, general template tasks can reuse task templates, and custom code tasks can reuse business environment templates, thereby achieving different levels of reuse and customization requirements and realizing low-cost batch migration of businesses.

[0118] In practical applications, the aforementioned business task system can enable the migration of custom code-based tasks from hundreds of original physical machines within a single business day. Furthermore, the average integration cycle for general template-based tasks is less than one hour, improving the stability and integration efficiency of business tasks.

[0119] Figure 6 This is a schematic structural block diagram of a task scheduling device according to an embodiment of the present disclosure.

[0120] like Figure 6As shown, the task scheduling device 600 may include a selection module 610, a creation module 620, and a scheduling module 630.

[0121] The selection module 610 is used to select a target template from a plurality of predetermined templates in response to receiving a selection instruction.

[0122] The creation module 620 is used to create a target task in response to received task information, based on the task information and the target template.

[0123] The scheduling module 630 is used to schedule the target task.

[0124] According to another embodiment of this disclosure, the scheduling module includes a first determining submodule, a first scheduling submodule, a first execution submodule, and a releasing submodule. The first determining submodule is used to determine whether a container meets predetermined pre-scheduling conditions based on the scheduling interval of the target task and the expected duration required for scheduling the container. The first scheduling submodule is used to pre-schedule the container before the target execution time, based on the expected duration and the target execution time of the target task, if the container meets the predetermined pre-scheduling conditions. The first execution submodule is used to execute the target task using the pre-scheduled container. The releasing submodule is used to release the container if the container has completed executing the target task.

[0125] According to another embodiment of this disclosure, the predetermined pre-scheduling conditions include: the scheduling interval duration is greater than the product of the expected duration and the predetermined coefficient, and the predetermined coefficient is greater than or equal to 1.

[0126] According to another embodiment of this disclosure, the first scheduling submodule includes a pre-scheduling unit for pre-scheduling containers at a predetermined time; wherein the time difference between the predetermined time and the target execution time of the target task is equal to the expected duration.

[0127] According to another embodiment of this disclosure, the apparatus further includes a second scheduling submodule, a second execution submodule, and a retention submodule. The second scheduling submodule is used to schedule the container according to the target execution time of the target task when it is determined that the container does not meet the predetermined pre-scheduling conditions; the second execution submodule is used to execute the target task using the scheduled container; and the retention submodule is used to retain the scheduled container when it is determined that the container has completed the execution of the target task.

[0128] According to another embodiment of this disclosure, the apparatus further includes a first acquisition submodule and a second determination submodule. The first acquisition submodule is used to acquire at least one actual duration required for scheduling the container from historical data of a predetermined time period; the second determination submodule is used to determine the expected duration based on at least one actual duration and a predetermined quantile value.

[0129] According to another embodiment of this disclosure, the apparatus further includes a third scheduling submodule, a second acquisition submodule, and a third execution submodule. The third scheduling submodule is used to schedule a new container in response to detecting that a container is in a fault state; the second acquisition submodule is used to acquire the execution status of the target task; and the third execution submodule is used to continue executing the target task based on the execution status using the new container.

[0130] According to another embodiment of this disclosure, the target template includes a business environment template, and the task information includes code related to the task.

[0131] According to another embodiment of this disclosure, the scheduling module includes an adding submodule, a third obtaining submodule, and a fourth execution submodule. The adding submodule is used to add the target task to a message queue in response to detecting that the current time is the trigger time of the target task; the third obtaining submodule is used to obtain the target task from the message queue using at least one of a plurality of worker nodes; and the fourth execution submodule is used to execute the target task using at least one worker node.

[0132] According to another embodiment of this disclosure, the apparatus further includes a first state determination submodule and a second state determination submodule. The first state determination submodule is configured to determine the state information of the target task as triggered in response to detecting that a target task has been added to a message queue; the second state determination submodule is configured to determine the state information of the target task as being processed in response to detecting that at least one worker node has retrieved the target task from the message queue; the third state determination submodule is configured to determine the state information of the target task as completed in response to detecting that at least one worker node has finished executing the target task; and a recording submodule is configured to record the state information of the target task.

[0133] According to another embodiment of this disclosure, the adding submodule includes a splitting unit and an adding unit. The splitting unit is used to split the target task into multiple subtasks in response to detecting that the target task meets predetermined splitting conditions; the adding unit is used to add the multiple subtasks to a message queue; wherein, the predetermined splitting conditions include at least one of the following: the data volume of the target task is greater than or equal to a first threshold, and the computational volume of the target task is greater than or equal to a second threshold; wherein, the fourth execution submodule is further used to execute the multiple subtasks in parallel using at least two worker nodes.

[0134] According to another embodiment of this disclosure, the target template includes a task template, the task template includes code related to the task, and the task information includes code parameters.

[0135] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0136] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0137] According to embodiments of this disclosure, this disclosure also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described task scheduling method.

[0138] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the above-described task scheduling method.

[0139] According to embodiments of this disclosure, this disclosure also provides a computer program product, including a computer program that implements the above-described task scheduling method when executed by a processor.

[0140] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0141] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0142] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0143] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as task scheduling methods. For example, in some embodiments, the task scheduling method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the task scheduling method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform task scheduling methods by any other suitable means (e.g., by means of firmware).

[0144] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0146] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0148] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0149] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0150] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0151] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A task scheduling method, comprising: In response to receiving a selection instruction, a target template is selected from a plurality of predetermined templates according to the selection instruction; In response to receiving task information, a target task is created based on the task information and the target template; as well as Scheduling the target task includes: Based on the scheduling interval of the target task and the expected time required to schedule the container, determine whether the container meets the predetermined pre-scheduling conditions. If it is determined that the container meets the predetermined pre-scheduling conditions, the container is pre-scheduled before the target execution time according to the expected duration and the target execution time of the target task. The target task is executed using pre-scheduled containers.

2. The method according to claim 1, wherein, The scheduling of the target task also includes: Once it is determined that the container has successfully completed the target task, the container is released.

3. The method according to claim 2, wherein, The predetermined pre-scheduling conditions include: The scheduling interval is longer than the product of the expected duration and the predetermined coefficient, where the predetermined coefficient is greater than or equal to 1.

4. The method according to claim 2, wherein, Based on the expected duration and the target execution time of the target task, pre-scheduling the container before the target execution time includes: Pre-schedule containers at predetermined times; Wherein, the time difference between the predetermined time and the target execution time of the target task is equal to the expected duration.

5. The method according to claim 2, further comprising: If it is determined that the container does not meet the predetermined pre-scheduling conditions, the container is scheduled according to the target execution time of the target task. The target task is executed using a scheduled container; as well as If it is determined that the container has completed the target task, the scheduled container is retained.

6. The method according to claim 2, further comprising: Obtain at least one actual duration required to schedule containers from historical data for a predetermined time period; as well as The expected duration is determined based on the at least one actual duration and the predetermined quantile value.

7. The method according to claim 2, further comprising: In response to the detection that a container is in a faulty state, a new container is scheduled; Obtain the execution status of the target task; as well as Using the new container, the target task continues to be executed based on the execution state.

8. The method according to any one of claims 1 to 7, wherein, The target template includes a business environment template, and the task information includes code related to the task.

9. The method according to claim 1, wherein, The scheduling of the target task includes: In response to the detection that the current time is the trigger time of the target task, the target task is added to the message queue; The target task is retrieved from the message queue using at least one of a plurality of worker nodes; and The target task is executed using at least one working node.

10. The method of claim 9, further comprising: In response to detecting that the target task has been added to the message queue, the status information of the target task is determined to be triggered; In response to detecting that at least one working node has obtained the target task from the message queue, the status information of the target task is determined to be in process; In response to detecting that at least one working node has completed the target task, the status information of the target task is determined to be completed; and Record the status information of the target task.

11. The method according to claim 9, wherein, Adding the target task to the message queue includes: In response to detecting that the target task meets a predetermined splitting condition, the target task is split into multiple subtasks; and Add the multiple subtasks to the message queue; The predetermined splitting conditions include at least one of the following: the data volume of the target task is greater than or equal to a first threshold, and the computational volume of the target task is greater than or equal to a second threshold. The step of using the at least one working node to execute the target task includes: The multiple subtasks are executed in parallel using at least two worker nodes.

12. The method according to any one of claims 9 to 11, wherein, The target template includes a task template, the task template includes code related to the task, and the task information includes code parameters.

13. A task scheduling device, comprising: The selection module is configured to, in response to receiving a selection instruction, select a target template from a plurality of predetermined templates according to the selection instruction; A creation module is used to create a target task in response to receiving task information, based on the task information and the target template; as well as The scheduling module is used to schedule the target task; The scheduling module includes: The first determining submodule is used to determine whether the container meets the predetermined pre-scheduling conditions based on the scheduling interval of the target task and the expected time required to schedule the container. The first scheduling submodule is used to perform pre-scheduling processing on the container before the target execution time, based on the expected duration and the target execution time of the target task, when it is determined that the container meets the predetermined pre-scheduling conditions. The first execution submodule is used to execute the target task using a pre-scheduled container.

14. The apparatus according to claim 13, wherein, The scheduling module also includes: The release submodule is used to release the container when it is determined that the container has completed the target task.

15. The apparatus according to claim 14, wherein, The predetermined pre-scheduling conditions include: The scheduling interval is longer than the product of the expected duration and the predetermined coefficient, where the predetermined coefficient is greater than or equal to 1.

16. The apparatus according to claim 14, wherein, The first scheduling submodule includes: The pre-scheduling unit is used to pre-schedule containers at a predetermined time. Wherein, the time difference between the predetermined time and the target execution time of the target task is equal to the expected duration.

17. The apparatus of claim 14, further comprising: The second scheduling submodule is used to schedule the container according to the target execution time of the target task when it is determined that the container does not meet the predetermined pre-scheduling conditions. The second execution submodule is used to execute the target task using a scheduled container; as well as The retention submodule is used to retain the scheduled container if it is determined that the container has completed the target task.

18. The apparatus of claim 14, further comprising: The first acquisition submodule is used to obtain at least one actual duration required for scheduling containers from historical data of a predetermined time period; as well as The second determining submodule is used to determine the expected duration based on the at least one actual duration and a predetermined quantile value.

19. The apparatus of claim 14, further comprising: The third scheduling submodule is used to schedule new containers in response to the detection that a container is in a fault state. The second acquisition submodule is used to acquire the execution status of the target task; as well as The third execution submodule is used to continue executing the target task based on the execution state using the new container.

20. The apparatus according to any one of claims 13 to 19, wherein, The target template includes a business environment template, and the task information includes code related to the task.

21. The apparatus according to claim 13, wherein, The scheduling module includes: Add a submodule to add the target task to the message queue in response to detecting that the current time is the trigger time of the target task; The third acquisition submodule is used to acquire the target task from the message queue using at least one of multiple worker nodes; and The fourth execution submodule is used to execute the target task using the at least one working node.

22. The apparatus of claim 21, further comprising: The first state determination submodule is used to determine the state information of the target task as triggered in response to detecting that the target task has been added to the message queue; The second state determination submodule is used to determine the state information of the target task as being processed in response to detecting that at least one working node has obtained the target task from the message queue; The third state determination submodule is used to determine the state information of the target task as completed in response to detecting that the at least one working node has completed the target task; as well as The recording submodule is used to record the status information of the target task.

23. The apparatus according to claim 21, wherein, Adding submodules includes: A splitting unit is configured to split the target task into multiple subtasks in response to detecting that the target task meets a predetermined splitting condition; and An adding unit is used to add the plurality of subtasks to the message queue; The predetermined splitting conditions include at least one of the following: the data volume of the target task is greater than or equal to a first threshold, and the computational volume of the target task is greater than or equal to a second threshold. The fourth execution submodule is further configured to execute the multiple subtasks in parallel using at least two working nodes.

24. The apparatus according to any one of claims 21 to 23, wherein, The target template includes a task template, the task template includes code related to the task, and the task information includes code parameters.

25. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 12.

27. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images