Task scheduling method and system, and storage medium

Abstract

The application provides a task scheduling method, a system and a storage medium. The task scheduling method is applied to a task scheduling system. The task scheduling system comprises a regional scheduling module having a core interaction submodule, a message generation submodule and a regional scheduling submodule. The task scheduling method comprises the following steps: obtaining a target task by the core interaction submodule; obtaining the target task from the core interaction submodule and generating a message queue by the message generation submodule; obtaining a target message queue from the message queue by the regional scheduling submodule; obtaining a target machine from the core interaction submodule by the regional scheduling submodule; and generating a regional scheduling strategy according to the target message queue and the target machine by the regional scheduling submodule. The average turnaround time of the job is reduced by means of two-stage scheduling, and the throughput capacity of the system is improved. In addition, there is a priority between the message queues and in the message queues, so that the high-priority task is executed preferentially.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a task scheduling method, system, and storage medium. Background Technology

[0002] A rendering platform consists of hundreds or even thousands of rendering nodes. The platform acquires rendering tasks and outputs high-quality, near-realistic images. The entire rendering process is mainly divided into scene file loading and image rendering, requiring significant resources such as network I / O, storage I / O, memory, CPU, and GPU.

[0003] Currently, traditional rendering platforms typically include machine management, task management, queue, and scheduling modules. The scheduling module is often implemented with a centralized scheduler; if the scheduler crashes, rendering tasks cannot be distributed, impacting system availability. Furthermore, queue modules generally use single or multiple queues, often resulting in high-priority tasks not being executed promptly. Summary of the Invention

[0004] The purpose of this application is to provide a task scheduling method, system, and storage medium. The region scheduling module is only responsible for region-level task scheduling, and the average turnaround time of jobs is reduced and the system throughput is improved through a two-level scheduling approach. Furthermore, priorities exist between and within message queues, ensuring that high-priority tasks are executed first.

[0005] In a first aspect, embodiments of this application provide a task scheduling method applied to a task scheduling system. The task scheduling system includes a region scheduling module comprising a core interaction submodule, a message generation submodule, a region scheduling submodule, and a data storage submodule. The method includes: the core interaction submodule acquiring a target task and storing it in the data storage submodule; the message generation submodule acquiring the target task from the data storage submodule and generating a message queue; the region scheduling submodule acquiring a target message queue from the message queue; the region scheduling submodule acquiring a target machine from the core interaction submodule; wherein the target machine is used to execute the target task, which includes a rendering task; and the region scheduling submodule generating a region scheduling strategy based on the target message queue and the target machine.

[0006] In the above implementation process, the task scheduling method provided in this application embodiment generates a target message queue by acquiring the target task in the regional scheduling module. The messages in this target message queue have priorities, which can ensure that when executing tasks, higher priority tasks are dequeued first and lower priority tasks are dequeued later, ensuring that high-priority tasks can be executed in a timely manner. Furthermore, the target message queue and target machine can select the correct availability zone, instead of directly scheduling a single machine; instead, the target machine is subsequently scheduled in the availability zone. In terms of scheduling, the status of the task, the status of the availability zone, and the status of the machine are all shared, which has extremely high availability.

[0007] Optionally, in this embodiment of the application, obtaining the target message queue from the message queue by the regional scheduling submodule includes: obtaining the ID of the message queue; obtaining the priority of the message queue according to the ID; and generating the target message queue according to the priority.

[0008] In the above implementation process, the task scheduling submodule can generate a target message queue with priority based on the message queue ID; when executing a task, the target message queue dequeues the task according to the priority of the message queue and the priority within the message queue; there are priorities between queues and priorities within queues, thus ensuring the timely execution of high-priority tasks.

[0009] Optionally, in this embodiment of the application, obtaining the target machine from the core interaction submodule by the regional scheduling submodule includes: obtaining the machine status from the core interaction submodule by the regional scheduling submodule; wherein, the machine status includes the status of the node lock; the node lock is a universally unique identifier of the machine; the machine status includes the IP of the node, the running status of the node and the status of the node lock; the node IP is a universally unique identifier of the machine.

[0010] In the above implementation process, this embodiment of the application identifies idle machines, i.e., target machines, by determining whether the node lock state in the machine status is empty; and determines the final region scheduling strategy by combining the number of target machines in the availability zone and the rendering tasks, the region scheduling strategy including the selected availability zone. This embodiment of the application first selects the availability zone, and then selects specific machines, making a scheduling strategy that maximizes machine resource utilization after reasonably evaluating the machines in all availability zones.

[0011] Optionally, in this embodiment, the task scheduling system further includes an application module; the regional scheduling module further includes a data storage submodule. After the regional scheduling submodule generates a regional scheduling policy based on the target message queue, machine state, and node lock state, the task scheduling method further includes: the regional scheduling submodule sending the regional scheduling policy to the core interaction submodule; the core interaction submodule sending the regional scheduling policy to the data storage submodule; and the application module obtaining the target task and regional scheduling policy from the data storage submodule.

[0012] In the above implementation process, by setting up a MySQL database (data storage module) and an application module, the application module can retrieve data from the MySQL database and realize real-time synchronized display of the data; and the existence of the application module can reduce the consumption of CPU resources, memory resources and network bandwidth resources.

[0013] Optionally, in this embodiment of the application, the regional scheduling module of the task scheduling system further includes a machine control submodule; the task scheduling method further includes: the machine control submodule obtaining a target message queue and idle machines from the core interaction submodule. It then determines whether the number of target machines corresponding to the target message queue is less than the number of idle machines; if the number of target machines corresponding to the target message queue is less than the number of idle machines, a specified number of idle machines are shut down; wherein, the specified number is the difference between the number of idle machines and the number of target machines.

[0014] In the above implementation process, the machine management submodule can be set to reasonably control the machines in the system, ensuring that a reasonable number of machines are online. If there are too many idle machines, the machine management submodule will shut down the excess idle machines to achieve reasonable utilization of resources.

[0015] Optionally, in this embodiment, the task scheduling system further includes a node scheduling module having a node scheduling submodule and a task execution submodule; the method further includes: the node scheduling submodule determining whether data can be retrieved from the core interaction submodule; if data can be retrieved from the core interaction submodule, obtaining the regional scheduling strategy from the core interaction submodule; the node scheduling submodule issuing scheduling instructions to the target machine according to the regional scheduling strategy; and the task execution submodule executing the target task according to the scheduling instructions.

[0016] In the above implementation process, the node scheduling submodule obtains the regional scheduling strategy from the regional scheduling submodule and further schedules the target machine to execute the rendering task. The task scheduling system provided in this application overcomes the defects of centralized scheduling through a two-level scheduling method. The regional scheduling submodule is responsible for the scheduling tasks at the regional level, and the node scheduling submodule is responsible for the scheduling tasks at the node level. This two-level scheduling method can reduce the average turnaround time of the job and improve the throughput of the system.

[0017] Optionally, in this embodiment, the task scheduling method further includes: if data cannot be retrieved from the core interaction submodule, the node scheduling submodule retrieves the target message queue from the message generation submodule. The target machine is then retrieved, and a regional scheduling strategy is generated based on the target message queue and the target machine.

[0018] In the above implementation process, the node scheduling submodule can directly retrieve tasks from the message queue for scheduling in the event of a complete failure of the regional scheduling submodule, without causing system paralysis or loss of machine resources.

[0019] Optionally, in this embodiment, after the task execution submodule executes the target task according to the scheduling instruction, the task scheduling method further includes: the task execution submodule obtaining the machine status and sending it to the node scheduling submodule; and the node scheduling submodule sending the machine status to the core interaction submodule.

[0020] In the above implementation process, after executing the target task, the task execution submodule reacquires the machine state and sends this machine state to the node scheduling submodule. Furthermore, the node scheduling submodule sends the machine state back to the core interaction submodule of the regional scheduling module. Thus, the core scheduling submodule updates the machine state, which can then serve as the basis for subsequent task scheduling, ensuring the efficient operation of the task scheduling system.

[0021] Secondly, embodiments of this application provide a task scheduling system, characterized in that it is applied to a task scheduling system; the task scheduling system includes a region scheduling module having a core interaction submodule, a message generation submodule, and a region scheduling submodule; the core interaction submodule is used to obtain a target task; the message generation submodule is used to obtain the target task from the core interaction submodule and generate a message queue. The region scheduling submodule is used to obtain a target message queue from the message queue; the region scheduling submodule is also used to obtain a target machine from the core interaction submodule; wherein, the target machine is used to execute the target task; the target task includes a rendering task. The region scheduling submodule is also used to generate a region scheduling strategy based on the target message queue and the target machine.

[0022] Thirdly, embodiments of this application provide an electronic device, which includes a memory and a processor. The memory stores program instructions, and when the processor reads and runs the program instructions, it executes the steps in any of the above implementation methods.

[0023] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which are read and executed by a processor to perform the steps in any of the above implementations. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of the regional scheduling process provided in this application embodiment;

[0026] Figure 2 A flowchart illustrating the target message queue generation process provided in this application embodiment;

[0027] Figure 3 A flowchart illustrating the target machine acquisition process provided in this application embodiment;

[0028] Figure 4 A machine control flowchart provided for an embodiment of this application;

[0029] Figure 5 A node scheduling flowchart provided for embodiments of this application;

[0030] Figure 6 This is a schematic diagram of task scheduling provided for an embodiment of this application;

[0031] Figure 7 This is a schematic diagram of the module of the task scheduling system provided in this embodiment;

[0032] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. For example, the flowcharts and block diagrams in the drawings illustrate the architecture, functions, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0034] During the research process, the applicant discovered that current traditional rendering platforms generally include machine management, task management, queue, and scheduling modules. The scheduling module is often implemented using a centralized scheduler; if the scheduler crashes, rendering tasks cannot be distributed, impacting system availability. Furthermore, the queue module typically uses single or multiple queues without implementing priority within the same queue, often resulting in high-priority tasks not being executed promptly.

[0035] Based on this, this solution provides a task scheduling method, system, and storage medium. The region scheduling module is only responsible for region-level task scheduling, reducing the average turnaround time of jobs and improving system throughput through a two-level scheduling approach. Furthermore, priorities exist between and within message queues, ensuring that high-priority tasks are executed first, resulting in high system availability.

[0036] Before introducing the specific solutions of the embodiments of this application, we will first give a unified introduction to the services executed by each module in the task scheduling system of this application.

[0037] The core interaction submodule can be used to execute the core-service, which is mainly used to provide global status, including machine status, task status, etc.

[0038] The message generation submodule can be used to execute the api-service. The api-service is the business gateway of the task scheduling system in this application embodiment, which can realize the function of protecting internal services and exposing interfaces to the outside. The interfaces exposed to the outside include, but are not limited to, task submission, task status query, real-time milestone log query (querying the health status of control plane components, querying the backlog of task queues, querying the memory and CPU usage of all machines, and disabling or enabling machine scheduling), etc.

[0039] The data storage submodule can be a MySQL database used to store data from the area scheduling module and the node scheduling module, such as machine status, task status, message queues, etc.

[0040] The regional scheduling submodule includes a scheduler service for scheduling rendering tasks. The scheduler service in the regional scheduling submodule is used to implement scheduling tasks and focuses on implementing availability zone-level scheduling.

[0041] The application module, including UI services and third-party services, supports integration with the Operation Support System (OSS). During runtime, scene files can be directly uploaded to OSS and downloaded during rendering tasks.

[0042] The machine control submodule can execute the node-scaler service, which is used to control the elastic power on / off of the machine, mainly to improve machine utilization and avoid unnecessary resource waste.

[0043] The node scheduling submodule mainly provides scheduler services within the node scheduling module, enabling node-level task scheduling.

[0044] The task execution submodule is used to execute rendering services.

[0045] Please refer to Figure 1 , Figure 1 This application provides a flowchart of a regional scheduling process; the method is applied to a task scheduling system, which includes a regional scheduling module having a core interaction submodule, a message generation submodule, and a regional scheduling submodule. The method includes:

[0046] Step S100: The core interaction submodule obtains the target task and stores the target task in the data storage submodule.

[0047] In step S100 above, the core interaction submodule core-service obtains the target task; core-service then synchronously updates the target task to the MySQL database. Each time the target task is updated, core-service synchronizes the updated target task to the MySQL database, ensuring that the database maintains the latest state. The task scheduling method provided in this application embodiment is mainly used in a contamination scheduling system capable of executing any rendering task, where the target task mainly includes a rendering task.

[0048] It should be noted that in the task scheduling system provided in this application embodiment, the core interaction submodule can be used to execute the core-service, which is mainly used to provide global status; wherein, the global status mainly includes machine status, machine node lock status, task status, etc.

[0049] Step S101: The message generation submodule retrieves the target task from the data storage submodule and generates a message queue.

[0050] In step S101 above, the message generation submodule retrieves the target task from the data storage submodule and generates a message queue based on the target task. The api-service retrieves the target task from the MySQL database and generates a message queue based on the target task (the api-service, acting as a Kafka producer, creates a new message in the Kafka queue).

[0051] It should be noted that the message generation submodule in the task scheduling system provided in this application embodiment can be used to execute the api-service. The api-service is the business gateway of the task scheduling system in this application embodiment, which can realize the function of protecting internal services and exposing interfaces to the outside. The interfaces exposed to the outside include, but are not limited to, task submission, task status query, real-time milestone log query (querying the health status of control plane components, querying the backlog of task queues, querying the memory and CPU usage of all machines, and disabling or enabling machine scheduling), etc.

[0052] Optionally, after generating the message queue, the API service pushes the task data to the message queue, which can be a Kafka message queue. Tasks of different priorities are submitted to different priority levels by the API service, ensuring that each priority level includes only one queue.

[0053] As those skilled in the art will understand, Kafka is a distributed, partitioned, and replica-based message system coordinated by ZooKeeper. Its greatest feature is its ability to process large amounts of data in real time to meet various needs and scenarios.

[0054] In the above implementation process, after the message queue is generated, the api-service pushes the task data to the message queue, which can ensure that high-priority tasks are executed first, and ensure that important and urgent rendering tasks are not covered by non-urgent and unimportant tasks (high-priority tasks are covered by low-priority tasks), thus preventing the delayed execution of important tasks.

[0055] Step S102: The regional scheduling submodule retrieves the target message queue from the message queue.

[0056] In step S102 above, the regional scheduling submodule retrieves the target message queue from the message queue. For example, the scheduler service pulls the task ID from the target message queue from the Kafka queue and stores it in the scheduler service's memory queue; it can be understood that there is a one-to-one correspondence between the message queues in the scheduler service and the message queues in Kafka.

[0057] Step S103: The regional scheduling submodule obtains the target machine from the core interaction submodule.

[0058] In step S103 above, the regional scheduling submodule scheduler service obtains the target machine from the core interaction submodule; it should be noted that the target machine is a machine used to execute the target task, and the target task in this embodiment includes a rendering task.

[0059] Step S104: The regional scheduling submodule generates a regional scheduling strategy based on the target message queue and the target machine.

[0060] In step S104 above, the regional scheduling submodule generates a regional scheduling policy based on the target message queue and the target machine. For example, if the number of machines required by the target message queue is greater than the number of idle machines in machine availability zone A, the scheduler service ignores machine availability zone A, that is, it does not schedule machines in region A. If the number of machines required by the target message queue is less than or equal to the number of idle machines in machine availability zone B, the scheduler service selects machine availability zone B. After the availability zone is selected, the scheduledZone field of the rendering task in the target message queue (which represents the address of the machine availability zone where the target message queue will be executed) is set to the region name of machine availability zone B. At the same time, the regional scheduling policy is sent back to the core-service, so that the regional scheduling policy exists in the core-service. Thus, the scheduling at the availability zone level is completed.

[0061] pass Figure 1 As can be seen, the task scheduling method provided in this application generates a target message queue by acquiring the target task in the regional scheduling module. The messages in this target message queue have priorities, which ensures that when executing tasks, higher-priority tasks are dequeued first and lower-priority tasks are dequeued later, ensuring that high-priority tasks can be executed in a timely manner. Furthermore, the target message queue and target machine can select the correct availability zone, instead of directly scheduling a single machine; instead, the target machine is subsequently scheduled within the availability zone. In terms of scheduling, the status of the task, the status of the availability zone, and the status of the machine are all shared, resulting in extremely high availability.

[0062] Please refer to Figure 2 , Figure 2 A flowchart for generating a target message queue is provided in this application embodiment; the method includes:

[0063] Step S200: Obtain the message queue ID.

[0064] In step S200 above, the scheduler service pulls the task ID from the target message queue from the Kafka queue and stores it in the scheduler service's memory. It should be noted that the message queues in the scheduler service and the message queues in Kafka are in one-to-one correspondence.

[0065] Step S201: Obtain the priority of the message queue based on the ID.

[0066] In step S201 above, the scheduler service obtains the priority of the message queue based on the ID.

[0067] Step S202: Generate the target message queue according to the priority.

[0068] In steps S201-S202 above, a target message queue can be generated based on the message queue ID. The target message queue has priorities. For example, messages in the target message queue are dequeued according to their priority; higher-priority elements are dequeued first, and lower-priority elements are dequeued last. For instance, if there are three message queues A, B, and C in the target message queue, with priorities of 3, 2, and 1 respectively, tasks in queue A are dequeued first, and tasks in queue C are dequeued last. Within each queue, tasks also have priorities, and the dequeueing order also matters.

[0069] pass Figure 2 As can be seen, the task scheduling submodule can generate a target message queue with priority based on the message queue ID; when executing a task, the target message queue dequeues tasks according to the priority of the message queue and the priority within the message queue; there are priorities between queues and priorities within queues, thus ensuring the timely execution of high-priority tasks.

[0070] Please refer to Figure 3 , Figure 3 This application provides a flowchart for obtaining a target machine according to an embodiment; the method includes:

[0071] Step S300: The regional scheduling submodule obtains the machine status from the core interaction submodule.

[0072] In step S300 above, the regional scheduling submodule scheduler service obtains the machine status from the core-service of the core interaction submodule; wherein, the machine status can be available or unavailable; unavailable cases include faulty machines and machines that are executing tasks.

[0073] It should be noted that machine state includes node lock state, and node lock can be understood as a field representing machine state.

[0074] Step S301: Determine whether the node lock status of the machine is empty.

[0075] In step S301 above, the node lock state can be empty or not empty; when the node lock state is not empty, it is determined that the machine corresponding to the non-empty node is running a task, and the scheduler service ignores this node; when the node lock value is empty and the machine state is normal, and the number of idle machines in the availability zone is greater than or equal to the number of machines required in the rendering task, the availability zone can be selected, and the final area scheduling result is written back to the core-service.

[0076] Step S302: If the node lock is empty, then the machine corresponding to the node lock is taken as the target machine.

[0077] In step S302 above, the scheduler service determines that the node lock is empty and uses the machine corresponding to the node lock as the target machine; if the number of target machines in the availability zone is greater than or equal to the number of machines required in the rendering task, the availability zone can be selected.

[0078] pass Figure 3 As can be seen, this embodiment of the application identifies idle machines, i.e., target machines, by determining whether the node lock state in the machine status is empty; and determines the final region scheduling strategy by combining the number of target machines in the availability zone and the rendering tasks, the region scheduling strategy including the selected availability zone. This embodiment of the application first selects the availability zone, and then selects specific machines, making a scheduling strategy that maximizes machine resource utilization after reasonably evaluating the machines in all availability zones.

[0079] In an optional embodiment, the task scheduling system provided in this application also includes an application module, which includes UI services and third-party services. It supports integration with the Operation Support System (OSS) and can directly upload scene files to OSS during runtime and download them when rendering tasks are executed. This can save server resources and costs to the greatest extent, such as CPU resources, memory resources, and network bandwidth resources.

[0080] After the regional scheduling submodule generates the regional scheduling policy based on the target message queue, machine status, and node lock status, the method further includes: the regional scheduling submodule sending the regional scheduling policy to the core interaction submodule; and the core interaction submodule sending the regional scheduling policy to the data storage submodule. Further, the application module obtains the target task and regional scheduling policy from the data storage submodule. For example, in this embodiment, the MySQL database synchronizes data after the core interaction submodule obtains the target task, obtains the machine status, and generates the regional scheduling policy. The application module can directly obtain data from the MySQL database and can pass it to a third-party service or UI interface; for example, it can directly obtain task progress, machine status, etc., from the UI interface, enabling real-time global monitoring of the task status.

[0081] Therefore, by setting up a MySQL database (data storage module) and an application module, the application module can retrieve data from the MySQL database and achieve real-time synchronized display of the data; and the existence of the application module can reduce the consumption of CPU resources, memory resources and network bandwidth resources.

[0082] Please refer to Figure 4 , Figure 4The machine control flowchart provided in this application embodiment; the regional scheduling module of the task scheduling system provided in this application embodiment further includes a machine control submodule; the method includes:

[0083] Step S400: The machine control submodule obtains the target message queue and idle machine from the core interaction submodule.

[0084] In step S400 above, the node-scaler service of the machine control submodule obtains the target message queue and idle machines from the core-service of the core interaction submodule; as mentioned earlier, the core-service can obtain machine status and task status, including idle machines.

[0085] Step S401: Determine whether the number of target machines corresponding to the target message queue is less than the number of idle machines.

[0086] Step S402: If the number of target machines corresponding to the target message queue is less than the number of idle machines, then shut down the specified number of idle machines.

[0087] In steps S401-S402 above, when executing a rendering task, if the number of target machines required in the generated target message queue is less than the number of idle machines in the region, then the node-scaler service shuts down a specified number of idle machines, leaving only the corresponding number of idle machines in the target message queue. The specified number of idle machines is the difference between the number of idle machines and the number of target machines.

[0088] For example, when many machines are idle, the node-scaler service calls the core-service interface to proactively take too many idle machines offline, preventing these idle machines from being scheduled for tasks; this can also allow the cloud provider's API to shut down machines, thereby avoiding waste of machine resources. On the other hand, when a large number of tasks are waiting to be executed, the cloud provider's API is called to turn on the shut-down machines, allowing more machines to execute rendering tasks and speeding up queue consumption.

[0089] pass Figure 4 It can be seen that the machine management submodule can be configured to reasonably control the machines in the system, ensuring that a reasonable number of machines are online. If there are too many idle machines, the machine management submodule will shut down the excess idle machines to achieve reasonable utilization of resources.

[0090] Please refer to Figure 5 , Figure 5 The node scheduling flowchart provided in this application embodiment; the task scheduling system provided in this application embodiment further includes a node scheduling module having a node scheduling submodule and a task execution submodule; the method includes:

[0091] Step S500: The node scheduling submodule determines whether data can be retrieved from the core interaction submodule.

[0092] In step S500 above, the scheduler service of the node scheduling submodule determines whether the scheduler service can pull data from the core interaction submodule core-service, which mainly involves the regional scheduling strategy.

[0093] Step S501: If data can be retrieved from the core interaction submodule, then obtain the regional task scheduling strategy from the core interaction submodule.

[0094] In step S501 above, if the scheduler service of the node scheduling submodule can pull data from the core interaction submodule core-service, it obtains the regional scheduling policy from the core-service. Based on the previous description, it is clear that the regional scheduling policy includes the availability zone corresponding to the target task.

[0095] Step S502: The node scheduling submodule issues scheduling instructions to the target machine according to the regional scheduling strategy.

[0096] In step S502 above, the scheduler service of the node scheduling submodule issues scheduling instructions to the target machine according to the regional scheduling policy, and only then does the scheduling of the target machine begin.

[0097] Step S503: The task execution submodule executes the target task according to the scheduling instructions.

[0098] In step S503 above, the task execution submodule causes the machine to execute the target task according to the scheduling instructions.

[0099] For example, the scheduler service of the node scheduling submodule obtains information on all machines in the local availability zone from the core-service in the regional scheduling policy; it then selects machines in regions where the machine status is normal, the node lock status is empty, and the number of idle machines is greater than the target number of machines; and sends the IPs of these selected idle machines back to the core-service to indicate that the scheduling task is complete. Further, after receiving the IPs of the selected idle machines, the core-service locks the node locks of these machines and sets the node lock value to the task ID. The task execution submodule then issues rendering tasks to the machines according to the task scheduling instructions.

[0100] pass Figure 5As can be seen, the scheduler service in the node scheduling submodule obtains the regional scheduling policy from the core-service and further schedules the target machine to execute the rendering task. The task scheduling system provided in this application overcomes the defects of centralized scheduling through a two-level scheduling method. The regional scheduling submodule is responsible for scheduling tasks at the regional level, and the node scheduling submodule is responsible for scheduling tasks at the node level. This two-level scheduling method can reduce the average turnaround time of the job and improve the throughput of the system.

[0101] In an optional embodiment, if data cannot be retrieved from the core interaction submodule, the node scheduling submodule retrieves the target message queue from the message generation submodule. The target machine is then retrieved, and a regional scheduling policy is generated based on the target message queue and the target machine. For example, if the node scheduler discovers through the service registry that the regional scheduler has been offline for more than a preset time, the node scheduling submodule retrieves the target message queue from the message generation submodule. The target machine is then retrieved, and a regional scheduling policy is generated based on the target message queue and the target machine. It should be noted that in this embodiment, the preset time can be 1 minute. In practical applications, it can be set according to actual needs, and the 1-minute preset time provided in this embodiment should not be considered a limitation of the preset time.

[0102] For example, if the scheduler service in the node scheduling submodule cannot obtain the regional scheduling policy from the core service, then the scheduler service in the node scheduling submodule will obtain the target message queue from the API service of the message generation submodule. For example, if the scheduler service in the node scheduling submodule discovers from the service registry that the regional scheduler has been offline for more than 1 minute, then the scheduler service in the node scheduling submodule will obtain messages from Kafka and generate the target queue in memory. Furthermore, a regional scheduling policy is generated based on the target message queue and the target machine; that is, even if the regional scheduling submodule fails, the node scheduling submodule can still independently complete the task scheduling. If the node scheduler discovers from the service registry that the regional scheduler has recovered, then it will no longer directly execute the regional scheduling logic; the responsibility for regional scheduling will be performed by the newly recovered regional scheduler.

[0103] Therefore, it can be seen that in the event of a complete failure of the regional scheduling submodule, the node scheduling submodule can directly retrieve tasks from the message queue for scheduling, without causing system paralysis or loss of machine resources.

[0104] In an optional embodiment, after the task execution submodule executes the target task according to the scheduling instructions, the task scheduling method further includes: the task execution submodule obtaining the machine status and sending it to the node scheduling submodule; and the node scheduling submodule sending the machine status to the core interaction submodule.

[0105] Therefore, after executing the target task, the task execution submodule re-acquires the machine's state and sends this state to the node scheduling submodule. Furthermore, the node scheduling submodule sends the machine state back to the core interaction submodule of the regional scheduling module. The core scheduling submodule then updates the machine state, which serves as the basis for subsequent task scheduling, ensuring the efficient operation of the task scheduling system.

[0106] Please refer to Figure 6 , Figure 6 This diagram illustrates task scheduling as provided in the embodiments of this application. The task scheduling method of this application will be described module by module. Before proceeding, it should be noted that the various modules provided in the embodiments of this application use the same registry center instance and configuration center instance, sharing the same registry center and configuration center for convenient management by operations and maintenance engineers.

[0107] The application module mainly includes UI services and third-party services. The UI service can directly interface with the OSS system; scene files can be directly uploaded to the OSS system and downloaded when tasks are executed; this can minimize the waste of CPU resources, memory resources, and network bandwidth resources.

[0108] The regional scheduling module primarily implements regional-level scheduling. It mainly includes a regional scheduling submodule, a message generation submodule, a data storage submodule, a machine management submodule, and a core interaction submodule. The rendering service pushes logs to the core interaction submodule, which then persists the rendering service data to the data storage submodule and notifies the message generation submodule of when new logs are added. Further, the message generation submodule reads the logs from the data storage submodule and forwards them to the application module. After the rendering task data is persisted to the data storage module, it is queried (primarily for tasks currently being rendered and completed historical tasks), and the task is pushed to the message queue of the message generation submodule, which can be a Kafka queue. The regional scheduling submodule retrieves the message queue and ultimately generates the regional scheduling policy. Simultaneously, when there are more idle machines in the system than the number required for the target task, the machine management submodule shuts down the excess idle machines; when a large backlog of tasks occurs in the system's task queue, the shut-down machines are restarted.

[0109] The node scheduling module primarily implements node-level task scheduling. It comprises a node scheduling submodule and a task execution submodule. The node scheduling submodule first obtains the regional scheduling policy from the regional scheduling submodule. After obtaining the policy, the task execution submodule executes the target task according to the task scheduling instructions, thus ensuring successful rendering task delivery. If rendering task delivery fails, the node scheduling submodule directly pulls tasks from the message queue to prevent task backlog in the Kafka queue due to failures in the regional scheduling submodule.

[0110] The task execution submodule is used to execute rendering tasks. It integrates various rendering engines and OSS storage to provide rendering capabilities. The scheduler service of the node scheduling module distributes tasks. After the rendering service obtains the task, it calls the rendering engine to perform rendering. Once rendering is complete, the output file is sent to OSS. Throughout the rendering process, important milestone logs from the rendering engine are written back to the core-service service in real time. After the task is completed, the final task status is returned to the core-service service, and the node lock release interface of the core-service service is called to release the lock, so that the region scheduling submodule knows that the machine is idle. At the same time, the rendering service periodically sends heartbeats (local CPU and memory usage, number of CPU cores, memory capacity, etc.) to the core-service service.

[0111] pass Figure 6 As can be seen, the task scheduling method provided in this application embodiment adopts a two-level scheduling approach, namely, the regional scheduling module implements regional-level scheduling and the node scheduling module implements node-level scheduling. When the regional scheduling module crashes and cannot implement regional-level scheduling, the node scheduling module detects this through the registration center and automatically obtains a task from the message queue for scheduling. At this time, the system still runs normally and there is no loss in machine resource utilization.

[0112] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the module of the task scheduling system provided in this embodiment; the task scheduling system 100 includes: a regional scheduling module 110 having a core interaction submodule 111, a message generation submodule 112 and a regional scheduling submodule 113.

[0113] Core interaction submodule 111 is used to obtain the target task;

[0114] The message generation submodule 112 is used to obtain the target task from the core interaction submodule 111 and generate a message queue;

[0115] The regional scheduling submodule 113 is used to obtain the target message queue from the message queue;

[0116] The regional scheduling submodule 113 is also used to obtain the target machine from the core interaction submodule 111; wherein the target machine is used to execute the target task; the target task includes a rendering task.

[0117] The regional scheduling submodule 113 is also used to generate regional scheduling strategies based on the target message queue and the target machine.

[0118] In an optional embodiment, the regional scheduling submodule 113 obtains the target message queue from the message queue, which includes: the regional scheduling submodule 113 obtaining the ID of the message queue; obtaining the priority of the message queue according to the ID; and the regional scheduling submodule 113 generating the target message queue according to the priority.

[0119] In an optional embodiment, the process of the regional scheduling submodule 113 obtaining the target machine from the core interaction submodule 111 includes: the regional scheduling submodule 113 obtaining the machine status from the core interaction submodule 111; wherein the machine status includes the status of the node lock; the node lock is a universally unique identifier for the machine. The regional scheduling submodule 113 determines whether the status of the node lock in the machine status is empty; if the status of the node lock is not empty, the regional scheduling submodule 113 takes the machine corresponding to the node lock as the target machine.

[0120] In an optional embodiment, the task scheduling system 100 further includes an application module 120, and the regional scheduling module 110 further includes a data storage submodule 114. After the regional scheduling submodule 113 generates a regional scheduling policy based on the target message queue, machine status, and node lock status, the method further includes: the regional scheduling submodule 113 sending the regional scheduling policy to the core interaction submodule 111; the core interaction submodule 111 sending the regional scheduling policy to the data storage submodule 114; and the application module 120 obtaining the target task and the regional scheduling policy from the data storage submodule 114.

[0121] In an optional embodiment, the regional scheduling module 110 of the task scheduling system 100 further includes a machine control submodule 150; the machine control submodule 150 obtains the target message queue and idle machines from the core interaction submodule 111. The machine control submodule 150 determines whether the number of target machines corresponding to the target message queue is less than the number of idle machines; if the number of target machines corresponding to the target message queue is less than the number of idle machines, the machine control submodule 150 shuts down a specified number of idle machines; wherein, the specified number is the difference between the number of idle machines and the number of target machines.

[0122] In an optional embodiment, the task scheduling system 100 further includes a node scheduling module 130 having a node scheduling submodule 131 and a task execution submodule 132. The node scheduling submodule 131 determines whether data can be retrieved from the core interaction submodule. If data can be retrieved from the core interaction submodule, the node scheduling submodule 131 obtains the regional scheduling policy from the core interaction submodule. The node scheduling submodule 131 issues scheduling instructions to the target machine according to the regional scheduling policy. The task execution submodule 132 executes the target task according to the scheduling instructions.

[0123] In an optional embodiment, if the node scheduling submodule 131 cannot retrieve data from the core interaction submodule 111, then the node scheduling submodule 131 obtains the target machine from the target message queue in the message generation submodule 112, and the node scheduling submodule 131 generates a regional scheduling strategy based on the target message queue and the target machine.

[0124] In an optional embodiment, after the task execution submodule 132 executes the target task according to the scheduling instruction, the task execution submodule 132 obtains the machine status and sends it to the node scheduling submodule 131; the node scheduling submodule 131 then sends the machine status to the core interaction submodule 111.

[0125] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes: a processor 301 and a memory 302. The memory 302 stores machine-readable instructions executable by the processor 301. When the machine-readable instructions are executed by the processor 301, the method described above is performed.

[0126] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the steps in any of the above implementations.

[0127] The computer-readable storage medium can be any medium capable of storing program code, such as Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM). The storage medium stores the program, and the processor executes the program after receiving an execution instruction. The method executed by the electronic terminal as defined in any embodiment of this invention can be applied to the processor or implemented by the processor.

[0128] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0129] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0131] It can be replaced and can be implemented, wholly or partially, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, wholly or partially, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated.

[0132] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0133] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0134] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A task scheduling method, characterized in that, The method is applied to a task scheduling system; The task scheduling system includes a region scheduling module comprising a core interaction submodule, a message generation submodule, a region scheduling submodule, and a data storage submodule; the method includes: The core interaction submodule obtains the target task and stores the target task in the data storage submodule. The message generation submodule retrieves the target task from the data storage submodule and generates a message queue. The regional scheduling submodule retrieves the target message queue from the message queue. The regional scheduling submodule obtains the target machine from the core interaction submodule; wherein the target machine is used to execute the target task, and the target task includes a rendering task; The regional scheduling submodule generates a regional scheduling strategy based on the target message queue and the target machine. The step of obtaining the target message queue from the message queue by the regional scheduling submodule includes: Obtain the ID of the message queue; The priority of the message queue is obtained based on the ID; The target message queue is generated according to the priority. The task scheduling system also includes an application module; After the regional scheduling submodule generates the regional scheduling policy based on the target message queue and the target machine, the method further includes: The regional scheduling submodule sends the regional scheduling strategy to the core interaction submodule; The core interaction submodule sends the regional scheduling strategy to the data storage submodule; The application module obtains the target task and the regional scheduling strategy from the data storage submodule. The task scheduling system further includes a node scheduling module with a node scheduling submodule and a task execution submodule; the method further includes: The node scheduling submodule determines whether data can be retrieved from the core interaction submodule. If data can be retrieved from the core interaction submodule, then the regional scheduling strategy can be obtained from the core interaction submodule. The node scheduling submodule issues scheduling instructions to the target machine according to the regional scheduling strategy; The task execution submodule executes the target task according to the scheduling instructions; The method further includes: If data cannot be retrieved from the core interaction submodule, the node scheduling submodule will retrieve the target message queue from the message generation submodule. Obtain the target machine and generate a regional scheduling policy based on the target message queue and the target machine.

2. The method according to claim 1, characterized in that, The step of obtaining the target machine from the core interaction submodule by the regional scheduling submodule includes: The regional scheduling submodule obtains the machine status from the core interaction submodule; wherein, the machine status includes the node IP, the node's running status, and the node lock status; the node IP is the machine's universally unique identifier; Determine whether the state of the node lock in the machine state is empty; If the node is running normally and the node lock is empty, then the machine corresponding to the node lock is taken as the target machine.

3. The method according to claim 1, characterized in that, in, The regional scheduling module of the task scheduling system further includes a machine control submodule; the method further includes: The machine control submodule obtains the target message queue and idle machine from the core interaction submodule; Determine whether the number of target machines corresponding to the target message queue is less than the number of idle machines; If the number of target machines corresponding to the target message queue is less than the number of idle machines, then a specified number of the idle machines are shut down; wherein, the specified number is the difference between the number of idle machines and the number of target machines.

4. The method according to claim 1, characterized in that, After the task execution submodule executes the target task according to the scheduling instruction, the method further includes: The task execution submodule obtains the machine status and sends it to the node scheduling submodule; The node scheduling submodule sends the machine transition status to the core interaction submodule.

5. A task scheduling system, characterized in that, It is applied to a task scheduling system; the task scheduling system includes a regional scheduling module with a core interaction submodule, a message generation submodule, a regional scheduling submodule, and a data storage submodule; The core interactive submodule is used to obtain the target task; The message generation submodule is used to obtain the target task from the core interaction submodule and generate a message queue; The regional scheduling submodule is used to obtain the target message queue from the message queue; The region scheduling submodule is further configured to obtain a target machine from the core interaction submodule; wherein the target machine is used to execute the target task; the target task includes a rendering task; The regional scheduling submodule is further configured to generate a regional scheduling strategy based on the target message queue and the target machine; The process of obtaining a target message queue from the message queue by the regional scheduling submodule includes: obtaining the ID of the message queue; obtaining the priority of the message queue based on the ID; and generating a target message queue based on the priority. The task scheduling system further includes an application module. After the regional scheduling submodule generates a regional scheduling policy based on the target message queue, machine status, and node lock status, the regional scheduling submodule sends the regional scheduling policy to the core interaction submodule; the core interaction submodule sends the regional scheduling policy to the data storage submodule, and the application module obtains the target task and regional scheduling policy from the data storage submodule. The task scheduling system further includes a node scheduling module with a node scheduling submodule and a task execution submodule; The node scheduling submodule determines whether it can retrieve data from the core interaction submodule; if it can retrieve data from the core interaction submodule, it obtains the regional scheduling strategy from the core interaction submodule. The node scheduling submodule issues scheduling instructions to the target machine according to the regional scheduling strategy; The task execution submodule executes the target task according to the scheduling instructions; If the node scheduling submodule cannot retrieve data from the core interaction submodule, then the node scheduling submodule retrieves the target message queue from the message generation submodule; retrieves the target machine; and generates a regional scheduling strategy based on the target message queue and the target machine.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, perform the steps of the method according to any one of claims 1-4.

Images