A task scheduling method, device, system and storage medium

Abstract

The application provides a task scheduling method, device and system and a storage medium, relates to the technical field of computers, and achieves decoupling of a scheduling platform and an executor, reduces the coupling degree of a task scheduling system, and ensures stable operation of the task scheduling system by the following steps: after a preset business template is completed, the scheduling platform sequentially caches a plurality of tasks to be processed in a task queue in a plurality of message queues in a preset queue cluster; the scheduling platform generates the number of completed upstream components of each task component in the preset business template according to the identifier of each task in the task queue and the preset business template; and the scheduling platform adds the number of completed upstream components of each task component to the scheduling instance of each task component that is cached in advance, so that the scheduling instance of each task component includes the number of upstream components and the number of completed upstream components of each task component.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a task scheduling method, apparatus, system, and storage medium. Background Technology

[0002] Task scheduling design is the skeleton of a system. It is one of the factors that determines whether the system can maximize the utilization of hardware performance. With the advent of localization and the wave of information technology innovation, the demand for task scheduling of business is increasing day by day. Standardized and automated task scheduling is particularly important.

[0003] In existing technologies, task scheduling methods are mainly divided into single-machine task scheduling and cluster-based task scheduling. Single-machine task scheduling is mainly suitable for streaming tasks and small-data-volume streaming tasks in small-scale office systems, but it cannot handle large data volumes and high concurrency. Although existing cluster-based task scheduling can handle large data volumes, the executor and scheduling platform cannot be completely decoupled, resulting in a high degree of coupling, which is not conducive to the stable operation of the task scheduling system. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of the prior art by providing a task scheduling method, apparatus, system, and storage medium to solve the technical problem of high coupling in existing cluster-based task scheduling systems.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0006] In a first aspect, embodiments of this application provide a task scheduling method, the method comprising:

[0007] After the preset business template is launched, the scheduling platform will sequentially cache multiple tasks to be processed into task queues in multiple message queues in the preset queue cluster. The multiple message queues are: message queues corresponding to each task component in the preset business template, and the task queue is: the message queue of the first task component. The message queue corresponding to each task component is used to enable each task component to process the messages to be processed in the corresponding message queue and cache the processing results of the messages to be processed into the message queues corresponding to downstream components.

[0008] The scheduling platform generates the number of upstream components completed for each task component in the preset business template based on the identifier of each task in the task queue and the preset business template.

[0009] The scheduling platform adds the number of upstream components completed for each task component to the pre-cached scheduling instance of each task component, so that the scheduling instance of each task component includes: the number of upstream components and the number of upstream components completed for each task component; the scheduling instance of each task component is used to enable the executor to cache the message processing results of the upstream components of each task component to the message queue corresponding to the downstream component.

[0010] Optionally, before the scheduling platform sequentially caches the multiple tasks to be processed into the task queues of multiple message queues in the preset queue cluster after the preset business template is launched, the method further includes:

[0011] The scheduling platform responds to the online operation of the preset service template and allocates a corresponding message queue to each task component in the preset queue cluster.

[0012] The scheduling platform parses the preset business template to generate scheduling instances for each task component. Each scheduling instance for a task component includes the number of upstream components for that task component.

[0013] The scheduling platform caches the scheduling instances of each task component to complete the online deployment of the preset business template.

[0014] Optionally, the online operation of the preset business template is: an online selection operation of the preset business template in at least one business template.

[0015] Optionally, before the scheduling platform responds to the online operation for the preset service template and allocates a corresponding message queue to each task component in the preset queue cluster, the method further includes:

[0016] The scheduling platform registers multiple task components, and after the multiple task components are registered, a business process template is created on the business orchestration page;

[0017] The scheduling platform generates a business template based on the component orchestration operation of the business process template input through the business orchestration page.

[0018] Optionally, the method further includes:

[0019] When the upstream component in each task component has finished processing, the executor determines the position of the preset upstream component and the corresponding downstream component based on the pre-cached preset business template.

[0020] The executor updates the upstream component completion count in the scheduling instance of the downstream component corresponding to the preset upstream component.

[0021] Optionally, the method further includes:

[0022] The executor determines whether the number of completed upstream components and the number of upstream components in the scheduling instance of the downstream component corresponding to the preset upstream component are equal;

[0023] If they are equal, the executor will cache the message processing result of the preset upstream component into the message queue corresponding to the downstream component of the preset upstream component.

[0024] Optionally, the method further includes:

[0025] The scheduling platform creates at least one consumer group for the message queue corresponding to each task component, so that the executor processes the message queue corresponding to each task component based on the at least one consumer group.

[0026] Secondly, embodiments of this application provide a task scheduling device, comprising:

[0027] The scheduling module includes: a caching unit, a generation unit, and an addition unit;

[0028] The caching unit is used to cache multiple tasks to be processed sequentially into the task queue of multiple message queues in a preset queue cluster after the preset business template is launched. The multiple message queues are: message queues corresponding to each task component in the preset business template, and the task queue is: the message queue of the first task component. The message queue corresponding to each task component is used to enable each task component to process the messages to be processed in the corresponding message queue and cache the processing results of the messages to be processed into the message queues corresponding to downstream components.

[0029] The generation unit is used to generate the number of upstream components completed for each task component in the preset business template based on the identifier of each task in the task queue and the preset business template.

[0030] The adding unit is used to add the number of upstream components completed by each task component to the pre-cached scheduling instance of each task component, so that the scheduling instance of each task component includes: the number of upstream components and the number of upstream components completed by each task component;

[0031] The execution module is used to cache the message processing results of the upstream component of each task component to the message queue corresponding to the downstream component.

[0032] Thirdly, embodiments of this application provide a task scheduling system, including: a scheduling platform and an executor, wherein the scheduling platform is used to execute the operation corresponding to the scheduling platform in the task scheduling method of the first aspect, and the executor is used to execute the operation corresponding to the executor in the task scheduling method of the first aspect.

[0033] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed, implements the task scheduling method of the first aspect described above.

[0034] Compared with the prior art, this application has the following beneficial effects:

[0035] This application provides a task scheduling method, apparatus, system, and storage medium. After a preset service template is launched, a scheduling platform sequentially caches multiple tasks to be processed into task queues within multiple message queues in a preset queue cluster. The scheduling platform generates the upstream component completion count for each task component in the preset service template based on the identifier of each task in the task queue and the preset service template. The scheduling platform adds the upstream component completion count of each task component to a pre-cached scheduling instance of each task component. This ensures that each scheduling instance of a task component includes: the number of upstream components and the upstream component completion count for each task component. Consequently, the executor schedules tasks according to the upstream component completion count. The number of upstream components and the number of completed upstream components of a task component determine whether to cache the message processing results of the upstream components of each task component to the message queue corresponding to the downstream component. In this application, the scheduling platform caches multiple tasks to be processed and the scheduling instances of each task component to a preset queue cluster. The executor caches the message processing results of the upstream components of each task component to the message queue in the preset queue cluster corresponding to the downstream component, so that each task component processes the messages to be processed in the corresponding message queue. Through the preset queue cluster, the scheduling platform and the executor are decoupled, the coupling degree of the task scheduling system is reduced, and thus the stable operation of the task scheduling system is ensured. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the structure of a task scheduling system provided in an embodiment of this application;

[0038] Figure 2A flowchart illustrating a task scheduling method provided in an embodiment of this application;

[0039] Figure 3 A flowchart illustrating another task scheduling method provided in an embodiment of this application;

[0040] Figure 4 A flowchart illustrating another task scheduling method provided in an embodiment of this application;

[0041] Figure 5 A flowchart illustrating another task scheduling method provided in an embodiment of this application;

[0042] Figure 6 A flowchart illustrating another task scheduling method provided in an embodiment of this application;

[0043] Figure 7 This is a schematic diagram of another task scheduling system provided in an embodiment of this application;

[0044] Figure 8 A schematic diagram of a task scheduling device provided in an embodiment of this application;

[0045] Figure 9 A schematic diagram of a scheduling platform provided in an embodiment of this application;

[0046] Figure 10 This is a schematic diagram of an actuator provided in an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0048] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0049] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0050] Existing task scheduling methods are mainly divided into single-machine task scheduling and cluster-based task scheduling. Among them, the single-machine method is relatively traditional and is mainly suitable for streaming tasks and small-data-volume streaming tasks in small-scale office systems. It is powerless to handle the large data volume and high concurrency required in today's data analysis field. It often encounters bottlenecks in online environments and requires continuous investment from operation and maintenance personnel. Although the cluster-based distributed computing method can handle large data volumes, the existing cluster-based task scheduling system has a high degree of coupling. Therefore, in order to reduce the coupling of the task scheduling system, this application provides a task scheduling system. The following specific examples will explain the task scheduling system provided by this application. Figure 1 This is a schematic diagram of the structure of a task scheduling system provided in an embodiment of this application. Figure 1 As shown, the system includes a scheduling platform 100 and an executor 200.

[0051] In this embodiment, the scheduling platform 100 and the executor 200 can interact with each other via a queue cluster 300. Specifically, the scheduling platform 100 can submit multiple tasks to be processed to the queue cluster 300, allowing the executor 200 to acquire and process these tasks, and then submit the processing results back to the queue cluster 300. The scheduling platform 100 can then retrieve these results from the queue cluster 300 for subsequent business processing.

[0052] The scheduling platform 100 is connected to the relational database 400, and the relevant data generated by the scheduling platform 100 can be stored in the relational database 400.

[0053] Optionally, the relevant data generated by the executor 200 can be stored in the cache data corresponding to the executor 200. Furthermore, the corresponding data stored in the cache data corresponding to the executor 200 can be updated based on the data stored in the relational database 400. Of course, the corresponding data stored in the relational database 400 can also be updated based on the relevant data stored in the cache data corresponding to the executor 200.

[0054] In this embodiment of the application, the scheduling platform 100 may also be referred to as a scheduling platform cluster, the executor 200 may also be referred to as an executor cluster, and the relational database 400 may also be referred to as a relational database cluster.

[0055] In this embodiment of the application, the queue cluster 300 can be implemented using a Redis cluster, and the Redis cluster can store the cached data corresponding to the executor 200.

[0056] Optionally, the queue technology corresponding to the queue cluster can also be implemented using other queue technologies, such as Kafka, Pulsar, MQ (Message Queue), and queue objects, etc., without specific limitations in this application embodiment.

[0057] Optionally, the data storage technology corresponding to the cached data of the executor 200 can also be implemented using other storage technologies. For example, the cached data corresponding to the executor 200 can also be implemented using other databases. No specific restrictions are imposed in this embodiment.

[0058] This application provides a task scheduling system, including a scheduling platform and an executor. Data interaction between the scheduling platform and the executor is achieved through a queue cluster, which realizes high cohesion and low coupling of the task scheduling system, reduces the coupling degree of the task scheduling system, and thus ensures the stable operation of the task scheduling system.

[0059] In the above Figure 1 Based on the aforementioned embodiments, in order to reduce the coupling of the task scheduling system, this application provides a task scheduling method. The following specific examples will explain the task scheduling method provided by the embodiments of this application. Figure 2 This is a flowchart illustrating a task scheduling method provided in an embodiment of this application. Figure 2 As shown, the method includes:

[0060] S201 After the preset business template is launched, the scheduling platform will cache the multiple tasks to be processed in sequence into the task queue of multiple message queues in the preset queue cluster.

[0061] Multiple message queues are defined as follows: each message queue corresponds to a task component in the preset business template, and the task queue is defined as the message queue of the first task component. The preset business template includes multiple task components and the relationships between them, which can be upstream components, downstream components, etc.

[0062] The first task component can be a producer-consumer or publish-subscribe component. The first component can retrieve multiple tasks from multiple message queues and process them.

[0063] The message queue corresponding to each task component is used to enable each task component to process the messages to be processed in the corresponding message queue and cache the processing results of the messages to be processed in the message queues of downstream components.

[0064] In this embodiment, the last task component processes the pending messages in the corresponding message queue and caches the processing results of the pending messages in the result queue in the preset queue cluster, so that the scheduling platform can obtain the processing results of the executor for multiple tasks based on the preset business template from the result queue.

[0065] S202, the scheduling platform generates the number of upstream components completed for each task component in the preset business template based on the identifier of each task in the task queue and the preset business template.

[0066] Since the preset business template includes multiple task components, the number of upstream components completed for each task component in the preset business template can be determined based on the preset business module. In this embodiment, the initial value of the number of upstream components completed for each task component is zero.

[0067] The upstream component completion count of each task component is associated with the corresponding task identifier, and the association is stored in the cache data in the Redis cluster to track the upstream component completion count of each task component in the preset business template corresponding to each task.

[0068] In the embodiments of this application, during the execution of each task component by the executor, the number of completed upstream components of each task component is continuously updated.

[0069] S203, the scheduling platform adds the number of upstream components completed by each task component to the pre-cached scheduling instance of each task component, so that the scheduling instance of each task component includes: the number of upstream components and the number of upstream components completed for each task component.

[0070] Since the preset business template includes multiple task components and the relationships between these multiple task components, the number of upstream components for each task component can be calculated based on the preset business template. The number of upstream components for each task component is a fixed value.

[0071] The number of upstream components for each task component is stored in the cached data of the Redis cluster in the form of scheduling instances of each task component. At the same time, after generating the number of upstream components completed for each task component in the preset business template, the number of upstream components completed for each task component is added to the pre-cached scheduling instances of each task component.

[0072] The scheduling instance of each task component is used to enable the executor to cache the message processing results of the upstream components of each task component to the message queue corresponding to the downstream component. In this embodiment, the executor can determine whether to cache the message processing results of the upstream components of each task component to the message queue corresponding to the downstream component based on the number of upstream components and the number of completed upstream components in the scheduling instance of each task component.

[0073] In this embodiment, the number of upstream components and the number of completed upstream components for each task component can be implemented using a counter. The counter can be implemented based on Redis. Of course, other implementation methods of the counter can also be used, and other counters can also be used. No specific restrictions are imposed in this embodiment.

[0074] Optionally, the counter can be in JSON data format. Of course, other data formats can also be defined, but no specific restrictions are imposed in this embodiment.

[0075] This application provides a task scheduling method in which the scheduling platform caches multiple tasks to be processed and the scheduling instances of each task component in a preset queue cluster, and the executor caches the message processing results of the upstream component of each task component in the message queue of the preset queue cluster corresponding to the downstream component, so that each task component processes the messages to be processed in the corresponding message queue. Through the preset queue cluster, the scheduling platform and the executor are decoupled, the coupling degree of the task scheduling system is reduced, and thus the stable operation of the task scheduling system is guaranteed.

[0076] In the above Figure 2 Based on the task scheduling method shown, this application embodiment also provides another task scheduling implementation method. Optionally, Figure 3 A flowchart illustrating another task scheduling method provided in this application embodiment is shown below. Figure 3 As shown, before method S201 above, after the preset business template is launched, before the scheduling platform sequentially caches the multiple tasks to be processed into the task queue of multiple message queues in the preset queue cluster, the method further includes:

[0077] S301, the scheduling platform responds to the online operation of the preset business template and allocates the corresponding message queue to each task component in the preset queue cluster.

[0078] The online operation of the preset business template is: the online selection operation of the preset business template in at least one business template.

[0079] After selecting a preset business template to be launched from at least one business template, the scheduling platform allocates a corresponding message queue to each task component of the preset business template in the preset queue cluster. After allocation, each task component of the preset business template has a corresponding message queue, enabling each task component to process the messages to be processed in its corresponding message queue. The message queue of the first task component can be referred to as the task queue.

[0080] In this embodiment, after the scheduling platform allocates a corresponding message queue to each task component of the preset business template in the preset queue cluster, it also needs to allocate a result queue to the preset business template. The result queue can also be included in the preset queue cluster. After the last task component processes the message to be processed in the corresponding message queue, it will cache the processing result of the message to be processed in the result queue, so that the scheduling platform can obtain the processing result of the executor for multiple tasks based on the preset business template from the result queue.

[0081] Optionally, each task component corresponds to a message queue. However, a message queue does not necessarily correspond to only one task component. A message queue can correspond to one task component or multiple task components. No specific limitation is made in the embodiments of this application.

[0082] S302, the scheduling platform parses the preset business template and generates scheduling instances for each task component.

[0083] The scheduling instance for each task component includes: the number of upstream components for each task component.

[0084] The preset business template includes multiple task components and the relationships between these task components. The scheduling platform parses the preset business template and can generate the number of upstream components for each task component based on the relationships between the multiple task components.

[0085] For example, if the upstream components of task component 4 include task component 2 and task component 3, then the number of upstream components of task component 4 is 2, indicating that task component 4 has two upstream components.

[0086] The number of upstream components for each task component is placed into the corresponding task component's scheduling instance; that is, the number of upstream components for each task component can be represented in the form of scheduling instances.

[0087] S303, the scheduling platform caches the scheduling instances of each task component to complete the online deployment of the preset business template.

[0088] The scheduling platform caches the scheduling instances of each task component, which can be cached in the cache data of the Redis cluster.

[0089] The scheduling platform can also cache preset business templates in the cache data of the Redis cluster, so that the executor can call the preset business templates and thus enable each task component to process the messages to be processed.

[0090] Optionally, if any of the above steps fail, the launch of the preset business template will fail.

[0091] This application provides a task scheduling method in which a scheduling platform responds to an online operation for a preset business template. In a preset queue cluster, a corresponding message queue is allocated to each task component. The scheduling platform parses the preset business template and generates scheduling instances for each task component. Each scheduling instance of a task component includes the number of upstream components for each task component. The scheduling platform caches the scheduling instances of each task component to complete the online deployment of the preset business template. This allows the scheduling platform to sequentially cache multiple tasks to be processed into task queues in multiple message queues within the preset queue cluster.

[0092] In the above Figure 3 Based on the task scheduling method shown, this application embodiment also provides another task scheduling implementation method. Optionally, Figure 4 A flowchart illustrating another task scheduling method provided in this application embodiment is shown below. Figure 4 As shown, before method S301 above, before the scheduling platform responds to the online operation for the preset business template and allocates the corresponding message queue to each task component in the preset queue cluster, the following steps are also included:

[0093] S401 involves the scheduling platform registering multiple task components, and then creating a business process template on the business orchestration page after the multiple task components are registered.

[0094] Registering multiple task components on the scheduling platform can be done by uploading software packages or accessing interface services. Both the software packages and interface services must conform to the unified scheduling specifications of the scheduling platform. Accessing interface services can be done by uploading a webpage address, i.e., a URL (Uniform Resource Locator) address.

[0095] When registering, task components need to upload not only software packages or interface services, but also corresponding test data. This allows the scheduling platform to perform usability and functional tests on the software packages or interface services based on the test data, confirming their compliance. After the tests are completed, the scheduling platform deletes the temporary data generated during the testing process and retains the test results, i.e., whether the test failed or succeeded. If the test fails, the registration fails; if the test succeeds, the task component information is persisted, i.e., stored in a relational database.

[0096] In this embodiment of the application, the information of the task component can be component identifier, component name, default configuration, and upstream dependencies. The component identifier is the unique number of the component, the component name is the name of the task component, the default configuration is the default configuration of the task component, and the upstream dependencies are the list of upstream dependent components of the task component. Of course, the information of the task component can also be other information that needs to be stored, and no specific restrictions are imposed in this embodiment of the application.

[0097] Successfully registered task components can be selected on the business orchestration page of the scheduling platform. The successful registered task components can be displayed as component identifier, component name, component input, and component output on the business orchestration page. Of course, other display formats are also possible, as long as the task components can be selected on the business orchestration page. No specific restrictions are imposed in this embodiment.

[0098] After multiple task components are successfully registered, a business process template is created on the business orchestration page. First, the basic information of the business template corresponding to the input operation should be received, such as the name and description of the business template. No specific restrictions are imposed in this embodiment.

[0099] S402, the scheduling platform generates a business template based on the component orchestration operation of the business process template entered through the business orchestration page.

[0100] After creating a business process template, based on the upstream dependencies of each task component, input the component orchestration operations of the business process template on the business orchestration page. Based on the component orchestration operations of the business process template input through the business orchestration page, a directed acyclic graph of business execution composed of various task components is formed.

[0101] In this embodiment of the application, after generating the directed acyclic graph, the configuration information of which task components needs to be modified is determined according to the specific business. Then, the configuration of the task components whose configuration information needs to be modified is customized, such as keyword filtering components, AND, OR, NOT decision components, etc. Other configurations are also possible, and no specific restrictions are made in this embodiment of the application.

[0102] A business template is generated based on the directed acyclic graph (DAG) and the configuration information of the task components. The data to be stored in the disk for this business template includes: business number, business name, component DAG, personalized component configuration, version number Ver, edit time Update_time, and creation time Create_time. The business number is the unique identifier of the business template, the business name is the name of the business template, the personalized component configuration corresponds to the independent configuration of each component in the component DAG, the version number is the version number of the business template, and the edit time is the latest modification time of the business template.

[0103] Optionally, the directed acyclic graph of components and personalized component configurations can be stored in JSON data format. Of course, other data formats can also be used, and no specific restrictions are imposed on them in this embodiment.

[0104] Optionally, the data stored in the business template may also include other data, which is not specifically limited in this embodiment.

[0105] This application provides a task scheduling method in which a scheduling platform registers multiple task components. After registration, a business process template is created on a business orchestration page. The scheduling platform generates a business template based on the component orchestration operation of the business process template entered through the business orchestration page. This method provides two different task type processors (software packages and interface services), enabling the task scheduling system to cope with different application scenarios, improving the scalability of the task scheduling system, reducing the workload of developers in scenario conversion, and improving the development efficiency of developers. At the same time, this method provides standardized task orchestration, which can improve the standardization and simplification of user operations, thereby improving the management efficiency of enterprises. Furthermore, this method uses a visual orchestration method to perform business process orchestration on the registered task components, improving the flexibility and scalability of the task scheduling system.

[0106] In the above Figure 2 Based on the task scheduling method shown, this application embodiment also provides another task scheduling implementation method. Optionally, Figure 5 A flowchart illustrating another task scheduling method provided in this application embodiment is shown below. Figure 5 As shown, the above method also includes:

[0107] S501, when the executor has completed the processing of the upstream component in each task component, it determines the position of the preset upstream component and the corresponding downstream component based on the pre-cached preset business template.

[0108] In this embodiment, the first task component performs data access, parsing, and filtering operations to generate data in a standardized format, which can then be used for specific task execution scheduling.

[0109] During scheduling, when the upstream component in each task component has finished processing, the executor determines the position of the upstream component based on the directed acyclic graph in the pre-cached pre-defined business template, and then determines the downstream component corresponding to the upstream component based on the directed acyclic graph in the pre-defined business template and the position of the upstream component.

[0110] For example, if the upstream component is preset to be task component 3, the downstream components corresponding to task component 3 are determined to be task component 4 and task component 5 according to the directed acyclic graph in the preset business template.

[0111] S502, the executor updates the upstream component completion count in the scheduling instance of the downstream component corresponding to the preset upstream component.

[0112] If the upstream component has processed the pending messages in the corresponding message queue, it means that the upstream component of the corresponding downstream component has completed its execution. At this time, the number of upstream components completed in the scheduling instance of the downstream component can be updated.

[0113] For example, if the upstream component is preset to be task component 3, the downstream components corresponding to task component 3 are determined to be task component 4 and task component 5 according to the directed acyclic graph in the preset business template. When the upstream component is finished, the completion count of the upstream component in the scheduling instance of task component 4 and task component 5 is incremented by one, indicating that one of the upstream components of task component 4 and task component 5 has been completed.

[0114] This application provides a task scheduling method in which the executor determines the position of the preset upstream component and the corresponding downstream component based on a pre-cached preset business template when the upstream component in each task component has completed processing. The executor then updates the upstream component completion count in the scheduling instance of the downstream component corresponding to the preset upstream component. This method uses a counter method, i.e., updating the upstream component completion count, to realize the dependency relationship of complex tasks. By parsing the directed acyclic graph, a business scheduling instance with a counter is generated, which enables the executor to realize the complete scheduling process by interpreting the scheduling instance, thereby improving scheduling efficiency.

[0115] In the above Figure 5 Based on the task scheduling method shown, this application embodiment also provides another task scheduling implementation method. Optionally, Figure 6 A flowchart illustrating another task scheduling method provided in this application embodiment is shown below. Figure 6 As shown, the above method also includes:

[0116] S601, the executor determines whether the number of completed upstream components and the number of upstream components in the scheduling instance of the downstream component corresponding to the preset upstream component are equal.

[0117] In this embodiment, task scheduling can be simplified to how the upstream component publishes tasks to the message queue of the downstream component.

[0118] The executor determines whether the number of completed upstream components and the number of upstream components in the scheduling instance of the downstream component corresponding to the preset upstream component are equal. If they are equal, it means that all upstream components of the downstream component have been completed. If they are not equal, it means that at least one of the upstream components of the downstream component has not been completed.

[0119] S602, if they are equal, the executor will cache the message processing result of the preset upstream component into the message queue corresponding to the downstream component of the preset upstream component.

[0120] In this application embodiment, the relationship between upstream and downstream components can be divided into three types. Among them, the case of multiple upstream components corresponding to multiple downstream components can be further broken down into a scenario where multiple upstream components correspond to one downstream component and a scenario where one upstream component corresponds to multiple upstream components. Therefore, task scheduling can be mainly abstracted into three scenarios: one-to-one, one-to-many, and many-to-one. The following details how these three scenarios are processed using a counter, and ultimately unified into the same processing logic.

[0121] When an upstream component corresponds to only one downstream component, after the upstream component finishes processing, the position of the upstream component and the position of the corresponding downstream component are found in the directed acyclic graph of the scheduling instance of the current upstream component in the cache data of the Redis cluster. Since it is a one-to-one scenario, only one downstream component can be found. At this time, the upstream component count of the downstream component is incremented by one, and it is checked whether the upstream component completion count of the downstream component is equal to the upstream component count. If they are equal, the message processing result of the upstream component is cached in the message queue corresponding to the downstream component.

[0122] When an upstream component corresponds to multiple downstream components, upon completion of processing by the upstream component, the position of the upstream component and its corresponding downstream component are found in the directed acyclic graph of the scheduling instance of the current upstream component in the Redis cluster's cache data. Since it is a one-to-many scenario, multiple downstream components can be found. The upstream component count of each of the multiple downstream components is incremented sequentially. The completion count of the upstream components of each of the multiple downstream components is checked sequentially to see if it equals the number of upstream components. If there is a downstream component whose upstream component completion count equals the number of upstream components, the message processing result of that upstream component is cached in the message queue corresponding to that downstream component. If there is a downstream component whose upstream component completion count does not equal the number of upstream components, the message processing result of that upstream component is cached in the Redis cluster's cache data and not placed in the message queue of that downstream component. This process continues until the upstream component completion count of that downstream component equals the number of upstream components. At this point, all message processing results of that downstream component cached in the Redis cluster are cached in the message queue of that downstream component, and all message processing results of that downstream component cached in the Redis cluster are cleared.

[0123] When multiple upstream components correspond to one downstream component, after the upstream component completes its processing, the position of the upstream component and the corresponding downstream component are found in the directed acyclic graph of the scheduling instance of the current upstream component in the Redis cluster's cache data. Since it is a many-to-one scenario, a downstream component can be found, and the upstream component completion count of the downstream component is incremented by one. It is then checked whether the upstream component completion count of the downstream component is equal to the number of upstream components. If they are not equal, the message processing result of the upstream component is cached in the Redis cluster's cache data and not put into the downstream component's message queue. This process continues until the upstream component completion count of the downstream component equals the number of upstream components. At this point, all message processing results of the downstream component cached in the Redis cluster are cached in the downstream component's message queue, and all message processing results of the downstream component cached in the Redis cluster are cleared.

[0124] Optionally, the scheduling platform can monitor task components through preset timeouts and heartbeats. The preset timeout determines whether the processing time for each task component to process the message exceeds a preset time; if it does, the task component may have a problem. The heartbeat mechanism involves the task component sending status information to the scheduling platform every preset time interval, allowing the platform to monitor the task component's status. Monitoring task components through preset timeouts and heartbeats effectively improves the stability of the task scheduling system.

[0125] The task scheduling method provided in this application embodiment determines whether the number of completed upstream components and the number of upstream components in the scheduling instance of the downstream component corresponding to the preset upstream component are equal. If they are equal, the executor caches the message processing result of the preset upstream component into the message queue corresponding to the downstream component of the preset upstream component. This method can realize complex dependency orchestration of task components in one-to-one, one-to-many, and many-to-one manner to achieve a complete scheduling process and improve scheduling efficiency.

[0126] In the above Figure 3 Based on the task scheduling method shown, this application embodiment also provides another task scheduling implementation method. Optionally, the above method further includes:

[0127] The scheduling platform creates at least one consumer group for the message queue corresponding to each task component, so that the executor processes the message queue corresponding to each task component based on at least one consumer group.

[0128] In this embodiment, a Redis stream consumer group is used as an example. A new data type can be set in Redis, which is a log structure built in an abstract way. To achieve fast parallel message consumption, a group of consumer clients can cooperate to consume different parts of the same message stream; this method can be called the consumer group approach. Specific consumer group commands are as follows:

[0129] XGROUP[CREATE key groupname id-or-$]: Creates a consumer group associated with the stream;

[0130] [SETID key groupname id-or-$]: Sets the next message to be delivered;

[0131] [DESTROY key groupname]: Destroys a consumer group;

[0132] [CREATECONSUMER key groupname consumername]: Adds the specified consumer to the consumer group;

[0133] [DELCONSUMER key groupname consumername]: Removes the specified consumer from the consumer group.

[0134] This method allows each task component to act as a consumer, using "CREATECONSUMER" and "DELCONSUMER" to achieve horizontal elastic scaling of each task component, thereby making full use of hardware resources.

[0135] When each task component acts as a consumer, it can be assumed that there are multiple nodes within each task component. These multiple nodes can operate in parallel, and each node can process the message to be processed. When the tasks to be processed are multiple consecutive tasks, i.e., when the task to be processed is a task flow, the multiple nodes in each task component can process multiple tasks separately, thereby improving the processing performance of each task component and greatly improving the processing efficiency of the task flow.

[0136] This application provides a task scheduling method in which the scheduling platform creates at least one consumer group for the message queue corresponding to each task component, so that the executor processes the message queue corresponding to each task component based on at least one consumer group. This method highly abstracts each task in the distributed task flow, and each task component achieves high cohesion and low coupling, which greatly enhances the scalability of each task component. It supports expansion in task event distribution, alarm center and other methods, and improves the performance of each task component.

[0137] Based on the task scheduling system and task scheduling method provided in the above embodiments of this application, the task scheduling system will be specifically explained and described below with reference to the accompanying drawings through a complete embodiment. Figure 7 This is a schematic diagram of another task scheduling system provided in an embodiment of this application. Figure 7 As shown, the task scheduling interface includes a registration interface, which allows task components to be registered by accessing the interface service.

[0138] Task components can be registered in the component registration section of the scheduling platform. Successfully registered components can be selected in the business editor to create business process templates on the business coding page of the business editor.

[0139] Once a preset business template is selected for deployment, and the deployment is successful, multiple tasks to be processed are sequentially cached into the task queues of multiple message queues in the Redis cluster through task submission. The first task component t1 of the preset business template in the executor cluster consumes the multiple tasks that have been accessed, i.e., data access, parsing and filtering, and generates data in a standardized format.

[0140] The message queues corresponding to task components t2 and t3 are component queues connected to task components t2 and t3 by dashed lines. After the first task component t1 receives the message processing result, it increments the completion count of the upstream components in its downstream components (task components t2 and t3). At this point, the completion count of the upstream components in task components t2 and t3 is equal to the total number of upstream components. Then, the first task component t1 puts the message processing result into the component queues corresponding to task components t2 and t3, allowing task components t2 and t3 to process the messages to be processed in their respective component queues. After task components t2 and t3 have finished processing all the messages to be processed, they put their respective message processing results into the component queue corresponding to the last task component t4, allowing the last task component t4 to process the messages to be processed in its corresponding component queue. After processing, the message processing result of the last task component t4 is put into the result queue through result feedback, so that the result consumer in the scheduling platform can obtain the message processing result from the result queue.

[0141] During execution, the executor updates the counters in the Redis cluster based on the execution results of each task component in the preset business template. The counters include the number of upstream components completed and the number of upstream components for each task component.

[0142] If task component t2 finishes processing the message to be processed, but task component t3 has not finished processing the message to be processed, then task component t2 caches the message processing result in the cache data in the Redis cluster until task component t3 has not finished processing the message to be processed. Then, the message processing result of task component t3 and the message processing result of task component t2 in the cache data are put into the component queue corresponding to the last task component t4.

[0143] This system decouples the executor and scheduling platform through a Redis cluster, reducing the coupling of the task scheduling system. It also provides different task type processors (task scheduling interfaces and uploaded software packages) for developers to implement various application scenarios, improving development efficiency. Furthermore, it provides standardized task orchestration through business editing, greatly improving enterprise management efficiency. Moreover, it uses counters (the number of upstream components completed versus the total number of upstream components) to implement dependencies on complex tasks. Through the business scheduling graph, it generates business scheduling instances with counters, allowing the executor to interpret these instances and complete the entire scheduling process, thus improving scheduling efficiency.

[0144] The following describes a task scheduling apparatus, device, and storage medium provided in this application for execution. The specific implementation process and technical effects are described above and will not be repeated below.

[0145] Figure 8 A schematic diagram of a task scheduling device provided in an embodiment of this application is shown below. Figure 8 As shown, the task scheduling device includes:

[0146] The scheduling module 801 includes: a cache unit 8011, a generation unit 8012, and an addition unit 8013.

[0147] The caching unit 8011 is used to cache multiple tasks to be processed sequentially into the task queue of multiple message queues in the preset queue cluster after the preset business template is launched. The multiple message queues are: the message queues corresponding to each task component in the preset business template, and the task queue is: the message queue of the first task component. The message queue corresponding to each task component is used to enable each task component to process the messages to be processed in the corresponding message queue and cache the processing results of the messages to be processed into the message queues corresponding to downstream components.

[0148] The generation unit 8012 is used to generate the number of upstream components completed for each task component in the preset business template based on the identifier of each task in the task queue and the preset business template.

[0149] Add unit 8013 to add the number of upstream components completed by each task component to the pre-cached scheduling instance of each task component, so that the scheduling instance of each task component includes: the number of upstream components and the number of upstream components completed by each task component.

[0150] The execution module 802 is used to cache the message processing results of the upstream component of each task component to the message queue corresponding to the downstream component.

[0151] Optionally, the caching unit 8011 is also used for the scheduling platform to respond to the online operation of the preset business template, allocate corresponding message queues for each task component in the preset queue cluster; the scheduling platform parses the preset business template to generate scheduling instances for each task component, and the scheduling instance of each task component includes: the number of upstream components of each task component; the scheduling platform caches the scheduling instances of each task component to complete the online deployment of the preset business template.

[0152] Optionally, the cache unit 8011 is specifically used for the online operation of the preset business template as follows: the online selection operation of the preset business template in at least one business template.

[0153] Optionally, the cache unit 8011 is also used for the scheduling platform to register multiple task components, and after the multiple task components are registered, to create a business process template on the business orchestration page; the scheduling platform generates a business template based on the component orchestration operation of the business process template entered through the business orchestration page.

[0154] Optionally, the execution module 802 is further configured to determine the position of the preset upstream component and the downstream component corresponding to the preset upstream component based on the pre-cached preset business template when the upstream component in each task component is completed by the executor; and update the number of upstream components completed in the scheduling instance of the downstream component corresponding to the preset upstream component by the executor.

[0155] Optionally, the execution module 802 is further configured to determine whether the number of completed upstream components and the number of upstream components in the scheduling instance of the downstream component corresponding to the preset upstream component are equal; if they are equal, the executor caches the message processing result of the preset upstream component in the message queue corresponding to the downstream component of the preset upstream component.

[0156] Optionally, the cache unit 8011 is also used to create at least one consumer group for the message queue corresponding to each task component by the scheduling platform.

[0157] Optionally, the execution module 802 is also configured to process the message queue corresponding to each task component based on at least one consumer group.

[0158] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0159] Figure 9 This is a schematic diagram of a scheduling platform provided in an embodiment of this application. The scheduling platform can be a computing device with computing processing capabilities.

[0160] The scheduling platform includes a first processor 901, a first storage medium 902, and a first bus 903. The first processor 901 and the first storage medium 902 are connected via the first bus 903.

[0161] The first storage medium 902 is used to store a program, and the first processor 901 calls the program stored in the first storage medium 902 to execute the above method embodiment. The specific implementation and technical effects are similar, and will not be described in detail here.

[0162] Figure 10 This is a schematic diagram of an actuator provided in an embodiment of this application. The actuator may be a computing device with computational processing capabilities.

[0163] The actuator includes a second processor 1001, a second storage medium 1002, and a second bus 1003. The second processor 1001 and the second storage medium 1002 are connected via the second bus 1003.

[0164] The second storage medium 1002 is used to store a program, and the second processor 1001 calls the program stored in the second storage medium 1002 to execute the above method embodiment. The specific implementation method and technical effect are similar, and will not be described again here.

[0165] Optionally, the present invention also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, is used to perform the above-described method embodiments.

[0166] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0167] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0169] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0170] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A task scheduling method, characterized in that, The method includes: After the preset business template is launched, the scheduling platform will sequentially cache multiple tasks to be processed into task queues in multiple message queues in the preset queue cluster. The multiple message queues are: message queues corresponding to each task component in the preset business template, and the task queue is: the message queue of the first task component. The message queue corresponding to each task component is used to enable each task component to process the messages to be processed in the corresponding message queue and cache the processing results of the messages to be processed into the message queues corresponding to downstream components. The scheduling platform generates the number of upstream components completed for each task component in the preset business template based on the identifier of each task in the task queue and the preset business template. The scheduling platform adds the completion count of upstream components of each task component to the pre-cached scheduling instances of each task component, such that each scheduling instance of a task component includes: the number of upstream components and the number of upstream component completions; the scheduling instance of each task component is used to enable the executor to cache the message processing results of the upstream components of each task component to the message queue corresponding to the downstream component; wherein, the executor determines whether the number of upstream component completions and the number of upstream components in the scheduling instance of the downstream component corresponding to the preset upstream component are equal; If they are equal, the executor will cache the message processing result of the preset upstream component into the message queue corresponding to the downstream component of the preset upstream component.

2. The task scheduling method according to claim 1, characterized in that, Before the scheduling platform sequentially caches multiple tasks to be processed into the task queue of multiple message queues in the preset queue cluster after the preset business template is launched, the method further includes: The scheduling platform responds to the online operation of the preset business template and allocates a corresponding message queue to each task component in the preset queue cluster. The scheduling platform parses the preset business template to generate scheduling instances for each task component. Each scheduling instance for a task component includes the number of upstream components for each task component. The scheduling platform caches the scheduling instances of each task component to complete the online deployment of the preset business template.

3. The task scheduling method according to claim 2, characterized in that, The online operation of the preset business template is: the online selection operation of the preset business template in at least one business template.

4. The task scheduling method according to claim 3, characterized in that, Before the scheduling platform responds to the online operation for the preset business template and allocates a corresponding message queue to each task component in the preset queue cluster, the method further includes: The scheduling platform registers multiple task components, and after the multiple task components are registered, a business process template is created on the business orchestration page; The scheduling platform generates a business template based on the component orchestration operation of the business process template input through the business orchestration page.

5. The task scheduling method according to claim 1, characterized in that, The method further includes: When the upstream component in each task component has finished processing, the executor determines the position of the preset upstream component and the corresponding downstream component based on the pre-cached preset business template. The executor updates the upstream component completion count in the scheduling instance of the downstream component corresponding to the preset upstream component.

6. The task scheduling method according to claim 2, characterized in that, The method further includes: The scheduling platform creates at least one consumer group for the message queue corresponding to each task component, so that the executor processes the message queue corresponding to each task component based on the at least one consumer group.

7. A task scheduling device, characterized in that, include: The scheduling module includes: a caching unit, a generation unit, and an addition unit; The caching unit is used to cache multiple tasks to be processed sequentially into the task queue of multiple message queues in a preset queue cluster after the preset business template is launched. The multiple message queues are: message queues corresponding to each task component in the preset business template, and the task queue is: the message queue of the first task component. The message queue corresponding to each task component is used to enable each task component to process the messages to be processed in the corresponding message queue and cache the processing results of the messages to be processed into the message queues corresponding to downstream components. The generation unit is used to generate the number of upstream components completed for each task component in the preset business template based on the identifier of each task in the task queue and the preset business template. The adding unit is used to add the number of upstream components completed by each task component to the pre-cached scheduling instance of each task component, so that the scheduling instance of each task component includes: the number of upstream components and the number of upstream components completed by each task component; The execution module is used to cache the message processing results of the upstream component of each task component to the message queue corresponding to the downstream component; the executor determines whether the number of completed upstream components and the number of upstream components in the scheduling instance of the downstream component corresponding to the preset upstream component are equal; if they are equal, the executor caches the message processing results of the preset upstream component to the message queue corresponding to the downstream component of the preset upstream component.

8. A task scheduling system, characterized in that, include: A scheduling platform and an executor, wherein the scheduling platform is used to execute the operation corresponding to the scheduling platform in any of the task scheduling methods described in claims 1-6, and the executor is used to execute the operation corresponding to the executor in any of the task scheduling methods described in claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when read and executed, implements the task scheduling method according to any one of claims 1-6.

Images