Distributed task scheduling method, task scheduling equipment, execution equipment and distributed scheduling platform system
By dynamically generating the total number and information of job shards through custom sharding rules, the rigidity problem of existing distributed task scheduling platforms is solved, enabling flexible task sharding and efficient business adaptation, and reducing operation and maintenance costs and resource waste.
Patent Information
- Application Number
- CN202511677458.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-02-17
AI Technical Summary
Existing distributed task scheduling platforms have fixed sharding mechanisms, making it difficult to adapt to diverse task sharding processing needs, resulting in high operation and maintenance costs and resource waste.
By obtaining custom sharding rules, a job triggering instruction is generated, and the execution device dynamically generates the total number of job shards. Based on this total number, sharding information is generated. The task scheduling device does not need to preset sharding strategies, thus achieving flexible task sharding.
It improves the adaptability of distributed task scheduling to different business scenarios, reduces operation and maintenance costs and resource waste, and improves processing efficiency.
Smart Images
Figure CN121541998A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer communication technology, and in particular to a distributed task scheduling method, task scheduling device, execution device, and distributed scheduling platform system. Background Technology
[0002] In the current era of big data, distributed computing has become one of the key technologies for processing massive amounts of data. As an important component of distributed computing, the core function of a distributed task scheduling platform is to decompose large tasks (jobs) into multiple subtasks, i.e., sharding, and allocate these shards to different computing nodes for execution, thereby improving the efficiency and speed of business data processing.
[0003] However, existing distributed task scheduling platforms generally have shortcomings in their sharding mechanisms: most platforms provide preset, general sharding algorithms, such as round-robin sharding, hash sharding, or average sharding. These strategies are disconnected from specific business logic, making it difficult to adapt to diverse task sharding processing needs. Summary of the Invention
[0004] The main purpose of this application is to provide a distributed task scheduling method, task scheduling device, execution device, and distributed scheduling platform system, aiming to solve the technical problem that the fragmentation mechanism provided by the existing distributed task scheduling platform is rigid and difficult to adapt to the diverse task fragmentation processing needs.
[0005] To achieve the above objectives, this application proposes a distributed task scheduling method, which is applied to a task scheduling device in a distributed scheduling platform system. The method includes: Obtain custom sharding rules and generate job triggering instructions based on the custom sharding rules, which are pre-configured by the user through platform components integrated on the execution device; The job triggering instruction is sent to the execution device, so that the execution device performs a custom job sharding operation in response to the job triggering instruction, and obtains the total number of job shards; Upon receiving the total number of job fragments sent by the execution device, fragment information is generated based on the total number of job fragments; The fragmentation information is sent to the execution device so that the execution device can determine the fragmented job data based on the fragmentation information and execute the job fragmentation according to the fragmented job data.
[0006] In one embodiment, the step of obtaining a custom sharding rule and generating a job triggering instruction based on the custom sharding rule includes: The platform component obtains the custom sharding rules and generates the initial job shards based on the custom sharding rules. The initial job fragments are numbered according to the preset initial numbering rules, and the numbered initial job fragments are stored in the initialized fragment information table; Upon completion of storage, a job triggering instruction is generated based on the initial job fragment numbered after it.
[0007] In one embodiment, the step of generating fragment information based on the total number of job fragments when receiving the total number of job fragments sent by the execution device includes: Upon receiving the total number of job fragments, a validity check is performed on the total number of job fragments. When the total number of job fragments passes the verification, fragment information is generated based on the total number of job fragments; After the step of generating shard information based on the total number of job shards, the method further includes: The initialized sharding information table is updated based on the sharding information to obtain a sharding information table with completed sharding. The sharding information table with completed sharding maintains several job sharding entries.
[0008] In one embodiment, after the step of sending the fragmentation information to the execution device, the method further includes: When the segmented operation is completed, segmented execution information is obtained from the execution device; The sharding information table is updated based on the sharding execution information to obtain the target sharding information table.
[0009] Furthermore, to achieve the above objectives, this application also proposes a distributed task scheduling method, which is applied to an execution device in a distributed scheduling platform system. The system further includes a task scheduling device electrically connected to the execution device. The method includes: Upon receiving a job triggering instruction from the task scheduling device, a custom job sharding operation is performed to obtain the total number of job shards. The job triggering instruction is generated by the task scheduling device when it obtains the custom sharding rules, which are pre-configured by the user through integrated platform components. The total number of job fragments is sent to the task scheduling device so that the task scheduling device can generate fragment information based on the total number of job fragments; Obtain the sharding information, determine the sharding job data based on the sharding information, and execute job sharding based on the sharding job data.
[0010] In one embodiment, the step of performing a custom job sharding operation to obtain the total number of job shards upon receiving a job triggering instruction from the task scheduling device includes: Upon receiving the job trigger instruction, the initial job fragmentation is executed, wherein the initial job fragmentation is generated by the task scheduling device according to the custom fragmentation rules; The application business data is sharded according to the custom sharding rules to obtain a data grouping table. The data grouping table is a mapping relationship between data group identifiers and job sharding sequence numbers. The data group identifier is used to identify different shards of business data obtained by sharding the application business data. The total number of job fragments is determined based on the data grouping table, and the initial job fragmentation is completed.
[0011] In one embodiment, the steps of obtaining the sharding information, determining sharded job data based on the sharding information, and performing job sharding based on the sharded job data include: The current job shard number is determined based on the sharding information, and the current business data group identifier corresponding to the current job shard number is queried according to the data grouping table; The corresponding sharded business data is retrieved from the database based on the current business data group identifier, and the sharded business data is identified as sharded job data. The job data is sharded according to a job sharding processing strategy, which is pre-configured by the user through the integrated platform components.
[0012] In addition, to achieve the above objectives, this application also proposes a task scheduling device, the device comprising: a first memory, a first processor, and a distributed task scheduler program stored in the first memory and executable on the first processor, the distributed task scheduler program being configured to implement the steps of the distributed task scheduling method of the task scheduling device applied to a distributed scheduling platform system as described above.
[0013] In addition, to achieve the above objectives, this application also proposes an execution device, the device comprising: a second memory, a second processor, and a distributed task scheduler stored in the second memory and executable on the second processor, the distributed task scheduler being configured to implement the steps of the distributed task scheduling method for the execution device applied to the distributed scheduling platform system described above.
[0014] In addition, to achieve the above objectives, this application also proposes a distributed scheduling platform system, which includes the task scheduling device and the execution device as described above.
[0015] This application discloses a distributed task scheduling method applied to a task scheduling device in a distributed scheduling platform system. The system also includes an execution device electrically connected to the task scheduling device. The method includes: obtaining custom sharding rules and generating a job triggering instruction based on the custom sharding rules, wherein the custom sharding rules are pre-configured by the user through a platform component integrated on the execution device; sending the job triggering instruction to the execution device so that the execution device performs a custom sharding operation in response to the job triggering instruction, thereby obtaining the total number of job shards; upon receiving the total number of job shards sent by the execution device, generating sharding information based on the total number of job shards; and sending the sharding information to the execution device so that the execution device determines the sharded job data based on the sharding information and performs job sharding based on the sharded job data.
[0016] This application enables the execution of user-defined sharding rules by the device to dynamically generate the total number of job shards and generate corresponding sharding information based on this total number. This allows the task scheduling device to achieve flexible task sharding without the need for pre-configured sharding strategies. This effectively overcomes the rigidity of traditional scheduling platform sharding strategies and significantly improves the adaptability of distributed task scheduling to different business scenarios. Furthermore, by refining the granularity of task execution control from the task level to the shard level, it helps reduce operational costs and resource waste, thereby improving processing efficiency. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart illustrating the first embodiment of the distributed task scheduling method of this application; Figure 2 Example diagram of the target fragment information table obtained after the job fragment execution is completed; Figure 3 This is a flowchart illustrating the second embodiment of the distributed task scheduling method of this application; Figure 4 A diagram illustrating the execution process of custom fragmentation operations for a job; Figure 5 A schematic diagram illustrating the execution process of job segmentation; Figure 6This is a schematic diagram illustrating the interaction between task scheduling devices and execution devices in a distributed scheduling platform system. Figure 7 This is a schematic diagram of the task scheduling equipment in this application; Figure 8 This is a schematic diagram of the structure of the device used in this application.
[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0023] The main solution of this application embodiment is as follows: obtain custom sharding rules and generate job triggering instructions according to the custom sharding rules. The custom sharding rules are pre-configured by the user through platform components integrated on the execution device; send the job triggering instructions to the execution device so that the execution device performs custom sharding operations in response to the job triggering instructions to obtain the total number of job shards; when the total number of job shards sent by the execution device is received, sharding information is generated according to the total number of job shards; send the sharding information to the execution device so that the execution device determines the sharded job data based on the sharding information and performs job sharding according to the sharded job data.
[0024] Current distributed task scheduling platforms typically offer several built-in, general-purpose sharding algorithms, such as "round-robin sharding," "hash sharding," or "average sharding." These strategies are static and decoupled from business logic. However, when faced with complex business scenarios, the platform's source code needs to be modified according to the specific business scenario, leading to platform code complexity and impacting platform stability. Furthermore, because the sharding algorithm is built into the scheduling platform, existing platforms usually only display the overall success or failure status of the task, making it impossible for users to perceive sharding details. That is, management operations (such as retries) can only be performed on the entire task as the smallest unit, failing to achieve task management from the sharding perspective, resulting in high operational costs.
[0025] Therefore, to address the aforementioned shortcomings, this embodiment provides a distributed task scheduling method applied to a task scheduling device within a distributed scheduling platform system. The system also includes an execution device electrically connected to the task scheduling device. Because the execution device can dynamically generate the total number of job shards by executing user-defined sharding rules, and generate corresponding sharding information based on this total number, the task scheduling device can achieve flexible task sharding without pre-setting sharding strategies. This effectively overcomes the rigidity of traditional scheduling platform sharding strategies and significantly improves the adaptability of distributed task scheduling to different business scenarios. Simultaneously, by refining the granularity of task execution control from the task level to the shard level, it helps reduce operational costs and resource waste, thereby improving processing efficiency.
[0026] Based on this, embodiments of this application provide a distributed task scheduling method, referencing Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the distributed task scheduling method of this application. In this embodiment, the method is applied to a task scheduling device in a distributed scheduling platform system. The system also includes an execution device electrically connected to the task scheduling device. The method includes steps S10 to S40: Step S10: Obtain custom sharding rules and generate job triggering instructions based on the custom sharding rules. The custom sharding rules are pre-configured by the user through platform components integrated on the execution device.
[0027] It should be noted that the executing entity in this embodiment can be a computing electronic device with data processing, network communication, program execution, and interactive control functions, such as a desktop computer, mainframe computer, or server. It can also be other electronic devices capable of implementing task scheduling functions in a distributed scheduling platform system. This embodiment will be specifically described here using a task scheduling device as an example; this task scheduling device can be considered as the platform side of the system.
[0028] It should be understood that the distributed scheduling platform system may also include one or more execution devices, which can be regarded as the business side of the system, and the execution device may integrate platform components provided by the platform side. Users (business side developers) can develop jobs on the business side through the platform components.
[0029] The aforementioned platform components are software development kits (SDKs) provided by the platform side. This allows the task scheduling device to interact with the execution device based on the SDK. The SDK contains one or more standard interfaces, such as the `SimpleJob` interface class. This interface class can provide multiple job strategies, including: `shardinghandler()` strategy (custom job sharding strategy); `preProcess()` strategy (unified pre-processing strategy for jobs); `beforeJob()` strategy (pre-processing strategy for job sharding); `execute()` strategy (job sharding processing strategy); `afterJob()` strategy (post-processing strategy for job sharding); and `postProcess()` strategy (unified post-processing strategy for jobs).
[0030] Business-side developers can write code with specific strategies on the execution device to implement the standard interfaces provided by the SDK and obtain job content. For example, if the business side uses Java jobs, business-side developers can implement the standard interfaces provided by the SDK in a Java class, which can then become a job, such as a TestJob job.
[0031] It should be understood that the custom sharding rule can be a specific shardinghandler() strategy in the standard interface. Therefore, the task scheduling device can start executing custom sharding when it obtains the shardinghandler() strategy written by the business side developers through the SDK: first, it generates an initial job shard for executing the above custom sharding rule, and then generates a job trigger instruction based on the initial shard, which enables the execution device to execute the initial job shard.
[0032] Furthermore, to specifically illustrate how to generate the job trigger instruction, step S10 includes: steps S101~S103: Step S101: Obtain custom sharding rules through platform components, and generate initial job shards according to the custom sharding rules.
[0033] It should be understood that when the execution device obtains the custom sharding rules through the SDK, it can generate an initial job shard, which has the highest priority (sharding level 0).
[0034] Step S102: Number the initial job fragments according to the preset initial numbering rules, and store the numbered initial job fragments in the initialized fragment information table.
[0035] It should be understood that, in order to prevent duplicate serial numbers, this preset initial numbering rule can be a rule to ensure that the serial number of the initial job fragment does not duplicate the serial numbers of each job fragment obtained when custom fragmenting is performed. For example, if the numbering rule when custom fragmenting is to start from 0 and number sequentially (e.g., 0, 1, 2, 3, 4, ...), then the job fragment serial number obtained by numbering the initial fragment job can be a value less than 0 (e.g., -3).
[0036] The sharding information table can be a pre-set information table used to maintain and manage all job shards in the system. The initial allocation information table can include the sharding information of the initial job shards mentioned above.
[0037] Step S103: Upon completion of storage, generate a job trigger instruction based on the initial job fragment numbered after it.
[0038] In a specific implementation, when the sharding information of the initial job sharding is stored in the sharding information table, a job triggering instruction can be generated. This job triggering instruction can carry job data for executing the initial job sharding.
[0039] Step S20: Send the job triggering instruction to the execution device so that the execution device responds to the job triggering instruction to perform a job custom sharding operation and obtain the total number of job shards.
[0040] It should be understood that the execution device can be one or more execution devices that are pre-registered with the task scheduling device in the system.
[0041] It is understandable that the total number of job shards can be the number of shards obtained by the execution device based on the sharding rules defined by the business side developers.
[0042] In the specific implementation, the task scheduling device can send a job triggering instruction carrying job data to each execution device. After each execution device responds to the job triggering instruction, one of the execution devices will specifically perform the job definition sharding operation, i.e., the initial job sharding, i.e., execute the shardinghandler() strategy to obtain the total number of job shards.
[0043] Step S30: Upon receiving the total number of job fragments sent by the execution device, generate fragment information based on the total number of job fragments.
[0044] It should be understood that after completing the initial job fragmentation, the execution device can return the total number of job fragments obtained to the task scheduling device.
[0045] When the task scheduling device receives the total number of job fragments, it can generate fragment information, which may include: job name ID, job code, job batch, fragment item number, application code, fragment level, fragment start time, fragment completion time, fragment status, and exception information.
[0046] Furthermore, to ensure the validity of the obtained total number of job fragments, step S30 further includes: steps S301~S302: Step S301: Upon receiving the total number of job fragments, perform a numerical validity check on the total number of job fragments.
[0047] Step S302: When the total number of job fragments passes the verification, fragment information is generated based on the total number of job fragments.
[0048] It should be noted that numerical validity verification can be used to determine whether the total number of job shards is a non-positive integer. If the received total number of job shards is negative, 0, or a non-integer, it can be determined that the total number of job shards has failed the verification, and a sharding exception alarm will be generated to notify the business side developers.
[0049] If the total number of job fragments received is a positive integer, it can be determined that the total number of job fragments has passed the verification. Then, different job fragments and their corresponding fragment information can be generated sequentially based on the total number of job fragments.
[0050] For example, if the total number of job fragments is 185, then job fragments with serial numbers 0, 1, 2, ..., 184 and their corresponding fragment information can be generated consecutively and uniquely.
[0051] Furthermore, after obtaining each job shard and its corresponding shard information, the initialized shard information table can be updated based on the shard information to obtain the shard information table after sharding is completed.
[0052] It should be noted that the sharding information table after the sharding is completed currently maintains several job sharding entries, each containing the sharding information for one job shard. Furthermore, at this point, apart from the sharding information corresponding to the initial job shard, the sharding start time, sharding completion time, sharding status, and exception information for each job shard in the completed sharding information table are all empty.
[0053] Step S40: Send the fragmentation information to the execution device so that the execution device determines the fragmented job data based on the fragmentation information and executes job fragmentation according to the fragmented job data.
[0054] It should be understood that the execution device can send the fragment information corresponding to the different job fragments mentioned above to the execution device, thereby triggering the execution device to execute the corresponding job fragment, that is, to execute the execute() strategy.
[0055] It should be noted that the sharded job data can be stored in a separate database server. During the initial job sharding process, the execution device can also group the complete application business data into several groups of sharded business data and maintain a mapping relationship between these groups and the sequence numbers of each job shard. Therefore, when executing a corresponding job shard, the corresponding sharded job data can be retrieved from the database server based on this mapping relationship.
[0056] In the specific implementation, the task scheduling device can send the sharding information to the execution device, triggering the execution device to start executing the corresponding job shards: obtaining sharded job data from the database server, performing the corresponding execute() strategy, and finally obtaining the sharding execution information and returning it to the task scheduling device.
[0057] Therefore, after step S40, the following steps are also included: steps S501~S502: Step S501: When the sharding operation is completed, obtain sharding execution information from the execution device.
[0058] It should be understood that the sharding execution information may include: sharding start time, sharding completion time, sharding job result, etc.
[0059] The sharding start time is the time when the execution device begins to execute the sharding operation, the sharding completion time is the time when the execution device completes the sharding operation, and the sharding operation result may include abnormal information during the sharding operation.
[0060] Step S502: Update the sharding information table of the completed sharding according to the sharding execution information to obtain the target sharding information table.
[0061] It should be noted that when the execution device starts executing the sharding job, the sharding start time value can be entered into the sharding start time field of the sharding information table for that job's sharding item. When the execution device completes the sharding job, the sharding completion time value and the exception information field can be entered into the sharding completion time field and the exception information field of the sharding information table for that job's sharding item, respectively. Furthermore, the sharding status of the sharding item for that job's sharding item in the sharding information table can be updated to SUCCESS or FAIL.
[0062] It should also be noted that the fragmentation status of the job fragmentation item in the fragmentation information table can be continuously updated as the fragmentation job progresses: initially set to NEW, updated to WAIT when the execution device receives fragmentation information, updated to RUNNING when the fragmentation job starts to be executed, updated to SUCCESS when the fragmentation job is successfully completed, and updated to FAIL when the fragmentation job is not successfully completed.
[0063] Furthermore, this can be referenced here. Figure 2 The update process of the sharding information table is explained in detail. Figure 2 An example diagram of the target fragment information table obtained after the job fragment execution is completed.
[0064] Depend on Figure 2 As can be seen, the sharding information table is the job_sharding_info table. Figure 2 The table shown is the target fragment information table when the job fragment execution is completed.
[0065] The target shard information table includes multiple job shard entries with job names IDs from id01 to id201. Each job shard entry maintains the following information: job name ID, job code, job batch (jobBatch0001), shard entry sequence number (-3, 0, 1, 2, ..., 198, 199), application code (LN-BATCH), shard level (0, 1), shard start time, shard completion time, shard status (SUCCESS), and exception information.
[0066] It should be noted that the task scheduling or execution device can also provide an interactive interface for business-side developers or other users to display the aforementioned sharding information table (target sharding information table), and provide filtering functions based on conditions such as job name ID, job code, job batch, sharding status, sharding start time, and sharding completion time. This allows users to understand the job execution process in detail from a sharding perspective, achieving transparency and visualization of the job execution process.
[0067] In addition, business-side developers can also extend the system's capabilities based on the aforementioned sharding information table: implement task management based on sharding dimensions (retry, skip, force success), which helps to achieve fine-grained control of tasks.
[0068] This embodiment dynamically generates the total number of job shards by executing user-defined sharding rules on the execution device, and generates corresponding sharding information based on this total number. This allows the task scheduling device to achieve flexible task sharding without pre-setting sharding strategies. This effectively overcomes the rigidity of traditional scheduling platform sharding strategies and significantly improves the adaptability of distributed task scheduling to different business scenarios. Furthermore, by refining the granularity of task execution control from the task level to the shard level, it helps reduce operational costs and resource waste, and improves processing efficiency.
[0069] Furthermore, to achieve the above objectives, this application also proposes a distributed task scheduling method, referring to... Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of the distributed task scheduling method of this application. In this embodiment, the method is applied to an execution device in a distributed scheduling platform system. The system also includes a task scheduling device electrically connected to the execution device. The method includes steps A10 to A30: Step A10: Upon receiving the job trigger instruction sent by the task scheduling device, perform a custom job sharding operation to obtain the total number of job shards. The job trigger instruction is generated by the task scheduling device when it obtains the custom sharding rules, which are pre-configured by the user through integrated platform components.
[0070] It should be noted that the execution entity in this embodiment can be a computing electronic device with data processing, network communication, and program execution functions, such as a desktop computer, mainframe computer, or server. It can also be other electronic devices capable of implementing task execution functions in a distributed scheduling platform system. This embodiment can be specifically described using an execution device as an example; this execution device can be considered as the business side of the system. Furthermore, in the system, there can be one or more execution devices; this embodiment can use a specific execution device in the system as the execution entity for illustrative purposes.
[0071] It should be understood that the execution device integrates a platform component, i.e., an SDK. Business-side developers can write interface class code based on this SDK, according to the business attributes of the task or the task sharding requirements. In particular, they can write the `shardinghandler()` strategy within the interface class to implement the standard interface provided by the SDK, thus creating a job with custom sharding rules. The custom sharding rules can be specific `shardinghandler()` strategies within the standard interface.
[0072] When the task scheduling device determines that the current job has a custom sharding rule, it can obtain the custom sharding rule and generate a job triggering instruction. When the execution device receives the job triggering instruction, it can enter the shardinghandler() strategy in the job policy, execute the shardinghandler() strategy, and obtain the total number of job shards.
[0073] Furthermore, to specifically illustrate the execution process of the shardinghandler() strategy by the execution device, step A10 specifically includes: steps A101~A103: Step A101: Upon receiving the job trigger instruction, the initial job sharding is executed. The initial job sharding is generated by the task scheduling device according to the custom sharding rules.
[0074] It should be noted that you can also refer to this section. Figure 4 The execution process of the initial job sharding, i.e., the shardinghandler() strategy processing process, is explained below. Figure 4 A diagram illustrating the execution process of custom fragmentation operations for a job.
[0075] It should be understood that the initial job shard is the job that executes the above-mentioned shardinghandler() strategy. It is generated by the task scheduling device when it obtains the custom sharding rules and maintains the corresponding sharding information in the sharding information table.
[0076] Step A102: Slice the application business data according to the custom sharding rules to obtain a data grouping table. The data grouping table is a mapping relationship between data group identifiers and job sharding sequence numbers. The data group identifier is used to identify different shards of business data obtained by sharding the application business data.
[0077] Step A103: Determine the total number of job fragments based on the data grouping table to complete the initial job fragmentation.
[0078] It should be noted that application business data refers to all business data of the tasks that the system currently needs to process, and it can be stored in the database server of the system, which is independent of the task scheduling device and the execution device.
[0079] Depend on Figure 4It can be seen that the execution device can shard application business data according to the shardinghandler() strategy, that is, split the application business data into different groups of sharded business data. Each group of sharded business data can correspond to a job shard, and different groups of sharded business data can be distinguished by data group identifiers, and different job shards can be distinguished by job shard sequence numbers, thereby establishing a mapping relationship between data group identifiers (such as batch number and group number) and job shard sequence numbers. This mapping relationship can be represented as a data grouping table, which is maintained by the execution device on the business side.
[0080] Since the job shard number can be generated based on a consecutive and unique numbering rule, the total number of job shards can be determined based on the job shard number in the data grouping table, thus completing the initial job sharding.
[0081] For example, such as Figure 4 The data grouping table shows that the total number of job fragments returned at this time is 2.
[0082] Step A20: Send the total number of job fragments to the task scheduling device so that the task scheduling device can generate fragment information based on the total number of job fragments.
[0083] It should be noted that the total number of job shards can be stored in a job context object. The job context is an in-memory data container created by the system for each job instance in a distributed architecture, used to achieve data transfer across policies and threads.
[0084] In a specific implementation, the execution device can store the total number of job fragments in the job context object and send it to the task scheduling device. This allows the task scheduling device to perform the actual custom fragmentation processing logic for the task after obtaining the total number of job fragments, generate the fragmentation information of the corresponding job fragments, and store it in the fragmentation information table.
[0085] Step A30: Obtain the sharding information, determine the sharding job data based on the sharding information, and execute job sharding based on the sharding job data.
[0086] It should be noted that the execution device can obtain the fragmentation information from the task scheduling device again to start executing the corresponding job fragment. Then, it can obtain the job fragmentation number from the fragmentation information and determine the data group identifier based on the job fragmentation number, thereby obtaining the corresponding fragmented job data from the database server. Finally, the job fragmentation is completed, and the fragmentation execution information is returned.
[0087] Furthermore, to illustrate in detail how to obtain sharded business data, step A30 specifically includes: steps A301~A303: Step A301: Determine the current job shard number based on the sharding information, and query the current business data group identifier corresponding to the current job shard number based on the data grouping table.
[0088] Step A302: Obtain the corresponding sharded business data from the database according to the current business data group identifier, and determine the sharded business data as sharded job data.
[0089] Step A303: Perform job sharding on the sharded job data according to the job sharding processing strategy, wherein the job sharding processing strategy is pre-configured by the user through the integrated platform components.
[0090] It should be noted that you can also refer to this section. Figure 5 The execution process of the following job fragmentation, i.e., the execution() strategy processing process, is explained. Figure 5 This is a schematic diagram illustrating the execution process of job segmentation.
[0091] Depend on Figure 5 It can be seen that when the execution device obtains the fragmentation information, it can obtain the job fragmentation number, i.e., the current job fragmentation number, and then query the current business data group identifier corresponding to the current job fragmentation number from the data grouping table maintained on the business side.
[0092] Next, the execution device can retrieve the corresponding fragmented business data from the database server based on the current business data group identifier.
[0093] Finally, the execution device can execute job sharding based on the sharded business data, that is, perform the corresponding execute() strategy, complete the job for the current job shard, and return the corresponding sharding execution information to the task scheduling device.
[0094] It should also be noted that since the number of execution devices can be less than the total number of job fragments, the same execution device needs to execute multiple different job fragments. That is, the execution device can repeatedly execute the above steps A301 to A303 until all job fragments have been executed, at which point the task is considered complete.
[0095] Furthermore, this can be referenced here. Figure 6 This document provides a complete explanation of the application process. Figure 6 This is a schematic diagram illustrating the interaction between task scheduling devices and execution devices in a distributed scheduling platform system.
[0096] exist Figure 6 In this context, the task scheduling device can be divided into a control engine (control-server) and a batch controller (worker-server); the execution device includes several agents, referred to as agent-server.
[0097] When a job (task) begins execution, and it is determined that the current job follows a custom sharding rule, the control engine can generate an initial job shard for that custom rule, assign a number to the initial sharded job, and store it in the sharding information table (job_sharding_info). At this point, the sharding information table only contains the job information for that initial sharded job.
[0098] Next, the control engine generates a job trigger instruction. After receiving the job trigger instruction, the batch controller sends it to the execution device, notifying the execution device to execute the initial job fragment.
[0099] After receiving the job trigger instruction, the execution device performs initial job sharding, i.e., executes the shardinghandler() strategy to group application business data and returns a total number of job shards.
[0100] The control engine can obtain the total number of job shards through the batch controller, and after verification, generate shard information for different job shards and store it in the aforementioned shard information table.
[0101] The execution device retrieves job information from the control engine again, obtains the corresponding sharded job data, and then executes the corresponding job sharding strategy, i.e., the execute() strategy, to obtain sharded execution information (exception information).
[0102] The execution device repeatedly performs the steps of obtaining job information and executing job segments until all job segments of the task within the system have been executed, at which point the task has been completed.
[0103] This embodiment achieves an effective combination of flexible customization of sharding rules on the business side and unified scheduling on the platform side by having the execution device respond to trigger commands to perform custom sharding operations and report the total number of shards. Compared with existing methods, since the task scheduling device does not need to pre-configure or understand specific sharding algorithms, it only needs to generate and schedule sharding information based on the total number of shards reported by the execution device, thereby enhancing the adaptability of the distributed scheduling platform system to different business scenarios.
[0104] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the distributed task scheduling method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.
[0105] This application also provides a task scheduling device, which includes: at least one first processor; and a first memory communicatively connected to the at least one first processor; wherein the first memory stores instructions executable by the at least one first processor, and the instructions are executed by the at least one first processor to enable the at least one first processor to execute the distributed task scheduling method in the first embodiment described above.
[0106] The following is for reference. Figure 7 , Figure 7 This is a schematic diagram of the task scheduling device of this application. The task scheduling device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The task scheduling device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0107] like Figure 7 As shown, the task scheduling device may include a first processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a first read-only memory (ROM) 1002 or a program loaded from a first storage device 1003 into a first random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the task scheduling device. The first processing unit 1001, the first ROM 1002, and the first RAM 1004 are interconnected via a first bus 1005. A first input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the first I / O interface 1006: a first input device 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a first storage device 1003 including, for example, magnetic tape, hard disk, etc.; and a first communication device 1009. The first communication device 1009 can allow the task scheduling device to communicate wirelessly or wiredly with other devices to exchange data.
[0108] The task scheduling device provided in this application employs the distributed task scheduling method in the first embodiment described above, which can solve the technical problems of distributed task scheduling methods. Compared with the prior art, the beneficial effects of the task scheduling device provided in this application are the same as those of the distributed task scheduling method provided in the first embodiment described above, and other technical features in the task scheduling device are the same as those disclosed in the method of the first embodiment, and will not be repeated here.
[0109] This application also provides an execution device, which includes: at least one second processor; and a second memory communicatively connected to the at least one second processor; wherein the second memory stores instructions executable by the at least one second processor, and the instructions are executed by the at least one second processor to enable the at least one second processor to execute the distributed task scheduling method in the second embodiment described above.
[0110] The following is for reference. Figure 8 , Figure 8 This is a schematic diagram of the structure of the execution device of this application. The execution device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The execution device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0111] like Figure 8As shown, the execution device may include a second processing unit 1001' (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a second read-only memory (ROM) 1002' or a program loaded from a second storage device 1003' into a second random access memory (RAM) 1004'. The RAM 1004' also stores various programs and data required for the operation of the execution device. The second processing unit 1001', the second ROM 1002', and the second RAM 1004' are interconnected via a second bus 1005'. A second input / output (I / O) interface 1006' is also connected to the bus. Typically, the following systems can be connected to the second I / O interface 1006': a second input device 1007' including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008' including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a second storage device 1003' including, for example, magnetic tape, hard disk, etc.; and a second communication device 1009'. The second communication device 1009' allows the device to communicate wirelessly or wiredly with other devices to exchange data.
[0112] The execution device provided in this application employs the distributed task scheduling method in the second embodiment described above, which can solve the technical problems of distributed task scheduling methods. Compared with the prior art, the beneficial effects of the execution device provided in this application are the same as those of the distributed task scheduling method provided in the second embodiment described above, and other technical features in the execution device are the same as those disclosed in the method of the second embodiment, and will not be repeated here.
[0113] Furthermore, this application also proposes a distributed scheduling platform system, which includes the task scheduling device and the execution device as described above.
[0114] The specific implementation of the distributed scheduling platform system in this embodiment can be referred to the descriptions of the above method embodiments, and will not be repeated in this embodiment.
[0115] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other elements in the process, method, article, or system that includes that element.
[0116] The sequence numbers of the above embodiments of the present invention are merely for description and do not represent the superiority or inferiority of the embodiments. They are only some embodiments of this application and are not intended to limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and based on the content of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.
Claims
1. A distributed task scheduling method, characterized in that, The method is applied to a task scheduling device in a distributed scheduling platform system, wherein the system further includes an execution device electrically connected to the task scheduling device, and the method includes: Obtain custom sharding rules and generate job triggering instructions based on the custom sharding rules, which are pre-configured by the user through platform components integrated on the execution device; The job triggering instruction is sent to the execution device, so that the execution device performs a custom job sharding operation in response to the job triggering instruction, and obtains the total number of job shards; Upon receiving the total number of job fragments sent by the execution device, fragment information is generated based on the total number of job fragments; The fragmentation information is sent to the execution device so that the execution device can determine the fragmented job data based on the fragmentation information and execute the job fragmentation according to the fragmented job data.
2. The method as described in claim 1, characterized in that, The step of obtaining custom sharding rules and generating job triggering instructions based on the custom sharding rules includes: The platform component obtains the custom sharding rules and generates the initial job shards based on the custom sharding rules. The initial job fragments are numbered according to the preset initial numbering rules, and the numbered initial job fragments are stored in the initialized fragment information table; Upon completion of storage, a job triggering instruction is generated based on the initial job fragment numbered after it.
3. The method as described in claim 2, characterized in that, The step of generating fragment information based on the total number of job fragments when receiving the total number of job fragments sent by the execution device includes: Upon receiving the total number of job fragments, a validity check is performed on the total number of job fragments. When the total number of job fragments passes the verification, fragment information is generated based on the total number of job fragments; After the step of generating shard information based on the total number of job shards, the method further includes: The initialized sharding information table is updated based on the sharding information to obtain a sharding information table with completed sharding. The sharding information table with completed sharding maintains several job sharding entries.
4. The method as described in claim 3, characterized in that, After the step of sending the fragment information to the execution device, the method further includes: When the segmented operation is completed, segmented execution information is obtained from the execution device; The sharding information table is updated based on the sharding execution information to obtain the target sharding information table.
5. A distributed task scheduling method, characterized in that, The method is applied to an execution device in a distributed scheduling platform system, wherein the system further includes a task scheduling device electrically connected to the execution device, and the method includes: Upon receiving a job triggering instruction from the task scheduling device, a custom job sharding operation is performed to obtain the total number of job shards. The job triggering instruction is generated by the task scheduling device when it obtains the custom sharding rules, which are pre-configured by the user through integrated platform components. The total number of job fragments is sent to the task scheduling device so that the task scheduling device can generate fragment information based on the total number of job fragments; Obtain the sharding information, determine the sharding job data based on the sharding information, and execute job sharding based on the sharding job data.
6. The method as described in claim 5, characterized in that, The step of performing a custom job sharding operation to obtain the total number of job shards upon receiving a job triggering instruction from the task scheduling device includes: Upon receiving the job trigger instruction, the initial job fragmentation is executed, wherein the initial job fragmentation is generated by the task scheduling device according to the custom fragmentation rules; The application business data is sharded according to the custom sharding rules to obtain a data grouping table. The data grouping table is a mapping relationship between data group identifiers and job sharding sequence numbers. The data group identifier is used to identify different shards of business data obtained by sharding the application business data. The total number of job fragments is determined based on the data grouping table, and the initial job fragmentation is completed.
7. The method as described in claim 6, characterized in that, The steps of obtaining the sharding information, determining sharded job data based on the sharding information, and executing job sharding based on the sharded job data include: The current job shard number is determined based on the sharding information, and the current business data group identifier corresponding to the current job shard number is queried according to the data grouping table; The corresponding sharded business data is retrieved from the database based on the current business data group identifier, and the sharded business data is identified as sharded job data. The job data is sharded according to a job sharding processing strategy, which is pre-configured by the user through the integrated platform components.
8. A task scheduling device, characterized in that, The device includes: a first memory, a first processor, and a distributed task scheduler stored in the first memory and executable on the first processor, the distributed task scheduler being configured to implement the steps of the distributed task scheduling method as described in any one of claims 1 to 4.
9. An execution device, characterized in that, The device includes: a second memory, a second processor, and a distributed task scheduler stored in the second memory and executable on the second processor, the distributed task scheduler being configured to implement the steps of the distributed task scheduling method as described in any one of claims 5 to 7.
10. A distributed scheduling platform system, characterized in that, The system includes the task scheduling device as described in claim 8 and the execution device as described in claim 9.