Distributed task scheduling method, device and equipment and readable storage medium

Abstract

The invention discloses a distributed task scheduling method, device and equipment and a readable storage medium, and relates to the technical field of databases. A task starting request is generated on a visual interface, a command object is converted and persisted through an application service, a master control node with a scheduling authority analyzes the command object to determine subtasks and a subtask dependency relationship, the subtasks are added into a priority queue, and the subtasks are distributed on the basis of a resource load state of an execution node. After the execution result is received, the sub-task state is updated, whether execution of the command object is completed or not is judged, and finally the execution result is fed back to the visual interface, so that full-process automatic scheduling and state closed-loop feedback from user operation to task execution completion are achieved.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a distributed task scheduling method, apparatus, device, and readable storage medium. Background Technology

[0002] In big data processing and offline modeling scenarios, methods can be highly complex, lacking a visual interface and relying on low-level code development and formula calculations, resulting in a steep learning curve for new users. Furthermore, in actual high-availability deployments, multiple master nodes are often deployed to form a redundant cluster to avoid single points of failure. In this case, without an effective coordination mechanism, multiple master nodes may simultaneously scan the database for pending task requests and repeatedly parse and distribute them, leading to the same business process being executed multiple times, causing data duplication, resource waste, and even business logic errors. Therefore, there is an urgent need for a distributed task scheduling method, device, equipment, and readable storage medium to address the problems of user learning difficulties and inflexible task scheduling. Summary of the Invention

[0003] To address the inflexibility of user operation and runtime scheduling in big data processing and offline modeling scenarios, this application provides the following technical solution: Firstly, a distributed task scheduling method is provided, including: Based on the user's drag-and-drop instructions, a task start request is generated in the visual interface.

[0004] The application service converts task start requests into command objects and persists these command objects to the database.

[0005] The master node with scheduling authority parses the command object obtained from the database, determines the subtasks and subtask dependencies corresponding to the command object, adds the subtasks to the priority queue, and persists the subtasks and subtask dependencies to the database.

[0006] The master node extracts subtasks from the priority queue, determines the execution node for the subtasks based on the resource load status of the execution nodes, and distributes the subtasks to the execution nodes.

[0007] It receives the execution results of the subtasks from the execution node to update the status of the subtasks stored in the database, and determines whether the command object has been executed successfully based on the subtask dependencies and the updated status of the subtasks.

[0008] In response to the completion of the command object execution, the result of the command object execution is fed back to the visual interface.

[0009] Furthermore, the task initiation request is converted into a command object through the application service, and the command object is persisted in the database, including: The application service encapsulates the startup request to obtain a command object.

[0010] The application service performs validation on the command object, and the validation content includes at least one of the following: field validation, process identifier validation, and parameter validity validation.

[0011] The application service sets system metadata for the verified command object, which includes at least the following: creation time, request source IP address, and user ID.

[0012] The application service will save the verified command object and system metadata to the database to achieve persistence and return a persistence success status.

[0013] Furthermore, the master node with scheduling authority parses the command object obtained from the database, determines the subtasks corresponding to the command object and their dependencies, adds the subtasks to a priority queue, and persists the subtasks and their dependencies to the database, including: The master node with scheduling privileges scans the database to obtain command objects.

[0014] The master node requests a distributed lock from the distributed coordination service.

[0015] In response to the master node acquiring the distributed lock, the command object is parsed to determine the subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies.

[0016] The master node persists the directed acyclic graph to the database and adds the subtasks to the priority queue.

[0017] Furthermore, in response to the master node acquiring the distributed lock, the command object is parsed to determine the subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies, including: The master node extracts the list of command nodes and the connection relationships between command nodes from the command object.

[0018] The master node performs topology verification on the connection relationships of the command nodes.

[0019] If the result of the topology check is that the command object conforms to the directed acyclic structure, then the command node is treated as a subtask, and the connection relationship between the command node is treated as the subtask dependency relationship to construct a directed acyclic graph.

[0020] Furthermore, the master node persists the directed acyclic graph to the database and adds subtasks to a priority queue, including: The master node iterates through the subtasks generated based on the command object, and does not initialize the state of any subtask.

[0021] The master node stores the directed acyclic graph constructed from subtasks and their dependencies in the database.

[0022] Based on the subtask status initialization result, add the subtasks with the status of ready to the priority queue.

[0023] Furthermore, the master node extracts subtasks from the priority queue, determines the execution node for the subtasks based on the resource load status of the execution nodes, and distributes the subtasks to the execution nodes, including: The master node extracts subtasks from the priority queue.

[0024] The load balancer obtains the load information of any execution node registered in the distributed coordination service.

[0025] Select the execution node for the subtask based on the load conditions.

[0026] Distribute subtasks to execution nodes.

[0027] Furthermore, the execution results are obtained in the following ways: The execution node receives and parses the subtasks to determine their task types.

[0028] The execution node determines the component that executes the subtask based on the task type of the subtask.

[0029] The execution node sets up the runtime environment for the execution components.

[0030] The execution node starts the execution component to process the subtask and records the processing of the subtask by the execution component in the log file.

[0031] Get the execution result of the subtask.

[0032] Secondly, a distributed task scheduling device is provided, comprising: The request generation module is used to generate task start requests in the visual interface based on user drag-and-drop instructions.

[0033] The request conversion module is used to convert task start requests into command objects through application services and persist the command objects to the database.

[0034] The command parsing module is used to parse command objects obtained from the database through the master control node with scheduling privileges, determine the subtasks and subtask dependencies corresponding to the command objects, add the subtasks to the priority queue, and persist the subtasks and subtask dependencies to the database.

[0035] The task distribution module is used to extract subtasks from the priority queue through the master node, determine the execution node to execute the subtasks based on the resource load status of the execution nodes, and distribute the subtasks to the execution nodes.

[0036] The command management module is used to receive the execution results of subtasks from the execution nodes, update the status of subtasks stored in the database, and determine whether the command object has been executed successfully based on the subtask dependencies and the updated status of the subtasks.

[0037] The result feedback module is used to provide feedback on the result of the command object's execution to the visual interface when the command object has been executed.

[0038] Thirdly, a computer device is provided, including a memory, a processor, and a distributed task scheduling computer program stored in the memory and executable on the processor. When the processor executes the distributed task scheduling computer program, it implements the distributed task scheduling method described in the first aspect.

[0039] Fourthly, a computer-readable storage medium is provided, on which a distributed task scheduling computer program is stored, wherein when the distributed task scheduling computer program is executed by a processor, the distributed task scheduling method described in the first aspect is implemented.

[0040] Fifthly, a computer program product is provided, including a computer program that, when executed by a processor, implements the distributed task scheduling method described in the first aspect.

[0041] The beneficial effects of the technical solution provided in this application are as follows: by generating a task start request in the visual interface, converting and persisting the command object through the application service, the master control node with scheduling authority parses the command object to determine the subtasks and subtask dependencies, adding the subtasks to the priority queue, distributing the subtasks based on the resource load status of the execution node, updating the subtask status after receiving the execution result and determining whether the command object has been executed, and finally feeding back the execution result to the visual interface, the fully automated scheduling and closed-loop status feedback from user operation to task execution completion are realized. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of a distributed task scheduling method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a distributed task scheduling device provided in an embodiment of this application; Figure 3 This is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0045] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The numbers in the accompanying drawings are only used to distinguish individual functional parts or modules and do not indicate logical relationships between parts or modules. The terms “comprising,” “including,” or “including,” and similar terms mean that the element or object preceding the term encompasses the element or object listed following the term and its equivalents, without excluding other elements or objects. The terms “connected,” “linked,” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. “Above,” “below,” “left,” “right,” etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0046] The various embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that in the drawings, the same reference numerals are assigned to components having substantially the same or similar structure and function, and repeated descriptions of them will be omitted.

[0047] To address the inflexibility of user operation and runtime scheduling in big data processing and offline modeling scenarios, this application provides the following technical solution: Firstly, a distributed task scheduling method is provided, including: S100: Generates a task start request in the visual interface based on the user's drag-and-drop instructions.

[0048] This is achieved by the user dragging and dropping command modules in the GUI interface. Each module is bound to a processing component. The processing component must include at least one of the following: Sqoop, MapReduce, shell, SQL, Python, Flink, Spark MLlib, or Flume.

[0049] To illustrate, users can drag and drop Sqoop collection nodes in the visual interface to collect data; add Spark cleaning nodes to perform data cleaning and preprocessing; configure Spark MLlib training nodes to perform data modeling and training; and add model deployment nodes to deploy and apply models.

[0050] Taking offline data modeling as an example: the data modeling formula can be summarized as: Model = Determine entities + Determine attributes + Determine relationships + Normalization + Naming conventions + Constraint settings + Performance considerations.

[0051] The entity identification method identifies entities in the system, with each entity corresponding to a database table.

[0052] Determining attributes means identifying the attributes of each entity, with each attribute corresponding to a field in the table.

[0053] Defining a relationship means defining the relationship between entities, including one-to-one, one-to-many, and many-to-many relationships.

[0054] Database normalization specifications refer to organizing data into a more rational form according to database normalization specifications in order to reduce data redundancy and improve data consistency.

[0055] Naming conventions mean using meaningful, descriptive names and following naming guidelines.

[0056] Constraint settings represent setting constraints based on business needs to ensure data integrity and consistency.

[0057] Performance considerations: Consider the performance requirements of the database and optimize performance by adding appropriate indexes and shards.

[0058] Configure scheduled tasks to extract offline data from different data sources and store it in the corresponding data warehouse. The application can integrate data acquisition components including Apache Sqoop, Flume, and various JDBC connection methods.

[0059] By integrating multiple data processing engines such as Spark, MapReduce, Tez, and Olap, and setting up scheduled tasks, the raw data can be cleaned, deduplicated, and missing value handled at the required frequency.

[0060] Schedule machine learning or statistical modeling tasks by setting up timed tasks, such as integrating Apache SparkMLlib or Scikit-learn for model training. Select the appropriate modeling algorithm based on business needs.

[0061] By setting up scheduled tasks, data models can be automatically deployed and updated. Trained and optimized data models can be deployed to the production environment, and scheduled tasks can be set up for real-time data prediction or analysis.

[0062] S200: Convert the task initiation request into a command object and persist the command object to the database.

[0063] The transformation of task initiation requests is achieved through application services.

[0064] S300: Parse the command object obtained from the database, determine the subtasks and subtask dependencies corresponding to the command object, add the subtasks to the priority queue, and persist the subtasks and subtask dependencies to the database.

[0065] Parsing command objects retrieved from the database is performed by the master node with scheduling privileges.

[0066] S400: Extract subtasks from the priority queue, determine the execution node for the subtasks based on the resource load status of the execution nodes, and distribute the subtasks to the execution nodes.

[0067] Extracting subtasks from the priority queue is done through the master node.

[0068] S500: Receives the execution results of the subtasks from the execution node to update the status of the subtasks stored in the database, and determines whether the command object has been executed successfully based on the subtask dependencies and the updated subtask status.

[0069] Receiving the execution results of subtasks from the execution nodes is done through the master node.

[0070] S600: In response to the completion of command object execution, the result of command object execution is fed back to the visual interface.

[0071] By generating a task start request through a visual interface, converting and persisting the command object via an application service, and having the master node with scheduling authority parse the command object to determine the subtasks and their dependencies, the subtasks are added to a priority queue. The subtasks are then distributed based on the resource load status of the execution node. After receiving the execution result, the subtask status is updated and it is determined whether the command object has been executed. Finally, the execution result is fed back to the visual interface, thus realizing fully automated scheduling and closed-loop status feedback from user operation to task completion.

[0072] Specifically, S200: Converts the task initiation request into a command object through the application service and persists the command object to the database, including: S210: The application service encapsulates the startup request to obtain a command object.

[0073] S220: The application service performs validation on the command object, wherein the validation content includes at least one of the following: field validation, process identifier validation, and parameter validity validation.

[0074] S230: The application service sets system metadata for the verified command object, wherein the system metadata includes at least the following: creation time, request source IP address, and user ID.

[0075] S240: The application service will save the verified command object and system metadata to the database to achieve persistence, and return a persistence success status.

[0076] By encapsulating the task initiation request to obtain a command object, and performing field validation, process identifier validation, and / or parameter validity validation on the command object, system metadata such as creation time, request source IP address, and / or user ID are set for the command object that passes the validation. Then, the command object and system metadata are persisted to the database, thereby ensuring the legality, integrity, and traceability of the command object and preventing illegal or missing data from entering the scheduling process.

[0077] Specifically, S300: The master control node with scheduling authority parses the command object obtained from the database, determines the subtasks corresponding to the command object and their dependencies, adds the subtasks to a priority queue, and persists the subtasks and their dependencies to the database, including: S310: The master node with scheduling authority scans the database to obtain command objects.

[0078] S320: The master node requests a distributed lock from the distributed coordination service.

[0079] S330: In response to the master node acquiring the distributed lock, the command object is parsed to determine the subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies.

[0080] S340: The master node persists the directed acyclic graph to the database and adds the subtasks to the priority queue.

[0081] Optionally, the distributed coordination service uses ZooKeeper. It is used to obtain registration information for the master node and execution nodes. By registering with ZooKeeper, the master node can acquire a distributed lock through contention, ensuring mutual exclusion in task execution. Execution nodes, by registering with ZooKeeper, can have their performance monitored by the master node, and subtasks can be allocated based on this performance status.

[0082] After the master node with scheduling authority scans the database to obtain the command object, it requests a distributed lock from the distributed coordination service. After successfully acquiring the distributed lock, it parses the command object to determine the subtasks and their dependencies, constructs a directed acyclic graph (DAG), persists the DAG to the database, and adds the subtasks to the priority queue. This avoids multiple master nodes processing the same command object repeatedly, ensuring scheduling consistency under a high-availability deployment.

[0083] Specifically, S330: In response to the master node acquiring the distributed lock, the command object is parsed to determine the subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies, including: S331: The master node extracts the list of command nodes and the connection relationships between command nodes from the command object.

[0084] S332: The master node performs topology verification on the connection relationship of the command node.

[0085] S333: In response to the result of topology verification that the command object conforms to a directed acyclic structure, the command node is treated as a subtask, and the connection relationship between the command node is treated as the subtask dependency relationship to construct a directed acyclic graph.

[0086] Extract the list of command nodes and their connections from the command object, perform topological verification on the connections, and after confirming that they conform to a directed acyclic structure, construct a directed acyclic graph by treating the command nodes as subtasks and their connections as subtask dependencies. This ensures that the scheduled process structure is valid and prevents tasks with cycles or invalid dependencies from entering the execution phase.

[0087] Specifically, S340: The master node persists the directed acyclic graph to the database and adds subtasks to a priority queue, including: S341: The master node traverses the subtasks generated based on the command object and initializes the state of any subtask.

[0088] S342: The master node stores the directed acyclic graph constructed from subtasks and their dependencies in the database.

[0089] S343: Based on the subtask status initialization result, add the subtasks with the status of ready to the priority queue.

[0090] Traverse subtasks and initialize their states. Save the directed acyclic graph constructed from the subtasks and their dependencies in the database. Based on the state initialization results, only add subtasks in the ready state to the priority queue. This ensures that subtasks are enqueued in the order of their dependencies, avoiding execution failures due to incomplete preceding tasks.

[0091] Specifically, S400: Extracts subtasks from the priority queue via the master node, determines the execution node for the subtasks based on the resource load status of the execution nodes, and distributes the subtasks to the execution nodes, including: S410: The master node extracts subtasks from the priority queue.

[0092] S420: Obtain the load status of any execution node registered in the distributed coordination service through the load balancer.

[0093] S430: Select the execution node for the subtask based on the load.

[0094] S440: Distribute subtasks to execution nodes.

[0095] The master node extracts subtasks from the priority queue, obtains the load status of the execution nodes registered in the distributed coordination service through the load balancer, selects the target execution node for the subtasks, and distributes the subtasks to the execution node, thereby realizing task allocation based on real-time resource load and improving cluster resource utilization and task execution efficiency.

[0096] Specifically, the execution result is obtained in the following way: S511: The execution node receives and parses the subtask to determine the task type of the subtask.

[0097] S512: The execution node determines the component to execute the subtask based on the task type of the subtask.

[0098] S513: Execution node sets the runtime environment for execution components.

[0099] S514: The execution node starts the execution component to process the subtask and records the processing of the subtask by the execution component in the log file.

[0100] S515: Obtain the execution result of the subtask.

[0101] The execution node receives and parses subtasks to determine the task type, determines the execution component based on the task type, sets the runtime environment of the execution component, starts the execution component to process the subtask and records the processing process in a log file, and finally obtains the execution result, thereby supporting the adaptive execution of multiple types of subtasks and realizing the standardization of the execution process and the traceability of logs.

[0102] The beneficial effects of the technical solution provided in this application are as follows: by generating a task start request in the visual interface, converting and persisting the command object through the application service, the master control node with scheduling authority parses the command object to determine the subtasks and subtask dependencies, adding the subtasks to the priority queue, distributing the subtasks based on the resource load status of the execution node, updating the subtask status after receiving the execution result and determining whether the command object has been executed, and finally feeding back the execution result to the visual interface, the fully automated scheduling and closed-loop status feedback from user operation to task execution completion are realized. Distributed parallel processing, through its scheduling engine, can simultaneously distribute multiple independent subtasks from a large task DAG to multiple nodes in the cluster for execution, achieving "multiple tasks running concurrently." This significantly shortens the overall task completion time and improves throughput. Through resource pooling and optimization, the scheduler provides a global view of the entire computing cluster's resources (CPU, memory), enabling intelligent resource allocation and load balancing. This prevents some machines from being idle while others are overloaded, thus fully maximizing the value of distributed computing resources and reducing computational costs. Furthermore, data processing components (such as those for data synchronization, sorting, and aggregation) are typically highly optimized at the underlying level for specific scenarios, resulting in execution efficiency far exceeding that of user-written general-purpose code, thus improving performance.

[0103] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Furthermore, Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0104] In other embodiments, such as Figure 2 As shown, a distributed task scheduling device includes: The request generation module is used to generate task start requests in the visual interface based on user drag-and-drop instructions.

[0105] The request conversion module is used to convert task start requests into command objects through application services and persist the command objects to the database.

[0106] The command parsing module is used to parse command objects obtained from the database through the master control node with scheduling privileges, determine the subtasks and subtask dependencies corresponding to the command objects, add the subtasks to the priority queue, and persist the subtasks and subtask dependencies to the database.

[0107] The task distribution module is used to extract subtasks from the priority queue through the master node, determine the execution node to execute the subtasks based on the resource load status of the execution nodes, and distribute the subtasks to the execution nodes.

[0108] The command management module is used to receive the execution results of subtasks from the execution nodes, update the status of subtasks stored in the database, and determine whether the command object has been executed successfully based on the subtask dependencies and the updated status of the subtasks.

[0109] The result feedback module is used to provide feedback on the result of the command object's execution to the visual interface when the command object has been executed.

[0110] For specific limitations regarding the distributed task scheduling device described above, please refer to the limitations of the distributed task scheduling method above, which will not be repeated here. Each module in the aforementioned distributed task scheduling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0111] In other embodiments, such as Figure 3 As shown, a computer device includes a memory, a processor, and a distributed task scheduling computer program stored in the memory and executable on the processor. When the processor executes the distributed task scheduling computer program, it implements the distributed task scheduling method described in the first aspect. Specifically, it includes: S100: Generates a task start request in the visual interface based on the user's drag-and-drop instructions.

[0112] S200: The application service converts the task start request into a command object and persists the command object to the database.

[0113] S300: The master control node with scheduling authority parses the command object obtained from the database, determines the subtasks and subtask dependencies corresponding to the command object, adds the subtasks to the priority queue, and persists the subtasks and subtask dependencies to the database.

[0114] S400: Extracts subtasks from the priority queue through the master node, determines the execution node to execute the subtask based on the resource load status of the execution node, and distributes the subtask to the execution node.

[0115] S500: Receives the execution results of the subtasks from the execution node to update the status of the subtasks stored in the database, and determines whether the command object has been executed successfully based on the subtask dependencies and the updated subtask status.

[0116] S600: In response to the completion of command object execution, the result of command object execution is fed back to the visual interface.

[0117] Specifically, S200: Converts the task initiation request into a command object through the application service and persists the command object to the database, including: S210: The application service encapsulates the startup request to obtain a command object.

[0118] S220: The application service performs validation on the command object, wherein the validation content includes at least one of the following: field validation, process identifier validation, and parameter validity validation.

[0119] S230: The application service sets system metadata for the verified command object, wherein the system metadata includes at least the following: creation time, request source IP address, and user ID.

[0120] S240: The application service will save the verified command object and system metadata to the database to achieve persistence, and return a persistence success status.

[0121] Specifically, S300: The master control node with scheduling authority parses the command object obtained from the database, determines the subtasks corresponding to the command object and their dependencies, adds the subtasks to a priority queue, and persists the subtasks and their dependencies to the database, including: S310: The master node with scheduling authority scans the database to obtain command objects.

[0122] S320: The master node requests a distributed lock from the distributed coordination service.

[0123] S330: In response to the master node acquiring the distributed lock, the command object is parsed to determine the subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies.

[0124] S340: The master node persists the directed acyclic graph to the database and adds the subtasks to the priority queue.

[0125] Specifically, S330: In response to the master node acquiring the distributed lock, the command object is parsed to determine the subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies, including: S331: The master node extracts the list of command nodes and the connection relationships between command nodes from the command object.

[0126] S332: The master node performs topology verification on the connection relationship of the command node.

[0127] S333: In response to the result of topology verification that the command object conforms to a directed acyclic structure, the command node is treated as a subtask, and the connection relationship between the command node is treated as the subtask dependency relationship to construct a directed acyclic graph.

[0128] Specifically, S340: The master node persists the directed acyclic graph to the database and adds subtasks to a priority queue, including: S341: The master node traverses the subtasks generated based on the command object and initializes the state of any subtask.

[0129] S342: The master node stores the directed acyclic graph constructed from subtasks and their dependencies in the database.

[0130] S343: Based on the subtask status initialization result, add the subtasks with the status of ready to the priority queue.

[0131] Specifically, S400: Extracts subtasks from the priority queue via the master node, determines the execution node for the subtasks based on the resource load status of the execution nodes, and distributes the subtasks to the execution nodes, including: S410: The master node extracts subtasks from the priority queue.

[0132] S420: Obtain the load status of any execution node registered in the distributed coordination service through the load balancer.

[0133] S430: Select the execution node for the subtask based on the load.

[0134] S440: Distribute subtasks to execution nodes.

[0135] Specifically, the execution result is obtained in the following way: S511: The execution node receives and parses the subtask to determine the task type of the subtask.

[0136] S512: The execution node determines the component to execute the subtask based on the task type of the subtask.

[0137] S513: Execution node sets the runtime environment for execution components.

[0138] S514: The execution node starts the execution component to process the subtask and records the processing of the subtask by the execution component in the log file.

[0139] S515: Obtain the execution result of the subtask.

[0140] In other embodiments, a computer-readable storage medium stores a distributed task scheduling computer program thereon, which, when executed by a processor, implements the distributed task scheduling method described above. Specifically, it includes: S100: Generates a task start request in the visual interface based on the user's drag-and-drop instructions.

[0141] S200: The application service converts the task start request into a command object and persists the command object to the database.

[0142] S300: The master control node with scheduling authority parses the command object obtained from the database, determines the subtasks and subtask dependencies corresponding to the command object, adds the subtasks to the priority queue, and persists the subtasks and subtask dependencies to the database.

[0143] S400: Extracts subtasks from the priority queue through the master node, determines the execution node to execute the subtask based on the resource load status of the execution node, and distributes the subtask to the execution node.

[0144] S500: Receives the execution results of the subtasks from the execution node to update the status of the subtasks stored in the database, and determines whether the command object has been executed successfully based on the subtask dependencies and the updated subtask status.

[0145] S600: In response to the completion of command object execution, the result of command object execution is fed back to the visual interface.

[0146] Specifically, S200: Converts the task initiation request into a command object through the application service and persists the command object to the database, including: S210: The application service encapsulates the startup request to obtain a command object.

[0147] S220: The application service performs validation on the command object, wherein the validation content includes at least one of the following: field validation, process identifier validation, and parameter validity validation.

[0148] S230: The application service sets system metadata for the verified command object, wherein the system metadata includes at least the following: creation time, request source IP address, and user ID.

[0149] S240: The application service will save the verified command object and system metadata to the database to achieve persistence, and return a persistence success status.

[0150] Specifically, S300: The master control node with scheduling authority parses the command object obtained from the database, determines the subtasks corresponding to the command object and their dependencies, adds the subtasks to a priority queue, and persists the subtasks and their dependencies to the database, including: S310: The master node with scheduling authority scans the database to obtain command objects.

[0151] S320: The master node requests a distributed lock from the distributed coordination service.

[0152] S330: In response to the master node acquiring the distributed lock, the command object is parsed to determine the subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies.

[0153] S340: The master node persists the directed acyclic graph to the database and adds the subtasks to the priority queue.

[0154] Specifically, S330: In response to the master node acquiring the distributed lock, the command object is parsed to determine the subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies, including: S331: The master node extracts the list of command nodes and the connection relationships between command nodes from the command object.

[0155] S332: The master node performs topology verification on the connection relationship of the command node.

[0156] S333: In response to the result of topology verification that the command object conforms to a directed acyclic structure, the command node is treated as a subtask, and the connection relationship between the command node is treated as the subtask dependency relationship to construct a directed acyclic graph.

[0157] Specifically, S340: The master node persists the directed acyclic graph to the database and adds subtasks to a priority queue, including: S341: The master node traverses the subtasks generated based on the command object and initializes the state of any subtask.

[0158] S342: The master node stores the directed acyclic graph constructed from subtasks and their dependencies in the database.

[0159] S343: Based on the subtask status initialization result, add the subtasks with the status of ready to the priority queue.

[0160] Specifically, S400: Extracts subtasks from the priority queue via the master node, determines the execution node for the subtasks based on the resource load status of the execution nodes, and distributes the subtasks to the execution nodes, including: S410: The master node extracts subtasks from the priority queue.

[0161] S420: Obtain the load status of any execution node registered in the distributed coordination service through the load balancer.

[0162] S430: Select the execution node for the subtask based on the load.

[0163] S440: Distribute subtasks to execution nodes.

[0164] Specifically, the execution result is obtained in the following way: S511: The execution node receives and parses the subtask to determine the task type of the subtask.

[0165] S512: The execution node determines the component to execute the subtask based on the task type of the subtask.

[0166] S513: Execution node sets the runtime environment for execution components.

[0167] S514: The execution node starts the execution component to process the subtask and records the processing of the subtask by the execution component in the log file.

[0168] S515: Obtain the execution result of the subtask.

[0169] In other embodiments, a computer program product includes a computer program that, when executed by a processor, implements the distributed task scheduling method described above. Specifically, it includes: S100: Generates a task start request in the visual interface based on the user's drag-and-drop instructions.

[0170] S200: The application service converts the task start request into a command object and persists the command object to the database.

[0171] S300: The master control node with scheduling authority parses the command object obtained from the database, determines the subtasks and subtask dependencies corresponding to the command object, adds the subtasks to the priority queue, and persists the subtasks and subtask dependencies to the database.

[0172] S400: Extracts subtasks from the priority queue through the master node, determines the execution node to execute the subtask based on the resource load status of the execution node, and distributes the subtask to the execution node.

[0173] S500: Receives the execution results of the subtasks from the execution node to update the status of the subtasks stored in the database, and determines whether the command object has been executed successfully based on the subtask dependencies and the updated subtask status.

[0174] S600: In response to the completion of command object execution, the result of command object execution is fed back to the visual interface.

[0175] Specifically, S200: Converts the task initiation request into a command object through the application service and persists the command object to the database, including: S210: The application service encapsulates the startup request to obtain a command object.

[0176] S220: The application service performs validation on the command object, wherein the validation content includes at least one of the following: field validation, process identifier validation, and parameter validity validation.

[0177] S230: The application service sets system metadata for the verified command object, wherein the system metadata includes at least the following: creation time, request source IP address, and user ID.

[0178] S240: The application service will save the verified command object and system metadata to the database to achieve persistence, and return a persistence success status.

[0179] Specifically, S300: The master control node with scheduling authority parses the command object obtained from the database, determines the subtasks corresponding to the command object and their dependencies, adds the subtasks to a priority queue, and persists the subtasks and their dependencies to the database, including: S310: The master node with scheduling authority scans the database to obtain command objects.

[0180] S320: The master node requests a distributed lock from the distributed coordination service.

[0181] S330: In response to the master node acquiring the distributed lock, the command object is parsed to determine the subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies.

[0182] S340: The master node persists the directed acyclic graph to the database and adds the subtasks to the priority queue.

[0183] Specifically, S330: In response to the master node acquiring the distributed lock, the command object is parsed to determine the subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies, including: S331: The master node extracts the list of command nodes and the connection relationships between command nodes from the command object.

[0184] S332: The master node performs topology verification on the connection relationship of the command node.

[0185] S333: In response to the result of topology verification that the command object conforms to a directed acyclic structure, the command node is treated as a subtask, and the connection relationship between the command node is treated as the subtask dependency relationship to construct a directed acyclic graph.

[0186] Specifically, S340: The master node persists the directed acyclic graph to the database and adds subtasks to a priority queue, including: S341: The master node traverses the subtasks generated based on the command object and initializes the state of any subtask.

[0187] S342: The master node stores the directed acyclic graph constructed from subtasks and their dependencies in the database.

[0188] S343: Based on the subtask status initialization result, add the subtasks with the status of ready to the priority queue.

[0189] Specifically, S400: Extracts subtasks from the priority queue via the master node, determines the execution node for the subtasks based on the resource load status of the execution nodes, and distributes the subtasks to the execution nodes, including: S410: The master node extracts subtasks from the priority queue.

[0190] S420: Obtain the load status of any execution node registered in the distributed coordination service through the load balancer.

[0191] S430: Select the execution node for the subtask based on the load.

[0192] S440: Distribute subtasks to execution nodes.

[0193] Specifically, the execution result is obtained in the following way: S511: The execution node receives and parses the subtask to determine the task type of the subtask.

[0194] S512: The execution node determines the component to execute the subtask based on the task type of the subtask.

[0195] S513: Execution node sets the runtime environment for execution components.

[0196] S514: The execution node starts the execution component to process the subtask and records the processing of the subtask by the execution component in the log file.

[0197] S515: Obtain the execution result of the subtask.

[0198] The beneficial effects of the technical solution provided in this application are as follows: By generating a task start request in the visual interface, converting and persisting the command object through the application service, the master control node with scheduling authority parses the command object to determine the subtasks and their dependencies, adds the subtasks to a priority queue, distributes the subtasks based on the resource load status of the execution node, updates the subtask status after receiving the execution result, determines whether the command object has been completed, and finally feeds back the execution result to the visual interface, thereby realizing fully automated scheduling and closed-loop status feedback from user operation to task completion. All the above optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0199] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program loaded on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from memory, or installed from ROM. When the computer program is executed by an external processor, it performs the functions defined in the methods of embodiments of this application.

[0200] It should be noted that the computer-readable medium in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0201] The aforementioned computer-readable medium may be included in the aforementioned server; or it may exist independently and not assembled into the server. The aforementioned computer-readable medium carries one or more programs that, when executed by the server, cause the server to: in response to detecting that the peripheral mode of the terminal is not activated, acquire the frame rate of the application on the terminal; when the frame rate meets the screen-off condition, determine whether the user is acquiring the terminal's screen information; and in response to the determination that the user is not acquiring the terminal's screen information, control the screen to enter an immediate dimming mode.

[0202] Computer program code for performing the operations of the embodiments of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0203] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0204] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

[0205] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A distributed task scheduling method, characterized in that, include: Based on the user's drag-and-drop instructions, a task start request is generated in the visual interface; The application service converts the task start request into a command object and persists the command object to the database. The master control node with scheduling authority parses the command object obtained from the database, determines the subtasks and subtask dependencies corresponding to the command object, adds the subtasks to the priority queue, and persists the subtasks and subtask dependencies to the database. The master node extracts subtasks from the priority queue, determines the execution node to execute the subtasks based on the resource load status of the execution nodes, and distributes the subtasks to the execution nodes. Receive the execution result of the subtask by the execution node to update the status of the subtask stored in the database, and determine whether the command object has been executed completely based on the subtask dependency relationship and the updated subtask status; In response to the completion of the command object execution, the result of the command object execution is fed back to the visualization interface.

2. The distributed task scheduling method according to claim 1, characterized in that, The step of converting the task start request into a command object through an application service and persisting the command object in the database includes: The application service encapsulates the startup request to obtain the command object; The application service performs validation on the command object, wherein the validation content includes at least one of the following: field validation, process identifier validation, and parameter validity validation; The application service sets system metadata for verified command objects, wherein the system metadata includes at least the following: creation time, request source IP address, and user ID; The application service will save the verified command object and the system metadata to the database to achieve persistence, and return a persistence success status.

3. The distributed task scheduling method according to claim 1, characterized in that, The step of parsing the command object obtained from the database through a master control node with scheduling authority, determining the subtasks and subtask dependencies corresponding to the command object, adding the subtasks to a priority queue, and persisting the subtasks and subtask dependencies to the database includes: The master control node with scheduling authority scans the database to obtain the command object; The master node requests a distributed lock from the distributed coordination service; In response to the master node acquiring the distributed lock, the command object is parsed to determine the subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies. The master node persists the directed acyclic graph to the database and adds the subtask to the priority queue.

4. The distributed task scheduling method according to claim 3, characterized in that, In response to the master node acquiring the distributed lock, the command object is parsed to determine subtasks and their dependencies, and a corresponding directed acyclic graph is created based on the subtasks and their dependencies, including: The master node extracts the list of command nodes and the connection relationship of command nodes from the command object; The master control node performs topology verification on the connection relationship of the command node; If the result of the topology check is that the command object conforms to a directed acyclic structure, then the command node is used as the subtask, and the connection relationship of the command node is used as the subtask dependency relationship to construct the directed acyclic graph.

5. The distributed task scheduling method according to claim 3, characterized in that, The master node persists the directed acyclic graph to the database and adds the subtasks to a priority queue, including: The master node traverses the subtasks generated based on the command object and initializes the state of any subtask. The master node stores the directed acyclic graph constructed from subtasks and their dependencies in the database; Based on the subtask status initialization result, add the subtasks with the status of ready to the priority queue.

6. The distributed task scheduling method according to claim 1, characterized in that, The step of retrieving subtasks from the priority queue via the master node, determining the execution node to execute the subtasks based on the resource load status of the execution nodes, and distributing the subtasks to the execution nodes includes: The master node extracts subtasks from the priority queue; The load balancer obtains the load status of any execution node registered in the distributed coordination service. Based on the load conditions, select the execution node to execute the subtask; The subtasks are distributed to the execution nodes.

7. The distributed task scheduling method according to claim 1, characterized in that, The execution result is obtained in the following way: The execution node receives and parses the subtask to determine the task type of the subtask; The execution node determines the component that executes the subtask based on the task type of the subtask; The execution node sets the runtime environment for the execution component; The execution node initiates the execution component to process the subtask and records the processing of the subtask by the execution component in a log file; Obtain the execution result of the subtask.

8. A distributed task scheduling device, characterized in that, include: The request generation module is used to generate task start requests in the visual interface based on user drag-and-drop instructions; The request conversion module is used to convert the task start request into a command object through the application service, and persist the command object to the database; The command parsing module is used to parse the command object obtained from the database through the master control node with scheduling authority, determine the subtasks and subtask dependencies corresponding to the command object, add the subtasks to the priority queue, and persist the subtasks and subtask dependencies to the database. The task distribution module is used to extract subtasks from the priority queue through the master node, determine the execution node to execute the subtasks according to the resource load status of the execution nodes, and distribute the subtasks to the execution nodes; The command management module is used to receive the execution results of the subtask by the execution node, update the status of the subtask stored in the database, and determine whether the command object has been executed completely based on the subtask dependencies and the updated status of the subtask. The result feedback module is used to respond to the completion of the command object execution by providing feedback on the result of the command object execution to the visualization interface.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the distributed task scheduling method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the distributed task scheduling method according to any one of claims 1 to 7.

Images