Intelligent parallel task scheduling and monitoring system

Abstract

The application discloses an intelligent parallel task scheduling and monitoring system, and relates to the field of intelligent parallel task scheduling and monitoring systems.The system comprises a resource monitoring module, which collects system data of a target system, obtains available nodes and resource vectors of the available nodes based on the system data; a task analysis module, which obtains a task dependency graph and a task feature vector of each target task based on input task information, obtains a graph node corresponding to the task feature vector based on the task dependency graph, takes the task feature vector as a node attribute of the graph node, and obtains a total dependency graph based on the graph node and the node attribute; and an intelligent scheduling module, which obtains a task scheduling strategy based on the total dependency graph, the available nodes and the resource vectors, and solves the problems of insufficient resource utilization, lack of task feature perception and rigid dependency management in existing parallel task scheduling methods, thereby reducing resource utilization, task scheduling efficiency and adaptability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent parallel task scheduling and monitoring systems, in particular, to an intelligent parallel task scheduling and monitoring system. BACKGROUND

[0002] With the popularity of cloud computing, big data and artificial intelligence technologies, distributed computing systems have become the core infrastructure for processing massive amounts of data. In such systems, thousands of computing tasks run on clusters composed of a large number of computing nodes. There are often complex dependencies between these tasks, for example, the execution of a subsequent task requires the output of a previous task as input.

[0003] Current mainstream task scheduling methods (such as the default strategy based on Hadoop YARN, Apache Mesos or simple Kubernetes scheduler) mostly use static rules or simple dynamic strategies (such as round robin, first come first served, or fixed priority-based scheduling). But these methods have the following inherent defects:

[0004] Inadequate resource utilization: static strategies cannot perceive real-time resource load (such as CPU hotspots, memory bottlenecks, network bandwidth fluctuations) of the system, leading to uneven utilization of cluster resources, with some nodes overloaded and others idle, resulting in resource waste;

[0005] Lack of task feature awareness: traditional schedulers treat tasks as black boxes, ignoring the heterogeneity of tasks (such as compute-intensive, I / O-intensive, memory-intensive), scheduling different types of tasks indiscriminately, which can easily cause resource competition and lead to overall system performance degradation;

[0006] Rigid dependency management: although existing systems support directed acyclic graph (DAG) task dependencies, their scheduling decisions are often made at the time of task submission, lacking the flexibility to dynamically adjust the execution order of dependent tasks and resource allocation based on system state at runtime. SUMMARY

[0007] To solve the problems of inadequate resource utilization, lack of task feature awareness and rigid dependency management in existing parallel task scheduling methods, which result in low resource utilization, low task scheduling efficiency and low adaptability, the present application provides an intelligent parallel task scheduling and monitoring system, which comprises:

[0008] a resource monitoring module for collecting system data of a target system, obtaining available nodes and resource vectors of the available nodes based on the system data, the system data including system load and resource utilization data;

[0009] a task analysis module configured to obtain a task dependency graph and a task feature vector of each target task based on input task information, obtain a graph node corresponding to the task feature vector based on the task dependency graph, take the task feature vector as a node attribute of the graph node, and obtain a total dependency graph based on the graph node and the node attribute;

[0010] an intelligent scheduling module configured to obtain a task scheduling strategy based on the total dependency graph, available nodes, and resource vectors.

[0011] The system resource monitoring module of the system can perceive system load in real time, the task analysis module can output a structured task dependency graph and attached task feature vectors, and the intelligent scheduling module can provide input. The intelligent scheduling module can intelligently match task features with node resource states, achieve fine-grained and dynamic load balancing, effectively reduce resource hotspots and idleness, improve overall resource utilization of the cluster, adopt hierarchical action selection, the model first outputs a probability distribution of all ready tasks, selects which task should be scheduled, then the model outputs a probability distribution of all available nodes for the selected task, selects the most suitable node for the task, introduces dynamic priority, generates optimal task-node mapping instructions, and obtains a task scheduling strategy.

[0012] Further, the task analysis module specifically includes:

[0013] a first module configured to obtain the task dependency graph based on the task information;

[0014] a second module configured to obtain the task feature vector of each target task based on the task information;

[0015] a third module configured to obtain the graph node corresponding to the task feature vector based on the task dependency graph, take the task feature vector as the node attribute of the graph node, and obtain the total dependency graph based on the graph node and the node attribute;

[0016] The first module is specifically configured to:

[0017] parse the task information to obtain a display dependency declaration and static metadata of each target task, generate a structured task base descriptor based on the display dependency declaration and the static metadata, and the static metadata includes a task name, a unique identifier, a command line parameter, a first declaration of an input data path, a second declaration of an output data path, and a predicted resource request;

[0018] obtain an upstream task and a downstream task based on the task base descriptor, and construct a directed edge based on the upstream task and the downstream task;

[0019] obtaining a first task and a second task from any two of the target tasks, if the first task and the second task do not have a time sequence, but an input path of the first task is an output path of the second task, then constructing a dependency edge from the second task to the first task based on the directed edge, or an input path of the second task is an output path of the first task, then constructing a dependency edge from the first task to the second task based on the directed edge;

[0020] obtaining the task dependency graph based on the target task, the directed edge and the dependency edge.

[0021] The task analysis module extracts explicit dependency declarations and static metadata of the tasks to form a set of structured task basic descriptors, and constructs a directed edge based on keywords such as upstream tasks and downstream tasks parsed from the task basic descriptors to form a skeleton of the dependency graph. Then, the input and output data paths of each task are analyzed. Even if two tasks do not have a direct sequence in the process definition, if the input path of task B is the output path of task A, the system will automatically create a dependency edge from A to B, which can solve the problem of missing dependencies caused by configuration omissions. The task nodes, directed edges and dependency edges are combined to form a standard task dependency graph. The task feature vector generated for each task is attached as the attribute of the corresponding graph node, and finally a total task dependency graph is output. This module does not simply parse the static dependency that task A runs after task B, but constructs an enhanced, quantitative and feature-driven task dependency model, which converts a set of original task descriptions into a dynamic knowledge graph that can be perceived and predicted by a deep-empowered intelligent scheduling engine.

[0022] Further, the second module is specifically configured to:

[0023] obtaining a first task and a second task from any two of the target tasks, if the first task and the second task do not have a time sequence, but an input path of the first task is an output path of the second task, then constructing a dependency edge from the second task to the first task based on the directed edge, or an input path of the second task is an output path of the first task, then constructing a dependency edge from the first task to the second task based on the directed edge;

[0024] obtaining the resource request specification of the target task based on the task information, standardizing and encoding the resource request specification to obtain a specification code, classifying and encoding the task type of the target task to obtain a dimension code, and obtaining a static coding feature based on the specification code and the dimension code;

[0025] construct a task library based on the historical running instances, if the target task does not exist in the task library, obtain a total similarity based on the target task and the task library, obtain a similar task based on the total similarity, weight historical features of the similar task, and obtain a dynamic estimated feature;

[0026] obtain the task feature vector of each target task based on the historical features, the static coding features, and the dynamic estimated features.

[0027] Multi-level task feature extraction and vectorization, which is the key to changing the task from a black box to a white box, provides a quantitative basis for intelligent scheduling. For the first time or no historical data tasks, the system creatively uses a similarity-based estimation method. The system compares the code features and resource requests of the new task with the tasks in the existing task library, finds multiple most similar tasks, and takes the weighted average of the historical features of these similar tasks as the estimated features of the new task, providing accurate resource and behavior estimation for tasks without historical records, which can solve the cold start problem in distributed task scheduling.

[0028] Further, the specific steps of obtaining the total similarity include:

[0029] construct tuples of the target task and the task library based on a tuple construction method, respectively obtain a plurality of target tuples and a plurality of historical tuples, obtain a subspace similarity of the target tuples and the historical tuples, the subspace similarity includes code similarity, specification similarity, and semantic similarity, and weight and fuse the subspace similarity to obtain the total similarity;

[0030] The tuple construction method is: parsing the source code of a task to obtain a source code abstract syntax tree, converting the source code abstract syntax tree into a numerical vector, encoding the third-party library and version number list of the task into a dependency vector, numerically processing the configuration parameters of the task to obtain a configuration vector, and obtaining a structural feature vector based on the numerical vector, the dependency vector, and the configuration vector;

[0031] obtain a resource request specification vector by obtaining the resource request of a task;

[0032] convert the task description text of a task into a semantic vector, classify the tasks based on users and project groups to obtain a plurality of task classifications, encode the task classifications to obtain a user vector, and obtain a context feature vector based on the semantic vector and the user vector;

[0033] obtain a tuple of a task based on the structural feature vector, the resource request specification vector, and the context feature vector.

[0034] The system constructs a feature vector of a task, not simply compares the task name, but constructs a comprehensive and quantifiable task feature portrait, and represents the portrait as a tuple for accurate similarity calculation; the total similarity adopts a weighted multi-modal similarity fusion algorithm, not a single-dimensional comparison, to improve the accuracy of matching.

[0035] Further, the intelligent scheduling module is specifically used for:

[0036] The trigger condition for constructing the task scheduling strategy includes:

[0037] A, the task information is input to the task analysis module, and the task analysis module generates the total dependency graph;

[0038] B, the resource monitoring module reports that a node releases resources;

[0039] C, a rescheduling request;

[0040] D, the system enters a new time scheduling period;

[0041] If any of the above trigger conditions is met, a ready task table is obtained based on the total dependency graph, and the ready task table includes a plurality of ready tasks;

[0042] A task-node mapping table is obtained based on a pre-trained intelligent decision model, and the task scheduling strategy is obtained based on the task-node mapping table;

[0043] The specific steps of obtaining the task-node mapping table based on the intelligent decision model include:

[0044] S1, aggregating the task feature vectors of the ready tasks in the ready task table to obtain an embedding vector, aggregating the resource vectors to obtain a system vector, obtaining an optimization target, and obtaining a scheduling context state vector based on the embedding vector, the system vector and the optimization target;

[0045] S2, based on the scheduling context state vector, obtaining a first probability distribution of the ready tasks, based on the first probability distribution, obtaining a first execution task, based on the first execution task, obtaining a second probability distribution of the available nodes, and based on the second probability distribution, obtaining a first node corresponding to the first execution task;

[0046] S3, deleting the first execution task in the ready task table, obtaining a new task based on the total dependency graph, obtaining a new task table based on the new task, obtaining a new vector based on the first node and updating the system vector, returning to the S1, updating the ready task table as the new task table and updating the system vector as the new vector, and repeating the S1 to the S3 until all the target tasks are executed;

[0047] S4, obtaining the task-node mapping table based on the first execution task and the first node.

[0048] The intelligent scheduling module constructs a highly structured data representation of the scheduling context state vector, which is not a simple list, and the optimization target allows the system to dynamically switch the optimization direction according to the operation and maintenance strategy at runtime; in order to cope with the problem of huge action space in large-scale clusters, the module adopts hierarchical action selection, the first layer: task selection: the model first outputs a probability distribution of all ready tasks, and selects which task should be scheduled; the second layer: node selection: then the model outputs a probability distribution of all available nodes for the selected task, and selects the most suitable node for the task; a dynamic priority is introduced to generate one or more pairing instructions, so as to obtain a task scheduling strategy.

[0049] Further, the first module is also used for:

[0050] Obtaining multi-source information of the target task, the multi-source information including a task execution script, a task configuration file and a task source code;

[0051] Obtaining a SQL statement and a target table name of the multi-source information, obtaining an operation type and a data entity based on the SQL statement, and obtaining a structured data source based on the target table name, the operation type and the data entity;

[0052] Obtaining a file system path based on a predefined regular expression pattern and the multi-source information, and obtaining a logical entity by endpoint identification of the multi-source information; obtaining an unstructured data path based on the file system path and the logical entity;

[0053] Obtaining a message queue and a cache based on the multi-source information, obtaining queue information of the message queue, the queue information including a topic name and a queue name, and obtaining a key mode of the cache;

[0054] Obtaining a data access mode of the data entity based on the structured data source, the unstructured data path, the queue information and the key mode;

[0055] Obtaining a third task and a fourth task from any two target tasks, and obtaining a reference entity from any one data entity;

[0056] if the third task writes the reference entity and the fourth task reads the reference entity, it is determined that the third task and the fourth task exist a producer-consumer logical dependency;

[0057] if the third task and the fourth task both write the reference entity, it is determined that the third task and the fourth task exist a competitive logical dependency;

[0058] if the third task and the fourth task both read the reference entity, it is determined that the third task and the fourth task exist a shared resource logical dependency;

[0059] the competitive logical dependency is marked as a strong logical dependency, and the producer-consumer logical dependency and the shared resource logical dependency are marked as weak logical dependencies;

[0060] the task dependency graph is updated based on the producer-consumer logical dependency, the shared resource logical dependency and the weak logical dependency.

[0061] By means of static analysis technology, the access intention of tasks and configurations to shared data resources or services is inferred from the task code and the configuration, so as to establish a non-mandatory but optimized and early warning dependent relationship, and the dependent relationship is taken as a weak dependency or a strong dependency. The dependency does not force blocking scheduling, but can be an important reference for decision-making of an intelligent scheduling engine, for example, tasks with logical dependencies are preferentially scheduled to the same rack to reduce network overhead. The dependency can automatically discover and establish an implicit association based on data entities and business logic, thereby realizing deeper scheduling optimization and risk early warning.

[0062] Further, the specific steps of obtaining the task-node mapping table further include:

[0063] a preset network topology distance table is provided, the network topology distance table including a plurality of network topology distances;

[0064] if the first execution task exists a producer-consumer logical dependency, a plurality of second execution tasks are obtained based on the producer-consumer logical dependency, if the second execution task has been scheduled, first data location information of the second execution task is obtained, the second node corresponding to the first execution task is obtained based on the first data location information and the network topology distance table, and the first node is updated to the second node; if the second execution task has not been scheduled, the first execution task is updated to the second execution task;

[0065] if the first execution task has a shared resource logical dependency, obtaining a plurality of third execution tasks based on the shared resource logical dependency, obtaining second data location information of the first execution task, and obtaining third nodes corresponding to the third execution tasks based on the second data location information and the network topology distance table;

[0066] if the first execution task has a competition logical dependency, obtaining a plurality of fourth execution tasks based on the competition logical dependency, obtaining a competition risk score based on the first execution task and the fourth execution task, if the competition risk score is greater than or equal to a first threshold, executing the fourth execution task after the first execution task based on a mutual exclusion mechanism, if the competition risk score is greater than or equal to a second threshold and less than the first threshold, obtaining warning information based on the first node;

[0067] updating the task-node mapping table based on the first node, the second node, and the third node.

[0068] The scheduling strategy is optimized. If there is a production-consumption logical dependency, the logical dependency is converted into a physical location affinity constraint to realize intelligent collaborative layout across tasks. If the dependency relationship has been satisfied, more intelligent timing optimization is performed based on data readiness. If there is a shared resource logical dependency, task combination optimization is performed to improve cache hit rate and resource reuse. If there is a competition logical dependency, a competition risk score is calculated to provide differentiated warning and processing, risk avoidance, and reduction of task failure and abnormal problems.

[0069] Further, the specific steps of obtaining the competition risk score include:

[0070] obtaining usage data of the first execution task and the fourth execution task, obtaining a report quantity and an application quantity of the usage data, obtaining a first score based on the report quantity and the application quantity, obtaining a second score based on a data service level agreement of the usage data, and obtaining a business impact score based on the first score and the second score;

[0071] obtaining an update time of the usage data, and obtaining freshness based on the update time;

[0072] obtaining historical execution data of the first execution task and the fourth execution task, the historical execution data including a planned trigger time, an actual start time deviation, and an execution duration distribution, obtaining an effective time window of the first execution task and the fourth execution task based on the historical execution data, and obtaining a window overlap probability based on the effective time window;

[0073] Based on the service impact score, the freshness, and the window overlap probability, obtain the competition risk score.

[0074] A multi-dimensional risk assessment is established, subjective risk judgment is converted into objective and quantifiable assessment, the accuracy of risk judgment is improved, potential hazards of write-after-write conflict are accurately predicted, and appropriate preventive measures are taken, which avoids over-conservative scheduling strategy and ensures data consistency and reliability of the system.

[0075] Considering the existing mainstream task scheduling method, there is also a problem of weak exception handling capability: when a task fails to execute or a computing node fails, the traditional rescheduling mechanism usually simply requeues or migrates the task to a random node. This process takes a long time to recover, and may fail again due to the failure of the root cause (such as insufficient resources) not being solved, causing task cascading delay; the system uses a root cause analysis model based on graph neural network to replace the traditional rule-based or decision tree-based method, when a task failure event is captured, the system automatically constructs an analysis context graph, inputs a pre-trained root cause analysis model, the model can capture complex and indirect dependency relationships, and then distinguish the root cause, dynamically adjust the resource or node selection, instead of simply requeueing, through root cause analysis and intelligent rescheduling, the system can quickly recover from failure, reduce invalid retries, and significantly improve the stability of the system and the success rate of the task.

[0076] Further, the system further comprises an exception handling module, the exception handling module comprises:

[0077] The listening module is configured to obtain real-time indicators based on the system data, and obtain output logs and task state change information of the target task.

[0078] The analysis module is configured to obtain a task failure event based on the real-time indicators, the output logs, and the task state change information, construct a root cause graph based on the task failure event, analyze the root cause graph based on a pre-trained root cause analysis model to obtain a root cause probability distribution and key evidence, and obtain a root cause result based on the root cause probability distribution.

[0079] The rescheduling module is configured to construct a root cause-strategy mapping library, obtain a rescheduling strategy based on the root cause result and the root cause-strategy mapping library, generate the rescheduling request based on the rescheduling strategy, and update the task scheduling strategy to the rescheduling strategy based on the rescheduling request.

[0080] Further, the specific steps of constructing a root cause graph based on the task failure event comprise:

[0081] The task failure event is taken as a center node, and associated nodes and edge relationships are obtained based on the task failure event and the total dependency graph, the associated nodes including a physical node of the task failure event, a data source relied on by the task failure event, a brother task running on a same node in a same time period and an upstream producer task of the task failure event;

[0082] Based on the center node, the associated nodes and the edge relationships, the root cause graph is constructed.

[0083] The one or more technical solutions provided by the application have at least the following technical effects or advantages:

[0084] 1. The system resource monitoring module perceives system load in real time, the task analysis module outputs a structured task dependency graph and an attached task feature vector, and provides input for the intelligent scheduling module; the intelligent scheduling module intelligently matches task features with node resource states, realizes fine-grained and dynamic load balancing, effectively reduces resource hotspots and idleness, improves overall resource utilization of the cluster, adopts hierarchical action selection, the model first outputs a probability distribution of all ready tasks, selects which task should be scheduled most; then the model outputs a probability distribution of all available nodes for the selected task, selects the most suitable node for the task, introduces dynamic priority, generates optimal task-node mapping instructions, and obtains a task scheduling strategy.

[0085] 2. The task analysis module extracts explicit dependency declarations and static metadata of tasks, forms a set of structured task basic descriptors, constructs a directed edge based on keywords such as upstream tasks and downstream tasks parsed from the task basic descriptors, forms a skeleton of a dependency graph, analyzes input and output data paths of each task, and even if two tasks have no direct relationship in process definition, if the input path of task B is the output path of task A, the system will automatically create a dependency edge from A to B, which can solve the problem of missing dependencies caused by configuration omissions; the task nodes, directed edges and dependency edges are combined to form a standard task dependency graph, the task feature vector generated for each task is attached as an attribute of the corresponding graph node, and finally a total task dependency graph is output. The module does not simply parse the static dependency that task A runs after task B, but constructs an enhanced, quantitative and feature-driven task dependency model, which converts a set of original task descriptions into a deep empowerment intelligent scheduling engine, a perceptible and predictable dynamic knowledge graph.

[0086] 3. Multi-level task feature extraction and vectorization, the key to turning tasks from black boxes into white boxes, provides quantitative basis for intelligent scheduling. For the first run or no historical data tasks, the system creatively adopts a similarity-based estimation method. The system compares the code features and resource requests of the new task with the tasks in the existing task library, finds multiple most similar tasks, and takes the weighted average of the historical features of these similar tasks as the estimated features of the new task, providing accurate resource and behavior estimation for tasks without historical records, which can solve the cold start problem in distributed task scheduling.

[0087] 4. The system constructs a structural feature vector, not simply compares task names, but constructs a comprehensive and quantifiable task feature portrait, and represents the portrait as a tuple for accurate similarity calculation; the total similarity uses a weighted multi-modal similarity fusion algorithm, not a single-dimensional comparison, to improve the accuracy of matching.

[0088] 5. The intelligent scheduling module constructs a highly structured data representation of the scheduling context state vector, which is not a simple list, and the optimization goal allows the system to dynamically switch the optimization direction according to the operation and maintenance strategy during runtime; to cope with the problem of huge action space in large-scale clusters, this module uses hierarchical action selection, the first layer: task selection: the model first outputs a probability distribution of all ready tasks, and selects which task should be scheduled; the second layer: node selection: then the model outputs a probability distribution of all available nodes for the selected task, and selects the most suitable node for the task; dynamic priority is introduced to generate one or more pairing instructions, thereby obtaining the task scheduling strategy.

[0089] 6. Through static analysis technology, the access intention of tasks to shared data resources or services is inferred from the task code and configuration, thereby establishing a non-mandatory but optimized and early warning value dependency, and it is treated as weak or strong dependency. This dependency does not forcibly block scheduling, but can be an important reference for the decision-making of the intelligent scheduling engine, for example, preferentially scheduling tasks with logical dependencies to the same rack to reduce network overhead, can override user-defined dependencies, automatically discover and establish implicit associations based on data entities and business logic, thereby achieving deeper scheduling optimization and risk warning.

[0090] 7. Scheduling strategy optimization, there is a production-consumption logical dependency, which converts logical dependency into physical location affinity constraints, realizes intelligent collaborative layout across tasks, and based on data readiness, performs more intelligent timing optimization when the dependency relationship is satisfied; there is a shared resource logical dependency, which improves cache hit rate and resource reuse through task combination optimization; there is a competitive logical dependency, which calculates the risk score, provides differentiated early warning and processing, risk avoidance, and reduces the problem of task failure and exception.

[0091] 8. The root cause analysis model based on graph neural network is used to replace the traditional rule-based or decision tree-based method. When a task failure event is captured, the system automatically constructs an analysis context graph, inputs a pre-trained root cause analysis model, the model can capture complex and indirect dependency relationships, and then distinguish the root cause, dynamically adjust the resource or node selection, and not only requeue. Through root cause analysis and intelligent rescheduling, the system can quickly recover from failure, reduce invalid retries, and significantly improve the stability of the system and the success rate of tasks. BRIEF DESCRIPTION OF DRAWINGS

[0092] The accompanying drawings, which are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0093] Fig. 1 is a flowchart of the intelligent parallel task scheduling and monitoring system in the present application;

[0094] Fig. 2 is a flowchart of obtaining a task-node mapping table in the present application. DETAILED DESCRIPTION

[0095] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0096] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, however, the present application can also be implemented in other ways different from the scope described herein, therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below.

[0097] Embodiment one

[0098] Reference Figs. 1-2 The present embodiment provides an intelligent parallel task scheduling and monitoring system, which comprises:

[0099] A resource monitoring module is used to collect system data of a target system, obtain available nodes and resource vectors of the available nodes based on the system data, the system data includes system load and resource utilization data (such as CPU utilization, memory usage, disk I / O, network bandwidth), and can also include system-level indicators (such as load average, container running state);

[0100] a task analysis module, configured to obtain a task dependency graph and a task feature vector of each target task based on input task information, obtain a graph node corresponding to the task feature vector based on the task dependency graph, take the task feature vector as a node attribute of the graph node, and obtain a total dependency graph based on the graph node and the node attribute;

[0101] In this embodiment, the task information can include but is not limited to:

[0102] a workflow definition file, such as a DAG description file, a Python script of Apache Airflow, and a Makefile;

[0103] a configuration file, such as a YAML / JSON file, in which resource requests and command parameters of tasks are declared;

[0104] program code / script, such as a JAR package and an entry class of a Spark / Flink job.

[0105] a smart scheduling module, configured to obtain a task scheduling strategy based on the total dependency graph, available nodes, and resource vectors.

[0106] The task analysis module specifically includes:

[0107] a first module, configured to obtain the task dependency graph based on the task information;

[0108] a second module, configured to obtain the task feature vector of each target task based on the task information;

[0109] a third module, configured to obtain the graph node corresponding to the task feature vector based on the task dependency graph, take the task feature vector as the node attribute of the graph node, and obtain the total dependency graph based on the graph node and the node attribute;

[0110] The first module is specifically configured to:

[0111] use a parser related to a workflow engine or a language (for example, a parser for Cron expressions, a Python AST parser, and an XML / JSON parser) to parse the task information to obtain display dependency declarations and static metadata of each target task, generate a structured task base descriptor based on the display dependency declarations and the static metadata, and the static metadata includes a task name, a unique identifier, command line parameters, a first declaration of an input data path, a second declaration of an output data path, and estimated resource requests (such as CPU and memory);

[0112] The display dependency declaration refers to an execution order relationship between tasks that is actively and explicitly specified by a user or a workflow designer through programming, a configuration file or a graphical interface, and is a kind of artificial preset and mandatory sequence constraint. It explicitly indicates a scheduling system, for example, task B must be executed after task A is successfully completed.

[0113] The data structure of the task base descriptor can include: identification information (such as a global unique identifier and a hash value of the task), execution specification information (such as an actual execution command or an entry point of the task, a container image address, script content or a fingerprint thereof), resource request information (such as a requested CPU core number, a requested memory size, a requested disk space, a requested GPU number and type), explicit dependency information (a list containing global unique identifiers of all direct upstream tasks of the task), and data path declaration information (a list declaring input data paths read by the task and another list declaring output data paths written by the task).

[0114] Based on the task base descriptor, upstream tasks (tasks that must be completed before the current task) and downstream tasks (tasks that depend on the current task) are obtained, and a directed edge is constructed based on the upstream tasks and the downstream tasks;

[0115] Any two target tasks are obtained to obtain a first task and a second task, if the first task and the second task do not exist in a time sequence, but an input path of the first task is an output path of the second task, then a dependency edge from the second task to the first task is constructed based on the directed edge; or an input path of the second task is an output path of the first task, then a dependency edge from the first task to the second task is constructed based on the directed edge;

[0116] The task dependency graph is obtained based on the target task, the directed edge and the dependency edge.

[0117] The second module is specifically configured to:

[0118] A historical execution log is obtained, and a plurality of historical running instances are obtained based on the historical execution log and a task identifier, such as all historical running instances of the task identifier (such as a job name, a code hash value) are retrieved, historical features of the historical running instances are obtained, and the historical features include a mean value of CPU occupancy, a peak value of CPU occupancy, a mean value of memory consumption, a peak value of memory consumption, disk I / O throughput, network data transmission and execution time distribution;

[0119] obtaining a resource request specification of the target task based on the task information (such as --executor-memory 4g), standardizing and encoding the resource request specification to obtain a specification code, classifying and encoding a task type of the target task to obtain a dimension code (such as marking it as machine learning-model training, ETL-data cleaning, or SQL query), and obtaining a static code feature based on the specification code and the dimension code;

[0120] constructing a task library based on the historical running instances, determining whether the target task exists in the task library by using a unique identifier, obtaining a total similarity based on the target task and the task library, obtaining similar tasks based on the total similarity (such as obtaining a plurality of most similar tasks based on K-nearest neighbor retrieval), and weighting historical features of the similar tasks to obtain a dynamic prediction feature;

[0121] obtaining the task feature vector of each target task based on the historical features, the static code features, and the dynamic prediction features (such as normalizing and splicing them into the task feature vector).

[0122] The specific steps of obtaining the total similarity include:

[0123] constructing tuples of the target task and the task library based on a tuple construction method, obtaining a plurality of target tuples and a plurality of historical tuples, obtaining a subspace similarity of the target tuples and the historical tuples, the subspace similarity including a code similarity (such as calculating a cosine similarity or an inverse of a Euclidean distance of the code), a specification similarity (such as calculating a cosine similarity of the specification), and a semantic similarity (such as a cosine similarity of the semantics), and weighting and fusing the subspace similarity to obtain the total similarity (such as fusing them based on a weight coefficient);

[0124] The tuple construction method is: parsing a source code of a task to obtain a source code abstract syntax tree (AST), converting the source code abstract syntax tree into a numerical vector through a tree-shaped neural network or a recurrent neural network to capture structural similarity of the code, encoding a third-party library and a version number list of the task into a dependency vector (such as a task that depends on tensorflow==2.9.1 and numpy==1.21.0 is likely to belong to the machine learning category), normalizing and numerizing configuration parameters (such as executor-cores and driver-memory of Spark) of the task to obtain a configuration vector, and obtaining a structural feature vector based on the numerical vector, the dependency vector, and the configuration vector.

[0125] Obtaining the resource request (such as request_cpus, request_memory_gb, request_disk_gb) of the task to obtain the resource request specification vector;

[0126] Converting the task description text of the task into a semantic vector, using a natural language processing model (such as BERT), embedding the task name and the description text filled by the user when submitting, and converting it into a semantic vector, which can capture the semantic association between image preprocessing and picture data cleaning tasks; classifying the tasks based on users and project groups to obtain a plurality of task classifications, encoding the task classifications to obtain a user vector, and encoding the user, project group or business line to which the task belongs. Tasks under the same user or project often have similar behavior patterns, and a context feature vector is obtained based on the semantic vector and the user vector;

[0127] Obtaining the tuple of the task based on the structural feature vector, the resource request specification vector and the context feature vector. The intelligent scheduling module is specifically configured to:

[0128] Constructing a trigger condition of the task scheduling strategy, the trigger condition comprising:

[0129] A. The task information is input to the task analysis module, and the task analysis module generates the total dependency graph; that is, a new task group is submitted and analyzed by the task analysis module;

[0130] B. The resource monitoring module reports that a node releases resources; that is, the resource monitoring module reports that a computing node releases resources (such as task completion);

[0131] C. Rescheduling request;

[0132] D. The system enters a new time scheduling period;

[0133] If any of the above trigger conditions is met, a ready task table is obtained based on the total dependency graph, and the ready task table includes a plurality of ready tasks; such as finding all ready tasks whose pre-depending dependencies have been met from the total dependency graph;

[0134] Obtaining a task-node mapping table based on a pre-trained intelligent decision model (such as a pre-trained deep reinforcement learning model), and obtaining the task scheduling strategy based on the task-node mapping table;

[0135] The specific steps of obtaining the task-node mapping table based on the intelligent decision model include:

[0136] S1, aggregate the task feature vectors of the ready tasks in the ready task table to obtain an embedding vector, such as being generated by a pre-trained encoder network, aggregate the resource vectors to obtain a system vector, that is, an aggregated representation of the resource vectors of all available nodes, obtain an optimization objective (such as maximizing throughput or minimizing delay), and obtain a scheduling context state vector based on the embedding vector, the system vector, and the optimization objective (such as direct splicing);

[0137] S2, based on the scheduling context state vector, obtain a first probability distribution of the ready tasks, obtain a first execution task based on the first probability distribution, select the one with the highest probability value as the first execution task, obtain a second probability distribution of the available nodes based on the first execution task, and obtain a first node (that is, the one with the highest probability value) corresponding to the first execution task based on the second probability distribution;

[0138] S3, delete the first execution task from the ready task table, obtain a new task based on the total dependency graph, obtain a new task table based on the new task, update the system vector based on the first node to obtain a new vector, return to S1, update the ready task table to the new task table, and update the system vector to the new vector, and repeat S1 to S3 until all target tasks are executed;

[0139] S4, based on the first execution task and the first node, obtain the task-node mapping table.

[0140] Embodiment two

[0141] Reference Fig. 2 On the basis of embodiment one, in this embodiment, the first module is further used to:

[0142] Obtain multi-source information of the target task, the multi-source information including task execution scripts, task configuration files, and task source codes;

[0143] Task execution command / script: for example, Shell command, Spark-submit command, Python script path, etc.

[0144] Task configuration file: such as YAML, Properties file, which may contain data source connection information;

[0145] Source code / bytecode: for JAR package or other resolvable binary tasks.

[0146] obtain an operation type and a data entity based on the SQL statement, and obtain a structured data source based on the target table name, the operation type, and the data entity; identify and parse the SQL statement in the task through SQL keywords (such as SELECT, INSERT, CREATE TABLE AS, etc.), and extract the operation type (read, write, delete, and modify) and the data entity (database name, table name, and partition name (for example, extract the table name db1.table1 and the partition dt='2023-10-01' from SELECT FROMdb1.table1 WHERE dt='2023-10-01');

[0147] obtain a file system path based on a predefined regular expression pattern and the multi-source information, perform endpoint identification on the multi-source information to obtain a logical entity, such as API endpoint identification by using an abstract syntax tree and regular expression matching: identify a call to a specific REST API or RPC service in the code, and extract a service endpoint URL or a service name as the logical entity, and obtain an unstructured data path based on the file system path and the logical entity;

[0148] obtain a message queue and a cache based on the multi-source information, obtain queue information of the message queue, the queue information including a topic name and a queue name, and obtain a key pattern of the cache; such as identifying a call to a message middleware such as Kafka, RabbitMQ, etc., extracting a topic (Topic) name and a queue name, and specifically based on a common Kafka client library: for a producer, finding a call to send a message, such as a send method, a parameter of which usually contains a topic name, a basicPublish method, and a parameter containing an exchange name and a routing key; for a consumer, finding a call to subscribe to a topic, such as a subscribe method, a queueDeclare, or a call to consume, such as a basicConsume, a parameter of which contains a queue name.

[0149] identify a call to a cache such as Redis, Memcached, etc., and extract a used key pattern (Key Pattern). Such as based on common client libraries: jedis, lettuce, redis-py, etc., finding common operations such as set, get, hget, etc., in which the first parameter is usually a build pattern.

[0150] obtaining a data access pattern of the data entity based on the structured data source, the unstructured data path, the queue information, and the key pattern;

[0151] The access mode of the task to the data resource (such as a file, a database table, a message queue, etc.) during execution includes reading, writing, updating, etc. operations, and the characteristics of the data entity, frequency, order, etc. involved in these operations.

[0152] Obtain a third task and a fourth task from any two of the target tasks, and obtain a reference entity from any one of the data entities;

[0153] If the third task writes the reference entity, and the fourth task reads the reference entity, it is judged that the third task and the fourth task exist production-consumption logical dependency;

[0154] If the third task and the fourth task both write the reference entity, it is judged that the third task and the fourth task exist competition logical dependency; this usually means that the task order is crucial, and needs to be serialized or specially coordinated;

[0155] If the third task and the fourth task both read the reference entity, it is judged that the third task and the fourth task exist shared resource logical dependency; there is no forced order between them, but the scheduler can execute them in parallel, or schedule them to the same node to take advantage of data locality.

[0156] Mark the competition logical dependency as a strong logical dependency, which may cause a competition relationship of data inconsistency or error, and the system marks it as a strong logical dependency, which can produce a strong warning on the user interface, prompting the user to check and possibly convert it to an explicit dependency; mark the production-consumption logical dependency and the shared resource logical dependency as weak logical dependencies; such dependencies will not block the execution of tasks, but will provide valuable optimization clues for the intelligent scheduling module.

[0157] Based on the production-consumption logical dependency, the shared resource logical dependency, and the weak logical dependency, update the task dependency graph. For example, create a new directed edge for the third task and the fourth task according to the logical relationship, and the edge attribute can include the logical type, the logical label, and the data entity.

[0158] The production-consumption logical dependency refers to the data flow relationship between two tasks, in which the output data of one task (producer) is directly used as the input data of another task (consumer).

[0159] The competition logical dependency refers to the fact that two or more tasks perform write operations on the same data entity, and there is no explicit execution order constraint, which has the risk of data coverage and state inconsistency.

[0160] Shared resource logical dependency refers to multiple tasks reading the same shared data resource, but no write conflict is involved, mainly affecting system performance and resource utilization efficiency.

[0161] The specific steps of obtaining the task-node mapping table further include:

[0162] A preset network topology distance table, the network topology distance table comprising a plurality of network topology distances;

[0163] Network topology distance refers to an index for measuring the communication cost or closeness between two nodes (usually referring to computing nodes) from the perspective of network structure. It is not only the physical distance, but also includes the positional relationship in the network hierarchy.

[0164] Same node: distance = 0 (highest priority); same rack: distance = 1; same data center: distance = 2; cross-data center: distance = 3 (lowest priority).

[0165] Data location information indicates that the system records the physical storage location of the output data of the producer task when it is completed (such as the DataNode list where the HDFS block is located, or the endpoint of the object storage).

[0166] Network topology distance refers to an index for measuring the communication cost or closeness between two nodes (usually referring to computing nodes) from the perspective of network structure. It is not only the physical distance, but also includes the positional relationship in the network hierarchy.

[0167] If the first execution task has a production-consumption logical dependency, a plurality of second execution tasks are obtained based on the production-consumption logical dependency, if the second execution task has been scheduled, the first data location information of the second execution task is obtained, the second node corresponding to the first execution task is obtained based on the first data location information and the network topology distance table, the first node is updated to the second node; if the second execution task has not been scheduled, the first execution task is updated to the second execution task; such as:

[0168] a. When the system is ready to schedule task Tb, check its weak logical dependency, find that it depends on the output of producer task Ta (i.e. Ta produces the data needed by Tb);

[0169] b. The system checks the node where Ta is currently scheduled (assuming Node_A), or if Ta has not been scheduled, determines the optimal node according to the data location information (if Ta has produced data, the data may have been stored on a certain node);

[0170] c. The system tries to schedule Tb on Node_A, if Node_A is not enough, the system selects a node which is in the same rack (or the nearest in the network topology) as Node_A to minimize the network transmission overhead;

[0171] d. If Ta is not running, the system can consider to schedule Ta and Tb together, i.e. schedule Ta first, and then schedule Tb on the same node immediately after Ta is finished.

[0172] If the first execution task has a shared resource logical dependency, a plurality of third execution tasks are obtained based on the shared resource logical dependency, second data location information of the first execution task is obtained, and a third node corresponding to the third execution task is obtained based on the second data location information and the network topology distance table. For example:

[0173] a. When two tasks Tc and Td have a shared resource logical dependency (for example, they both need to read the same large data set), the system can bind them to run on the same node;

[0174] b. In this way, the node only needs to load the data once to the memory or local disk, and the two tasks can share the data, reducing the repeated data loading overhead and network transmission;

[0175] c. In specific implementation, when scheduling Tc, the system selects node Node_C according to the data location, and then schedules Td to Node_C (if the resources allow), if the resources do not allow, the adjacent node is selected, and the node with data copy is selected by using the copy mechanism of the distributed storage system.

[0176] If the first execution task has a competitive logical dependency, a plurality of fourth execution tasks are obtained based on the competitive logical dependency, a competitive risk score is obtained based on the first execution task and the fourth execution task, if the competitive risk score is greater than or equal to a first threshold value, the fourth execution task is executed after the first execution task is executed based on a mutual exclusion mechanism, if the competitive risk score is greater than or less than a second threshold value and less than the first threshold value, warning information is obtained based on the first node. For example:

[0177] a. In the task dependency graph construction stage, the system identifies strong logical dependencies (for example, two tasks Tx and Ty both write to the same data entity E, and there is no explicit dependency to define the execution order);

[0178] b. The system marks such strong logical dependencies as potential risk points, and highlights them with a prominent color (such as red) on the operation and maintenance monitoring interface, and attaches warning information, for example: tasks Tx and Ty have write-after-write competition, which may cause data inconsistency;

[0179] c.The system can automatically suggest the user to add explicit dependencies, e.g. let Tx execute after Ty, or use a mutual exclusion mechanism, i.e. when both Tx and Ty are in ready state, only one of the tasks is selected for scheduling and the other is suspended until the former is completed.

[0180] updating the task-node mapping table based on the first node, the second node, and the third node.

[0181] The specific steps of obtaining the competition risk score include:

[0182] Obtain the usage data of the first execution task and the fourth execution task, such as automatically obtaining the business attributes of the data entity through the data asset directory or metadata management system connected with the enterprise, obtaining the number of reports and the number of applications (referring to software systems or service modules such as microservices and API backends whose business logic execution directly depends on the data entity) of the usage data, obtaining a first score based on the number of reports and the number of applications, reading the number of downstream applications and reports of the data, the more the number is, the wider the influence is, and the higher the score is, the report is the downstream report, which refers to the visual data product or decision support tool whose content is directly generated or driven by the data entity; obtaining a data service level agreement of the usage data, obtaining a second score based on the data service level agreement, such as core transaction data (5 points), operation analysis data (3 points), and temporary test data (1 point); obtaining a business impact score based on the first score and the second score, such as fusion based on a weight coefficient;

[0183] The data service level agreement refers to the service level agreement reached between the data owner (such as the business party) and the data service provider (such as the data platform team) on the key quality attributes of data such as availability, timeliness, accuracy, and consistency, which can be quantitatively measured.

[0184] Obtain the update time of the usage data, and obtain the freshness based on the update time; such as real-time data: requiring minute-level update (5 points), near real-time data: requiring hour-level update (3 points), and batch data: day-level update (1 point).

[0185] Obtain the historical execution data of the first execution task and the fourth execution task, the historical execution data including the planned trigger time, the actual start time deviation, and the execution duration distribution; obtain the effective time window of the first execution task and the fourth execution task based on the historical execution data, the effective time window being [planned trigger time + actual start time deviation, planned trigger time + actual start time deviation + execution duration distribution], and obtain the window overlap probability based on the effective time window using the existing interval overlap probability formula;

[0186] Based on the business impact score, the freshness, and the window overlap probability, obtain the competition risk score, such as fusion based on a weight coefficient.

[0187] Embodiment Three

[0188] On the basis of the above-mentioned embodiments, in this embodiment, the system further comprises an exception processing module, the exception processing module comprises:

[0189] The monitoring module is configured to obtain real-time indicators (such as CPU>95% for 5 minutes, node heartbeat loss) based on the system data, obtain output logs (standard output / error logs of the task container) and task state change information (such as failure, eviction, unknown) of the target task.

[0190] The analysis module is configured to obtain a task failure event based on the real-time indicators, the output logs, and the task state change information, construct a root cause graph based on the task failure event, analyze the root cause graph based on a pre-trained root cause analysis model (such as a pre-trained GNN model) to obtain a root cause probability distribution and key evidence, and obtain a root cause result based on the root cause probability distribution.

[0191] The GNN aggregates the states and attributes of each node in the graph through message passing, and finally outputs a root cause probability distribution on the center node (failed task):

[0192] P(root cause = resource shortage) = 0.75, P(root cause = data loss) = 0.15, P(root cause = code defect) = 0.08, and P(root cause = unknown) = 0.02.

[0193] At the same time, the key evidence for the judgment is output, such as: for the judgment of resource shortage, the evidence is that the node memory usage rate reaches 99% at the task failure moment, and for the judgment of data loss, the evidence is that FileNotFoundException is thrown in the task log.

[0194] The rescheduling module is configured to construct a root cause-strategy mapping library, such as:

[0195] Root cause: memory shortage; strategy: resource expansion rescheduling; action: apply for an increase of 20% of the memory limit for the task, and preferentially schedule to a node with sufficient memory.

[0196] Root cause: node failure; strategy: topology avoidance rescheduling; action: schedule the task to a healthy node in a different failure domain (different racks, different availability zones).

[0197] Root cause: loss of data locality; strategy: data affinity rescheduling; action: recalculate the current location of the input data block and schedule to the node with the most data blocks.

[0198] Root cause: temporary contention; strategy: delay backoff reschedule; action: wait for a random backoff time, then retry as is.

[0199] Obtain a rescheduling strategy based on the root cause result and the root cause-strategy mapping library, generate the rescheduling request based on the rescheduling strategy, and update the task scheduling strategy to the rescheduling strategy based on the rescheduling request.

[0200] The specific steps of constructing a root cause graph based on the task failure event include:

[0201] The task failure event is taken as a center node, and associated nodes and edge relationships (describing the relationship between nodes, such as running on, consuming data from, and competing for resources with) are obtained based on the task failure event and the total dependency graph. The associated nodes include the physical node of the task failure event, the data source depended on by the task failure event, the sibling task running on the same node at the same time, and the upstream producer task of the task failure event.

[0202] Based on the center node, the associated nodes, and the edge relationships, the root cause graph is constructed.

[0203] Embodiment Four

[0204] On the basis of the above-mentioned embodiments, in this embodiment, the system further includes a user interaction module: for converging the system data of each node into a global resource state view; displaying the global resource state view and the total dependency graph on the screen, and different types / colors of lines represent different meanings, such as solid arrow: explicit dependency; dashed green arrow: weak logical dependency (optimization opportunity); dashed red arrow: strong logical dependency (contention risk).

[0205] The strong logical dependency is marked as a potential risk point, and is highlighted in a prominent color (such as red) on the operation and maintenance monitoring interface, and warning information is attached, a pop-up box of early warning information, and the user can selectively operate:

[0206] One-time ignore: only this execution allows contention;

[0207] Add explicit dependency: the system automatically adds a sequential dependency between Tx and Ty;

[0208] Convert to weak dependency: the user confirms the risk and wants to retain the performance benefits of concurrent execution;

[0209] Join the whitelist: permanently ignore such contention warnings for such tasks.

[0210] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims include all such modifications and variations as fall within the scope of the present application.

[0211] It is apparent that those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. It is therefore intended that the present application cover all such changes and modifications that are within its scope.

Claims

1. An intelligent parallel task scheduling and monitoring system, characterized in that, The system includes: Resource monitoring module: used to collect system data of the target system, and obtain available nodes and resource vectors of the available nodes based on the system data. The system data includes system load and resource utilization data. Task analysis module: used to obtain a task dependency graph and a task feature vector for each target task based on the input task information; based on the task dependency graph, obtain graph nodes corresponding to the task feature vectors; use the task feature vectors as node attributes of the graph nodes; and obtain a total dependency graph based on the graph nodes and the node attributes. Intelligent scheduling module: used to obtain task scheduling strategy based on the overall dependency graph, the available nodes, and the resource vector; The task analysis module specifically includes: First module: used to obtain the task dependency graph based on the task information; The second module is used to obtain the task feature vector of each target task based on the task information. The third module is used to obtain the graph node corresponding to the task feature vector based on the task dependency graph, use the task feature vector as the node attribute of the graph node, and obtain the total dependency graph based on the graph node and the node attribute. The first module is specifically used for: Parse the task information to obtain the explicit dependency declaration and static metadata of each target task, and generate a structured task base descriptor based on the explicit dependency declaration and the static metadata. The static metadata includes the task name, unique identifier, command line parameters, first declaration of input data path, second declaration of output data path, and estimated resource request. Based on the task base descriptor, upstream and downstream tasks are obtained, and directed edges are constructed based on the upstream and downstream tasks. Obtain any two target tasks to obtain a first task and a second task. If the first task and the second task do not have a time order, but the input path of the first task is the output path of the second task, then construct a dependency edge from the second task to the first task based on the directed edge; or if the input path of the second task is the output path of the first task, then construct a dependency edge from the first task to the second task based on the directed edge. The task dependency graph is obtained based on the target task, the directed edges, and the dependent edges; The second module is specifically used for: Obtain historical execution logs, and based on the historical execution logs and task identifiers, obtain several historical running instances. Obtain the historical characteristics of the historical running instances, including the average CPU utilization, peak CPU utilization, average memory consumption, peak memory consumption, disk I / O throughput, network data transmission volume, and execution duration distribution. Based on the task information, the resource request specification of the target task is obtained, the resource request specification is standardized and encoded to obtain the specification code, the task type of the target task is classified and encoded to obtain the dimension code, and static encoding features are obtained based on the specification code and the dimension code. A task library is constructed based on the historical running instances. If the target task does not exist in the task library, the total similarity is obtained based on the target task and the task library. Similar tasks are obtained based on the total similarity. The historical features of the similar tasks are weighted to obtain dynamic prediction features. Based on the historical features, the static coding features, and the dynamic prediction features, the task feature vector for each target task is obtained; The intelligent scheduling module is specifically used for: The triggering conditions for constructing the task scheduling strategy include: A. The task information is input into the task analysis module, and the task analysis module generates the overall dependency graph; B. The resource monitoring module reports that a node is releasing resources; C. Rescheduling request; D. The system enters a new time scheduling cycle; If any of the above triggering conditions are met, a ready task table is obtained based on the total dependency graph, and the ready task table includes several ready tasks. A task-node mapping table is obtained based on a pre-trained intelligent decision-making model, and the task scheduling strategy is obtained based on the task-node mapping table. The specific steps for obtaining the task-node mapping table based on the intelligent decision-making model include: S1. Aggregate the task feature vectors of the ready tasks in the ready task table to obtain an embedding vector, aggregate the resource vectors to obtain a system vector, obtain an optimization target, and obtain a scheduling context state vector based on the embedding vector, the system vector and the optimization target. S2. Based on the scheduling context state vector, obtain the first probability distribution of the ready tasks, obtain the first execution task based on the first probability distribution, obtain the second probability distribution of the available nodes based on the first execution task, and obtain the first node corresponding to the first execution task based on the second probability distribution. S3. Delete the first execution task from the ready task table, obtain a new task based on the total dependency graph, obtain a new task table based on the new task, update the system vector based on the first node to obtain a new vector, return to S1, update the ready task table to the new task table, update the system vector to the new vector, and loop from S1 to S3 until all the target tasks have been executed. S4. Based on the first execution task and the first node, obtain the task-node mapping table; The first module is also used for: Obtain multi-source information about the target task, including the task execution script, task configuration file, and task source code; The SQL statement and target table name for obtaining the multi-source information are obtained; the operation type and data entity are obtained based on the SQL statement; and the structured data source is obtained based on the target table name, the operation type, and the data entity. The file system path is obtained based on a predefined regular expression pattern and the multi-source information; the logical entity is obtained by endpoint identification of the multi-source information; and the unstructured data path is obtained based on the file system path and the logical entity. Based on the multi-source information, a message queue and a cache are obtained; the queue information of the message queue is obtained, including the topic name and the queue name; and the key pattern of the cache is obtained. The data access pattern of the data entity is obtained based on the structured data source, the unstructured data path, the queue information, and the key pattern; Obtain any two of the target tasks to obtain the third and fourth tasks, and obtain any one of the data entities to obtain the reference entity; If the third task writes to the reference entity and the fourth task reads the reference entity, then it is determined that the third task and the fourth task have a production-consumption logic dependency. If both the third task and the fourth task are written to the reference entity, then it is determined that the third task and the fourth task have a competing logical dependency. If both the third task and the fourth task read the reference entity, it is determined that the third task and the fourth task have a shared resource logical dependency. The competing logical dependency is marked as a strong logical dependency, and the producer-consumer logical dependency and the shared resource logical dependency are marked as weak logical dependencies. Update the task dependency graph based on the production-consumption logical dependency, the shared resource logical dependency, and the weak logical dependency. The specific steps to obtain the task-node mapping table also include: A preset network topology distance table is provided, which includes several network topology distances. If the first execution task has a producer-consumer logical dependency, then a number of second execution tasks are obtained based on the producer-consumer logical dependency. If the second execution task has been scheduled, then the first data location information of the second execution task is obtained, and the second node corresponding to the first execution task is obtained based on the first data location information and the network topology distance table, and the first node is updated to the second node. If the second execution task has not been scheduled, then the first execution task is updated to the second execution task. If the first execution task has a shared resource logical dependency, then a number of third execution tasks are obtained based on the shared resource logical dependency. Then, the second data location information of the first execution task is obtained, and the third node corresponding to the third execution task is obtained based on the second data location information and the network topology distance table. If the first execution task has a competitive logical dependency, then a number of fourth execution tasks are obtained based on the competitive logical dependency. A competition risk score is obtained based on the first execution task and the fourth execution task. If the competition risk score is greater than or equal to a first threshold, then the fourth execution task is executed after the first execution task is completed based on a mutual exclusion mechanism. If the competition risk score is greater than or equal to a second threshold and less than the first threshold, then a warning information is obtained based on the first node. Update the task-node mapping table based on the first node, the second node, and the third node; The system also includes an exception handling module, which includes: Monitoring module: used to obtain real-time metrics based on the system data, acquire output logs and task status change information of the target task; Analysis module: used to obtain task failure events based on the real-time metrics, the output logs and the task status change information, construct a root cause graph based on the task failure events, analyze the root cause graph based on a pre-trained root cause analysis model to obtain root cause probability distribution and key evidence, and obtain root cause results based on the root cause probability distribution; Rescheduling module: used to construct a root cause-policy mapping library, obtain a rescheduling policy based on the root cause results and the root cause-policy mapping library, generate a rescheduling request based on the rescheduling policy, and update the task scheduling policy to the rescheduling policy based on the rescheduling request.

2. The intelligent parallel task scheduling and monitoring system according to claim 1, characterized in that, The specific steps for obtaining the total similarity include: Based on the tuple construction method, tuples of the target task and the task library are constructed, and several target tuples and several historical tuples are obtained respectively. The subspace similarity of the target tuples and the historical tuples is obtained. The subspace similarity includes code similarity, specification similarity and semantic similarity. The total similarity is obtained by weighted fusion of the subspace similarity. The tuple construction method is as follows: the source code of the task is parsed to obtain the source code abstract syntax tree, the source code abstract syntax tree is converted into a numerical vector, the third-party library and version number list of the task are encoded into a dependency vector, the configuration parameters of the task are numerated to obtain a configuration vector, and a structural feature vector is obtained based on the numerical vector, the dependency vector and the configuration vector. Obtain the resource request specification vector from the resource request of the task; The task description text is converted into a semantic vector. The task is classified according to the user and the project team respectively to obtain several task classifications. The task classification is encoded to obtain a user vector. The context feature vector is obtained based on the semantic vector and the user vector. The task tuple is obtained based on the structural feature vector, the resource request specification vector, and the context feature vector.

3. The intelligent parallel task scheduling and monitoring system according to claim 1, characterized in that, The specific steps for obtaining the competition risk score include: Obtain usage data for the first and fourth execution tasks; obtain the number of reports and applications using the usage data; obtain a first score based on the number of reports and applications; obtain the data service level agreement (GSLA) of the usage data; obtain a second score based on the GSLA; and obtain a business impact score based on the first and second scores. Obtain the update time of the usage data, and determine the freshness based on the update time; Obtain historical execution data for the first and fourth execution tasks, including planned trigger time, actual start time deviation, and execution duration distribution; obtain effective time windows for the first and fourth execution tasks based on the historical execution data, and obtain window overlap probability based on the effective time windows; The competition risk score is obtained based on the business impact score, the freshness, and the window overlap probability.

4. The intelligent parallel task scheduling and monitoring system according to claim 1, characterized in that, The specific steps for constructing a root cause graph based on the task failure events include: Using the task failure event as the central node, associated nodes and edge relationships are obtained based on the task failure event and the overall dependency graph. The associated nodes include the physical node of the task failure event, the data source that the task failure event depends on, sibling tasks running on the same node at the same time, and the upstream producer task of the task failure event. The root cause graph is constructed based on the central node, the associated nodes, and the edge relationships.

Images