A method and system for scheduling tasks based on mesh data processing

Abstract

This invention provides a mesh-based data processing task scheduling method and system, belonging to the field of task scheduling technology. The method extracts task description data and task dependencies from a set of data processing tasks, constructs a task execution network using a directed weighted graph, adds tasks to an execution queue, and uses data processing tasks without prerequisite dependencies as the root tasks of the queue. It then calculates the task priority of each data processing task in the queue, optimizes the queue based on system resource status and task priorities, and obtains a task scheduling sequence. Finally, it performs task dependency verification on the scheduling sequence; if the verification passes, the data processing tasks are executed sequentially. After a task is completed, the task description data is updated, and the system resource status is monitored in real time. In case of emergencies, the system is re-optimized. This invention can achieve precise resource allocation and rapid response to emergencies, significantly improving task scheduling efficiency and stability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of task scheduling, in particular to a mesh data processing task scheduling method and system. BACKGROUND

[0002] In the big data era, mesh data is unstructured or semi-structured data with many-to-many cross-relationships between data elements, without fixed hierarchy or linear relationship, and presents a mesh interwoven association form. Therefore, mesh data processing tasks have the characteristics of high complexity, strong correlation and diverse resource requirements.

[0003] At present, the existing task scheduling technology has the following defects in mesh task scenarios such as big data processing, multi-database collaborative job, and complex business process task arrangement:

[0004] 1. Low scheduling accuracy: Lack of accurate task priority calculation model, only scheduling according to single or small number of dimensions (such as task type, simple time rule), unable to adapt to complex mesh task relationships and dynamic resource changes, leading to key task delay and unreasonable resource allocation.

[0005] 2. Weak response to sudden situations: In the face of sudden situations such as data volume mutation and resource fluctuation, it is difficult to quickly adjust resource allocation, prone to resource waste (idling) or overload (task blocking, system crash), and unable to guarantee timely completion of all tasks.

[0006] 3. Poor scene adaptability: Traditional scheduling methods do not fully consider task network characteristics and resource dynamic constraints, and the scheduling efficiency drops sharply in complex mesh task scenarios, unable to meet large-scale, high-concurrency data processing requirements. SUMMARY

[0007] Therefore, the present application provides a mesh data processing task scheduling method and system to improve task scheduling accuracy in complex scenarios.

[0008] The technical solution adopted by the present application is:

[0009] The present application provides a mesh data processing task scheduling method, comprising:

[0010] Extracting task description data and task dependency relationships of data processing tasks in the task set, and constructing a task running network in the form of a directed weighted graph; the task dependency relationships include pre-dependency relationships and post-dependency relationships;

[0011] Adding data processing tasks in the task running network to an execution queue according to the task dependency relationships, and taking a data processing task with no pre-dependency relationship as a root task of the execution queue;

[0012] Based on the task description data, the task priority of each data processing task in the to-be-executed queue is calculated;

[0013] The system resource state is acquired, and based on the system resource state and the task priority, the to-be-executed queue is scheduled and optimized by using a preset scheduling algorithm to obtain a task scheduling sequence;

[0014] The task scheduling sequence is subjected to task dependency verification, and after the verification is passed, the data processing tasks are executed in sequence according to the task scheduling sequence, and after the tasks are completed, the task description data is updated and the system resource state is monitored in real time, and when a burst condition is triggered, the priority scheduling optimization is re-executed.

[0015] Further, the task description data and the task dependency relationship of the data processing tasks in the task set are extracted, and a task running network is constructed in a directed weighted graph, and the following steps are executed:

[0016] The task description data of each data processing task in the task set is collected, and the task description data includes the task type, the task metadata, the task resource requirement and the task estimated duration;

[0017] The sequence and the dependency condition of each data processing task in the task set are extracted to obtain the task dependency relationship of the data processing tasks;

[0018] The task ID of the data processing task is taken as a node, the task dependency relationship is taken as a directed edge, a directed weighted graph is used to construct a task graph structure, and an adjacency list is used to store the task graph structure;

[0019] The task description data of the data processing task is used to define the node feature of the task graph structure, and based on the data processing task interaction quantity and the topological position of the data processing task in the task graph structure, the task importance is calculated by using the PageRank algorithm, the task importance is fused into the edge weight of the directed edge or the node attribute, and the task running network is obtained.

[0020] Further, the task importance is calculated by using the PageRank algorithm, and the following steps are executed:

[0021] A transition matrix is constructed, and each element in the matrix corresponds to the execution probability from one task to another task;

[0022] An importance vector is initialized, and the importance vectors of all nodes are set to 1 / A, A being the total number of tasks;

[0023] The importance vector is updated by left multiplication iteration, and a damping coefficient and a random jump factor are introduced in each iteration left multiplication calculation, and the iteration is stopped until the convergence condition is met, and the current importance vector is output as the task importance.

[0024] Further, the method further comprises: performing a circular dependency detection on the constructed task running network by using a depth-first search, triggering an alarm if a circular dependency is detected, updating the task graph structure and the task running network after a user corrects the task dependency relationship.

[0025] Further, based on the task description data, the task priority of each data processing task in the to-be-executed queue is calculated, and the following steps are performed:

[0026] Starting from the root task, based on the task description data of each data processing task, in combination with the task importance of the data processing task, a preset dynamic priority calculation formula is used to evaluate the task priority of each data processing task in the to-be-executed queue in real time.

[0027] Further, the system resource state is obtained, and based on the system resource state and the task priority, a preset scheduling algorithm is used to perform scheduling optimization on the to-be-executed queue to obtain a task scheduling sequence, and the following steps are performed:

[0028] The system resource state is collected in real time, and a reverse learning chaotic mapping adaptive particle swarm optimization model is used as the preset scheduling algorithm to establish a particle swarm optimization model;

[0029] Based on the system resource state and the task priority of each data processing task, the particle swarm optimization model is used to perform scheduling optimization on the to-be-executed queue, and a globally optimal task scheduling scheme is output as the task scheduling sequence.

[0030] Further, the particle swarm optimization model is used to perform scheduling optimization on the to-be-executed queue, and a globally optimal task scheduling scheme is output as the task scheduling sequence, and the following steps are performed:

[0031] The task scheduling scheme is used as a particle, and an initial particle swarm is randomly generated according to a reverse learning strategy, and the initial particle swarm is merged and screened to obtain an initial population;

[0032] An adaptability function value of each particle in the initial population is calculated by using an adaptability function;

[0033] The particle velocity and position are adjusted by using a chaotic mapping, and the inertia weight and learning factor of the particle swarm optimization model are adaptively adjusted;

[0034] The individual optimal solution and the global optimal solution of the population are iteratively updated, and when the iteration number reaches a preset iteration threshold or the adaptability improvement is less than a preset adaptability threshold, the iteration is terminated, and the task scheduling sequence corresponding to the global optimal solution is output.

[0035] Further, the initial particle swarm is randomly generated according to the reverse learning strategy, and the initial particle swarm is merged and screened to obtain the initial population, including:

[0036] N initial particles are randomly generated in the solution space to obtain an initial particle group;

[0037] The inverse particle of each particle in the initial particle group is calculated to obtain an inverse particle group;

[0038] The initial particle group and the inverse particle group are combined to obtain a candidate population containing 2N particles, and the fitness of each particle in the candidate population is calculated, and the first Q particles with the best fitness are selected to form an initial population.

[0039] Further, the task scheduling sequence is checked for task dependency, and after the check is passed, the data processing tasks are executed in turn according to the task scheduling sequence, and the system resource state is monitored in real time, and when a burst condition is triggered, the priority scheduling optimization is re-executed, and the following steps are executed:

[0040] According to the execution order of the data processing tasks in the task scheduling sequence, the preconditions required for the execution of each data processing task are verified in turn, if the preconditions required for the execution are met, the check is passed, otherwise the check fails;

[0041] After the check is passed, the data processing tasks are executed in turn according to the execution order;

[0042] The system resource state is monitored in real time, and when the system resource state triggers a preset burst condition interruption condition, the priority scheduling optimization is re-executed to determine a new task scheduling sequence;

[0043] After the data processing tasks are executed, the task description data corresponding to the data processing tasks is updated.

[0044] On the basis of the above-mentioned mesh data processing task scheduling method, the application also provides a mesh data processing task scheduling system, which comprises:

[0045] A task relationship configuration center is configured to configure the task description data and the task dependency relationship of the data processing tasks, and to store the topology data of the task running network;

[0046] A dynamic scheduling engine is configured to receive the task description data and the task dependency relationship synchronized by the task relationship configuration center to construct the task running network and the to-be-executed queue, and to update the task running network in real time based on the topology data of the task running network, and to calculate the task priority of the data processing tasks in real time, and to schedule and optimize the to-be-executed queue based on the system resource state and the task priority by using a preset scheduling algorithm;

[0047] A data access processing module is configured to receive the task execution instructions issued by the dynamic scheduling engine, to access a plurality of different types of data sources, and to call a relational database to store the data extracted from the different types of data sources;

[0048] A resource monitoring and feedback module is configured to collect system resource status and task execution data in real time, synchronize the system resource status and the task execution data to the dynamic scheduling engine, and send a burst condition alarm to the dynamic scheduling engine when the system resource status triggers a preset burst condition interrupt condition.

[0049] In summary, the beneficial effects of the present application are as follows:

[0050] The present application provides a mesh data processing task scheduling method, which first extracts the description data and the pre-post dependency relationship of the data processing task, constructs a task running network with a directed weighted graph, quantifies the association and attribute characteristics between tasks, and initializes the to-be-executed queue with a root task without pre-post dependency, to ensure that the first scheduled task is not blocked by dependency; then calculates the priority of each task based on the task description data, accurately associates the task attribute and the scheduling priority, and combines the real-time system resource status to optimize the to-be-executed queue through a preset scheduling algorithm to obtain a scheduling sequence, realizing accurate matching of task demand and resource supply. Finally, the scheduling sequence is checked one by one for task dependency legality, and the data is updated after the task is completed, and a dynamic closed loop is formed again under a burst condition to further correct the subsequent scheduling decision, continuously improving the accuracy and adaptability of scheduling in complex scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. For those skilled in the art, other drawings can also be obtained without creative labor on the premise of these drawings, and these are within the protection scope of the present application.

[0052] Figure 1 A flow chart of a mesh data processing task scheduling method according to the present application;

[0053] Figure 2 A scheduling optimization flow chart of a task scheduling queue according to the present application;

[0054] Figure 3 A module relationship diagram of a mesh data processing task scheduling system according to the present application;

[0055] Figure 4 A work flow chart of a mesh data processing task scheduling system according to the present application. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Unless otherwise specified, the present invention and the various features in the embodiments can be combined with each other, all of which are within the protection scope of the present invention.

[0057] Example 1: Refer to Figure 1 As shown, Figure 1 This is a schematic flowchart of a task scheduling method based on mesh data processing according to an embodiment of the present invention. The method of this embodiment includes:

[0058] S1: Extract the task description data and task dependencies of the data processing tasks in the task set, and construct a task execution network using a directed weighted graph; the task dependencies include pre-dependencies and post-dependencies;

[0059] S2: Add the data processing tasks in the task execution network to the execution queue according to the task dependency relationship, and take the data processing task without the preceding dependency relationship as the root task of the execution queue;

[0060] S3: Based on the task description data, calculate the task priority of each data processing task in the queue to be executed;

[0061] S4: Obtain the system resource status, and based on the system resource status and task priority, use a preset scheduling algorithm to optimize the scheduling of the queue to be executed, and obtain a task scheduling sequence;

[0062] S5: Perform task dependency verification on the task scheduling sequence. If the verification passes, execute the data processing tasks sequentially according to the task scheduling sequence. After the task is completed, update the task description data and monitor the system resource status in real time. If an emergency occurs, re-execute the priority scheduling optimization.

[0063] In this embodiment of the invention, task description data and pre- and post-dependencies are extracted first to construct a task execution network using a directed weighted graph. This clearly quantifies the relationships and attribute characteristics between tasks, avoiding incorrect selection of scheduling objects due to ambiguous relationship descriptions under complex dependencies, thus laying a topological foundation for accurate scheduling. Secondly, a root task without pre-dependencies is used as the initial object of the execution queue, ensuring that all tasks scheduled initially meet the legal execution prerequisite of no dependency blocking, reducing invalid scheduling attempts. Then, priority calculation based on task description data accurately associates task attributes (such as resource requirements and execution characteristics) with scheduling priorities, avoiding delays in critical tasks due to single priority determination in complex scenarios, and ensuring accurate scheduling ordering. Simultaneously, combining real-time system resource status and priorities, the execution queue is optimized using a preset scheduling algorithm to achieve precise matching of task requirements and resource supply, avoiding execution interruptions or efficiency losses caused by resource mismatches in complex scenarios. Finally, the legality of task dependencies is verified one by one in the scheduling sequence to ensure that each executed task meets the preconditions. After the task is completed, data is updated and re-optimized in case of emergencies, forming a dynamic closed loop. This further corrects subsequent scheduling decisions and continuously improves the accuracy and adaptability of scheduling in complex scenarios.

[0064] In this embodiment, step S1 extracts the task description data and task dependencies of the data processing tasks in the task set, and constructs a task execution network using a directed weighted graph, performing the following sub-steps:

[0065] S11: Collect task description data for each data processing task in the task set. The task description data includes task type, task metadata, task resource requirements, and estimated task duration. The task type includes any one of the following: data interface access, file access, database metadata collection, database data reading, stored procedure execution, and data statistics. Resource requirements include information such as the number of CPU cores and memory capacity, and are dynamically updated after the task runs.

[0066] S12: Extract the sequence and dependencies of each data processing task in the task set to obtain the task dependencies of the data processing tasks;

[0067] S13: Using the task ID of the data processing task as the node and the task dependency relationship as the directed edge, a directed weighted graph is used to construct the task graph structure, and an adjacency list is used to store the task graph structure.

[0068] S14: Define the node features of the task graph structure using the task description data of the data processing task, and calculate the task importance based on the task data interaction volume and the topological position of the data processing task in the task graph structure using the PageRank algorithm. Then, merge the task importance into the edge weight of the directed edge or the node attribute to obtain the task running network.

[0069] In this embodiment, the node features of each graph node in the graph structure are updated to construct a task feature matrix; the task feature matrix and the resource usage data of each virtual machine are input into the particle swarm optimization model to determine the target task scheduling scheme for directing each task to the corresponding virtual machine; the task feature matrix and the resource usage data of each virtual machine are used to initialize the particle swarm, and each particle in the particle swarm is defined by a corresponding potential task scheduling scheme, which includes the pre-allocation relationship between each task and each virtual machine.

[0070] The priority constraints between tasks are represented using a directed acyclic graph (DAG) model, i.e., G =<V ,E> Where V is the task set, E is the set of directed edges between tasks, representing the priority constraints between tasks. Indicates in the task The task can only be executed after the previous one is completed. Each task ,in, CPU cycles required to complete the task Enter the data size for the task.

[0071] For a known DAG model, the task The set of predecessor nodes is The set of successor nodes is The layer value to which each task in the task set belongs can be determined by the following formula:

[0072]

[0073] Indicates if task If there is no predecessor node, it is the first node; if there is a predecessor node, it is the next node. This indicates that the set is not empty.

[0074] In this embodiment, step S14 calculates the task importance using the PageRank algorithm and performs the following steps:

[0075] Construct the transition matrix M, where each element... This corresponds to the execution probability of moving from one task to another.

[0076] Initialize importance vector And set the importance vector of all nodes to 1 / A, where A is the total number of tasks. The importance vector is represented as:

[0077]

[0078] The importance vector is updated by iterative left multiplication. The formula for iterative left multiplication is as follows: .

[0079] In each iteration of the left multiplication calculation, a damping coefficient d (usually d is 0.85) and a random jump factor (1-d) / n are introduced. At this point, the importance vector is represented as: .

[0080] The iteration continues until the convergence condition is met, at which point the iteration stops and the current importance vector is output as the task importance. The task importance is represented as... The convergence condition is expressed as: , This is the preset convergence threshold.

[0081] Alternatively, the transpose matrix can be stored in row-major order. And perform parallel computation in blocks by row, the computation of each row can be represented as:

[0082]

[0083] in, This represents the importance score of the task after the (k+1)th iteration.

[0084] Furthermore, to avoid subsequent execution errors and ensure the smooth execution of tasks, the method also includes using depth-first search to detect circular dependencies in the constructed task execution network. If a circular dependency is detected, an alarm is triggered, and the task graph structure and task execution network are updated after the user corrects the task dependencies.

[0085] In step S2 of this embodiment, the task execution network is scanned to filter out "root tasks" without preceding tasks, which are then selected as the first batch of tasks to be executed. The root tasks are added to the queue to be executed according to the default initial priority (configurable basic weight) to trigger the first scheduling process, so as to avoid errors when executing the first task.

[0086] In this embodiment, step S3 calculates the task priority of each data processing task in the queue to be executed based on the task description data, and performs the following steps:

[0087] Starting from the root task, based on the task description data of each data processing task and combined with the task importance of the data processing task, the task priority of each data processing task in the queue to be executed is evaluated in real time using a preset dynamic priority calculation formula. The task priority is represented as Priority.

[0088] The specific formula for calculating the preset dynamic priority is as follows:

[0089]

[0090] in: This represents the historical average runtime of the task, reflecting the task execution efficiency. Task importance (calculated based on the association weights of tasks on the critical path of the task in the task execution network) reflects the impact of the task on the overall process. The importance of the task is calculated using the PageRank algorithm. Estimate resource consumption (such as CPU and memory usage) for tasks and prioritize scheduling tasks with low resource consumption; W1, W2, and W3 are configurable weight coefficients that are dynamically adjusted according to business scenarios (such as resource-sensitive or efficiency-sensitive).

[0091] The PageRank algorithm for calculating task importance evaluates task importance by analyzing the link structure between tasks. In this embodiment, the PageRank of each task is calculated iteratively, and the specific calculation process is as described in step S14 of the above embodiment.

[0092] In this embodiment, refer to Figure 2 As shown, in step S4, the system resource status is obtained, and based on the system resource status and task priority, the queue to be executed is optimized using a preset scheduling algorithm to obtain a task scheduling sequence, and the following steps are performed:

[0093] S41: Real-time acquisition of system resource status, and establishment of a particle swarm optimization model using the reverse learning chaotic mapping adaptive particle swarm algorithm as the preset scheduling algorithm.

[0094] The Inverse Learning Chaotic Map Adaptive Particle Swarm Optimization (OAPSO) algorithm is used to solve for the globally optimal task execution network. OAPSO, or Object-Oriented Adaptive Particle Swarm Optimization, is an improvement on Particle Swarm Optimization (PSO). Its core idea borrows from APSO's evolutionary state evaluation, system adaptive parameters (inertia weights, local and global acceleration), and elite learning strategy, incorporating object-oriented concepts. This allows particles to dynamically adjust themselves based on their own and the group's experience while searching for the optimal solution, achieving fast global convergence. The algorithm mainly includes the following processing steps:

[0095] Reverse learning initialization: In the particle swarm initialization stage, a reverse learning strategy is introduced to generate initial particles, expand the population diversity, and avoid premature convergence of the algorithm.

[0096] Chaotic Map Update: Utilize chaotic maps (such as Logistic maps) to dynamically adjust the particle velocity and position update formulas, thereby enhancing the algorithm's global search capability in complex task networks.

[0097] Adaptive parameter adjustment: Based on the population evolution state, the inertia weight and learning factor are adaptively adjusted to balance the algorithm's exploration (global search) and development (local optimization) capabilities, optimize task-resource matching, and achieve efficient resource utilization and load balancing.

[0098] The modeling process of the particle swarm optimization model specifically includes the following:

[0099] Particle Objects: Each particle is encapsulated as a "task scheduling object," containing attributes such as: position x (representing the allocation strategy between tasks and server nodes), velocity v (the direction and magnitude of task migration between nodes), individual optimal solution p_Best (the object's own historical best scheduling scheme), and fitness evaluation method (built-in cost calculation logic). ParticleSize corresponds to the number of candidate scheduling schemes in a distributed scenario. Considering the core objectives of task scheduling (reducing total execution time, balancing load, and prioritizing high-priority tasks), a suitable fitness function is selected to determine the merits of the scheduling scheme.

[0100] Fitness function: The fitness function is the "guiding principle" for OAPSO optimization. It needs to take into account the core objectives of task scheduling (reducing total execution time, balancing load, and prioritizing high-priority tasks), and is defined as follows:

[0101] fitness = α × (1 / total execution time) + β × (1 / load standard deviation) + γ × (high priority task completion rate)

[0102] Among them, total execution time: the total time taken for all tasks from start to finish (the smaller the better); load standard deviation: the standard deviation of CPU utilization of each executor (the smaller the better, the more balanced the load); high priority task completion rate: the percentage of high priority tasks completed within the specified time (the larger the better); α, β, γ are weighting coefficients (which can be dynamically adjusted according to the business scenario, such as γ having a higher weight in core task scenarios).

[0103] Parameter initialization: Adaptive parameter baseline value: Initial value of inertia weight Initial value of local acceleration factor Initial value of global acceleration factor (This allows for future dynamic adjustments);

[0104] Adaptive parameter adjustment: Dynamically adjust the inertia weight w, local acceleration factor c1, and global acceleration factor c2 based on the real-time task queue and executor status (as described in the strategy above).

[0105] Exploration control parameters: initial exploration level T covers diverse combinations of distributed node clusters, and chaotic perturbation coefficient (used to enhance search diversity).

[0106] Elite Particle Pool: Reserved storage space for recording the top Q best-performing elite particles in the population (foundation of OAPSO elite learning strategy).

[0107] Initial fitness calculation: Each particle object calculates its fitness f using a built-in method, initializes f to the individual's best value p_BestValue, and records the corresponding position as p_Best.

[0108] Global Optimum and Evolutionary State Initialization: Select the population optimum g_BestValue and position g_Best from all particles, and update the elite particle pool synchronously; initialize the evolutionary state evaluation index (such as the population fitness standard deviation, reflecting the current search diversity), and set the current global optimum as the historical optimum BestValue.

[0109] Iterative optimization: Particles update their velocity / position (adjusting task allocation), calculate fitness, and update pbest (their own best) and gbest (global best).

[0110] S42: Based on system resource status and the task priorities of each data processing task, the particle swarm optimization model is used to optimize the scheduling of the queue to be executed, and the globally optimal task scheduling scheme is output as the task scheduling sequence. After the task scheduling sequence is determined, the tasks can be assigned to suitable servers for execution.

[0111] In this embodiment, step S41 uses a particle swarm optimization model to optimize the scheduling of the queue to be executed, and outputs the globally optimal task scheduling scheme as the task scheduling sequence, and then performs the following steps:

[0112] Using the task scheduling scheme as particles, an initial particle swarm is randomly generated according to the reverse learning strategy, and the initial particle swarm is merged and filtered to obtain the initial population.

[0113] The fitness function value of each particle in the initial population is calculated using the fitness function.

[0114] By utilizing chaotic mappings (such as Logistic mappings) to adjust particle velocity and position, the algorithm's global search capability in complex task networks is enhanced. Furthermore, the inertia weight and learning factor of the particle swarm optimization model are adaptively adjusted to balance the algorithm's exploration (global search) and development (local optimization) capabilities, optimizing task-resource matching and achieving efficient resource utilization and load balancing. Specifically, the inertia weight w has an initial value of 0.7. When the population fitness standard deviation > 0.2 (high diversity), w = 0.9 (enhancing global search); when the standard deviation < 0.1 (convergence), w = 0.4 (strengthening local optimization).

[0115] Local acceleration factor c1, global acceleration factor c2: Initially c1=2.0, c2=2.0; in the early stage of iteration c1>c2 (emphasizing individual experience), in the later stage c2>c1 (emphasizing group experience).

[0116] The process iteratively updates the individual particle optimal solution and the global optimal solution of the population. When the number of iterations reaches a preset iteration threshold (e.g., 50 times to ensure real-time performance) or the fitness improvement is less than a preset fitness threshold (e.g., 0.1%), the iteration terminates and the task scheduling sequence corresponding to the global optimal solution is output.

[0117] Specifically, an initial particle swarm is randomly generated based on a reverse learning strategy, and the initial particle swarm is merged and filtered to obtain an initial population, which includes:

[0118] N initial particles are randomly generated in the solution space to obtain the initial particle swarm. Each particle (e is the dimension of the solution space), and .

[0119] Calculate each particle in the initial particle swarm antiparticle This yields the inverse particle swarm:

[0120] ;

[0121] Antiparticle The components can be defined as: .in, and These are the minimum and maximum values ​​of the node numbers, respectively. For example, the node numbers are {1, 2, 3, 4}. =1, =4), if a task is assigned to node 2 ( = 2), then its reverse node is 1+4-2=3.

[0122] The initial particle swarm X and the reverse particle swarm Merge the particles to obtain a candidate population containing 2N particles, calculate the fitness of each particle in the candidate population, and select the top Q particles with the best fitness (i.e., elite particles) to form the initial population.

[0123] In this embodiment, step S5 performs task dependency verification on the task scheduling sequence. After the verification passes, the data processing tasks are executed sequentially according to the task scheduling sequence, and the system resource status is monitored in real time. If an emergency occurs, priority scheduling optimization is re-executed, and the following steps are performed:

[0124] S51: According to the order of execution of data processing tasks in the task scheduling sequence, check whether the preconditions required for the execution of each data processing task are met in turn. If the preconditions required for execution are met, the check passes; otherwise, the check fails.

[0125] S52: After the verification is passed, the data processing tasks will be executed sequentially according to the order of execution;

[0126] S53: Monitor system resource status in real time. When the system resource status triggers a preset emergency interruption condition, re-execute priority scheduling optimization and determine a new task scheduling sequence.

[0127] S54: After the data processing task is completed, update the task description data corresponding to the data processing task.

[0128] In this embodiment, during the task execution phase, dependency checks and task execution are performed first to verify whether the preconditions required for task execution are met. If the conditions are met, task execution proceeds. During execution, if an unexpected situation occurs, a special handling process is quickly initiated to resolve issues such as faults and resource conflicts. If no unexpected situation occurs, the task continues to proceed according to optimized scheduling. After the unexpected situation is handled, feedback is sent to the task scheduling sequence optimization stage for adjustment and correction of subsequent task scheduling strategies.

[0129] During feedback adjustments, we can first select the highest-priority uninvoked tasks in each task graph (i.e., the task execution network) to form a candidate scheduling set. Then, we select the task with the highest priority in the scheduling set and, based on the optimal scheduling machine derived from the particle swarm optimization algorithm, schedule the subtasks to their corresponding positions for execution. We repeat the above process until all tasks are completed.

[0130] Finally, this embodiment continuously focuses on resource monitoring and feedback optimization, monitoring resource usage status in real time, collecting task execution data, and continuously optimizing task relationship configuration, priority calculation, and scheduling algorithms based on feedback results. This forms a closed loop of planning-execution-monitoring-optimization, ensuring that task scheduling continuously adapts to actual needs and improving overall execution efficiency. Current resource usage information, including bandwidth, computing resources (e.g., CPU utilization), and storage resources (e.g., disk space usage), is periodically obtained from each server. For example, a distributed monitoring system or resource management tool (such as Prometheus or Nagios) can be used for data collection to ensure the real-time nature and accuracy of the data.

[0131] Example 2: In this example, referring to Figure 3 As shown, based on the above embodiment 1, a mesh data processing task scheduling system is also provided, the system comprising:

[0132] The task relationship configuration center is used to configure the task description data and task dependencies of data processing tasks, as well as store the topology data of the task running network.

[0133] The dynamic scheduling engine receives task description data and task dependencies synchronized from the task relationship configuration center to build a task execution network and a queue to be executed. It updates the task execution network in real time based on the task execution network topology data, calculates the task priority of data processing tasks in real time, and optimizes the scheduling of the queue to be executed based on the system resource status and task priority using a preset scheduling algorithm.

[0134] The data access and processing module is used to receive task execution instructions issued by the dynamic scheduling engine, access various data sources of different types, and call the relational database to store the data extracted from different data sources.

[0135] The resource monitoring and feedback module is used to collect system resource status and task execution data in real time, synchronize the system resource status and task execution data to the dynamic scheduling engine, and send an emergency alarm to the dynamic scheduling engine when the system resource status triggers the preset emergency interruption conditions.

[0136] In this embodiment, the task scheduling system uses a dynamic scheduling engine as its central hub, with each module working collaboratively. The specific collaborative content is as follows:

[0137] 1. Task Input: The task relationship configuration center sorts out information such as task dependencies and basic configurations, and accurately inputs it into the dynamic scheduling engine to lay a solid logical foundation for scheduling and execution, defining what to do and in what order.

[0138] 2. Task Execution: The data access and processing module receives scheduling instructions from the dynamic scheduling engine and executes task operations; it also synchronously feeds back execution data (such as task progress and results) to the engine, building a closed loop of "instruction-execution-feedback" to ensure accurate and controllable scheduling.

[0139] 3. Resource Management: The resource monitoring and feedback module continuously collects resource status and task data, transmitting it to the dynamic scheduling engine in real time. When the data reflects resource shortages or task deviations, the engine is triggered to initiate scheduling optimization, adjusting execution strategies to form a closed-loop management system of monitoring-optimization-execution, ensuring the system efficiently adapts to resource and task requirements. All modules collaborate around the dynamic scheduling engine, using a two-way closed-loop mechanism to achieve precise task scheduling and intelligent resource management, guaranteeing efficient system operation.

[0140] The task relationship configuration center is the cornerstone of the system's stable operation. It provides users with a user-friendly interface, allowing them to flexibly configure task dependencies and define the order of task execution. It also allows users to set basic task attributes, such as task type and initial resource requirements. This center stores the topology data of the task execution network, which forms the foundation for task scheduling throughout the system. Furthermore, the task relationship configuration center boasts powerful real-time configuration information updates and version management capabilities, adapting to constantly changing business needs and ensuring that the system can schedule tasks based on the latest and most accurate configuration information at any time.

[0141] The dynamic scheduling engine is the core "power source" of the system. It deeply integrates a dynamic priority calculation model with an improved OAPSO algorithm, constructing and updating the task execution network in real time based on data provided by the task relationship configuration center. During task execution, it calculates the dynamic priority of tasks in real time, driving the OAPSO algorithm to optimize task-resource scheduling. Simultaneously, this engine closely integrates information from the resource monitoring module, enabling rapid response and dynamic adjustment of scheduling strategies when system resource status or task execution changes. This flexibly addresses various unforeseen circumstances, ensuring efficient and orderly task execution even in complex environments.

[0142] The data access and processing module acts as the system's data entry point, boasting strong compatibility and supporting data extraction from various data sources. Whether it's structured data from relational databases (such as MySQL and Oracle), semi-structured or unstructured data from non-relational databases (such as MongoDB and Redis), data provided by other systems through API interfaces, or data stored on file servers, all can be efficiently accessed to meet diverse data processing needs.

[0143] In the data processing stage, the multi-source data access and processing module can call stored procedures in relational databases to complete a series of complex data processing operations such as data transformation, cleaning, and aggregation. This function is deeply integrated with the task execution process, ensuring that the data processing logic can be executed efficiently during task execution, providing accurate and high-quality data support for the task, and guaranteeing the data processing capabilities and efficiency of the entire system.

[0144] The resource monitoring and feedback module acts as the system's "health monitor" and "information feedback device." It collects real-time data on system resource status, including the usage of critical resources such as CPU, memory, storage, and network, while also gathering task execution data, such as task runtime and actual resource consumption. This data provides crucial input for dynamic priority calculation and OAPSO algorithm optimization, enabling the system to make more rational scheduling decisions based on real-time data. Furthermore, when resource status exceeds pre-set thresholds (e.g., CPU utilization > 80%), it immediately triggers the dynamic scheduling engine for rescheduling, achieving closed-loop resource management and ensuring the system maintains efficient and stable operation even with limited resources.

[0145] Through the organic integration of the aforementioned layered architecture and functional modules, this task scheduling system forms a complete closed-loop system encompassing task configuration, scheduling, execution, resource monitoring, and feedback. The close collaboration and cooperation between each layer and module effectively improves the accuracy of task scheduling, resource adaptability, and emergency response capabilities in complex scenarios, providing reliable technical support for various complex business scenarios.

[0146] Reference Figure 4 As shown, the task scheduling system in this embodiment focuses on efficient task scheduling and dynamic response, ensuring the orderliness and adaptability of task execution. Its specific workflow is as follows:

[0147] First, the task relationships are configured, clarifying the logical connections between tasks such as their sequence and dependencies, and identifying which tasks need to be completed first, thus laying a solid foundation for subsequent processes. Based on this, a runtime network is constructed, presenting the tasks in a network structure, clearly showing the task flow paths and interaction relationships, and making the overall execution framework visible.

[0148] Then, the task dependency network is scanned, and root tasks without prerequisite tasks are selected as the first batch of tasks to be run. The root tasks are added to the queue to be run according to the default initial priority (configurable base weight), triggering the first scheduling process.

[0149] Next, we move into the dynamic optimization phase, where dynamic priorities are calculated. Based on factors such as task attributes, resource requirements, and time requirements, task priorities are evaluated in real time to ensure that critical tasks are prioritized.

[0150] Subsequently, the OAPSO algorithm was used to optimize scheduling, and the algorithm was used to intelligently plan task execution sequences and resource allocation to improve the scientific nature and efficiency of scheduling.

[0151] During the task execution phase, dependency checks and task execution are carried out first to verify whether the prerequisites for task execution are met. If the conditions are met, task execution is promoted.

[0152] At the end of the process, resource monitoring and feedback optimization continue to exert their effects. Real-time monitoring of resource usage status and collection of task execution data are used to continuously optimize task relationship configuration, priority calculation, and scheduling algorithms based on feedback results, forming a closed loop of "planning-execution-monitoring-optimization." This ensures that task scheduling continuously adapts to actual needs and improves overall execution efficiency. Current resource usage information, including bandwidth, computing resources (e.g., CPU utilization), and storage resources (e.g., disk space usage), is periodically obtained from each server. For example, distributed monitoring systems or resource management tools (such as Prometheus or Nagios) can be used for data collection to ensure the real-time nature and accuracy of the data.

[0153] The system in this embodiment has the following technical advantages:

[0154] Improved scheduling accuracy: The dynamic priority model integrates multi-dimensional task features (runtime, resources, importance) and uses the OAPSO algorithm to optimize task-resource matching, significantly improving the accuracy of scheduling complex mesh tasks and ensuring that critical tasks are executed first.

[0155] Adaptive to unforeseen circumstances: Real-time resource monitoring and dynamic scheduling triggering mechanism can quickly adjust scheduling strategies in the face of sudden changes in data volume, resource fluctuations, etc., to avoid resource waste and overload, and ensure the overall task completion rate.

[0156] Algorithm innovation advantages: The OAPSO algorithm incorporates reverse learning, chaotic mapping, and adaptive parameter adjustment, enhancing its optimization capabilities in complex task networks. Compared to the traditional particle swarm optimization algorithm, it achieves faster convergence and better scheduling results.

[0157] High efficiency through multi-component collaboration: The task relationship configuration center, dynamic scheduling engine, and multi-source processing modules work together to adapt to diverse data sources and complex task logic, supporting efficient scheduling in scenarios such as big data processing and multi-database collaboration.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for scheduling a mesh-based data processing task, characterized in that, The method comprises the following steps: extracting task description data and task dependency relationship of data processing tasks in a task set, and constructing a task running network in the form of a directed weighted graph; the task dependency relationship comprises a preceding dependency relationship and a subsequent dependency relationship; adding data processing tasks in the task running network to a to-be-executed queue according to the task dependency relationship, and taking a data processing task without a preceding dependency relationship as a root task of the to-be-executed queue; calculating task priorities of the data processing tasks in the to-be-executed queue based on the task description data, which comprises: calculating task importance by using a PageRank algorithm; starting from the root task, evaluating the task priority of each data processing task in the to-be-executed queue in real time by using a preset dynamic priority calculation formula based on the task description data of each data processing task and the task importance of the data processing task; obtaining a system resource state, and performing scheduling optimization on the to-be-executed queue by using a preset scheduling algorithm based on the system resource state and the task priority, to obtain a task scheduling sequence; performing task dependency checking on the task scheduling sequence, and executing the data processing tasks in sequence according to the task scheduling sequence after the checking is passed, updating the task description data and monitoring the system resource state in real time after the execution of the tasks is completed, and re-executing the priority scheduling optimization when a sudden situation occurs.

2. The mesh-based data processing task scheduling method according to claim 1, characterized in that, The method of extracting task description data and task dependency relationship of data processing tasks in a task set, and constructing a task running network in the form of a directed weighted graph comprises the following steps: collecting task description data of each data processing task in the task set, wherein the task description data comprises a task type, task metadata, task resource demand and task estimated duration; extracting the sequence and dependency condition of each data processing task in the task set to obtain the task dependency relationship of the data processing tasks; constructing a task graph structure by taking the task ID of the data processing task as a node and the task dependency relationship as a directed edge, and storing the task graph structure by using an adjacency list; defining the node features of the task graph structure by using the task description data of the data processing task, calculating the task importance based on the task data interaction amount and the topological position of the data processing task in the task graph structure, and fusing the task importance into the edge weight of the directed edge or the node attribute to obtain the task running network.

3. The mesh-based data processing task scheduling method according to claim 2, wherein, The method of calculating the task importance by using a PageRank algorithm comprises the following steps: constructing a transition matrix, wherein each element in the matrix corresponds to the execution probability from one task to another task; initializing an importance vector, and setting the importance vector of all nodes as 1 / A, wherein A is the total number of tasks; iteratively performing left multiplication to update the importance vector, and introducing a damping coefficient and a random jump factor in each iteration of the left multiplication, and stopping the iteration when a convergence condition is met, and outputting the current importance vector as the task importance.

4. The mesh-based data processing task scheduling method according to claim 2, wherein, The method further comprises the following steps: performing loop dependency detection on the constructed task running network by using a depth-first search, and triggering an alarm if loop dependency is detected, and updating the task graph structure and the task running network after the user corrects the task dependency relationship.

5. The mesh-based data processing task scheduling method according to claim 1, wherein, The system resource state is acquired, and based on the system resource state and the task priority, a preset scheduling algorithm is used to perform scheduling optimization on the to-be-executed queue to obtain a task scheduling sequence, and the following steps are performed: The system resource state is acquired in real time, and a reverse learning chaotic mapping adaptive particle swarm optimization model is used as the preset scheduling algorithm to establish a particle swarm optimization model; Based on the system resource state and the task priority of each data processing task, the particle swarm optimization model is used to perform scheduling optimization on the to-be-executed queue, and a globally optimal task scheduling scheme is output as the task scheduling sequence.

6. The mesh-based data processing task scheduling method according to claim 5, characterized in that, The particle swarm optimization model is used to perform scheduling optimization on the to-be-executed queue, and a globally optimal task scheduling scheme is output as the task scheduling sequence, and the following steps are performed: The task scheduling scheme is used as a particle, and an initial particle swarm is randomly generated according to a reverse learning strategy, and the initial particle swarm is merged and screened to obtain an initial population; An adaptive function is used to calculate the adaptive function value of each particle in the initial population; The chaotic mapping is used to adjust the particle speed and position, and the inertia weight and learning factor of the particle swarm optimization model are adaptively adjusted; The particle individual optimal solution and the population global optimal solution are iteratively updated, and when the iteration number reaches a preset iteration threshold or the fitness improvement is less than a preset fitness threshold, the iteration is terminated, and the task scheduling sequence corresponding to the global optimal solution is output.

7. The mesh-based data processing task scheduling method according to claim 6, characterized in that, The initial particle swarm is randomly generated according to the reverse learning strategy, and the initial particle swarm is merged and screened to obtain the initial population, including: N initial particles are randomly generated in the solution space to obtain an initialization particle swarm; The reverse particle of each particle in the initial particle swarm is calculated to obtain a reverse particle swarm; The initial particle swarm and the reverse particle swarm are merged to obtain a candidate population containing 2N particles, and the fitness of each particle in the candidate population is calculated, and the first Q particles with the best fitness are selected to form the initial population.

8. The mesh-based data processing task scheduling method of claim 1, wherein, The task scheduling sequence is verified for task dependency, and after the verification passes, the data processing tasks are executed in sequence according to the task scheduling sequence, and the system resource state is monitored in real time, and when a burst condition is triggered, the priority scheduling optimization is re-executed, and the following steps are performed: According to the to-be-executed order of the data processing tasks in the task scheduling sequence, the preconditions required for the execution of each data processing task are verified in sequence, and if the preconditions required for the execution are met, the verification passes, otherwise the verification fails; After the verification passes, the data processing tasks are executed in sequence according to the to-be-executed order; The system resource state is monitored in real time, and when the system resource state triggers a preset burst condition interruption condition, the priority scheduling optimization is re-executed to determine a new task scheduling sequence; After the data processing task is executed, the task description data corresponding to the data processing task is updated.

9. A mesh data processing task scheduling system based on the mesh data processing task scheduling method according to any one of claims 1-8, characterized in that, It includes: A task relationship configuration center is configured to configure the task description data and the task dependency relationship of the data processing task, and to store the topology data of the task running network; The dynamic scheduling engine is configured to receive task description data and task dependency relationship from the task relationship configuration center, construct a task running network and a to-be-executed queue, update the task running network based on network topology data in real time, calculate task priorities of data processing tasks in real time, and schedule and optimize the to-be-executed queue based on system resource states and the task priorities by using a preset scheduling algorithm. The data access processing module is configured to receive task execution instructions issued by the dynamic scheduling engine, access multiple different types of data sources, and call a relational database to store data extracted from the different types of data sources. The resource monitoring and feedback module is configured to collect system resource states and task execution data in real time, synchronize the system resource states and the task execution data to the dynamic scheduling engine, and send a burst condition alarm to the dynamic scheduling engine when a preset burst condition interruption condition is triggered by the system resource states.

Images