Civil aviation industry-level data service platform task scheduling method and device, equipment and medium

Abstract

The application provides a civil aviation industry-level data service platform task scheduling method and device, equipment and medium, relates to the technical field of task scheduling, and the method comprises the following steps: determining the task type of each task in a task set to be scheduled; if it is determined that the resource requirement of any one computing-intensive analysis task exceeds a preset resource threshold and the computing-intensive analysis task is decomposable, the computing-intensive analysis task is decomposed into a plurality of subtasks; a to-be-scheduled queue containing all routine data processing tasks, all subtasks and all undecomposed computing-intensive analysis tasks is constructed, and the attributes of the tasks in the to-be-scheduled queue are recorded; based on the attributes of the tasks, the real-time available resource state and topology information of the computing nodes in the civil aviation industry-level data service platform, the tasks are dynamically distributed to a plurality of computing nodes, so that the problems of resource contention, damaged performance of routine data processing tasks, low efficiency of computing-intensive analysis tasks and the like are solved.

Description

Technical Field

[0001] This application relates to the field of task scheduling technology, and provides a task scheduling method, apparatus, equipment and medium for a civil aviation industry-level data service platform. Background Technology

[0002] The civil aviation industry-level data service platform is a key infrastructure of the modern civil aviation operation system. Its core functions include aggregating massive amounts of heterogeneous data (such as flight plans, real-time dynamics, weather reports, radar data, passenger information, etc.) from airlines, airports, air traffic control and other parties; performing data governance (such as cleaning, verification and standardization) and storage on this heterogeneous data; and providing data forwarding, sharing and statistical analysis services to various users (such as airline operation control centers, airport command centers, air traffic control units, etc.) as needed.

[0003] The civil aviation industry-level data service platform simultaneously runs two main types of tasks: routine data processing tasks and computationally intensive analysis tasks. The platform employs common resource scheduling strategies (such as queue-based priority scheduling, simple resource threshold scheduling, or the default intelligent scheduler of cloud platform / container orchestration systems like Kubernetes) for resource allocation and node selection.

[0004] When a large, computationally intensive analytical task is scheduled for execution, it may consume significant amounts of CPU, memory, or network bandwidth resources over a long period. Simultaneously running routine data processing tasks may experience significantly increased latency or even timeouts due to insufficient resources. Civil aviation industry-level data service platforms typically schedule large, computationally intensive analytical tasks as a whole, resulting in excessively long queuing times and low execution efficiency. Summary of the Invention

[0005] This application provides a task scheduling method, apparatus, equipment, and medium for a civil aviation industry-level data service platform, which is used to solve problems such as resource contention, performance degradation of routine data processing tasks, and low efficiency of computationally intensive analysis tasks in the civil aviation industry-level data service platform.

[0006] Firstly, this application provides a task scheduling method for a civil aviation industry-level data service platform, including:

[0007] Obtain the set of tasks to be scheduled, and the real-time available resource status and topology information of computing nodes in the civil aviation industry-level data service platform;

[0008] Determine the task type for each task in the task set; the task type includes routine data processing tasks and computationally intensive analysis tasks;

[0009] If it is determined that the resource requirement of any computationally intensive analysis task exceeds a preset resource threshold and the computationally intensive analysis task is decomposable, then the computationally intensive analysis task is decomposed into multiple subtasks.

[0010] Construct a scheduling queue containing all routine data processing tasks, all subtasks, and all undecomposed computationally intensive analysis tasks, and record the attributes of each task in the scheduling queue; the attributes include task type, priority, resource requirements, SLA requirements, and dependencies between tasks / subtasks.

[0011] Based on the attributes of each task, the real-time available resource status, and the topology information, each task in the scheduling queue is dynamically allocated to multiple computing nodes.

[0012] Optionally, the step of determining the task type of each task in the task set includes:

[0013] If no historical execution data is pre-stored in the local database, the task type of each task is determined based on the metadata of each task in the task set; the metadata includes type tags, resource requirements, and service level agreements.

[0014] If historical execution data is pre-stored in the local database, the task type of each task is determined based on the historical execution data of similar tasks in the task set; the historical execution data includes historical resource consumption data, historical execution time, and number of failure retries.

[0015] Optionally, the step of determining the task type of each task based on the metadata of each task in the task set includes:

[0016] Based on whether the type label is realtime_forwarding, or whether the resource requirement is greater than a preset resource threshold, or whether the maximum latency in the service level agreement is greater than a preset time, the task type of the corresponding task is determined to be either a computationally intensive analysis task or a routine data processing task.

[0017] Optionally, the step of determining the task type of each task based on the historical execution data of similar tasks in the task set includes:

[0018] The task type is determined as either a computationally intensive analysis task or a routine data processing task based on whether the duration of CPU utilization exceeding 80% for the same type of task is greater than a preset duration, or whether the historical execution duration of the same type of task exceeds a preset execution duration, or whether the number of failed retries for the same type of task within a preset time period is greater than a preset number.

[0019] Optionally, the resource requirements include the requested number of CPU cores, the requested RAM size, the requested GPU type and number of GPUs, and the estimated runtime; the step of determining that the resource requirements of any compute-intensive analysis task exceed a preset resource threshold includes:

[0020] Determine that the number of CPU cores requested is greater than a preset CPU threshold;

[0021] Alternatively, determine that the requested RAM size is greater than a preset RAM threshold;

[0022] Alternatively, determine that the number of GPUs for the target type of the request is greater than the preset number of GPUs corresponding to the target type;

[0023] Alternatively, determine that the expected runtime is greater than the preset runtime.

[0024] Optionally, the step of decomposing the computationally intensive analysis task into multiple sub-tasks includes:

[0025] The dataset that the computationally intensive analysis task needs to process is split into multiple data subsets;

[0026] Generate multiple subtasks with the same processing logic as the computationally intensive analysis task; each subtask is used to independently process a different subset of data.

[0027] Optionally, the step of decomposing the computationally intensive analysis task into multiple sub-tasks includes:

[0028] The computationally intensive analysis task is decomposed into multiple serially dependent subtasks by executing the steps in the order they are performed.

[0029] Optionally, the step of dynamically allocating each task in the scheduling queue to multiple computing nodes based on the attributes of each task, the real-time available resource status, and the topology information includes:

[0030] Each task in the queue to be scheduled is treated as an intelligent agent;

[0031] Each agent selects a computing node for task scheduling based on its current state information; the current state information includes the attributes of the corresponding task, the real-time available resource status, and the topology information.

[0032] A multi-agent learning algorithm is used to train and iteratively optimize the task scheduling strategy based on historical experience until the reward function is maximized; the historical experience includes the previous state information, the previous selected computing node, and the previous reward.

[0033] Optionally, each intelligent agent selects a computing node for task scheduling based on its current state information, including:

[0034] Each agent selects a computing node for task scheduling based on its current state information and hard constraints; the hard constraints include that all routine data processing tasks must meet SLA requirements and that tasks cannot be scheduled to nodes that do not have the required hardware.

[0035] Optionally, the reward function is constructed based on several optimization objectives:

[0036] All routine data processing tasks meet SLA requirements;

[0037] Minimize the overall completion time of all subtasks and all undecomposed computationally intensive analysis tasks;

[0038] Maximize the overall resource utilization of the civil aviation industry-level data service platform;

[0039] Minimize the load imbalance among the computing nodes of the civil aviation industry-level data service platform;

[0040] Minimize the resource consumption cost of the civil aviation industry-level data service platform.

[0041] Optionally, the reward function is as follows:

[0042]

[0043] Among them, SLA_Score t This represents the score for all routine data processing tasks meeting SLA requirements within time step t; Makespan t The overall completion time for all subtasks and all undecomposed computationally intensive analysis tasks; This refers to the overall resource utilization rate; A metric for measuring the load imbalance between computing nodes; The resource consumption cost associated with this scheduling action; This is a penalty for serious violations of non-hard constraints resulting from this task scheduling. These are the weight factors corresponding to each optimization objective.

[0044] Optionally, the SLA_Score t The calculation formula is as follows:

[0045]

[0046] Where N is the total number of tasks within time step t. Δt iIt is the delay time of the i-th task;

[0047] It is a function based on a delay time, defined as follows:

[0048]

[0049] in, Δt i It is the delay time of the i-th task. Δt SLA This is the maximum delay time required by the SLA. Δt max It is the maximum delay tolerance time.

[0050] Secondly, this application provides a task scheduling device for a civil aviation industry-level data service platform, comprising:

[0051] The acquisition module is used to acquire the set of tasks to be scheduled, the real-time available resource status and topology information of computing nodes in the civil aviation industry-level data service platform;

[0052] An analysis module is used to determine the task type of each task in the task set; the task type includes routine data processing tasks and computationally intensive analysis tasks.

[0053] The task decomposition module is used to decompose the computationally intensive analysis task into multiple sub-tasks if it is determined that the resource requirement of any computationally intensive analysis task exceeds a preset resource threshold and the computationally intensive analysis task is decomposable.

[0054] The building module is used to construct a scheduling queue containing all routine data processing tasks, all subtasks, and all undecomposed computationally intensive analysis tasks, and to record the attributes of each task in the scheduling queue; the attributes include task type, priority, resource requirements, SLA requirements, and dependencies between tasks / subtasks.

[0055] The scheduling module is used to dynamically allocate each task in the scheduling queue to multiple computing nodes based on the attributes of each task, the real-time available resource status, and the topology information.

[0056] Thirdly, this application provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the task scheduling method of the civil aviation industry-level data service platform described in the first aspect.

[0057] Fourthly, this application provides a computer-readable storage medium storing a computer program, on which a processor executes the computer program to implement the task scheduling method of the civil aviation industry-level data service platform described in the first aspect.

[0058] Compared with the prior art, the beneficial effects of the embodiments of this application are as follows:

[0059] 1. Improve the execution efficiency of computationally intensive analysis tasks.

[0060] By decomposing computationally intensive analysis tasks into multiple independently executable subtasks, it is possible to effectively prevent tasks from slowing down the overall progress due to insufficient resources. In particular, when the resource requirements of a task exceed a preset threshold, the computational burden on individual nodes can be reduced by decomposition, thereby improving the execution efficiency of computationally intensive analysis tasks.

[0061] 2. Guarantee service quality

[0062] By recording the attributes of each task, including Service Level Agreement (SLA) requirements, tasks can be prioritized and scheduled based on these requirements. This ensures that routine data processing tasks meet SLA requirements and that routine data processing tasks perform in accordance with the expected quality of service standards.

[0063] 3. Reduce dependency conflicts between tasks / subtasks

[0064] During scheduling, task dependencies are explicitly recorded, taking into account dependencies between tasks or subtasks, thus avoiding scheduling conflicts or deadlocks. Clear management of task dependencies effectively controls the order of task execution and reduces execution delays caused by dependency conflicts.

[0065] 4. Optimize resource utilization

[0066] This method can classify tasks based on task type (such as routine data processing tasks and computationally intensive analysis tasks). In particular, for computationally intensive analysis tasks, it can dynamically decompose them into multiple sub-tasks according to their resource requirements. By acquiring the available resource status and topology information of computing nodes in real time, task scheduling can be performed in a dynamic environment based on the resource requirements, priority, and other attributes of each task, and dynamic allocation can be performed among multiple computing nodes to avoid waste or overload of computing resources and ensure optimal resource utilization. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0068] Figure 1 A schematic diagram of the computing device structure of the hardware operating environment involved in the embodiments of this application;

[0069] Figure 2 A flowchart illustrating a task scheduling method for a civil aviation industry-level data service platform provided in this application embodiment;

[0070] Figure 3 Another flowchart illustrating the task scheduling method of the civil aviation industry-level data service platform provided in this application embodiment;

[0071] Figure 4 A schematic diagram of the task scheduling device of the civil aviation industry-level data service platform provided in this application embodiment.

[0072] The diagram is labeled as follows: 101-Processor, 102-Communication bus, 103-Network interface, 104-User interface, 105-Memory. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, other embodiments obtained by those skilled in the art without creative effort are all within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0074] Civil aviation industry-level data service platforms typically operate in cloud or hybrid cloud environments, utilizing virtualization or containerization technologies to deploy various data processing services. The platform simultaneously runs two main types of tasks: routine data processing tasks and computationally intensive analysis tasks. These are described below:

[0075] Routine data processing tasks are numerous and typically have high latency requirements (e.g., needing to be completed within seconds or minutes), but the resource consumption of a single task is relatively low. Routine data processing tasks can include receiving and verifying real-time flight dynamic data, format conversion and rapid forwarding according to subscription rules, parsing and distributing meteorological reports, statistics of basic operational data, and simple alarms, etc.

[0076] Computationally intensive analytical tasks are relatively few in number, but a single task may require significant computing resources and a long processing time (potentially tens of minutes to several hours). Computational resources include, for example, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and Random Access Memory (RAM). Examples of computationally intensive analytical tasks include training and inference of large-scale flight delay prediction models based on historical and real-time data; complex analysis and situational assessment of airspace traffic flow; and in-depth analysis of long-term operational efficiency and bottlenecks.

[0077] The civil aviation industry-level data service platform faces the following problems when handling the aforementioned mixed workloads, particularly when coordinating computationally intensive analysis tasks with routine data processing tasks:

[0078] 1. Resource contention leads to performance degradation in routine data processing tasks.

[0079] When a large, computationally intensive analytical task is scheduled for execution, it may consume a large amount of CPU, memory, or network bandwidth resources for a long time. This can cause concurrently running, latency-sensitive routine data processing tasks to experience significantly increased processing latency due to insufficient resources, or even time out and fail. This directly affects the decision-making efficiency and operational safety of downstream users (such as airline operations control). Consequently, it makes it difficult for civil aviation industry-level data service platforms to effectively isolate the competitive impact of the two types of tasks on resources.

[0080] 2. Low execution efficiency and resource waste in computationally intensive tasks

[0081] When scheduling large, computationally intensive analytics tasks as a whole, it may be difficult to find a single computing node that can meet its peak resource requirements, leading to excessively long task queuing times. Alternatively, even if a suitable node is found, the task may not fully utilize its resources (e.g., a CPU-intensive task is scheduled to a node with abundant GPU resources). Or, the task's execution pattern may have inherent parallelism, but the overall scheduling cannot leverage the parallel processing capabilities of multiple nodes, resulting in inefficient execution of computationally intensive analytics tasks and the idleness and waste of computing resources.

[0082] 3. Poor platform scalability and cost-effectiveness.

[0083] When new, more complex, computationally intensive analytical tasks need to be introduced, platform administrators often need to reserve or configure a large amount of additional, dedicated computing resources to avoid impacting existing routine data processing tasks. This over-provisioning approach to resource expansion is costly and results in low resource utilization during off-peak analysis periods, making it difficult to cost-effectively expand the platform's analytical capabilities while maintaining service quality.

[0084] In summary, existing civil aviation industry-level data service platforms suffer from problems such as resource contention, performance degradation during routine data processing tasks, low execution efficiency for computationally intensive analysis tasks, and high platform expansion costs. Therefore, this application provides a task scheduling method for a civil aviation industry-level data service platform, which can be executed by computer equipment. Please refer to... Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiments of this application.

[0085] like Figure 1 As shown, the computer device may include: a processor 101, a communication bus 102, a network interface 103, a user interface 104, and a memory 105. The communication bus 102 is used to enable communication between these components. The user interface 104 may include a display screen or an input unit such as a keyboard; the user interface 104 may include standard wired or wireless interfaces. The network interface 103 may include standard wired or wireless interfaces (such as a Wi-Fi interface). The memory 105 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. Optionally, the memory 105 may also be a storage device independent of the aforementioned processor 101.

[0086] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0087] like Figure 1 As shown, the memory 105, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a task scheduling device for a civil aviation industry-level data service platform.

[0088] exist Figure 1In the computer device shown, the network interface 103 is mainly used for data communication with the network server; the user interface 104 is mainly used for data interaction with the user; the processor 101 and the memory 105 in the computer device of the present invention can be set in the computer device, and the computer device calls the task scheduling device of the civil aviation industry-level data service platform stored in the memory 105 through the processor 101, and executes the task scheduling method of the civil aviation industry-level data service platform provided in the embodiment of this application.

[0089] based on Figure 1 The computer equipment shown below, in conjunction with Figure 2 The following describes a task scheduling method for a civil aviation industry-level data service platform, based on embodiments of this application. The steps are as follows:

[0090] S201. Obtain the set of tasks to be scheduled, and the real-time available resource status and topology information of computing nodes in the civil aviation industry-level data service platform.

[0091] In practical implementation, the set of tasks to be scheduled includes multiple tasks awaiting execution without allocated resources. The civil aviation industry-level data service platform comprises multiple computing nodes, such as physical machines, virtual machines, and containers. Real-time available resource status of computing nodes includes CPU utilization percentage, available physical / virtual machine RAM size (GB), available GPU memory size (GB), network interface received / sent bytes per second, and disk I / O latency. Real-time resource usage of each computing node, such as CPU utilization, memory usage, and disk I / O, can be collected using system monitoring tools (such as Prometheus, Nagios, and Zabbix). Network monitoring tools (such as NetFlow and sFlow) can be used to collect real-time network interface traffic data (such as received and sent bytes per second). The topology information of the computing nodes reflects the connection relationships between the various computing nodes in the civil aviation industry-level data service platform. The topology information of the computing nodes can be obtained using network topology discovery tools (such as OpenDaylight and Cisco Prime).

[0092] S202. Determine the task type for each task in the task set.

[0093] In the specific implementation process, the task types include routine data processing tasks and computationally intensive analysis tasks. The task type of each task in the task set can be determined based on metadata and historical operation data.

[0094] In one possible embodiment, if no historical running data is pre-stored in the local database, the task type of each task is determined based on the metadata of each task in the task set; if historical running data is pre-stored in the local database, the task type of each task is determined based on the historical running data of similar tasks in the task set.

[0095] The metadata includes type tags, resource requirements, and service level agreements (SLAs). Resource requirements refer to the resources such as CPU, RAM, and GPU needed to perform tasks, while service level agreements refer to the specific agreements and standards between service providers and customers regarding service quality, response time, availability, and other aspects.

[0096] Tasks of the same type refer to tasks with the same type label. Historical execution data includes historical resource consumption data, historical execution duration, and number of retries. Historical resource consumption data refers to the CPU, memory, and I / O usage records of similar tasks in the past, including task ID, task type, start / end time, average / peak CPU utilization, and average / peak memory usage. Historical execution duration refers to the average historical execution duration of all similar tasks. Number of retries refers to the number of times task execution failed and was retried, which can be obtained through Kubernetes event streams or job scheduling logs.

[0097] In one possible embodiment, determining the task type of each task based on the metadata of each task in the task set includes the following methods:

[0098] Method 1: Determine if the type label is realtime_forwarding to identify whether the corresponding task is a computationally intensive analysis task or a routine data processing task.

[0099] Specifically, if the type label is realtime_forwarding, the task type is determined to be routine data processing task; if the type label is hourly_statistics, ml_training_daily, or complex_airspace_analysis, the task type is determined to be computationally intensive analysis task.

[0100] Method 2: Determine whether the resource demand exceeds the preset resource threshold, and then determine whether the corresponding task is a computationally intensive analysis task or a routine data processing task.

[0101] Specifically, if the resource requirement exceeds a preset resource threshold, the task type is determined to be a compute-intensive analysis task. For example, a task requesting more than 16 CPU cores, more than 64GB of RAM, or a specific GPU is considered a compute-intensive analysis task. If the resource requirement is less than or equal to the preset resource threshold, the task type is determined to be a routine data processing task. For example, a task requesting less than 1 CPU cores and less than 2GB of RAM is considered a routine data processing task.

[0102] Method 3: Determine whether the maximum delay time in the service level agreement is greater than the preset time to determine whether the corresponding task is a computationally intensive analysis task or a routine data processing task.

[0103] Specifically, if the maximum latency in the service level agreement is greater than the preset time, the task type is determined to be a compute-intensive analysis task. For example, a task requiring completion within 8 hours is a compute-intensive analysis task. If the maximum latency in the service level agreement is less than or equal to the preset time, the task type is determined to be a routine data processing task. For example, a task requiring a latency of <500ms is a routine data processing task.

[0104] In this embodiment, when new tasks or historical running data are missing, automatic classification is performed based on metadata to avoid unsolvable problems or task delays caused by relying entirely on historical running data, thereby ensuring the normal operation of the system. Furthermore, compared to historical running data, metadata is static and can be quickly read, allowing for rapid determination of the task type.

[0105] In one possible embodiment, determining the task type of each task based on historical execution data of similar tasks within the task set includes the following methods:

[0106] Method 1: Determine whether the duration of CPU utilization exceeding 80% for similar tasks is greater than the preset duration to identify whether the corresponding task is a computationally intensive analysis task or a routine data processing task.

[0107] Specifically, if the duration of CPU utilization of similar tasks is greater than 80% and is longer than the preset duration (e.g., 30 minutes), then the task type of the target task is determined to be a computationally intensive analysis task. If the CPU utilization of similar tasks is less than or equal to 80%, or if the duration of CPU utilization of similar tasks is less than or equal to the preset duration, then the task type of the corresponding task is determined to be a routine data processing task.

[0108] Method 2: Determine whether the historical execution time of similar tasks exceeds the preset execution time, and determine whether each task is a computationally intensive analysis task or a routine data processing task.

[0109] If the historical execution time of a similar task exceeds the preset execution time (e.g., 30 minutes), the task type is determined to be a computationally intensive analysis task; if the historical execution time of a similar task does not exceed the preset execution time (e.g., 30 minutes), the task type is determined to be a routine data processing task.

[0110] Method 3: Determine whether the number of failed retries for the same type of task within a preset time period is greater than the preset number, and determine whether the corresponding task is a computationally intensive analysis task or a routine data processing task.

[0111] If the number of retries for a similar task exceeds a preset limit, the task type is determined to be a computationally intensive analysis task. If the number of retries for a similar task is less than or equal to a preset limit, the task type is determined to be a routine data processing task.

[0112] In this embodiment, considering that metadata typically contains pre-defined information that may differ from the actual execution of a task, historical execution data provides the actual execution status of the task and more realistically reflects its computational requirements, resource consumption, and execution characteristics. Therefore, by combining historical execution data, the assumptions in the metadata can be calibrated, avoiding incorrect task classification. This is especially true when the task execution process is complex or resource-intensive, where historical data provides a more reliable basis. For example, a significant increase in the failure / retry frequency of a certain type of task indicates that it may be sensitive to resource contention or have hidden resource bottlenecks. Even if its declared resource requirements are not high, it should still be marked as a computationally intensive analysis task.

[0113] S203. If it is determined that the resource requirement of any computationally intensive analysis task exceeds a preset resource threshold and the computationally intensive analysis task is decomposable, then the computationally intensive analysis task is decomposed into multiple subtasks.

[0114] In practical implementation, for each computationally intensive analysis task, resource requirement assessment and decomposability assessment can be performed. Resource requirement assessment determines whether the resource requirements of the computationally intensive analysis task exceed a preset resource threshold. Resource requirements include the requested number of CPU cores, requested RAM size, requested GPU type and number, and expected runtime. Decomposability assessment analyzes the internal logic or processing pattern of the computationally intensive analysis task to determine whether the task can be decomposed into multiple subtasks that can be executed in parallel or sequentially. For example, whether the task can handle fragmented data, or whether it contains multiple independent computation stages, etc.

[0115] In one possible embodiment, determining that the resource requirements of any computationally intensive analysis task exceed a preset resource threshold includes the following methods:

[0116] Method (1): Determine whether the number of CPU cores requested in the resource requirements is greater than the preset CPU threshold.

[0117] If the requested number of CPU cores is determined to be greater than the preset CPU threshold (e.g., 32 cores), then the resource requirements of the computationally intensive analysis task are determined to exceed the preset resource threshold.

[0118] Method (2): Determine whether the requested RAM size in the resource requirement is greater than the preset RAM threshold.

[0119] If the requested RAM size is determined to be greater than a preset RAM threshold (e.g., 128GB), then the resource requirements of the compute-intensive analysis task are determined to exceed the preset resource threshold.

[0120] Method (3): Determine whether the number of GPUs requested for the target type in the resource requirements is greater than the preset number of GPUs corresponding to the target type.

[0121] If it is determined that the number of GPUs for the target type requested is greater than the preset number of GPUs for the target type (e.g., requesting 2 V100 GPUs), then it is determined that the resource requirements of the compute-intensive analysis task exceed the preset resource threshold.

[0122] Method (4): Determine whether the expected runtime in the resource requirements is greater than the preset runtime.

[0123] If the estimated runtime is determined to be longer than the preset runtime (e.g., 1 hour), then the resource requirements of the compute-intensive analysis task are determined to exceed the preset resource threshold.

[0124] In the embodiments of this application, different types of tasks may have different resource requirements. For example, some tasks may rely more on the CPU, while others may have higher requirements for RAM or GPU. By comparing the resource requirements such as the number of CPU cores, RAM size, GPU, and runtime with corresponding preset thresholds, the resource requirements of a task can be assessed more accurately.

[0125] In one possible embodiment, decomposing the computationally intensive analysis task into multiple subtasks includes the following methods:

[0126] The first approach is to split the dataset that the computationally intensive analysis task needs to process into multiple data subsets; generate multiple subtasks with the same processing logic as the computationally intensive analysis task; and have each subtask independently process a different data subset.

[0127] In practice, large datasets are split into multiple subsets and processed in parallel. This approach is suitable for situations where computational tasks can be executed independently of specific data subsets. For example, a terabyte-level file containing daily operational data for all airports nationwide can be divided into multiple subfiles based on airport or region.

[0128] In this embodiment, when the dataset is very large, splitting it into multiple subsets and processing them in parallel can effectively reduce the computational burden of a single task and improve the overall execution efficiency. This approach has excellent horizontal scalability; if the amount of data to be processed increases, horizontal scaling can be achieved by further splitting the dataset, utilizing more computing nodes or resources.

[0129] The second approach is to decompose the computationally intensive analysis task into multiple sequentially dependent subtasks, following the steps executed in the order they are performed.

[0130] In practice, tasks are broken down into multiple subtasks in sequence, with the execution of each subtask depending on the result of the previous subtask. This approach is suitable for tasks that contain sequential dependencies or phased tasks. For example, a computationally intensive analysis task consisting of three sequentially executed steps—data loading and cleaning, feature engineering, and model training—can be decomposed into three sequentially dependent subtasks.

[0131] In the embodiments of this application, it is possible to process tasks with sequential dependencies and ensure that tasks are executed in the correct steps. Sometimes, splitting a task into serially dependent subtasks can reduce the storage requirements of intermediate data because the results of the subtasks are used sequentially, and it is not necessary to store multiple intermediate results at the same time.

[0132] S204. Construct a scheduling queue containing all routine data processing tasks, all subtasks, and all undecomposed computationally intensive analysis tasks, and record the attributes of each task in the scheduling queue.

[0133] In practical implementation, a scheduling queue can be constructed to store all routine data processing tasks, all undecomposed computationally intensive analysis tasks, and all decomposed subtasks from the task set to be scheduled. The task attributes are recorded in the scheduling queue, including task type, priority, resource requirements, SLA requirements, and dependencies between tasks / subtasks.

[0134] It should be noted that routine data processing tasks have a higher priority than subtasks and computationally intensive analysis tasks. By assigning higher priority to routine data processing tasks, their low-latency SLA requirements are ensured to be met first.

[0135] S205. Based on the attributes of each task, the real-time available resource status, and topology information, dynamically allocate each task in the scheduling queue to multiple computing nodes.

[0136] In practical implementation, an intelligent scheduler can be designed. This scheduler needs to simultaneously consider all routine data processing tasks, decomposed subtasks, and undecomposed computationally intensive analysis tasks on the civil aviation industry-level data service platform. The intelligent scheduler's input includes the attributes of each task in the scheduling queue, the real-time available resource status of computing nodes, and topology information. The intelligent scheduler employs an optimized allocation strategy to determine which task / subtask is assigned to which computing node for execution.

[0137] In one possible embodiment, step S205 includes:

[0138] Each task in the queue to be scheduled is treated as an agent; each agent selects a computing node for task scheduling based on its current state information; the current state information includes the attributes of the corresponding task, the real-time available resource status, and topology information; a multi-agent learning algorithm is used to train and iteratively optimize the task scheduling strategy based on historical experience until the reward function is maximized; historical experience includes the previous state information, the previously selected computing node, and the previous reward.

[0139] In the specific implementation process, the first step is to model the agents, treating each unit that needs to be scheduled and deployed (whether it is a routine data processing task, a subtask, or an undecomposed computationally intensive analysis task) as an agent. Each agent needs to make or influence decisions about its deployment location. Secondly, the actions each agent (task / subtask) needs to take in its current state primarily involve selecting the most suitable target computing node d (from the set of available nodes D) for deployment and execution. Then, a reward function is designed to evaluate the global effect produced by a series of scheduling actions, quantitatively reflecting the contribution of scheduling decisions to key performance indicators.

[0140] Finally, by employing multi-agent learning algorithms suitable for collaborative tasks (such as those based on value function decomposition, policy gradient, and the Actor-Critic framework), the task scheduling strategy can be trained and iteratively optimized based on historical experience (state, action, reward). This enables the intelligent scheduler to learn how to make trade-offs in complex dynamic environments and to make scheduling decisions that maximize cumulative rewards in the long run (i.e., achieve the optimization goal).

[0141] In this embodiment, a multi-agent learning algorithm (MARL) is used for task scheduling. Each task is treated as an independent agent, and intelligent decisions are made based on real-time resources and task attributes. This effectively improves the automation, flexibility, intelligence, and resource utilization efficiency of the scheduling process. Through feedback from historical experience and continuous learning, the task scheduling strategy can be continuously optimized to maximize overall performance, improve the reliability and fault tolerance of task scheduling, and ensure the efficient operation of each computing node.

[0142] In one possible embodiment, the step of each agent selecting a computing node for task scheduling based on its current state information includes:

[0143] Each agent selects a computing node for task scheduling based on its current state information and hard constraints; the hard constraints include that all routine data processing tasks must meet SLA requirements and that tasks cannot be scheduled to nodes that do not have the required hardware.

[0144] In this embodiment, the boundary of the MARL agent's decision space is defined by hard constraints. The multi-agent learning algorithm cannot violate these hard constraints when exploring and selecting actions. By using SLA requirements as hard constraints, the task scheduling system is ensured to strictly adhere to business objectives during scheduling, guaranteeing that each task is completed on time, thereby improving customer satisfaction and system reliability. By forcibly constraining tasks to be scheduled to nodes with the necessary hardware resources, task scheduling failures or system errors are avoided, reducing operational errors caused by hardware incompatibility or insufficient resources.

[0145] In one possible implementation, the reward function is constructed based on several optimization objectives:

[0146] All routine data processing tasks meet SLA requirements;

[0147] Minimize the overall completion time of all subtasks and all undecomposed computationally intensive analysis tasks;

[0148] Maximize the overall resource utilization of the civil aviation industry-level data service platform;

[0149] Minimize the load imbalance among computing nodes of the civil aviation industry-level data service platform;

[0150] Minimize the resource consumption cost of the civil aviation industry-level data service platform.

[0151] In this embodiment, a reward function is designed by comprehensively considering multiple factors such as task timeliness, resource efficiency, and cost control. By ensuring high-priority tasks, critical tasks are completed on time to avoid impacting business operations. By efficiently completing computational tasks, the platform's performance under high load conditions is improved. By maximizing resource utilization and minimizing costs, the system's economy is enhanced. By load balancing, the system is ensured to maintain stable operation under various load conditions.

[0152] In one possible implementation, the reward function is as follows:

[0153]

[0154] in, These are the weight factors corresponding to each optimization objective.

[0155] SLA_Score t This represents the score (or SLA satisfaction score) for all routine data processing tasks meeting SLA requirements within time step t. A higher score results in a higher reward. The SLA satisfaction score can be calculated using several methods, which are described below:

[0156] The first method calculates the percentage of all routine tasks that meet the low-latency SLA requirements within time step t, thus obtaining an SLA satisfaction score.

[0157] The specific formula is as follows:

[0158] SLA_Score t =SLA_n / N

[0159] Among them, SLA_Score t The SLA satisfaction score is represented by SLA_n, where SLA_n represents the number of tasks that meet the SLA within time step t, and N represents the total number of tasks within time step t.

[0160] In the embodiments of this application, the first method is simple and suitable for quickly assessing SLA compliance. However, it can only obtain a ratio value, which may ignore the specific value of the delay (such as the length of the delay).

[0161] The second method, based on a continuous function with delay, obtains the SLA satisfaction score.

[0162] The following are continuous functions based on delay:

[0163]

[0164] Among them, SLA_Score t This represents the SLA satisfaction score, where N is the number of all tasks within time step t. Δti It is the delay time of the i-th task.

[0165] It is a function based on a delay time, defined as follows:

[0166]

[0167] in, Δt i It is the delay time of the i-th task. Δt SLA This is the maximum delay time required by the SLA. Δt max This is the maximum delay tolerance time; tasks exceeding this time will no longer receive any points.

[0168] In this embodiment, the second method provides a more detailed evaluation, quantifying the deviation of latency from SLA for each task. However, the calculation process is more complex than the first method and requires a predefined maximum latency tolerance time.

[0169] Makespan t This represents the overall completion time for all subtasks and all undecomposed computationally intensive analysis tasks. The reward is a reciprocal; shorter times result in higher rewards.

[0170] η uti,t The overall resource utilization rate is equal to the average utilization rate of all resources (CPU, RAM, GPU, etc.) in the civil aviation industry-level data service platform. The higher the overall resource utilization rate, the higher the reward. The specific formula is as follows:

[0171]

[0172] Where M represents the number of resource types in the civil aviation industry-level data service platform, such as CPU, RAM, GPU, etc. It is the actual usage of the i-th resource. It is the maximum available quantity of the i-th type of resource.

[0173] σ load,t To measure the load imbalance among compute nodes, a metric such as the standard deviation of CPU utilization on each compute node could be used, represented by a negative number. σ load,t The smaller the size, the higher the reward.

[0174] C rest,t This represents the resource consumption costs associated with this scheduling action, such as the amount of paid resources used or energy consumption estimates. It is represented by negative numbers. Crest,t The smaller the size, the higher the reward.

[0175] P viol,t This is a penalty term for severe violations of non-hard constraints caused by this task scheduling. Non-hard constraints include, for example, scheduling tasks to nodes with lower loads whenever possible. The penalty term is 1 for violating a non-hard constraint and 0 otherwise. Negative numbers are used to represent these penalties. P viol,t The smaller the size, the higher the reward.

[0176] In one possible implementation, the execution status (running, completed, failed), resource consumption (real-time CPU / memory utilization), and SLA compliance (end-to-end processing latency) of all tasks on the civil aviation industry-level data service platform can be continuously monitored. If performance deviations are detected (such as the average latency of routine tasks consistently exceeding the SLA threshold, analysis task progress being much slower than expected, or long-term severe unevenness in node load), these monitoring indicators and event information can be fed back to the intelligent scheduler.

[0177] For running tasks, if the underlying platform supports it, the intelligent scheduler can try to adjust its resource limits (such as CPU shares / limits); or based on preset rules or learned strategies, it can pause some low-priority analysis subtasks to make way for high-priority tasks.

[0178] Intelligent schedulers can also use feedback performance data (e.g., which scheduling decisions resulted in good / bad rewards) to drive their learning process and update their internal models or policy parameters. This allows intelligent schedulers to learn from actual operational results and continuously improve their future scheduling decision-making capabilities.

[0179] Please refer to Figure 3 This is another flowchart illustrating the task scheduling method of the civil aviation industry-level data service platform provided in the embodiments of this application.

[0180] S301, Receive task requests from the platform.

[0181] Receive task requests from the civil aviation industry-level data service platform, including routine data processing tasks and computationally intensive analysis tasks.

[0182] S302. Identify and evaluate computationally intensive analytical tasks.

[0183] By combining task metadata and historical execution data, task types can be distinguished, and computationally intensive analysis tasks can be identified.

[0184] S303. Determine whether computationally intensive analysis tasks can be decomposed.

[0185] Determine whether the resource requirements of a computationally intensive analysis task exceed a preset resource threshold and whether the computationally intensive analysis task can be decomposed.

[0186] If yes, then execute S304; otherwise, execute S305.

[0187] S304, Execute task decomposition.

[0188] Decompose computationally intensive analysis tasks into multiple subtasks that can be executed in parallel or sequentially.

[0189] S305. Construct a task scheduling strategy.

[0190] Task scheduling strategies can be built based on the MARL principle.

[0191] S306, Execution of task scheduling and deployment.

[0192] The intelligent scheduler makes scheduling decisions based on task scheduling strategies, generates allocation schemes for tasks / subtasks to computing nodes, and starts execution instances of tasks / subtasks on selected nodes by interacting with the underlying resource management platform.

[0193] S307, Monitoring and Feedback Adjustment.

[0194] The system continuously monitors the execution status, resource consumption, and SLA compliance of all tasks on the civil aviation industry-level data service platform. If performance deviations are detected, these monitoring indicators and event information can be fed back to the intelligent scheduler. The intelligent scheduler then adjusts and optimizes its task scheduling strategy based on the feedback.

[0195] In summary, this application also provides a task scheduling method for a civil aviation industry-level data service platform. This method can intelligently perceive the characteristics of computationally intensive analysis tasks, effectively decompose them, and collaboratively optimize the scheduling mechanism of their sub-tasks and a large number of routine data processing tasks in a shared resource pool. This solves the problems of resource contention, performance degradation of routine tasks, low efficiency of analysis tasks, and high expansion costs in civil aviation industry-level data service platforms.

[0196] Based on the same inventive concept, such as Figure 4 As shown in the embodiments of this application, a task scheduling device for a civil aviation industry-level data service platform is also provided, including:

[0197] The acquisition module is used to acquire the set of tasks to be scheduled, the real-time available resource status and topology information of computing nodes in the civil aviation industry-level data service platform;

[0198] The analysis module is used to determine the task type of each task in the task set; task types include routine data processing tasks and computationally intensive analysis tasks.

[0199] The task decomposition module is used to decompose a computationally intensive analysis task into multiple subtasks if it is determined that the resource requirements of any computationally intensive analysis task exceed a preset resource threshold and the computationally intensive analysis task is decomposable.

[0200] The building module is used to construct a scheduling queue containing all routine data processing tasks, all subtasks, and all undecomposed computationally intensive analysis tasks, and to record the attributes of each task in the scheduling queue. The attributes include task type, priority, resource requirements, SLA requirements, and dependencies between tasks / subtasks.

[0201] The scheduling module is used to dynamically allocate tasks in the scheduling queue to multiple computing nodes based on the attributes of each task, the real-time available resource status, and topology information.

[0202] It should be noted that each module in the task scheduling device of the civil aviation industry-level data service platform in this embodiment corresponds one-to-one with each step in the task scheduling method of the civil aviation industry-level data service platform in the aforementioned embodiment. Therefore, the specific implementation of this embodiment can refer to the implementation of the task scheduling method of the aforementioned civil aviation industry-level data service platform, and will not be repeated here.

[0203] In addition, in one embodiment, this application also provides a computer device, the computer device including a processor, a memory and a computer program stored in the memory, the computer program being executed by the processor to implement the aforementioned task scheduling method of the civil aviation industry-level data service platform.

[0204] In addition, in one embodiment, this application also provides a computer storage medium storing a computer program, which is executed by a processor to implement the aforementioned task scheduling method of the civil aviation industry-level data service platform.

[0205] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a device including one or any combination of the above-mentioned memories. The computer may be a variety of computing devices, including smart terminals and servers.

[0206] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0207] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborative files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0208] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0209] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0210] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0211] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a multimedia terminal device (which may be a mobile phone, computer, television receiver, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0212] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A task scheduling method for a civil aviation industry-level data service platform, characterized in that, include: Obtain the set of tasks to be scheduled, and the real-time available resource status and topology information of computing nodes in the civil aviation industry-level data service platform; Determine the task type for each task in the task set; the task type includes routine data processing tasks and computationally intensive analysis tasks; If it is determined that the resource requirement of any computationally intensive analysis task exceeds a preset resource threshold and the computationally intensive analysis task is decomposable, then the computationally intensive analysis task is decomposed into multiple subtasks. Construct a scheduling queue containing all routine data processing tasks, all subtasks, and all undecomposed computationally intensive analysis tasks, and record the attributes of each task in the scheduling queue; the attributes include task type, priority, resource requirements, SLA requirements, and dependencies between tasks / subtasks. Based on the attributes of each task, the real-time available resource status, and the topology information, each task in the scheduling queue is dynamically allocated to multiple computing nodes; the dynamic allocation of each task in the scheduling queue to multiple computing nodes based on the attributes of each task, the real-time available resource status, and the topology information includes: Each task in the queue to be scheduled is treated as an intelligent agent; Each agent selects a computing node for task scheduling based on its current state information; the current state information includes the attributes of the corresponding task, the real-time available resource status, and the topology information. A multi-agent learning algorithm is employed to train and iteratively optimize the task scheduling strategy based on historical experience until the reward function is maximized. This historical experience includes the previous state information, the previously selected computing node, and the previous reward. The reward function is constructed based on the following optimization objectives: All routine data processing tasks meet SLA requirements; Minimize the overall completion time of all subtasks and all undecomposed computationally intensive analysis tasks; Maximize the overall resource utilization of the civil aviation industry-level data service platform; Minimize the load imbalance among the computing nodes of the civil aviation industry-level data service platform; Minimize the resource consumption cost of the civil aviation industry-level data service platform.

2. The task scheduling method for the civil aviation industry-level data service platform as described in claim 1, characterized in that, The step of determining the task type of each task in the task set includes: If no historical execution data is pre-stored in the local database, the task type of each task is determined based on the metadata of each task in the task set; the metadata includes type tags, resource requirements, and service level agreements. If historical execution data is pre-stored in the local database, the task type of each task is determined based on the historical execution data of similar tasks in the task set; the historical execution data includes historical resource consumption data, historical execution time, and number of failure retries.

3. The task scheduling method for a civil aviation industry-level data service platform as described in claim 2, characterized in that, The step of determining the task type of each task based on the metadata of each task in the task set includes: Based on whether the type label is realtime_forwarding, or whether the resource requirement is greater than a preset resource threshold, or whether the maximum latency in the service level agreement is greater than a preset time, the task type of the corresponding task is determined to be either a computationally intensive analysis task or a routine data processing task.

4. The task scheduling method for a civil aviation industry-level data service platform as described in claim 2, characterized in that, The step of determining the task type of each task based on the historical execution data of similar tasks in the task set includes: The task type is determined as either a computationally intensive analysis task or a routine data processing task based on whether the duration of CPU utilization exceeding 80% for the same type of task is greater than a preset duration, or whether the historical execution duration of the same type of task exceeds a preset execution duration, or whether the number of failed retries for the same type of task within a preset time period is greater than a preset number.

5. The task scheduling method for a civil aviation industry-level data service platform as described in claim 1, characterized in that, The resource requirements include the requested number of CPU cores, the requested RAM size, the requested GPU type and number of GPUs, and the estimated runtime; the step of determining that the resource requirements of any compute-intensive analysis task exceed a preset resource threshold includes: Determine that the number of CPU cores requested is greater than a preset CPU threshold; Alternatively, determine that the requested RAM size is greater than a preset RAM threshold; Alternatively, determine that the number of GPUs for the target type of the request is greater than the preset number of GPUs corresponding to the target type; Alternatively, determine that the expected runtime is greater than the preset runtime.

6. The task scheduling method for a civil aviation industry-level data service platform as described in claim 1, characterized in that, The step of decomposing the computationally intensive analysis task into multiple sub-tasks includes: The dataset that the computationally intensive analysis task needs to process is split into multiple data subsets; Generate multiple subtasks with the same processing logic as the computationally intensive analysis task; each subtask is used to independently process a different subset of data.

7. The task scheduling method for a civil aviation industry-level data service platform as described in claim 1, characterized in that, The step of decomposing the computationally intensive analysis task into multiple sub-tasks includes: The computationally intensive analysis task is decomposed into multiple serially dependent subtasks by executing the steps in the order they are performed.

8. The task scheduling method for a civil aviation industry-level data service platform as described in claim 1, characterized in that, The step of each agent selecting a computing node for task scheduling based on its current state information includes: Each agent selects a computing node for task scheduling based on its current state information and hard constraints; the hard constraints include that all routine data processing tasks must meet SLA requirements and that tasks cannot be scheduled to nodes that do not have the required hardware.

9. The task scheduling method for a civil aviation industry-level data service platform as described in claim 1, characterized in that, The reward function is as follows: Among them, SLA_Score t This represents the score for all routine data processing tasks meeting SLA requirements within time step t; Makespan t The overall completion time for all subtasks and all undecomposed computationally intensive analysis tasks; This refers to the overall resource utilization rate; A metric for measuring the load imbalance among computing nodes; The resource consumption cost associated with this scheduling action; This is a penalty for serious violations of non-hard constraints resulting from this task scheduling. These are the weight factors corresponding to each optimization objective.

10. The task scheduling method for a civil aviation industry-level data service platform as described in claim 9, characterized in that, The SLA_Score t The calculation formula is as follows: Where N is the total number of tasks within time step t. Δt i It is the delay time of the i-th task; It is a function based on a delay time, defined as follows: in, Δt i It is the delay time of the i-th task. Δt SLA This is the maximum delay time required by the SLA. Δt max It is the maximum delay tolerance time.

11. A task scheduling device for a civil aviation industry-level data service platform, characterized in that, include: The acquisition module is used to acquire the set of tasks to be scheduled, the real-time available resource status and topology information of computing nodes in the civil aviation industry-level data service platform; An analysis module is used to determine the task type of each task in the task set; the task type includes routine data processing tasks and computationally intensive analysis tasks. The task decomposition module is used to decompose the computationally intensive analysis task into multiple sub-tasks if it is determined that the resource requirement of any computationally intensive analysis task exceeds a preset resource threshold and the computationally intensive analysis task is decomposable. The building module is used to construct a scheduling queue containing all routine data processing tasks, all subtasks, and all undecomposed computationally intensive analysis tasks, and to record the attributes of each task in the scheduling queue; the attributes include task type, priority, resource requirements, SLA requirements, and dependencies between tasks / subtasks. A scheduling module is used to dynamically allocate tasks in the scheduling queue to multiple computing nodes based on the attributes of each task, the real-time available resource status, and the topology information; the dynamic allocation of tasks in the scheduling queue to multiple computing nodes based on the attributes of each task, the real-time available resource status, and the topology information includes: Each task in the queue to be scheduled is treated as an intelligent agent; Each agent selects a computing node for task scheduling based on its current state information; the current state information includes the attributes of the corresponding task, the real-time available resource status, and the topology information. A multi-agent learning algorithm is employed to train and iteratively optimize the task scheduling strategy based on historical experience until the reward function is maximized. This historical experience includes the previous state information, the previously selected computing node, and the previous reward. The reward function is constructed based on the following optimization objectives: All routine data processing tasks meet SLA requirements; Minimize the overall completion time of all subtasks and all undecomposed computationally intensive analysis tasks; Maximize the overall resource utilization of the civil aviation industry-level data service platform; Minimize the load imbalance among the computing nodes of the civil aviation industry-level data service platform; Minimize the resource consumption cost of the civil aviation industry-level data service platform.

12. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the task scheduling method of the civil aviation industry-level data service platform as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the processor executes the computer program to implement the task scheduling method of the civil aviation industry-level data service platform as described in any one of claims 1-10.

Images