A multi-level task scheduling method based on a cloud heterogeneous architecture

Abstract

The application discloses a kind of multi-level task scheduling methods based on cloud heterostructure, it is related to big data analysis technical field, the original task is obtained by task feature analysis engine quantitative analysis subtask split feature parameter set, in combination with cloud heterostructure real-time resource load distribution, by intelligent scheduling decision system and carry out cross-level dependence check and resource conflict detection, realize task and resource accurate matching;By creating subtask coordination management factory, generate the unique identification containing parent identification and subtask serial number, subtask is classified and collected according to the architecture level on cloud heterostructure and forms execution set and writes distributed task list, realize subtask ordered management;By analyzing system log to estimate resource occupation, in combination with historical data analysis and prediction resource variation trend, according to the predicted result, migrate subtask to adaptive level and update task list, realize resource intelligent balance, avoid overload risk, improve system efficiency and stability, play the advantage of cloud heterostructure.

Description

Technical Field

[0001] This invention relates to the field of big data analytics, specifically a multi-level task scheduling method based on a cloud-based heterogeneous architecture. Background Technology

[0002] Cloud-based heterogeneous architecture integrates diverse computing units such as CPUs, GPUs, and FPGAs. It can precisely allocate computing power based on task characteristics, allowing each unit to fully leverage its strengths. For example, GPUs help efficiently process parallel computing tasks, thereby specifically improving overall computing performance and meeting the high computing power requirements of scenarios such as machine learning and scientific computing. By allocating tasks to suitable processors, it effectively optimizes energy efficiency and significantly reduces energy consumption in data centers. At the same time, it ensures stable system operation through intelligent resource allocation and adapts to diverse application scenarios with its excellent flexibility and scalability. It is the key to breaking through the bottlenecks of traditional cloud computing and expanding application boundaries.

[0003] In cloud-based heterogeneous architectures, traditional multi-level task scheduling mechanisms lack in-depth analysis of task characteristics and adaptability to diverse computing units, resulting in inaccurate resource matching and dynamic balancing. This leads to an imbalance where some computing units become overloaded, creating a "busier is busier" scenario. Furthermore, limitations in subtask decomposition and hierarchical coordination further exacerbate uneven resource utilization, causing wasted computing power, reduced efficiency, and decreased system stability, thus hindering the full realization of the advantages of cloud-based heterogeneous architectures. Therefore, a multi-level task scheduling method based on cloud-based heterogeneous architectures is urgently needed. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-level task scheduling method based on a cloud-based heterogeneous architecture to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-level task scheduling method based on a cloud-based heterogeneous architecture, the multi-level task scheduling method comprising the following steps:

[0006] Step S1: Quantitatively analyze the original task using the task feature analysis engine to obtain the subtask splitting feature parameter set of each subtask of the original task; synchronously obtain the heterogeneous resource status in the cloud; use the intelligent scheduling decision system to obtain the subtask splitting set of the original task through the subtask splitting feature parameter set and the heterogeneous resource status in the cloud.

[0007] Step S1-1: Select any original task to be executed in the cloud heterogeneous as the research object, and use the task feature analysis engine to quantify and analyze the priority, data processing volume, computational complexity and constraints of the original task to generate a subtask splitting feature parameter set containing task latency threshold, data throughput and computing power requirements.

[0008] Step S1-2: Use monitoring protocols to obtain CPU utilization, memory usage, GPU computing power load and network bandwidth status data of each architecture level in the cloud heterogeneous architecture, and obtain the real-time resource load distribution of each architecture level in the cloud heterogeneous architecture.

[0009] Steps S1-3: Decompose the feature parameter set of the subtask and the real-time resource load distribution of each architecture layer in the cloud heterogeneous architecture, and input them into the intelligent scheduling decision system to obtain the initial allocation set of the subtasks of the original task in the edge layer, regional cloud layer and core cloud layer.

[0010] Steps S1-4: Perform cross-level dependency verification and resource conflict detection on the initial allocation set of subtasks of the original task, use a directed acyclic graph to obtain the execution order and constraints between each subtask, and finally output the subtask split set of the original task through the intelligent scheduling decision system.

[0011] The task feature parsing engine quantifies and analyzes the original task to obtain a set of subtask splitting feature parameters. Simultaneously, the status of heterogeneous resources in the cloud is obtained. The intelligent scheduling decision system then obtains a set of subtask splits, which can achieve accurate matching between the original task and heterogeneous resources in the cloud. Cross-level dependency verification and resource conflict detection ensure the rationality of subtask splitting, laying the foundation for subsequent scheduling and improving the scientificity and adaptability of subtask allocation.

[0012] Step S2: Create a subtask coordination and management factory. Initialize the subtask coordination and management factory according to the subtask splitting scheme of the original task, and extract subtask data information according to the splitting scheme. Generate a unique identifier for each subtask through the subtask coordination and management factory to obtain the subtask execution set of each architecture layer on the cloud heterogeneous platform.

[0013] Step S2-1: Create a subtask coordination management factory, traverse and read the subtask split set of the original task, analyze and extract the business logic and target execution architecture hierarchy information of each subtask, and enter it into the subtask coordination management factory configuration module to complete the initialization parameter configuration of the subtask coordination management factory.

[0014] Step S2-2: Based on the data field definitions in the subtask splitting set, extract the input parameters, output data format and dependency information of each subtask from the metadata of the original task to obtain the subtask data information table.

[0015] Step S2-3: The subtask coordination management factory calls the identifier generation module to assign task execution IDs to the subtask data information table obtained in step S2-2. The task execution ID contains a unique identifier of the parent task and a unique identifier of the subtask sequence number combination.

[0016] Step S2-4: Read the hierarchical attributes of the target execution architecture of each subtask. The hierarchical attributes include the hierarchical identifiers of the edge layer, regional cloud layer, and core cloud layer. For each subtask, use the unique identifier and the corresponding execution hierarchical identifier to classify and collect the subtask information according to the edge layer, regional cloud layer, and core cloud layer.

[0017] Step S2-5: Sort the classified and grouped subtasks according to their task numbers and combine them to form subtask execution sets of corresponding levels. Each set contains a unique identifier for the subtask, an input parameter index, an output data storage path, and an identifier for dependent subtasks.

[0018] Step S2-6: Write the subtask execution sets of the edge layer, regional cloud layer and core cloud layer into the corresponding partitions of the distributed task list. The distributed task list adopts a key-value pair storage structure, selecting the level identifier as the key and the corresponding subtask execution set as the value for storage.

[0019] A subtask coordination and management factory is created and initialized. Subtask data information is extracted, and a unique identifier containing the parent identifier and the subtask sequence number is generated through the factory. Subtasks are classified and collected according to the edge layer, regional cloud layer, and core cloud layer, and sorted to form corresponding hierarchical execution sets. These sets are written to the corresponding partitions of the distributed task list, realizing the orderly management and hierarchical organization of subtasks. This provides a clear task index and execution basis for subsequent accurate scheduling, ensuring the standardization and traceability of subtask execution.

[0020] Step S3: Obtain and parse the system logs of each architecture on the cloud heterogeneous system to obtain the status data of the tasks executed at each architecture level on the cloud heterogeneous system; analyze the status data to obtain the execution status of each subtask in the subtask execution set, estimate the resource consumption based on the subtask execution status, and obtain the total memory resource consumption of the architecture level where the subtask is executed.

[0021] Step S3-1: Select any architecture layer on the cloud heterogeneous architecture as the research object, and call the log collection interface to obtain the system logs of the edge layer, regional cloud layer and core cloud layer in the cloud heterogeneous architecture. The system logs include task execution timestamp, unique identifier of the executed task, resource usage record, task execution progress and task execution result.

[0022] Step S3-2: Perform structured parsing on the system logs, extract subtasks with normal execution results from each architecture level, and form a hierarchical resource usage record table by associating the CPU usage time, task execution progress, and memory consumption with the unique identifier of the extracted subtasks.

[0023] Step S3-3: Match the CPU timestamps and memory allocation records in the system logs using regular expressions, and perform unit standardization on the extracted CPU usage time and memory consumption. The unit standardization means that the extracted CPU usage time and memory consumption are converted into a unified time unit and storage unit, respectively.

[0024] Step S3-4: Based on the CPU usage time of the subtasks in the current architecture layer, the execution time of the subtasks is obtained. The time required for the subtasks to be completed is estimated using the task execution progress extracted in step S3-2, and recorded as the subtask execution time.

[0025] The formula for calculating the execution time of a subtask is as follows:

[0026] T=T cpu ÷P;

[0027] In the formula, T represents the time required for the subtask to execute; T cpu This represents the CPU usage time of the subtask; P represents the execution progress of the subtask.

[0028] Step S3-5: Based on the memory consumption analysis of the subtasks executed in the current architecture layer, obtain the average memory required for the execution of the subtask, which is recorded as the average memory of the subtask execution. Multiply the execution time of the subtask by the average memory of the subtask execution to obtain the total memory resource usage of the current architecture layer for executing the subtask.

[0029] By acquiring and parsing system logs from various heterogeneous architecture layers in the cloud, extracting status data such as task execution timestamps and resource usage records, and performing structured parsing and unit standardization processing, the execution time of subtasks is obtained by combining CPU usage time with task execution progress analysis. Then, the average memory usage is obtained through memory consumption analysis. Finally, the total memory resource usage of the architecture layer where the subtask is executed is calculated, achieving accurate prediction of resource usage. This provides reliable data support for subsequent resource scheduling and optimization, ensuring the reasonable allocation and efficient utilization of system resources.

[0030] Step S4: Obtain the historical execution task system logs of the architecture layer where the subtask is located to construct the architecture layer resource set. Analyze the architecture layer resource set to obtain the resource utilization curve of the architecture layer where the subtask is located. Based on the resource utilization curve of the architecture layer where the subtask is located, combined with the resource fluctuation characteristics of historical execution tasks and the current resource occupancy data, predict the trend of resource utilization changes through a time series prediction model.

[0031] Step S4-1: Extract the historical execution task system logs of the subtask's architecture layer from the log storage system, filter out multiple task records containing the complete execution lifecycle, and construct an architecture layer resource set containing task execution timestamps, unique identifiers of the executed tasks, and resource usage records according to the time sequence.

[0032] Step S4-2: Select the timestamp of task execution in the resource set of the architecture layer as the horizontal axis and the resource usage record corresponding to the timestamp as the vertical axis to construct the resource utilization curve of the architecture layer where the subtask is located.

[0033] Step S4-3: Select the unique identifier of the task to be executed in the set of resources at the architecture level as the index, and calculate the total memory resource usage required for each task to be executed at the architecture level based on the task execution timestamp. The specific calculation process is as follows: For each task record corresponding to the unique identifier, sort by timestamp and extract the internal resource usage value, and sum the internal resource usage values ​​corresponding to all timestamps to obtain the total memory resource usage required for the task to be executed at the architecture level.

[0034] The formula for calculating the total memory resource usage required for a task to complete execution at the architectural level is as follows:

[0035] ;

[0036] In the formula, Msum represents the total amount of memory resources required for the task to complete at the architecture level; Mi represents the resource usage value corresponding to different timestamps at the architecture level; and m represents the total number of timestamps.

[0037] Step S4-4: Calculate the complete execution time of each task. Divide the execution cycle of each task into N equally proportional time nodes. Calculate the memory resource usage value corresponding to each time node using linear interpolation. Select the execution progress as the horizontal axis and the interpolated memory resource usage value as the vertical axis to construct a normalized resource utilization curve. Input the normalized resource utilization curve and the real-time execution progress and memory usage value of the current subtask into the time series prediction model to output the future memory resource utilization trend of the architecture layer where the subtask is located.

[0038] By extracting historical execution task system logs from the architecture layer where the subtask resides, an architecture-level resource set is constructed. A resource utilization curve is built with timestamps as the horizontal axis and resource usage records as the vertical axis. The total memory resource usage is calculated by combining the task execution timestamps. Then, a normalized resource utilization curve is constructed using linear interpolation. This curve, along with the real-time execution progress and memory usage of the current subtask, is input into a time-series prediction model. This enables accurate prediction of the changing trend of architecture-level resource utilization, providing a scientific basis for forward-looking resource scheduling, enhancing the system's ability to cope with resource fluctuations, and optimizing overall resource allocation efficiency.

[0039] Step S5: The subtask coordination and management factory schedules the execution of subtasks in the subtask execution set based on the predicted trend of resource utilization changes at the architecture level where the subtasks are located.

[0040] Step S5-1: Simultaneously use step S4 to process other architectural layers on the cloud heterogeneous architecture to obtain the memory resource utilization trend of each layer on the cloud heterogeneous architecture.

[0041] Step S5-2: Obtain the upper limit of resource capacity when executing tasks at each architecture level in the cloud heterogeneous architecture through the hardware configuration interface, and set it as the memory resource utilization threshold accordingly.

[0042] Step S5-3: Monitor and judge the memory resource utilization threshold of each architecture level by calling the subtask coordination management factory, compare the memory resource utilization trend of the corresponding architecture level with the memory resource utilization threshold, mark the architecture level whose memory resource utilization trend exceeds the memory resource utilization threshold, and the corresponding subtask execution set.

[0043] Step S5-3: For the marked architecture level, extract the unfinished subtasks in the corresponding subtask execution set through the subtask coordination and management factory, analyze and extract the subtask dependencies and target execution architecture level attributes, and generate a migration subtask list.

[0044] Step S5-4: Use the subtask coordination and management factory to call the task migration interface to migrate the subtasks in the migration subtask list to the architecture level that satisfies the execution of the subtasks. The architecture level that satisfies the execution of the subtasks is specifically the architecture level whose remaining memory resource capacity satisfies the total memory resource usage required for the execution of the tasks in the migration subtask list.

[0045] Step S5-5: After the migration of subtasks in the migration subtask list is completed, update the subtask execution set at the corresponding architecture level in the distributed task list, as well as the associated hierarchy information of the unique identifier of the subtask.

[0046] By coordinating and managing subtasks, the factory predicts resource utilization trends at each architecture level and simultaneously processes memory resource utilization trends at other levels. It sets memory resource utilization thresholds based on the resource capacity limits obtained from the hardware configuration interface. Through monitoring and comparison, it identifies the levels exceeding the threshold and their corresponding subtask sets. It extracts incomplete subtasks, generates a migration list, and migrates them to the architecture level that meets memory resource requirements. At the same time, it updates the distributed task list and the associated hierarchy information of the subtasks. This achieves dynamic migration scheduling of subtasks and intelligent resource balancing, effectively avoiding the risk of resource overload and improving the overall operating efficiency and stability of the system.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] 1. This invention uses a task feature parsing engine to quantify and analyze the original task to obtain a set of subtask splitting feature parameters. Combined with the real-time heterogeneous resource load distribution in the cloud, a subtask splitting set is generated by an intelligent scheduling decision system. Simultaneously, cross-level dependency verification and resource conflict detection are performed to achieve accurate matching of tasks and resources, avoid allocation imbalance, improve the scientificity and adaptability of subtask allocation, and lay a solid foundation for multi-level scheduling.

[0049] 2. This invention creates a subtask coordination and management factory, generates a unique identifier containing a parent identifier and a subtask sequence number, and classifies and collects subtasks according to the edge layer, regional cloud layer, and core cloud layer to form an execution set and write it into a distributed task list. This achieves orderly management and hierarchical organization of subtasks, provides a clear index for scheduling, and ensures execution standardization and traceability.

[0050] 3. This invention estimates resource usage by analyzing system logs, constructs resource utilization curves by combining historical data, obtains resource change trends through a time-series prediction model, and dynamically migrates sub-tasks exceeding the threshold to the adaptation level and updates the task list through the sub-task coordination and management factory, thereby achieving intelligent resource balancing, avoiding overload risks, improving system efficiency and stability, and fully leveraging the advantages of cloud heterogeneity. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating a multi-level task scheduling method based on a cloud-based heterogeneous architecture according to the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Example 1: As Figure 1 As shown, the present invention provides a technical solution, a multi-level task scheduling method based on a cloud-based heterogeneous architecture, the multi-level task scheduling method comprising the following steps:

[0054] Step S1: Quantitatively analyze the original task using the task feature analysis engine to obtain the subtask splitting feature parameter set of each subtask of the original task; synchronously obtain the heterogeneous resource status in the cloud; use the intelligent scheduling decision system to obtain the subtask splitting set of the original task through the subtask splitting feature parameter set and the heterogeneous resource status in the cloud.

[0055] Step S1-1: Select any original task to be executed in the cloud heterogeneous as the research object, and use the task feature analysis engine to quantify and analyze the priority, data processing volume, computational complexity and constraints of the original task to generate a subtask splitting feature parameter set containing task latency threshold, data throughput and computing power requirements.

[0056] Step S1-2: Use monitoring protocols to obtain CPU utilization, memory usage, GPU computing power load and network bandwidth status data of each architecture level in the cloud heterogeneous architecture, and obtain the real-time resource load distribution of each architecture level in the cloud heterogeneous architecture.

[0057] Steps S1-3: Decompose the feature parameter set of the subtask and the real-time resource load distribution of each architecture layer in the cloud heterogeneous architecture, and input them into the intelligent scheduling decision system to obtain the initial allocation set of the subtasks of the original task in the edge layer, regional cloud layer and core cloud layer.

[0058] Steps S1-4: Perform cross-level dependency verification and resource conflict detection on the initial allocation set of subtasks of the original task, use a directed acyclic graph to obtain the execution order and constraints between each subtask, and finally output the subtask split set of the original task through the intelligent scheduling decision system.

[0059] In practice, the task feature parsing engine calls the task parsing API to quantify and parse the features of the original task, such as priority and data processing volume. Simultaneously, the real-time resource status of the edge layer, regional cloud layer and core cloud layer is obtained through the SNMP protocol. The intelligent scheduling decision system based on reinforcement learning is used to realize the subtask splitting and initial allocation. The subtask splitting must meet the cross-level dependency relationship and avoid resource conflicts, ensuring that the split subtasks are adapted to the resource characteristics of the corresponding level.

[0060] Step S2: Create a subtask coordination and management factory. Initialize the subtask coordination and management factory according to the subtask splitting scheme of the original task, and extract subtask data information according to the splitting scheme. Generate a unique identifier for each subtask through the subtask coordination and management factory to obtain the subtask execution set of each architecture layer on the cloud heterogeneous platform.

[0061] Step S2-1: Create a subtask coordination management factory, traverse and read the subtask split set of the original task, analyze and extract the business logic and target execution architecture hierarchy information of each subtask, and enter it into the subtask coordination management factory configuration module to complete the initialization parameter configuration of the subtask coordination management factory.

[0062] Step S2-2: Based on the data field definitions in the subtask splitting set, extract the input parameters, output data format and dependency information of each subtask from the metadata of the original task to obtain the subtask data information table.

[0063] Step S2-3: The subtask coordination management factory calls the identifier generation module to assign task execution IDs to the subtask data information table obtained in step S2-2. The task execution ID contains a unique identifier of the parent task and a unique identifier of the subtask sequence number combination.

[0064] Step S2-4: Read the hierarchical attributes of the target execution architecture of each subtask. The hierarchical attributes include the hierarchical identifiers of the edge layer, regional cloud layer, and core cloud layer. For each subtask, use the unique identifier and the corresponding execution hierarchical identifier to classify and collect the subtask information according to the edge layer, regional cloud layer, and core cloud layer.

[0065] Step S2-5: Sort the classified and grouped subtasks according to their task numbers and combine them to form subtask execution sets of corresponding levels. Each set contains a unique identifier for the subtask, an input parameter index, an output data storage path, and an identifier for dependent subtasks.

[0066] Step S2-6: Write the subtask execution sets of the edge layer, regional cloud layer and core cloud layer into the corresponding partitions of the distributed task list. The distributed task list adopts a key-value pair storage structure, selecting the level identifier as the key and the corresponding subtask execution set as the value for storage.

[0067] In practice, a subtask coordination and management factory is created, and initialization is completed through the factory's configuration interface. The data extraction tool is called to extract information from the subtask split set. A unique ID containing the parent identifier is generated using a UUID generator. After being classified by level, it is written to the Redis distributed task list through the distributed storage interface. The uniqueness of the unique identifier is used to ensure traceability and ensure the accuracy of the key-value correspondence.

[0068] Step S3: Obtain and parse the system logs of each architecture on the cloud heterogeneous system to obtain the status data of the tasks executed at each architecture level on the cloud heterogeneous system; analyze the status data to obtain the execution status of each subtask in the subtask execution set, estimate the resource consumption based on the subtask execution status, and obtain the total memory resource consumption of the architecture level where the subtask is executed.

[0069] Step S3-1: Select any architecture layer on the cloud heterogeneous architecture as the research object, and call the log collection interface to obtain the system logs of the edge layer, regional cloud layer and core cloud layer in the cloud heterogeneous architecture. The system logs include task execution timestamp, unique identifier of the executed task, resource usage record, task execution progress and task execution result.

[0070] Step S3-2: Perform structured parsing on the system logs, extract subtasks with normal execution results from each architecture level, and form a hierarchical resource usage record table by associating the CPU usage time, task execution progress, and memory consumption with the unique identifier of the extracted subtasks.

[0071] Step S3-3: Match the CPU timestamps and memory allocation records in the system logs using regular expressions, and perform unit standardization on the extracted CPU usage time and memory consumption. The unit standardization means that the extracted CPU usage time and memory consumption are converted into a unified time unit and storage unit, respectively.

[0072] Step S3-4: Based on the CPU usage time of the subtasks in the current architecture layer, the execution time of the subtasks is obtained. The time required for the subtasks to be completed is estimated using the task execution progress extracted in step S3-2, and recorded as the subtask execution time.

[0073] Step S3-5: Based on the memory consumption analysis of the subtasks executed in the current architecture layer, obtain the average memory required for the execution of the subtask, which is recorded as the average memory of the subtask execution. Multiply the execution time of the subtask by the average memory of the subtask execution to obtain the total memory resource usage of the current architecture layer for executing the subtask.

[0074] In practice, the system logs at each level are obtained by calling the log collection interface through Logstash. After being structured and standardized by Fluentd, the execution time is estimated by combining the usage time and progress data obtained by the CPU monitoring tool. The average memory usage and total usage are calculated by the memory analysis tool. The log parsing needs to ensure completeness and unify the time and storage units to ensure data accuracy.

[0075] Step S4: Obtain the historical execution task system logs of the architecture layer where the subtask is located to construct the architecture layer resource set. Analyze the architecture layer resource set to obtain the resource utilization curve of the architecture layer where the subtask is located. Based on the resource utilization curve of the architecture layer where the subtask is located, combined with the resource fluctuation characteristics of historical execution tasks and the current resource occupancy data, predict the trend of resource utilization changes through a time series prediction model.

[0076] Step S4-1: Extract the historical execution task system logs of the subtask's architecture layer from the log storage system, filter out multiple task records containing the complete execution lifecycle, and construct an architecture layer resource set containing task execution timestamps, unique identifiers of the executed tasks, and resource usage records according to the time sequence.

[0077] Step S4-2: Select the timestamp of task execution in the resource set of the architecture layer as the horizontal axis and the resource usage record corresponding to the timestamp as the vertical axis to construct the resource utilization curve of the architecture layer where the subtask is located.

[0078] Step S4-3: Select the unique identifier of the task to be executed in the set of resources at the architecture level as the index, and calculate the total memory resource usage required for each task to be executed at the architecture level based on the task execution timestamp. The specific calculation process is as follows: For each task record corresponding to the unique identifier, sort by timestamp and extract the internal resource usage value, and sum the internal resource usage values ​​corresponding to all timestamps to obtain the total memory resource usage required for the task to be executed at the architecture level.

[0079] Step S4-4: Calculate the complete execution time of each task. Divide the execution cycle of each task into N equally proportional time nodes. Calculate the memory resource usage value corresponding to each time node using linear interpolation. Select the execution progress as the horizontal axis and the interpolated memory resource usage value as the vertical axis to construct a normalized resource utilization curve. Input the normalized resource utilization curve and the real-time execution progress and memory usage value of the current subtask into the time series prediction model to output the future memory resource utilization trend of the architecture layer where the subtask is located.

[0080] In practice, historical logs are extracted from the Elasticsearch log storage system to construct a resource set, resource utilization curves are plotted using Matplotlib, and normalized curves are constructed using linear interpolation through the SciPy toolkit. The results are then input into the LSTM time series prediction model to obtain resource change trends. Complete lifecycle records are selected, and time nodes are reasonably divided to improve prediction accuracy.

[0081] Step S5: The subtask coordination and management factory schedules the execution of subtasks in the subtask execution set based on the predicted trend of resource utilization changes at the architecture level where the subtasks are located.

[0082] Step S5-1: Simultaneously use step S4 to process other architectural layers on the cloud heterogeneous architecture to obtain the memory resource utilization trend of each layer on the cloud heterogeneous architecture.

[0083] Step S5-2: Obtain the upper limit of resource capacity when executing tasks at each architecture level in the cloud heterogeneous architecture through the hardware configuration interface, and set it as the memory resource utilization threshold accordingly.

[0084] Step S5-3: Monitor and judge the memory resource utilization threshold of each architecture level by calling the subtask coordination management factory, compare the memory resource utilization trend of the corresponding architecture level with the memory resource utilization threshold, mark the architecture level whose memory resource utilization trend exceeds the memory resource utilization threshold, and the corresponding subtask execution set.

[0085] Step S5-3: For the marked architecture level, extract the unfinished subtasks in the corresponding subtask execution set through the subtask coordination and management factory, analyze and extract the subtask dependencies and target execution architecture level attributes, and generate a migration subtask list.

[0086] Step S5-4: Use the subtask coordination and management factory to call the task migration interface to migrate the subtasks in the migration subtask list to the architecture level that satisfies the execution of the subtasks. The architecture level that satisfies the execution of the subtasks is specifically the architecture level whose remaining memory resource capacity satisfies the total memory resource usage required for the execution of the tasks in the migration subtask list.

[0087] Step S5-5: After the migration of subtasks in the migration subtask list is completed, update the subtask execution set at the corresponding architecture level in the distributed task list, as well as the associated hierarchy information of the unique identifier of the subtask.

[0088] In practice, the subtask coordination and management factory calls the resource monitoring interface to obtain trends at each level, obtains resource thresholds through the hardware configuration API, marks items exceeding the threshold, generates a migration list using the task dependency analysis tool, calls the Kubernetes task migration interface to migrate to the level that meets the requirements, and simultaneously modifies the distributed task list through the list update interface, analyzes dependencies to avoid execution anomalies, and ensures that the resources at the target level meet the requirements.

[0089] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A multi-level task scheduling method based on a cloud-based heterogeneous architecture, characterized in that: The multi-level task scheduling method includes the following steps: Step S1: Quantitatively analyze the original task using the task feature analysis engine to obtain the subtask splitting feature parameter set of each subtask of the original task; synchronously obtain the heterogeneous resource status in the cloud; use the intelligent scheduling decision system to obtain the subtask splitting set of the original task through the subtask splitting feature parameter set and the heterogeneous resource status in the cloud. Step S1-1: Select any original task to be executed in the cloud heterogeneous as the research object, and use the task feature analysis engine to quantify and analyze the priority, data processing volume, computational complexity and constraints of the original task to generate a subtask splitting feature parameter set containing task latency threshold, data throughput and computing power requirements. Step S2: Create a subtask coordination and management factory. Initialize the subtask coordination and management factory according to the subtask splitting scheme of the original task, and extract subtask data information according to the splitting scheme. Generate a unique identifier for each subtask through the subtask coordination and management factory to obtain the subtask execution set of each architecture layer on the cloud heterogeneous platform. Step S2-1: Create a subtask coordination management factory, traverse and read the subtask split set of the original task, analyze and extract the business logic and target execution architecture hierarchy information of each subtask, and enter it into the subtask coordination management factory configuration module to complete the initialization parameter configuration of the subtask coordination management factory. Step S2-2: Based on the data field definitions in the subtask splitting set, extract the input parameters, output data format and dependency information of each subtask from the metadata of the original task to obtain the subtask data information table. Step S2-3: The subtask coordination management factory calls the identifier generation module to assign task execution IDs to the subtask data information table obtained in step S2-2. The task execution ID contains a unique identifier of the parent task and a unique identifier of the subtask sequence number combination. Step S2-4: Read the hierarchical attributes of the target execution architecture of each subtask. The hierarchical attributes include the hierarchical identifiers of the edge layer, regional cloud layer, and core cloud layer. For each subtask, use the unique identifier and the corresponding execution hierarchical identifier to classify and collect the subtask information according to the edge layer, regional cloud layer, and core cloud layer. Step S2-5: Sort the classified and grouped subtasks according to their task numbers and combine them to form subtask execution sets of corresponding levels. Each set contains a unique identifier for the subtask, an input parameter index, an output data storage path, and an identifier for dependent subtasks. Step S2-6: Write the subtask execution sets of the edge layer, regional cloud layer and core cloud layer into the corresponding partitions of the distributed task list. The distributed task list adopts a key-value pair storage structure, selecting the level identifier as the key and the corresponding subtask execution set as the value for storage. Step S3: Obtain and parse the system logs of each architecture on the cloud heterogeneous system to obtain the status data of the tasks executed at each architecture level on the cloud heterogeneous system; analyze the status data to obtain the execution status of each subtask in the subtask execution set, estimate the resource consumption based on the subtask execution status, and obtain the total memory resource consumption of the architecture level where the subtask is executed. Step S4: Obtain the historical execution task system logs of the architecture layer where the subtask is located to construct the architecture layer resource set. Analyze the architecture layer resource set to obtain the resource utilization curve of the architecture layer where the subtask is located. Based on the resource utilization curve of the architecture layer where the subtask is located, combined with the resource fluctuation characteristics of historical execution tasks and the current resource occupancy data, predict the trend of resource utilization changes through a time series prediction model. Step S5: The subtask coordination and management factory schedules the execution of subtasks in the subtask execution set based on the predicted trend of resource utilization changes at the architecture level where the subtasks are located.

2. The multi-level task scheduling method based on a cloud-based heterogeneous architecture according to claim 1, characterized in that: The specific steps of step S1 are as follows: Step S1-2: Use monitoring protocols to obtain CPU utilization, memory usage, GPU computing load and network bandwidth status data of each architecture level in the cloud heterogeneous architecture, and obtain the real-time resource load distribution of each architecture level in the cloud heterogeneous architecture.

3. The multi-level task scheduling method based on a cloud-based heterogeneous architecture according to claim 2, characterized in that: Step S1 also includes: Steps S1-3: Decompose the feature parameter set of the subtask and the real-time resource load distribution of each architecture layer in the cloud heterogeneous architecture, and input them into the intelligent scheduling decision system to obtain the initial allocation set of the subtasks of the original task in the edge layer, regional cloud layer and core cloud layer. Steps S1-4: Perform cross-level dependency verification and resource conflict detection on the initial allocation set of subtasks of the original task, use a directed acyclic graph to obtain the execution order and constraints between each subtask, and finally output the subtask split set of the original task through the intelligent scheduling decision system.

4. The multi-level task scheduling method based on a cloud-based heterogeneous architecture according to claim 3, characterized in that: The specific steps of step S3 are as follows: Step S3-1: Select any architecture layer on the cloud heterogeneous architecture as the research object, and call the log collection interface to obtain the system logs of the edge layer, regional cloud layer and core cloud layer in the cloud heterogeneous architecture. The system logs include task execution timestamp, unique identifier of the executed task, resource usage record, task execution progress and task execution result. Step S3-2: Perform structured parsing on the system logs, extract subtasks with normal execution results from each architecture level, and form a hierarchical resource usage record table by associating the CPU usage time, task execution progress, and memory consumption with the unique identifier of the extracted subtasks.

5. The multi-level task scheduling method based on a cloud-based heterogeneous architecture according to claim 4, characterized in that: Step S3 also includes: Step S3-3: Match the CPU timestamps and memory allocation records in the system logs using regular expressions, and perform unit standardization on the extracted CPU usage time and memory consumption. The unit standardization means that the extracted CPU usage time and memory consumption are converted into a unified time unit and storage unit, respectively. Step S3-4: Based on the CPU usage time of the subtasks in the current architecture layer, the execution time of the subtasks is obtained. The time required for the subtasks to be completed is estimated using the task execution progress extracted in step S3-2, and recorded as the subtask execution time. Step S3-5: Based on the memory consumption analysis of the subtasks executed in the current architecture layer, obtain the average memory required for the subtasks to be executed, and record it as the average memory of the subtasks. Multiply the execution time of the subtasks by the average memory of the subtasks to obtain the total memory resource usage of the current architecture layer for executing the subtasks.

6. The multi-level task scheduling method based on a cloud-based heterogeneous architecture according to claim 5, characterized in that; The specific steps of step S4 are as follows: Step S4-1: Extract the historical execution task system logs of the subtask's architecture layer from the log storage system, filter out multiple task records containing the complete execution lifecycle, and construct an architecture layer resource set containing task execution timestamps, unique identifiers of the executed tasks, and resource usage records according to the time sequence. Step S4-2: Select the timestamp of task execution in the resource set of the architecture layer as the horizontal axis and the resource usage record corresponding to the timestamp as the vertical axis to construct the resource utilization curve of the architecture layer where the subtask is located. Step S4-3: Select the unique identifier of the task to be executed in the set of resources at the architecture level as the index, and calculate the total memory resource usage required for each task to be executed at the architecture level based on the task execution timestamp. The specific calculation process is as follows: For each task record corresponding to the unique identifier, sort by timestamp and extract the internal resource usage value, and sum the internal resource usage values ​​corresponding to all timestamps to obtain the total memory resource usage required for the task to be executed at the architecture level. Step S4-4: Calculate the complete execution time of each task, divide the execution cycle of each task into N equally proportional time nodes, and calculate the memory resource usage value corresponding to each time node using linear interpolation. Execution progress is selected as the horizontal axis, and interpolated memory resource usage is selected as the vertical axis to construct a normalized resource utilization curve. The normalized resource utilization curve, along with the real-time execution progress and memory usage of the current subtask, is input into the time series prediction model to output the future memory resource utilization trend of the architecture layer where the subtask is located.

7. The multi-level task scheduling method based on a cloud-based heterogeneous architecture according to claim 6, characterized in that: The specific steps of step S5 are as follows: Step S5-1: Simultaneously use step S4 to process other architectural layers on the cloud heterogeneous architecture to obtain the memory resource utilization trend of each layer on the cloud heterogeneous architecture. Step S5-2: Obtain the upper limit of resource capacity when executing tasks at each architecture level in the cloud heterogeneous architecture through the hardware configuration interface, and set it as the memory resource utilization threshold accordingly. Step S5-3: Monitor and judge the memory resource utilization threshold of each architecture level by calling the subtask coordination management factory, compare the memory resource utilization trend of the corresponding architecture level with the memory resource utilization threshold, mark the architecture level whose memory resource utilization trend exceeds the memory resource utilization threshold, and the corresponding subtask execution set.

8. The multi-level task scheduling method based on a cloud-based heterogeneous architecture according to claim 7, characterized in that: Step S5 also includes: Step S5-3: For the marked architecture level, extract the unfinished subtasks in the corresponding subtask execution set through the subtask coordination and management factory, analyze and extract the subtask dependencies and target execution architecture level attributes, and generate a migration subtask list. Step S5-4: Use the subtask coordination and management factory to call the task migration interface to migrate the subtasks in the migration subtask list to the architecture level that satisfies the execution of the subtasks. The architecture level that satisfies the execution of the subtasks is specifically the architecture level whose remaining memory resource capacity satisfies the total memory resource usage required for the execution of the tasks in the migration subtask list. Step S5-5: After the migration of subtasks in the migration subtask list is completed, update the subtask execution set at the corresponding architecture level in the distributed task list, as well as the associated hierarchy information of the unique identifier of the subtask.

Images