Intelligent elastic scheduling method and system for space weather numerical operation

Abstract

The embodiment of the present disclosure provides an intelligent flexible scheduling method and system for space weather numerical operation. Applied to the field of high-performance computing scheduling technology, the method comprises the following steps: obtaining space weather numerical operation historical resource consumption data and workflow structure, constructing resource feature set, task dependency topology and key path task set; screening current period to-be-executed tasks and operation parameters, completing mapping analysis to obtain scheduling mapping relationship and executable task set; evaluating computing resource demand, combining HPC available resources and business priority to perform flexible allocation to generate task resource scheme and execution time period; dynamically scheduling and managing according to the scheme, and adjusting remaining task execution order and resource allocation in real time until completion. The scheme predicts and allocates resources by using historical resources and task dependencies, improves HPC utilization and operation efficiency; at the same time, the scheme optimizes task order and resource allocation through dynamic scheduling, and ensures that key tasks are completed on time.

Description

Technical Field

[0001] This disclosure relates to the field of high-performance computing scheduling technology, and in particular to an intelligent flexible scheduling method and system for space weather numerical operations. Background Technology

[0002] Numerical space weather operations are of great significance for meteorological monitoring and disaster prevention. They rely on complex numerical models, and different models are closely interdependent. To ensure efficient and accurate execution of these operations, it is necessary to delve into resource consumption, task dependencies, and scheduling management to achieve rational resource allocation and efficient task scheduling.

[0003] The current scheduling process for space weather numerical operations involves first collecting historical operational data from numerical models to construct a workflow structure; then, based on the current task list and parameters, mapping executable tasks using the workflow; next, evaluating task requirements and allocating resources according to historical resource data; and finally, executing tasks according to the plan and monitoring their status.

[0004] However, relying on historical data to assess resource needs makes it difficult to accurately match the actual workload of current tasks, resulting in resource allocation deviations. Furthermore, the workflow and scheduling sequence are fixed and lack dynamic adjustment capabilities, making it impossible to optimize based on the real-time status of tasks and resource availability, thus affecting job execution efficiency and resource utilization. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this disclosure provides an intelligent and flexible scheduling method and system for space weather numerical operations. This disclosure solves the problems of existing technologies relying on historical data to assess resource requirements, making it difficult to accurately match the actual workload of the current task and causing resource allocation deviations. Furthermore, existing technologies have fixed workflows and scheduling sequences, lacking dynamic adjustment capabilities and failing to optimize based on real-time task status and resource availability, thus affecting operation efficiency and resource utilization.

[0006] According to a first aspect of this disclosure, an intelligent elastic scheduling method for space weather numerical operations is provided, comprising: acquiring historical resource consumption data and a preset workflow structure for each numerical mode in space weather numerical operations; and constructing a resource consumption feature set, task dependency topology, and critical path task set for each numerical mode based on the historical resource consumption data and the preset workflow structure.

[0007] Obtain the space weather numerical tasks to be executed in the current scheduling cycle and their operation parameters. Based on the operation parameters and task dependency topology, perform operation mapping analysis on the space weather numerical tasks to obtain the scheduling mapping relationship of the space weather numerical tasks in the preset workflow structure and the set of executable tasks.

[0008] Obtain the numerical mode type of each executable task in the executable task set, and perform resource demand assessment based on the numerical mode type and resource consumption feature set to obtain the computational resource demand data of each executable task in the executable task set in the current scheduling cycle.

[0009] Obtain available resource data and business priorities of the HPC system. Based on computing resource demand data, scheduling mapping relationship and critical path task set, and combined with available resource data and business priorities, perform elastic resource allocation to obtain the task resource allocation scheme and execution time period of each executable task in the executable task set.

[0010] Based on the task resource allocation scheme and execution time period, the system schedules and manages the execution of each executable task, obtains the actual scheduling status and resource usage data of each executable task in real time, and dynamically adjusts the execution order and resource allocation of the remaining tasks in the execution task sequence based on the actual scheduling status and resource usage data until all executable tasks in the executable task set are completed.

[0011] According to a second aspect of this disclosure, an intelligent elastic scheduling system for space weather numerical operations is provided, which is used to execute the method described in the first aspect, including: a resource dependency analysis module, which is used to acquire historical resource consumption data and preset workflow structure of each numerical mode in space weather numerical operations, and construct resource consumption feature set, task dependency topology and critical path task set of each numerical mode based on the historical resource consumption data and preset workflow structure.

[0012] The mapping analysis module is used to obtain the space weather numerical tasks to be executed in the current scheduling cycle and their operation parameters. Based on the operation parameters and task dependency topology, the module performs operation mapping analysis on the space weather numerical tasks to obtain the scheduling mapping relationship of the space weather numerical tasks in the preset workflow structure and the set of executable tasks.

[0013] The evaluation module is used to obtain the numerical mode type of each executable task in the executable task set, and to evaluate the resource requirements based on the numerical mode type and the resource consumption feature set to obtain the computational resource requirements data of each executable task in the executable task set in the current scheduling cycle.

[0014] The resource elastic allocation module is used to obtain available resource data and business priorities of the HPC system. Based on computing resource demand data, scheduling mapping relationship and critical path task set, and combined with available resource data and business priorities, it performs resource elastic allocation to obtain the task resource allocation scheme and execution time period of each executable task in the executable task set.

[0015] The dynamic adjustment module is used to schedule and manage the execution of each executable task based on the task resource allocation scheme and the execution time period. It obtains the actual scheduling status and resource usage data of each executable task in real time, and dynamically adjusts the execution order and resource allocation of the remaining tasks in the execution task sequence based on the actual scheduling status and resource usage data, until all executable tasks in the executable task set are completed.

[0016] According to a third aspect of this disclosure, an electronic device is provided, comprising a processor, a memory, and a program or instructions stored in the memory and executable on the processor, the program or instructions being loaded and executed by the processor to perform the steps of the intelligent flexible scheduling method for space weather numerical operations described above.

[0017] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the steps of the intelligent flexible scheduling method for space weather numerical operations described above.

[0018] In the intelligent elastic scheduling method and system for space weather numerical operations provided above, the embodiments of this disclosure utilize historical resources and task-dependent prediction and allocation of resources to improve HPC utilization and operational efficiency; at the same time, by dynamically scheduling and optimizing task sequence and resource allocation, it ensures that critical tasks are completed on time. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic flowchart of an intelligent flexible scheduling method for space weather numerical operations according to an embodiment of the present disclosure is shown.

[0021] Figure 2 A schematic flowchart of an intelligent flexible scheduling method for space weather numerical operations according to an embodiment of the present disclosure is shown.

[0022] Figure 3 A schematic block diagram of an intelligent flexible scheduling system for space weather numerical operations according to an embodiment of the present disclosure is shown.

[0023] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0024] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0025] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them. It should also be understood that in the embodiments of this disclosure, "multiple" can refer to two or more, and "at least one" can refer to one, two, or more. It should also be understood that any component, data, or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless explicitly limited or given a contrary indication in the context. Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this disclosure generally indicates that the related objects before and after are in an "or" relationship. It should also be understood that the descriptions of the various embodiments in this disclosure emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be elaborated upon one by one.

[0026] Furthermore, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. Techniques, methods, and apparatus known to those skilled in the art will not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] Figure 1This is a schematic flowchart illustrating an intelligent, flexible scheduling method for space weather numerical operations provided in this embodiment. The method of this embodiment aims to achieve accurate detection of both large and small targets in images.

[0029] S101, acquire historical resource consumption data and preset workflow structure of each numerical mode in space weather numerical operation, and construct resource consumption feature set, task dependency topology and critical path task set of each numerical mode based on the historical resource consumption data and preset workflow structure.

[0030] Numerical space weather operations refer to a set of computational tasks performed on high-performance computing systems to simulate and predict the evolution of space weather. It typically includes multiple numerical models, each responsible for simulating different physical processes or space regions.

[0031] Numerical models are computational models used in space weather numerical operations to simulate specific physical phenomena. Each model defines independent computational tasks, parameter settings, and operational logic. For example, a solar wind dynamics model might focus on the evolution of the solar wind flow field, while an ionospheric model might simulate changes in ionospheric electron density.

[0032] Historical resource consumption data refers to the actual computing resource information used by each numerical model during past space weather numerical operations. This data includes indicators such as CPU or computing core usage, memory usage, storage I / O and data transfer volume, and task execution time.

[0033] A predefined workflow structure refers to a predefined framework for executing tasks, used to plan the execution order and dependencies of tasks. It clarifies which tasks must be completed first, which tasks can be executed in parallel, and the logical relationships between tasks.

[0034] A resource consumption feature set is a collection derived from the analysis of historical resource consumption data. It describes the typical resource usage of each numerical model in terms of computation, memory, and data transfer. It typically includes information such as the average consumption, peak consumption, and distribution range for each model.

[0035] Task dependency topology is a model that represents the dependencies between tasks in space weather numerical operations in the form of a graph structure. Nodes in the graph represent tasks, and edges represent the dependency constraints between tasks.

[0036] The critical path task set is the set of tasks along the longest path in the task dependency topology. The completion time of these tasks directly affects the total completion time of the entire job.

[0037] To obtain historical resource consumption data and preset workflow structures for each numerical model in space weather numerical operations, it is first necessary to extract the computation records of each numerical model executed in the past from the job logs and resource monitoring system of the high-performance computing system. These records include the CPU or computing core usage time, memory usage, storage read / write rate, and data transfer volume on the network or bus for each task. Simultaneously, by combining the job execution order information and dependency constraints stored in the job scheduling system, the preset workflow structure can be obtained, i.e., the predefined execution logic and interdependencies of each task within the job.

[0038] After obtaining the above data, based on the historical resource usage of each numerical model, statistical methods are used to quantitatively analyze indicators such as computing, memory, and data transmission, calculate the average value, peak value, and distribution range, and construct a resource consumption feature set for each numerical model to reflect the typical resource requirements and fluctuation characteristics of different models during operation.

[0039] Based on the execution order and dependency constraints of each task in the preset workflow structure, a task dependency topology is constructed. Each task in the job is regarded as a node in the topology graph, and the dependencies are regarded as directed edges of the graph, thus forming a complete task dependency network. On the basis of the task dependency topology, critical path analysis is further performed. By calculating the cumulative execution time of tasks on the longest execution path in the topology, the set of tasks that have the greatest impact on the overall job completion time is identified, namely the critical path task set.

[0040] Based on the above technical solution, optionally, based on the historical resource consumption data and the preset workflow structure, a resource consumption feature set, task dependency topology, and critical path task set for each numerical mode are constructed, including:

[0041] Based on the historical resource consumption data, feature extraction is performed to obtain resource consumption feature sets for each numerical pattern;

[0042] Based on the preset workflow structure and resource consumption feature set, task dependency analysis is performed to obtain the task dependency topology corresponding to each numerical mode.

[0043] Based on the resource consumption feature set and task dependency topology, critical path analysis is performed on the tasks corresponding to each numerical model to obtain the critical path task set for each numerical model.

[0044] In this scheme, features are extracted from each numerical pattern based on historical resource consumption data to form a complete resource consumption feature set. Specifically, the CPU or compute core usage, memory usage, storage I / O rate, and network transmission volume of each numerical pattern during HPC system execution are statistically analyzed, and typical resource usage patterns are analyzed in conjunction with task execution time. By calculating the average, peak, variance, and distribution range of these indicators, the resource demand fluctuations of different numerical patterns during execution can be quantified, forming a resource consumption feature set that can be used for prediction, thus providing an accurate data foundation for subsequent scheduling and resource allocation.

[0045] After resource feature extraction, task dependency analysis is performed on each numerical model based on a pre-defined workflow structure to generate corresponding task dependency topologies. In this process, each task in the workflow is considered a node in the topology graph, and the order and dependency constraints between tasks are mapped to directed edges in the graph. Resource consumption characteristics of each node are also appended to its attributes. During topology construction, potential parallelism and dependency chains between tasks are analyzed. By traversing and sorting the dependencies between nodes, it is determined which tasks must be executed sequentially and which tasks can be executed in parallel. This ensures that the generated topology fully reflects the workflow structure and resource usage characteristics of each numerical model, providing a logical basis for critical path identification and scheduling optimization.

[0046] After constructing the task dependency topology and obtaining the resource consumption feature set, critical path analysis is performed on the tasks of each numerical model to obtain the critical path task set for each numerical model. Specifically, the longest execution time path from the start node to the end node is calculated in the topology graph, and tasks along this path are identified as critical path tasks. Simultaneously, the resource consumption characteristics of each task are combined to assess its impact on the overall job completion time. By prioritizing and analyzing the resource consumption of critical path tasks, it is possible to determine which tasks have a decisive impact on the total job completion time, thus providing a clear priority scheduling basis for subsequent scheduling and elastic resource allocation.

[0047] This solution can accurately capture the resource requirements and task dependencies of each numerical model, improving the scientific nature of job scheduling and resource allocation. It also identifies critical path tasks to ensure timely job completion and optimize overall execution efficiency.

[0048] S102, obtain the space weather numerical tasks to be executed in the current scheduling cycle, their operation parameters and numerical mode types, perform operation mapping analysis on the space weather numerical tasks based on the operation parameters and task dependency topology, and obtain the scheduling mapping relationship of the space weather numerical tasks in the preset workflow structure and the set of executable tasks.

[0049] The current scheduling period refers to a fixed time window in an HPC or space weather operation scheduling system used for executing and allocating computing tasks. Within this time period, the system determines which tasks need to be started, which computing resources need to be allocated, and the execution order of the tasks.

[0050] A space weather numerical mission refers to a single computational unit executed in space weather numerical operations to achieve a specific numerical model or computational objective. Each mission corresponds to a specific simulation calculation, such as simulating the solar wind field, magnetic field disturbances, or ionospheric state at a certain moment.

[0051] Operational parameters refer to the configuration information and operating conditions required for each space weather numerical mission to be executed. These parameters include the mission's time step, spatial grid resolution, initial input conditions, model physical parameters, and output data requirements.

[0052] The scheduling mapping relationship refers to the mapping of the task location, execution order, and resource allocation scheme of the space weather numerical tasks to be executed within a preset workflow structure. It clarifies the time period in which each task should be executed, which computing resources should be used, and its dependencies on other tasks.

[0053] The executable task set refers to the set of space weather numerical tasks that, within the current scheduling cycle, can be immediately allocated resources and executed after dependency analysis, resource requirement assessment, and scheduling mapping. This set excludes tasks that cannot be executed temporarily due to incomplete dependencies on other tasks or insufficient resources, and is a subset of tasks that the scheduling system can directly operate on.

[0054] When obtaining the space weather numerical tasks to be executed and their operational parameters within the current scheduling period, it is first necessary to define the start and end points of the current time window in the scheduling system; this window is defined as the current scheduling period. Within this time range, by querying the space weather numerical task queue and task history records, all tasks that have not yet been completed and whose dependent tasks were completed in the previous period can be filtered out. These tasks are the space weather numerical tasks to be executed in this period. For each filtered task, its detailed configuration and operating conditions, i.e., operational parameters, need to be read.

[0055] After obtaining the job parameters, and combining them with the task dependency topology constructed in the previous stage, the job mapping analysis for each space weather numerical task to be executed first matches the dependency constraints of each task with the nodes in the topology structure to determine the task's position in the entire preset workflow, and automatically calculates the execution order based on the sequential relationships between tasks. During this process, the task with the greatest impact on the overall job completion time is identified along the longest dependency path in the topology, and the tasks are prioritized accordingly. Simultaneously, task groups that can be executed in parallel and task chains that must be executed sequentially are identified based on task type and dependency relationships, ensuring that the task mapping conforms to dependency constraints while fully utilizing the system's parallel processing capabilities. Subsequently, referring to the job parameters of each task, the computation time, memory usage, storage read / write volume, and data transfer requirements of each task are estimated, and these resource requirements are combined with the task's position in the workflow to form a complete task mapping scheme, ensuring that the mapping results meet the job scheduling logic and system resource constraints.

[0056] After completing the job mapping analysis, a scheduling mapping relationship can be generated, which explicitly represents the position, execution order, resource requirements, and potential parallel relationships of each space weather numerical task to be executed within the preset workflow structure. This relationship indicates in which time period each task should be started, which tasks it should execute in parallel with or depend on, and provides a direct basis for resource allocation and scheduling. Based on the scheduling mapping relationship, combined with the current available system resources and the completion status of dependent tasks, a set of tasks that can be executed immediately within the current cycle can be further filtered out, namely the executable task set. This set ensures that the dependency constraints of each task are met, resource allocation is feasible, and it can directly enter the scheduling queue for execution, achieving efficient and dynamic task scheduling management.

[0057] The space weather numerical tasks to be executed can be combined with the task dependency topology constructed in the previous stage because these tasks belong to known numerical models, and their execution order and dependencies are clearly represented in the topology. The task dependency topology provides the sequential constraints and parallel possibilities between tasks, enabling the scheduling system to accurately determine which tasks can be executed immediately. The set of tasks selected in this way is the executable task set, which is a subset of the space weather numerical tasks to be executed that meet the dependency constraints and can be started within the current scheduling cycle. The two are directly correlated through dependencies and resource availability.

[0058] Based on the above technical solution, optionally, a job mapping analysis is performed on the space weather numerical tasks based on the job parameters and task dependency topology to obtain the scheduling mapping relationship of the space weather numerical tasks in the preset workflow structure and the set of executable tasks, including:

[0059] Based on space weather numerical missions and their operational parameters, a list of missions to be mapped is obtained through mission filtering and analysis.

[0060] Based on the list of tasks to be mapped and the task dependency topology, the mapping position and dependency relationship of space weather numerical tasks in the preset workflow structure are determined, and the scheduling mapping relationship of space weather numerical tasks in the preset workflow structure is determined based on the mapping position and dependency relationship.

[0061] Based on the mapping location and dependency relationship, identify the set of logically parallelizable tasks, and based on the set of logically parallelizable tasks, generate the set of executable tasks for space weather numerical tasks in the preset workflow structure.

[0062] In this scheme, the list of tasks to be mapped is a set of space weather numerical tasks that have been filtered and parsed within the current scheduling cycle. These tasks have clearly defined their respective operation parameters, such as calculation requirements, priorities, and execution times, but their specific positions and execution orders in the preset workflow structure have not yet been determined.

[0063] The mapping location is the specific node position of each space weather data task in the preset workflow structure, that is, the logical position of the task in the entire operation process.

[0064] Dependency is a constraint on the order of execution and a logical dependency between tasks, meaning that some tasks can only begin execution after other tasks have been completed.

[0065] A logically parallelizable task set is a set of tasks that can be executed simultaneously without violating dependency constraints, based on task dependency topology and mapping location analysis.

[0066] Based on the space weather numerical tasks and their operational parameters within the current scheduling cycle, tasks are filtered and analyzed to generate a set that can be used for subsequent mapping analysis. In this process, each space weather numerical task is traversed, and its computational load, input data volume, estimated execution time, and priority are obtained from its operational parameters. The CPU, memory, storage, and data transmission requirements of each task are analyzed in conjunction with historical resource consumption characteristics, eliminating tasks that cannot be met by current resources. Simultaneously, the pre-constraints of the tasks are checked, removing tasks that cannot be executed sequentially or may cause resource conflicts. As each task is evaluated and filtered, a list of tasks to be mapped is gradually formed and updated. This list is continuously improved through task checking and filtering, providing a foundation for the next step of mapping analysis.

[0067] Based on the generated list of tasks to be mapped and the task dependency topology, mapping analysis is performed on each task to determine its execution position and task constraints within the preset workflow structure. During the analysis, tasks in the list are matched with topology nodes, identifying the preceding and succeeding tasks for each task. Simultaneously, execution time and resource consumption are estimated by combining job parameters and historical resource consumption characteristics to ensure that the task mapping satisfies both job logic and system resources. As each task mapping is completed, the corresponding mapping position and dependencies are gradually generated and improved. With each completed mapping, the scheduling system obtains the position and constraints of a portion of the tasks within the workflow, providing a foundation for parallelism analysis.

[0068] Based on the generated mapping locations and dependencies, parallelizable task combinations are progressively identified to form an executable framework. While scanning all mapped tasks, the dependencies and resource availability of each task are examined. Tasks that can be started simultaneously are selected, organized into task groups, and the resource allocation and execution time of tasks within each group are evaluated to ensure that parallel execution does not cause conflicts or delays. As task groups are generated, a logically parallelizable task set is gradually formed and improved, providing a complete foundation for the final executable task set. The scheduling system updates the executable task set each time a parallelizable task group is added.

[0069] In this scheme, key data is generated step by step to improve scheduling accuracy and resource utilization efficiency; the identification logic allows for parallel task sets to provide a basis for dynamic scheduling, ensuring that key tasks are completed on time.

[0070] S103, obtain the numerical mode type of each executable task in the executable task set, perform resource demand assessment based on the numerical mode type and resource consumption feature set, and obtain the computational resource demand data of each executable task in the executable task set in the current scheduling cycle.

[0071] An executable task refers to a space weather numerical task that has satisfied all dependency constraints and can be immediately allocated computing resources for execution within the current scheduling cycle. These tasks have undergone scheduling mapping analysis and dependency topology verification to ensure that their execution is not affected by incomplete preceding tasks or insufficient resources. They are task units that the scheduling system can directly schedule and run.

[0072] Numerical model type refers to the category of space weather numerical simulation model corresponding to each executable mission. Different types of numerical models correspond to different physical calculation processes and algorithms, such as solar wind dynamics models, ionospheric evolution models, or magnetic field perturbation models.

[0073] Computing resource requirements data refers to the estimated computing resources needed for each executable task within the current scheduling cycle, including indicators such as the number of CPUs or computing cores, memory usage, storage read / write volume, network transmission load, and estimated execution time.

[0074] When obtaining the numerical model type of each executable task in the executable task set, it is first necessary to read the operation parameters of each task, including the simulation time range, spatial grid resolution, time step, physical process settings, input initial conditions, and output data requirements. Simultaneously, the execution status of similar tasks in historical execution records is referenced to determine the physical phenomena and computational objectives to be simulated. This information is then matched against the numerical model library defined in space weather numerical operations. Based on the computational objectives, simulation area, grid size, time step, and physical process characteristics of the task, the numerical model type corresponding to each task is identified, ensuring that the operation logic and computational method of each task are consistent with its type and accurately reflect the resource load characteristics required by the task. After completing the type identification, a detailed resource requirement assessment is performed on each executable task, based on the resource consumption feature set constructed in the previous stage. During the evaluation process, historical resource consumption data is mapped and proportionally adjusted with the current task's parameters. For example, CPU and memory estimates are increased for tasks with increased grid resolution, and computational load and storage I / O are adjusted for tasks with shortened time steps. Statistical analysis methods and empirical models are used to calculate the task's expected CPU usage, memory usage, storage read / write volume, network transmission requirements, and expected execution time for the current cycle. This process also considers task parallelism and dependencies, combining task parameters with historical peak, average, and distribution ranges of resource consumption to perform a refined estimate of resource usage, generating complete computational resource requirements data for each task.

[0075] When obtaining the resource requirements of an executable task, a resource consumption feature set built on the basis of historical resource consumption data of each numerical model in space weather numerical operations and the preset workflow structure can be used. This feature set has quantified the typical CPU, memory, storage and data transmission consumption of each numerical model under different computing scales and simulation conditions, which can directly provide a reference for the current task, enabling the scheduling system to accurately estimate its resource requirements in the current scheduling cycle based on the numerical model type of the task.

[0076] Based on the above technical solution, optionally, resource demand assessment can be performed based on the numerical mode type and resource consumption feature set to obtain the computational resource demand data of each executable task in the executable task set during the current scheduling cycle, including:

[0077] Based on the numerical pattern type, each executable task is associated with the computing resource features corresponding to the resource consumption feature set to obtain the pattern resource feature parameters corresponding to each executable task.

[0078] Based on the aforementioned pattern resource characteristic parameters, the computational resource and data transmission requirements of each executable task in the current scheduling cycle are evaluated to obtain the computational resource requirement data of each executable task in the executable task set.

[0079] In this scheme, the computing resource characteristics are resource consumption characteristics that describe the typical use of computing resources during the execution of the numerical model, including indicators such as CPU / computing core usage, memory usage, storage I / O rate, network or bus data transfer volume, and average and peak consumption.

[0080] The mode resource characteristic parameter is a specific set of parameters obtained by associating each executable task with the computational resource characteristics of its corresponding numerical mode. This parameter describes the expected computational resource usage and data transfer requirements of the task within the current scheduling cycle.

[0081] For each executable task in the executable task set, its corresponding numerical pattern type is first obtained, and then the computational resource characteristics matching that numerical pattern type are searched in the resource consumption feature set. During the matching process, the task's job parameters are combined with the computational resource characteristics. For example, memory requirements are estimated based on the task's input data volume; the required number of CPU cores and time slice allocation are calculated based on the task's expected execution time and historical CPU usage data; and storage I / O and data transfer rates are calculated based on the task's output and intermediate data volume. Simultaneously, the task's priority and task dependency topology information are combined to analyze whether the task can be executed in parallel with other tasks, and resource characteristics are adjusted to reflect shared resource consumption and potential conflicts in a parallel environment. After the above processing, each executable task generates corresponding pattern resource characteristic parameters, which include the number of CPU cores, memory usage, storage read / write rate, network or bus data transfer volume, and possible peak and average values, reflecting the typical resource consumption behavior of the task within the current scheduling cycle.

[0082] Based on the generated pattern resource characteristic parameters, the computational resource requirements and data transmission requirements of each executable task in the current scheduling cycle are quantitatively evaluated. The evaluation steps include: first, combining task dependencies and scheduling mapping relationships to determine the execution order and possible parallel segments of tasks in the entire workflow; then, based on task execution time and resource consumption characteristics, calculating the CPU time slice, peak memory usage, data read / write rate, and total transmission volume required for each task; next, considering shared resources between tasks, scheduling windows, and system load constraints, adjusting the resource requirements of each task to minimize resource conflicts and waste while ensuring performance. As each task completes its evaluation, the computational resource requirements data of each executable task in the executable task set are gradually obtained.

[0083] In this solution, the resource requirements of each executable task can be accurately predicted, ensuring the rational allocation of HPC system resources and improving job completion efficiency; at the same time, the quantified computing resource requirement data provides a basis for dynamic scheduling, thereby optimizing the task execution order and resource utilization.

[0084] S104. Obtain available resource data and business priorities of the HPC system. Based on computing resource demand data, scheduling mapping relationship and critical path task set, and combined with available resource data and business priorities, perform elastic resource allocation to obtain the task resource allocation scheme and execution time period of each executable task in the executable task set.

[0085] HPC systems are computing platforms used to perform large-scale computing tasks. They typically consist of a large number of computing nodes, processors, high-speed networks, storage systems, and scheduling and management systems. These systems can run multiple complex tasks simultaneously and provide high throughput and low latency computing capabilities, making them the core computing infrastructure upon which space weather numerical operations rely.

[0086] Available resource data refers to the computing resources that an HPC system can allocate to tasks during the current scheduling cycle, including the number of idle CPUs or computing cores, available memory capacity, storage I / O bandwidth, network bandwidth, and node availability status.

[0087] Business priority refers to the relative importance or urgency of each space weather numerical task in the HPC system scheduling. Priorities are usually set based on the application scenario of the task, the forecast timeliness requirements, user needs, or system strategies. Tasks with higher priority will be given priority in resource allocation and execution order, thereby meeting the timely computing needs of critical tasks.

[0088] A task resource allocation scheme refers to the specific arrangement of computing resources allocated to each executable task within the current scheduling cycle. This includes detailed resources such as the number of CPUs or computing cores allocated to the task, memory capacity, storage I / O, and network bandwidth, as well as the execution order of the task and its parallel relationship with other tasks.

[0089] The execution time period refers to the actual time range during which each executable task is started and completed in the HPC system. It is usually dynamically determined by the scheduling system based on the task's resource requirements, dependencies, business priorities, and system resource status.

[0090] When acquiring available resources for an HPC system, it is necessary to perform real-time statistics on the CPU or core usage, memory consumption, storage I / O rate, and network bandwidth of all computing nodes in the system. This data, combined with the health status and load of the nodes, determines the allocatable resources available to each node within the current scheduling cycle. This data forms the available resource data for the current scheduling cycle, reflecting not only the total system resource capacity but also revealing potential bottleneck nodes. It provides fundamental information for subsequent task resource allocation while ensuring that the allocation process does not exceed the system's actual carrying capacity.

[0091] By combining the computational resource requirements of each executable task with its position in the scheduling mapping and its priority within the critical path task set, the actual resource consumption and time sensitivity of each task are assessed, and the business priority of each task is set accordingly. During allocation, high-priority tasks are preferentially matched with idle computing cores and memory resources. Simultaneously, parallel task groups are dynamically combined to ensure that tasks satisfy dependency constraints while fully utilizing the system's parallel processing capabilities. In resource-constrained situations, the allocation is adjusted appropriately based on the task's computational load and estimated execution time to ensure that critical path tasks are completed on time, while allowing other tasks to run reasonably within the available resource range, thus achieving elastic scheduling.

[0092] Based on the resource matching and priority analysis described above, a complete task resource allocation scheme is generated for each task, clearly defining the number of CPUs or computing cores, memory capacity, storage and network resources allocated to each task, as well as the parallel relationships and scheduling order between tasks. Simultaneously, the execution time period for each task is determined based on its resource consumption and estimated execution time; that is, the actual time range for startup and completion in the HPC system. This provides a clear basis for scheduling management, resource monitoring, and subsequent dynamic adjustments.

[0093] When scheduling tasks and allocating resources, a set of critical path tasks can be built based on the historical resource consumption data of each numerical model in space weather numerical operations and the preset workflow structure. This set has identified the tasks with the longest execution time in the entire operation dependency topology that directly affect the total operation completion time. By referring to this set, the resource allocation and execution order of these critical tasks can be prioritized, thereby improving the scheduling efficiency and system utilization of the entire operation.

[0094] Based on the above technical solution, optionally, based on computing resource demand data, scheduling mapping relationships, and critical path task sets, and combined with available resource data and business priorities, elastic resource allocation is performed to obtain the task resource allocation scheme and execution time period for each executable task in the executable task set, including:

[0095] Based on computing resource demand data, the resource consumption scale and duration of each executable task in the current scheduling cycle are analyzed to obtain resource constraint information.

[0096] Based on resource constraint information, scheduling mapping relationships, and critical path task sets, the execution order of each executable task and resource occupation conflicts are determined to determine the allocatable time range of each executable task.

[0097] Based on available resource data and business priorities, elastic resource allocation is implemented for each executable task within the allocable time range to obtain the task resource allocation scheme and execution time period for each executable task in the executable task set.

[0098] In this scheme, the resource consumption scale is the total amount of computing resources expected to be occupied by each executable task in the current scheduling cycle, including indicators such as CPU core usage, memory usage, storage I / O rate, and data transfer volume.

[0099] Duration is the length of time required for each executable task to complete from the start of execution, which is the time window during which the task actually occupies system resources.

[0100] Resource constraint information is the system resource limitation information obtained by analyzing the resource consumption scale and duration of each executable task. It includes the amount of computing, memory, storage and data transmission resources available within a specific time period, as well as the resource constraints that may be imposed on task execution.

[0101] The execution order determines the order in which executable tasks are executed within the current scheduling cycle, under the constraints of task dependencies and workflow logic. In other words, it determines which tasks must be executed first, and which tasks can be executed later or in parallel.

[0102] Resource contention occurs when multiple tasks may compete for the same type of resource within the same time period, such as simultaneously requiring the same CPU core, memory, or storage I / O bandwidth.

[0103] The allocatable time range is the time interval during which each executable task can be scheduled and allocated after taking into account the execution order, resource conflicts, and system resource constraints.

[0104] Based on the computational resource requirements of each executable task, the resource consumption scale and duration of each task within the current scheduling cycle are analyzed. Specifically, this involves calculating the demand intensity of each task in terms of CPU cores, memory, storage I / O, and data transfer, and estimating the actual duration of each task in the current scheduling cycle by combining historical execution times and job parameters. By comparing the resource requirements of each task with the available resources of the system, the total amount of resources each task may consume within a given time period and its fluctuation range can be identified, thus obtaining complete resource constraint information. This provides a quantitative basis for subsequent scheduling.

[0105] Based on the obtained resource constraint information, combined with the scheduling mapping relationship of tasks in the workflow and the critical path task set, the execution order and resource contention of each executable task are determined. The steps include arranging tasks according to topological dependencies and critical path priorities, identifying the set of tasks that may generate resource contention within the same time window, and marking tasks that must be executed sequentially and task groups that can be executed in parallel. In this way, the safe execution order of each task throughout the entire scheduling cycle can be clearly defined, and the types and periods of potential resource conflicts can be identified, providing a basis for the next step of time window delineation, thus forming the allocatable time range for each task.

[0106] Based on the available resource data of the HPC system and the business priorities of each task, elastic resource allocation is implemented for each executable task within the aforementioned allocatable time range. Specific operations include: dynamically allocating CPU cores, memory, and I / O bandwidth within the time range to prioritize resources for critical tasks, while rationally arranging the execution order of parallel tasks to reduce resource conflicts and waiting times. Through this elastic resource scheduling, a task resource allocation scheme and corresponding execution time period are generated for each task.

[0107] In this solution, flexible allocation and conflict management can optimize the completion time of critical tasks and improve the scheduling efficiency of the entire executable task set.

[0108] S105, based on the task resource allocation scheme and execution time period, performs scheduling and execution management of each executable task, obtains the actual scheduling status and resource usage data of each executable task in real time, and dynamically adjusts the execution order and resource allocation of the remaining tasks in the execution task sequence based on the actual scheduling status and resource usage data, until all executable tasks in the executable task set are completed.

[0109] Actual scheduling status refers to the real-time running status of each executable task in the HPC system during execution, including whether the task has been started, its current running stage, whether it is waiting for resources, whether it has been suspended or paused, and the execution progress of the task.

[0110] Resource usage data refers to the actual computing resources used by each task during its operation, including CPU or computing core utilization, memory usage, storage I / O read / write speed, and network bandwidth usage.

[0111] Remaining tasks refer to the set of executable tasks that have not yet been completed within the current scheduling cycle, including tasks that are waiting for resources, queued, or are being executed but have not yet been completed.

[0112] Execution order refers to the order in which executable tasks in an HPC system are run, determined by task dependencies, priorities, and resource availability.

[0113] Resource allocation refers to the specific computing resources allocated by the system to each task, including the number of CPUs or computing cores, memory capacity, storage I / O and network bandwidth, as well as the task's usage time and parallel execution strategy within the scheduling cycle.

[0114] At the start of scheduling and execution management, the scheduling system uses the generated task resource allocation scheme and execution time period as an initial reference. Each executable task is loaded into the HPC system according to its allocated computing cores, memory, and storage resources, as well as its predetermined start and completion times. Simultaneously, a real-time monitoring mechanism is established to continuously collect data on the task's running status, obtaining the actual scheduling status of each task, including whether the task has started as planned, its current running stage, whether there are any waiting or blocking issues, the percentage of progress completed, and whether the task has been suspended or prematurely terminated. In parallel, the system collects resource usage data for each task, recording the actual number of CPUs or computing cores used, memory usage, storage I / O rate, and network bandwidth consumption during execution. This data is compared with the estimated resource requirements of the task to determine if there is any resource waste or bottleneck, providing data support for subsequent dynamic adjustments.

[0115] During task execution, the scheduling system continuously analyzes all remaining unfinished tasks, dynamically optimizing the execution order based on their position in the task sequence, dependencies, and the status of completed tasks. This involves adjusting the order in which tasks are started and their parallel combinations, ensuring that critical path tasks receive necessary resources first, while non-critical tasks are executed in parallel to improve throughput, provided system resources allow. Simultaneously, based on actual scheduling status and resource usage data, resource allocation for tasks is updated in real time. For example, underutilized computing cores, memory, or I / O bandwidth are reallocated to waiting tasks, or resource supply is increased when task execution rates are lower than expected, ensuring the overall job operates efficiently with limited resources.

[0116] Throughout the process, the system continuously monitors, evaluates, and adjusts, capturing the status changes and resource usage of each task in real time. Based on the changes, it adjusts the execution order and resource allocation of the remaining tasks in real time until all tasks in the entire executable task set are completed in the optimized scheduling order. This ensures that critical tasks are completed on time and maximizes the utilization of HPC system resources, while avoiding task blocking, resource waste, or scheduling conflicts.

[0117] The execution of remaining tasks depends on the resource usage and scheduling of current tasks in the system, as there are dependencies and resource-sharing constraints between tasks. By acquiring the actual scheduling status and resource usage data of executable tasks that have started or are currently executing, it is possible to accurately determine which resources are occupied, which tasks have been completed or are about to be completed, and thus determine when the remaining tasks will be able to acquire sufficient resources and the optimal time period for execution. This allows the remaining tasks to dynamically adjust their execution order and resource allocation to adapt to the real-time changing system state, achieving reasonable scheduling and efficient execution.

[0118] In this embodiment, historical resources and task dependencies are used to predict and allocate resources, thereby improving HPC utilization and job efficiency; at the same time, dynamic scheduling optimizes task order and resource allocation to ensure that critical tasks are completed on time.

[0119] Based on the above technical solution, optionally, the execution order and resource allocation of the remaining tasks in the task sequence can be dynamically adjusted based on the actual scheduling status and resource usage data until all executable tasks in the executable task set are completed, including:

[0120] Based on the actual scheduling status and resource usage data, the running status of the remaining tasks in the executable task set is analyzed to obtain the schedulable status information of the remaining tasks.

[0121] Based on the schedulable state information, the execution order constraints and available resource matching relationships among the remaining tasks are analyzed to obtain the scheduling constraints and feasible resource allocation relationships of the remaining tasks.

[0122] Based on the aforementioned scheduling constraints and feasible resource allocation relationships, the execution order and resource allocation status of the remaining tasks are dynamically controlled to obtain the real-time scheduling status and actual resource occupancy information of the remaining tasks.

[0123] Based on the real-time scheduling status and actual resource usage information, the execution priorities of critical path tasks and non-critical tasks in the remaining tasks are adjusted to obtain a dynamically optimized task scheduling control strategy.

[0124] Based on the task scheduling and control strategy, the remaining tasks are dynamically scheduled and resource managed until all executable tasks in the executable task set are completed.

[0125] In this scheme, the schedulable status information is the feasibility and readiness status of the remaining tasks in the executable task set at the current time point, including whether the task meets the preconditions, whether the required resources are available, and whether the task can immediately enter the scheduling queue.

[0126] The execution order constraint is the dependency constraint between the remaining tasks, that is, which tasks must be executed after other tasks are completed, and which tasks can be executed in parallel. These constraints ensure that the task execution order conforms to the job logic and workflow requirements.

[0127] The available resource matching relationship is the correspondence between the remaining tasks' demand for available computing, storage, network, and other resources and the currently available resources of the HPC system.

[0128] Scheduling constraints are a set of constraints derived from a comprehensive analysis of the scheduling feasibility of the remaining tasks, based on execution order constraints and available resource matching relationships. These constraints include the time order that tasks must adhere to, resource limitations, and conditions for parallel execution.

[0129] The feasible resource allocation relationship is the actual combination of resources that each remaining task can obtain under the condition of satisfying scheduling constraints, including CPU, memory, storage bandwidth, etc., to ensure that there are no resource conflicts or shortages during task execution.

[0130] Resource allocation status refers to the actual computing, storage, and network resources occupied by the remaining tasks at a certain moment, reflecting the current resource usage and allocation effect of the tasks.

[0131] Real-time scheduling status refers to the current running status of remaining tasks in the scheduling system, including information such as being executed, waiting for scheduling, or already completed.

[0132] Actual resource usage information represents the actual amount of computing resources, storage I / O, and data transfer consumed during task execution, reflecting the real-time load of system resources.

[0133] Critical path tasks are those located on the longest execution path in the task dependency topology. The completion time of these tasks directly determines the total completion time of the entire job, and their resources and scheduling need to be prioritized.

[0134] Non-critical tasks are tasks that are not on the critical path. Their delays will not directly affect the overall job completion time, but they still need to be scheduled reasonably to optimize resource utilization and parallel execution efficiency.

[0135] Execution priority is the scheduling order of remaining tasks based on task attributes, criticality, and resource requirements. It is used to determine the order of task preemption and resource allocation in dynamic scheduling.

[0136] Task scheduling control strategy is a set of rules for the execution order, resource allocation and scheduling of remaining tasks that are formulated by comprehensively considering schedulable status information, real-time scheduling status, resource occupancy information, priority of key tasks and feasible resource allocation during dynamic scheduling.

[0137] In the initial stage of scheduling and execution, the system first collects the execution metrics of each remaining task in the executable task set in real time, including CPU utilization, memory usage, storage space usage, network bandwidth utilization, and the amount of data currently being processed. Simultaneously, it records the execution progress of the task and the completion status of its predecessor tasks, and matches this dynamic data with the position of each task in the scheduling mapping relationship. By comparing the current state of the task with the pre-calculated resource requirements, the system evaluates whether each task meets the execution conditions within the current time window and determines whether it is subject to dependency constraints. Based on these analyses, the system generates schedulable status information for each remaining task, clarifying its status as immediately executable, waiting for resources, or waiting for the completion of predecessor tasks. It also derives the execution sequence constraints between tasks, i.e., which tasks must be executed sequentially and which tasks can be scheduled in parallel, laying the foundation for resource allocation and execution control. Furthermore, considering the current resource pool status of the HPC system, the system maps the resource requirements of tasks to available resources, forming an available resource matching relationship for each task, i.e., which tasks can be allocated which types of resources, thus obtaining complete scheduling constraints and feasible resource allocation relationships, providing precise input for dynamic scheduling.

[0138] During the execution phase of the remaining tasks, the system dynamically controls the execution order and resource consumption of tasks in real time based on the scheduling constraints and resource allocation relationships obtained in the previous step. Specifically, the system locks the task chains that must be executed sequentially and groups parallel tasks according to available resources, ensuring that each task can run normally within its allocated resource range. Simultaneously, the system monitors the actual resource consumption of tasks in real time, including the number of CPU cores, memory usage, storage I / O, and network throughput, updating the task's resource allocation status with the latest measured data and synchronously updating the real-time scheduling status and actual resource consumption information of each task. If the actual resource consumption of a task deviates from expectations, the system immediately adjusts the resource allocation of other tasks within the same time period to avoid resource conflicts and optimize overall job efficiency. Through this fine-grained control, the execution order of each remaining task is dynamically optimized, while resource utilization is maximized.

[0139] In the final stage of dynamic adjustment, the system identifies the tasks with the greatest impact on the overall job completion time—the critical path tasks—and the non-critical tasks with a smaller impact on the overall schedule. Based on the real-time scheduling status and actual resource usage information of each task, the system adjusts its execution priority, prioritizing the completion of critical path tasks as planned, while flexibly adjusting the execution order of non-critical tasks to fill resource gaps. The priorities and resource allocation schemes of all tasks are combined to form an optimized task scheduling and control strategy. The system implements dynamic scheduling and resource management according to this strategy, making real-time adjustments to the remaining tasks and continuously monitoring task completion until all tasks in the executable task set are completed.

[0140] This solution can dynamically adjust the execution order and resource allocation of remaining tasks based on real-time scheduling status and resource usage, thereby improving task completion efficiency and system resource utilization. At the same time, priority is given to critical path tasks to ensure that jobs are completed on schedule, thus optimizing the scheduling effect of the entire set of executable tasks.

[0141] Figure 2 This is a flowchart illustrating an intelligent and flexible scheduling method for space weather numerical operations provided in this embodiment of the disclosure. The method may include the following steps:

[0142] S201, acquire historical resource consumption data and preset workflow structure of each numerical mode in space weather numerical operation, and construct resource consumption feature set, task dependency topology and critical path task set of each numerical mode based on the historical resource consumption data and preset workflow structure.

[0143] S202, obtain the space weather numerical tasks to be executed in the current scheduling cycle and their operation parameters, perform operation mapping analysis on the space weather numerical tasks based on the operation parameters and task dependency topology, and obtain the scheduling mapping relationship of the space weather numerical tasks in the preset workflow structure and the set of executable tasks.

[0144] S203, obtain the numerical mode type of each executable task in the executable task set, and perform resource demand assessment based on the numerical mode type and resource consumption feature set to obtain the computational resource demand data of each executable task in the executable task set in the current scheduling cycle.

[0145] S204: Obtain available resource data and business priorities of the HPC system. Based on computing resource demand data, scheduling mapping relationship and critical path task set, and combined with available resource data and business priorities, perform elastic resource allocation to obtain the task resource allocation scheme and execution time period of each executable task in the executable task set.

[0146] S205, based on the task resource allocation scheme and execution time period, performs scheduling and execution management of each executable task, obtains the actual scheduling status and resource usage data of each executable task in real time, and dynamically adjusts the execution order and resource allocation of the remaining tasks in the execution task sequence based on the actual scheduling status and resource usage data, until all executable tasks in the executable task set are completed.

[0147] S206, acquire the running status data of each executable task in real time, and determine whether there is any running abnormality based on the running status data.

[0148] Runtime status data refers to the collection of various information reflecting the current status of each executable task during the scheduling and execution process, including whether the task has started, the task execution progress, CPU and memory usage, I / O operation status, task response time, error or exception logs, etc.

[0149] After each executable task begins execution, the system continuously monitors its operation, periodically collecting data such as CPU utilization, memory usage, disk I / O, network transfer rate, task execution progress, and log information to form complete operational status data. This data includes not only resource consumption but also task response time, error or anomaly records, etc., to comprehensively reflect the real-time performance of tasks within the HPC system. The system organizes the collected data by time series and compares it with historical execution data to identify abnormal trends and potential risks in task execution, ensuring dynamic tracking of the status changes of each executable task.

[0150] After acquiring the runtime status data, the system evaluates the actual performance of the task based on data analysis algorithms to determine whether it deviates from the expected behavior. For example, when CPU or memory usage exceeds a set threshold for an extended period, the task progress lags behind the scheduled execution time, or errors or abnormal information appear in the logs, it can be determined that the task is experiencing runtime abnormalities.

[0151] S207, If an operational abnormality exists, an alarm message is generated based on the operational status data and sent to the control center.

[0152] Alarm information is structured data generated when an executable task encounters an abnormality. It describes the specific circumstances of the abnormality, including the type of abnormality, such as CPU overload, abnormal memory usage, task delay, etc., the abnormal task identifier, the time of occurrence of the abnormality, the usage of related resources, the scope of the abnormality, the priority level, and the recommended handling measures.

[0153] The control center is the central system or platform responsible for overall task scheduling, monitoring, and resource management. It can receive alarm information, monitor and handle abnormal events, and trigger operations such as resource adjustments, task rescheduling, or notification of maintenance personnel.

[0154] Upon detecting an anomaly in an executable task, the system extracts anomaly-related information from the acquired operational status data, including the anomaly type, time of occurrence, task identifier, and associated resource usage. Based on this information, the system constructs standardized alarm messages. These alarm messages not only record the specific manifestations of the anomaly but also include the potential impact scope, priority level, and suggested handling measures for rapid anomaly localization and response. After generating the alarm message, the system sends it to the control center via network communication.

[0155] In this embodiment, abnormal situations of executable tasks can be detected in a timely manner, improving the system's response speed to sudden problems. At the same time, by centrally transmitting alarm information to the control center, it is helpful to quickly locate anomalies and take targeted measures to ensure the reliability and continuity of task execution.

[0156] Figure 3 A schematic block diagram of an intelligent flexible scheduling system for space weather numerical operations provided in this disclosure. The system is characterized by comprising:

[0157] The resource dependency analysis module 301 is used to acquire historical resource consumption data and preset workflow structure of each numerical mode in space weather numerical operations, and to construct resource consumption feature set, task dependency topology and critical path task set of each numerical mode based on the historical resource consumption data and preset workflow structure.

[0158] The mapping analysis module 302 is used to obtain the space weather numerical tasks to be executed in the current scheduling cycle and their operation parameters, and to perform operation mapping analysis on the space weather numerical tasks based on the operation parameters and task dependency topology to obtain the scheduling mapping relationship of the space weather numerical tasks in the preset workflow structure and the set of executable tasks.

[0159] The evaluation module 303 is used to obtain the numerical mode type of each executable task in the executable task set, and to evaluate the resource requirements based on the numerical mode type and the resource consumption feature set to obtain the computational resource requirements data of each executable task in the executable task set in the current scheduling cycle.

[0160] The resource elastic allocation module 304 is used to obtain the available resource data and business priorities of the HPC system, and to perform resource elastic allocation based on the computing resource demand data, scheduling mapping relationship and critical path task set, combined with the available resource data and business priorities, to obtain the task resource allocation scheme and execution time period of each executable task in the executable task set.

[0161] The dynamic adjustment module 305 is used to schedule and manage the execution of each executable task based on the task resource allocation scheme and the execution time period. It obtains the actual scheduling status and resource usage data of each executable task in real time, and dynamically adjusts the execution order and resource allocation of the remaining tasks in the execution task sequence based on the actual scheduling status and resource usage data until all executable tasks in the executable task set are completed.

[0162] The intelligent flexible scheduling system for space weather numerical operations provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0163] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0164] Furthermore, the present invention also provides an electronic device 400, including a processor 401, a memory 402, and a program or instructions stored in the memory 402 and executable on the processor 401. When the program or instructions are executed by the processor 401, they implement the various processes of the above-described embodiment of the intelligent flexible scheduling method for space weather numerical operations and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0165] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0166] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program for executing a smart flexible scheduling method for space weather numerical operations according to the above-described method embodiments. This program can be loaded and run by a processor to implement the aforementioned smart flexible scheduling method for space weather numerical operations. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0167] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, hardware, or a combination of software and hardware. Therefore, the number of modules shown in the figures is merely illustrative.

[0168] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of the present invention; therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.

[0169] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for intelligent and flexible scheduling of space weather numerical operations, characterized in that, The method includes: Historical resource consumption data and preset workflow structure of each numerical model in space weather numerical operations are obtained. Based on the historical resource consumption data and preset workflow structure, resource consumption feature set, task dependency topology and critical path task set of each numerical model are constructed. Obtain the space weather numerical tasks to be executed in the current scheduling cycle and their operation parameters. Based on the operation parameters and task dependency topology, perform operation mapping analysis on the space weather numerical tasks to obtain the scheduling mapping relationship of the space weather numerical tasks in the preset workflow structure and the set of executable tasks. Obtain the numerical mode type of each executable task in the executable task set, and perform resource demand assessment based on the numerical mode type and resource consumption feature set to obtain the computational resource demand data of each executable task in the executable task set in the current scheduling cycle. Obtain available resource data and business priorities of the HPC system. Based on computing resource demand data, scheduling mapping relationship and critical path task set, and combined with available resource data and business priorities, perform elastic resource allocation to obtain the task resource allocation scheme and execution time period of each executable task in the executable task set. Based on the task resource allocation scheme and execution time period, the system schedules and manages the execution of each executable task, obtains the actual scheduling status and resource usage data of each executable task in real time, and dynamically adjusts the execution order and resource allocation of the remaining tasks in the execution task sequence based on the actual scheduling status and resource usage data until all executable tasks in the executable task set are completed.

2. The intelligent flexible scheduling method for space weather numerical operations according to claim 1, characterized in that, in, Based on the historical resource consumption data and the preset workflow structure, a resource consumption feature set, task dependency topology, and critical path task set for each numerical mode are constructed, including: Based on the historical resource consumption data, feature extraction is performed to obtain resource consumption feature sets for each numerical pattern; Based on the preset workflow structure and resource consumption feature set, task dependency analysis is performed to obtain the task dependency topology corresponding to each numerical mode. Based on the resource consumption feature set and task dependency topology, critical path analysis is performed on the tasks corresponding to each numerical model to obtain the critical path task set for each numerical model.

3. The intelligent flexible scheduling method for space weather numerical operations according to claim 1, characterized in that, in, Based on the aforementioned operational parameters and task dependency topology, a job mapping analysis is performed on space weather numerical tasks to obtain the scheduling mapping relationship of space weather numerical tasks in a preset workflow structure and the set of executable tasks, including: Based on space weather numerical missions and their operational parameters, a list of missions to be mapped is obtained through mission filtering and analysis. Based on the list of tasks to be mapped and the task dependency topology, determine the mapping position and dependency relationship of space weather numerical tasks in the preset workflow structure, and determine the scheduling mapping relationship of space weather numerical tasks in the preset workflow structure based on the mapping position and dependency relationship. Based on the mapping location and dependency relationship, identify the set of logically parallelizable tasks, and based on the set of logically parallelizable tasks, generate the set of executable tasks for space weather numerical tasks in the preset workflow structure.

4. The intelligent flexible scheduling method for space weather numerical operations according to claim 1, characterized in that, in, Based on the numerical pattern type and resource consumption feature set, resource demand assessment is performed to obtain the computational resource demand data of each executable task in the executable task set during the current scheduling cycle, including: Based on the numerical pattern type, each executable task is associated with the computing resource features corresponding to the resource consumption feature set to obtain the pattern resource feature parameters corresponding to each executable task. Based on the aforementioned pattern resource characteristic parameters, the computational resource and data transmission requirements of each executable task in the current scheduling cycle are evaluated to obtain the computational resource requirement data of each executable task in the executable task set.

5. The intelligent flexible scheduling method for space weather numerical operations according to claim 1, characterized in that, in, Based on computing resource demand data, scheduling mapping relationships, and critical path task sets, and combined with available resource data and business priorities, elastic resource allocation is performed to obtain the task resource allocation scheme and execution time period for each executable task in the executable task set, including: Based on computing resource demand data, the resource consumption scale and duration of each executable task in the current scheduling cycle are analyzed to obtain resource constraint information. Based on resource constraint information, scheduling mapping relationships, and critical path task sets, the execution order of each executable task and resource occupation conflicts are determined to determine the allocatable time range of each executable task. Based on available resource data and business priorities, elastic resource allocation is implemented for each executable task within the allocable time range to obtain the task resource allocation scheme and execution time period for each executable task in the executable task set.

6. The intelligent flexible scheduling method for space weather numerical operations according to claim 1, characterized in that, in, The execution order and resource allocation of the remaining tasks in the task sequence are dynamically adjusted based on the actual scheduling status and resource usage data until all executable tasks in the executable task set are completed, including: Based on the actual scheduling status and resource usage data, the running status of the remaining tasks in the executable task set is analyzed to obtain the schedulable status information of the remaining tasks. Based on the schedulable state information, the execution order constraints and available resource matching relationships among the remaining tasks are analyzed to obtain the scheduling constraints and feasible resource allocation relationships of the remaining tasks. Based on the aforementioned scheduling constraints and feasible resource allocation relationships, the execution order and resource allocation status of the remaining tasks are dynamically controlled to obtain the real-time scheduling status and actual resource occupancy information of the remaining tasks. Based on the real-time scheduling status and actual resource usage information, the execution priorities of critical path tasks and non-critical tasks in the remaining tasks are adjusted to obtain a dynamically optimized task scheduling control strategy. Based on the task scheduling and control strategy, the remaining tasks are dynamically scheduled and resource managed until all executable tasks in the executable task set are completed.

7. The intelligent flexible scheduling method for space weather numerical operations according to claim 1, characterized in that, in, After scheduling and managing the execution of each executable task based on the task resource allocation scheme and execution time period, the method further includes: Real-time acquisition of runtime status data for each executable task; determination of whether runtime anomalies exist based on the runtime status data. If an operational anomaly is detected, an alarm message is generated based on the operational status data and sent to the control center.

8. An intelligent and flexible scheduling system for space weather numerical operations, characterized in that, The system includes: The resource dependency analysis module is used to acquire historical resource consumption data and preset workflow structure of each numerical model in space weather numerical operations. Based on the historical resource consumption data and preset workflow structure, it constructs resource consumption feature set, task dependency topology and critical path task set of each numerical model. The mapping analysis module is used to obtain the space weather numerical tasks to be executed in the current scheduling cycle and their operation parameters. Based on the operation parameters and task dependency topology, the module performs operation mapping analysis on the space weather numerical tasks to obtain the scheduling mapping relationship of the space weather numerical tasks in the preset workflow structure and the set of executable tasks. The evaluation module is used to obtain the numerical mode type of each executable task in the executable task set, and to evaluate the resource requirements based on the numerical mode type and the resource consumption feature set to obtain the computational resource requirements data of each executable task in the executable task set in the current scheduling cycle. The resource elastic allocation module is used to obtain available resource data and business priorities of the HPC system. Based on computing resource demand data, scheduling mapping relationship and critical path task set, and combined with available resource data and business priorities, it performs resource elastic allocation to obtain the task resource allocation scheme and execution time period of each executable task in the executable task set. The dynamic adjustment module is used to schedule and manage the execution of each executable task based on the task resource allocation scheme and the execution time period. It obtains the actual scheduling status and resource usage data of each executable task in real time, and dynamically adjusts the execution order and resource allocation of the remaining tasks in the execution task sequence based on the actual scheduling status and resource usage data, until all executable tasks in the executable task set are completed.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of an intelligent flexible scheduling method for space weather numerical operations as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of an intelligent flexible scheduling method for space weather numerical operations as described in any one of claims 1-7.

Images