A task scheduling and data interaction system and method for large sample simulation

Abstract

A task scheduling and data interaction system and method for large-sample simulation includes a three-layer system of "center-region nodes". The center layer is used to batch allocate tasks to the most suitable regions according to the computational characteristics of the tasks. The region layer performs fine-grained allocation based on the real-time resource status of the nodes in the node layer. The computational results generated on different nodes flow to the distributed stream processing of the center layer to complete the pre-deployed analysis job. Data interaction between different regions in the region layer is carried out using the DDS data exchange bus. The flow of computational results to the distributed stream processing of the center layer is carried out using the DDS data exchange bus or a regional data aggregator. The node layer can automatically and seamlessly switch between microsecond-level "shared memory communication" and DDS-based real-time "network communication" depending on whether the models involved in the interactive data are located on the same physical node.

Description

Technical Field

[0001] This invention relates to a task scheduling and data interaction system and method, and more particularly to a hierarchical distributed task scheduling and data interaction system and method for large-sample simulation, belonging to the field of simulation data processing. Background Technology

[0002] In the development of complex equipment such as aviation, aerospace, shipbuilding, and automobiles, model-based systems engineering (MBSE) and digital twin technology have become core development trends. To comprehensively evaluate equipment performance, verify system reliability, and explore optimal parameter combinations during the design phase, engineers need to perform large-scale simulation calculations, also known as "large-sample simulations." This typically means automating and batch-running simulations of thousands of parameter combinations. Examples include: parameter scanning of aircraft wings at different angles of attack and Mach numbers; full-condition analysis of fuel economy and propulsion efficiency of ship propulsion systems under different power levels, speeds, and sea states; and Monte Carlo simulation of the durability of automobile chassis under different road surfaces, loads, and speeds.

[0003] The existing technical solution is as follows: Solution 1: Task scheduling based on a workflow engine. In industry and academia, workflow management systems such as Apache Airflow, Celery, and AWS Step Functions are widely used for orchestrating and scheduling computational tasks. The typical approach is to define each simulation scenario as an independent task node. A centralized scheduler distributes tasks to distributed worker nodes for execution based on the task dependencies defined by a directed acyclic graph (DAG). After the worker nodes complete their execution, they feed back their status and results to the scheduler.

[0004] Option 2: Job scheduling based on high-performance computing clusters. Traditional supercomputing centers and large enterprise computing centers commonly use job scheduling systems such as Slurm, PBS Pro, and LSF to manage large-scale computing resources. Users write a job script for each workload and submit it to the queue via the sbatch or qsub command. The scheduler matches and allocates resources based on the requested resources (number of CPU cores, memory size, number of GPUs) and the available resources on each computing node in the cluster, then starts the job on the selected node for execution.

[0005] Option 3: Batch task processing based on message queues. Message queue systems commonly used in the internet industry and distributed systems (such as Apache Kafka, RabbitMQ, and AWS SQS) are also used to build batch processing systems. The architecture is as follows: all the operational condition descriptions to be simulated are published as messages to one or more message queues; multiple computing nodes act as consumers, concurrently pulling messages from the queues, parsing the operational parameters, and starting the local simulation model; after execution, the result data is sent as a new message to another result queue or written to a database.

[0006] Option 4: Distributed Simulation Based on Co-simulation Standards. In the field of complex system simulation, HLA (High-Level Architecture) and DIS (Distributed Interactive Simulation) are classic standards. They achieve interconnection and timing synchronization between multiple simulation applications (federation members) through a runtime support environment (RTI). RTI typically provides services such as declaration management, object management, time management, and ownership management to ensure the consistency of the entire simulation federation in time and space. Summary of the Invention

[0007] I. The problem solved by the core technology of this invention 1. High-concurrency scheduling and management of massive deterministic tasks: Existing technologies (such as workflow engines, HPC job scheduling systems, message queues, etc.) are mainly geared towards task flows with complex dependencies or coarse-grained independent jobs. However, when faced with a large number of predefined independent subtasks (such as parameter scanning conditions and Monte Carlo simulation samples), there is a lack of efficient batch task automatic splitting, high-throughput distribution, full lifecycle status tracking, and fine-grained failure retry mechanisms, resulting in slow task distribution speed, limited concurrency, and high management complexity.

[0008] 2. Cross-Domain Resource Collaborative Management and Control in Hierarchical Complex Systems: For large-scale equipment simulation with a clearly defined "overall-system-subsystem" hierarchical structure, existing solutions (whether centralized control or a fully peer-to-peer distributed architecture) struggle to achieve the optimal balance between unified global control and high-frequency real-time local interaction. In centralized architectures, the global control center easily becomes a performance bottleneck and a single point of failure, resulting in significant communication latency across regions (cross-clusters, cross-data center). Meanwhile, fully peer-to-peer distributed architectures lack global coordination capabilities, making it difficult to achieve cross-system, cross-professional resource optimization and global data aggregation and analysis.

[0009] 3. The problem of accurate matching of heterogeneous resources based on task characteristics: In large-sample simulations, the computational characteristics and resource requirements of different working conditions vary greatly. For example, computational fluid dynamics (CFD) requires GPU acceleration, while control system logic verification is more suitable for high-frequency CPUs, and structural finite element analysis (CSM) may be sensitive to memory bandwidth. Existing technologies lack a mechanism that can automatically identify the computational characteristics (type, complexity) of subtasks and accurately and dynamically match them with heterogeneous resource pools such as GPUs / CPUs, resulting in low resource utilization and a large amount of idle or misallocated computing resources.

[0010] 4. The problem of streaming processing and real-time analysis feedback of simulation results: The traditional "simulate first, process later" model (writing all simulation results to disk files and starting the analysis program only after all tasks are completed) leads to a serious lag in the analysis of massive amounts of data. In large-sample simulation scenarios, this means that engineers may have to wait for hours or even days to obtain preliminary conclusions, making it impossible to achieve real-time feedback and rapid decision-making in design iterations, which seriously restricts R&D efficiency.

[0011] 5. The challenge of optimizing performance for high-frequency coupling and interaction among multiple models within a node: In distributed simulations, multiple highly coupled models often need to exchange data at extremely high frequencies (milliseconds or even microseconds) (e.g., co-simulation of engine and transmission models). Existing communication methods based on network protocol stacks (even using standard protocols such as DDS and HLA / RTI) still have significant latency and CPU overhead, failing to meet the performance requirements of such scenarios.

[0012] 6. A task-oriented shared-memory node and cross-node integrated heterogeneous graph neural network is constructed to predict the optimal path during task execution based on historical accumulated data. This allows for pre-computation based on task characteristics before simulation-on-demand, thus obtaining the data potentially needed in the task ahead of time. When the simulation requires further computation, it can access any node on the optimal path to check if pre-computed data is available. This ensures that key data needed for actual execution is prepared before the simulation begins, saving actual operation time and significantly improving task completion efficiency. This scheme relies on the existing strategy of simulation before computation, treating traditional robust computing architectures as a database of computational results for use in this invention. This coexistence of traditional and new systems can meet the needs of users in different computing modalities (including the volume of data, the size of the model, and the complexity of the computation) and improve the utilization rate of traditional systems in a low-cost way (i.e. without completely replacing the traditional system with the new system of this invention). It also helps the new system to improve quality and efficiency, especially for high-order computing modal tasks (i.e., large data volume, large model size, and high complexity).

[0013] II. Specific Technical Solution of the Invention This invention provides a hierarchical distributed task scheduling and data interaction system for large-sample simulation, comprising a three-tiered system of "center-region-node". The center layer (global control center) is deployed in the core data center and is responsible for global data management and task decision-making. It contains the system's operating software. The regional layer (regional coordination center) is used as a logical control unit divided according to physical location, network topology, or logical organization; The node layer (model execution node) includes simulation model execution units; The central layer allocates tasks in batches to the most suitable regions based on their computational characteristics. The regional layer then performs fine-grained allocation based on the real-time resource status of nodes in the node layer. Computational results generated on different nodes flow to the distributed stream processing engine in the central layer to complete the pre-deployed analysis job (i.e., a distributed computing job). Data interaction between different regions in the regional layer uses the DDS data exchange bus, while the flow of computational results to the distributed stream processing engine in the central layer uses either the DDS data exchange bus or a regional data aggregator. The node layer can automatically and seamlessly switch between microsecond-level "shared memory communication" and real-time "network communication" based on DDS, depending on whether the models involved in the interactive data are located on the same physical node.

[0014] Optionally, the physical location includes R&D centers located in different geographical locations and different computer rooms in the same data center; the network topology includes a local area network; and the logical organization includes the power system subsystem and the avionics system subsystem.

[0015] Optionally, the central layer further includes: a global task manager, a global resource and status monitor, a global time base generator, and a global result database interface, wherein the global task manager provides a Web UI or API interface to receive large sample tasks defined by the user through the "parameterized configuration" interface. The global task manager is responsible for generating a unique ID and a subtask description file containing specific parameter values ​​for each subtask and storing them in a distributed task queue. The global resource and status monitor continuously monitors the health status and real-time load information of all registered region and node layers through a heartbeat mechanism, and builds and maintains a global dynamic resource view. For simulation scenarios with strong cross-region timing consistency requirements (generally, more than half of the data interaction requires cross-region interaction), the global time base generator is responsible for generating a globally unique, high-precision time synchronization signal and distributing it to each region layer and node layer through NTP or its enhanced protocol to ensure the uniformity of the simulation time of the entire system. The global results database interface provides a standard interface for receiving the final simulation results aggregated from various regions and persistently storing them in a distributed file system or time series database for global analysis and visualization.

[0016] Optionally, the regional layer further includes: a regional task scheduler, which pulls a batch of subtasks allocated to the regional area from the global task queue. It maintains a real-time local dynamic resource view of all nodes within the regional area and performs secondary fine-grained scheduling. The scheduling algorithm not only considers the idle CPU / memory of nodes, but more importantly, combines the computational feature tags of the tasks to accurately deliver the tasks to the most suitable nodes; The regional fault handler is responsible for ensuring the reliability of its region. It continuously monitors the heartbeat of each node within the region. If a node fails to respond to the heartbeat three times consecutively, it is determined that the node has failed. The regional fault handler immediately executes the following local fault recovery process: the subtasks that are running or waiting to run on the node are re-added to the regional task queue and scheduled to be executed on other healthy nodes in the same region.

[0017] The regional data aggregator is responsible for the initial processing and aggregation of simulation data within its region. It receives real-time result data streams from nodes within the region, performs data format standardization, outlier cleaning, and data compression. The compressed data is then packaged according to time windows or data volume thresholds and uploaded in batches to the global result database at the central layer, effectively reducing cross-regional network bandwidth pressure and the write pressure at the central layer. Inter-regional communication gateways are responsible for handling cross-domain data interaction requests between their own region and other regions, and ensuring the security of cross-regional control commands and data transmission through encrypted channels.

[0018] Optionally, the node layer further includes: a model access adapter constituting a pluggable component library, providing adaptability to various mainstream simulation model interface specifications, including but not limited to: a) FMI / FMU Standard: Supports importing models that conform to the functional model interface standard; b) Simulink Model: Supports encapsulating Simulink-generated code or models into executable units; c) Modelica models: Supports importing models based on the Modelica language; d) Self-developed code model: Provides APIs to support users in integrating models written in languages ​​such as C++, Fortran, and Python after encapsulation; The local scheduler, which constitutes a lightweight scheduling component, is responsible for receiving instructions from the regional scheduler and controlling the execution of the local model according to those instructions. It is also responsible for managing the process or thread lifecycle of the local model.

[0019] The data caching module is a component used to achieve high-performance communication within a node. It opens a high-speed shared buffer in the node's memory and adopts a "publish-subscribe" local memory version mode. When a model running on the same node needs to output data, it only needs to write the data to a specific area of ​​the corresponding "topic" in the shared buffer. Another model that has subscribed to the "topic" can directly read the data through memory mapping.

[0020] The entire process is completed entirely in user space, without going through any network protocol stack or system calls, thus achieving microsecond-level communication latency.

[0021] The local resource monitoring agent is responsible for collecting real-time resource usage data for this node and periodically reporting it to the regional fault handler via heartbeat packets.

[0022] Optionally, the system further includes a pre-computation system for calculating the optimal computation path for completing the task using a pre-built heterogeneous graph neural network based on the task's computational characteristics, and pre-compiling the path for use in the central layer data analysis operation. The physical layer of the heterogeneous graph neural network consists of multiple memory nodes distributed in different regional layers. Its logical layer uses these memory nodes as network nodes. By using the computational characteristics of historical tasks and the data interaction paths between nodes, it achieves node aggregation and path aggregation. Different node importances are used to obtain the optimal computational path through path aggregation. A mapping relationship is established between the optimal computational path and the corresponding task computational characteristics. When a new task is received, the computational characteristics of the new task are used to obtain the current optimal computational path. The analysis jobs on the path nodes are calculated using the pre-computation system to form pre-computation results. When the central layer performs distributed computing jobs, it first searches in the memory of the pre-computation system to see if the required pre-computation results exist. If they do, they are adopted; otherwise, the node data reception and analysis jobs are performed normally.

[0023] Optionally, node aggregation and path aggregation of network nodes are achieved through the computational characteristics of historical tasks and the data interaction paths between nodes. The optimal computational path is obtained through path aggregation based on the importance of different nodes. The method for establishing a mapping relationship between the optimal computational path and the corresponding task computational characteristics is as follows: The first step is to obtain the computational characteristics of all computational tasks within a single historical task. The second step is to obtain the nodes in the node layer that are finally assigned based on computational characteristics, as well as the data interaction flow between nodes, forming multiple paths. It is understandable that for the same computational feature, due to different task objectives, it may be assigned to different nodes to complete the task (for example, the computational complexity is quite large, but the models and algorithms involved are different, so it needs to be specifically assigned to different nodes, which may be the same region or different regions in the region layer), rather than a fixed path.

[0024] The third step involves performing node aggregation based on the acquired nodes to obtain the importance features of the nodes corresponding to the calculated features. Furthermore, based on the acquired multiple paths, path aggregation is performed to obtain the weight of each path corresponding to the calculated features. The fourth step is to select the path with the highest weight as the optimal computation path and construct a mapping from computational features to the optimal computation path. It should be understood that optimal path selection is equivalent to choosing the path with the highest probability of computation when faced with the same computational features.

[0025] The fifth step involves constructing multiple mappings from computational features to optimal computational paths for multiple historical tasks, following all the steps outlined above. This mapping set serves as the mapping relationship between the optimal computational path and the corresponding task computational features.

[0026] A single historical task can be a phased historical task within a preset period of time, or it can be a complete historical task that achieves the predetermined task objective throughout the entire simulation process.

[0027] Preferably, when the pre-calculation system is not used, the system is run independently to obtain independent running results. When the pre-calculation system is used, the pre-calculation adoption results that actually occur in the system are statistically analyzed, and the independent running results are compared with the pre-calculation adoption results. If the difference between the two is within a preset threshold range, the pre-calculation adoption results are used as the final calculation results; otherwise, the independent running results are adopted as the final calculation results.

[0028] It is easy to understand that, since the pre-computation system relies on the traditional system, it may make calculation errors when faced with highly complex tasks, which may lead to incorrect calculation results after adoption. Therefore, it is necessary to run the system independently to ensure the reliability of the results, thereby evaluating the reliability of the pre-computation.

[0029] Since the calculation is derived from the analysis of the optimal path in historical tasks, its calculation order is clear and does not require real-time allocation. It only needs to clarify the calculation characteristics to achieve resource allocation and one-time calculation, so its efficiency is extremely high. However, when inputting data, it is necessary to go through two data input paths: the system and the pre-system.

[0030] Optionally, the current confidence level is set for each mapping in the mapping relationship by using the pre-calculated adoption result as the current proportion of the final calculation result, and the current confidence level is continuously and dynamically updated based on the accumulation of historical tasks.

[0031] Preferably, if the confidence level of the calculated feature-corresponding mapping does not meet the preset range, the required pre-calculated result is not adopted in the execution of the new task, and the normal node data reception and analysis operation is performed; otherwise, the required pre-calculated result is adopted.

[0032] Another aspect of the present invention provides a hierarchical distributed task scheduling and data interaction method for large-sample simulation based on the aforementioned hierarchical distributed task scheduling and data interaction system for large-sample simulation, comprising the following steps: Step S1: Parameterized definition and automatic splitting of large sample tasks. The system engineer logs into the web interface of the system software, selects the "System Simulation" scenario template, and defines at least one variable to be scanned in the parameter configuration panel; Click the "Generate Task" button and set the task properties. The global task manager in the system background will automatically perform the Cartesian product operation to generate multiple independent subtasks. Each subtask description file contains a unique ID for that working condition, variable values, and a path pointing to the basic simulation model. Step S2: Secondary intelligent scheduling based on task characteristics and resource profiles, including (1) first-level scheduling from the central layer to the regional layer, and second-level scheduling from the regional layer to the node layer; (2) after the regional task scheduler receives a batch of sub-tasks, it combines the real-time resource status of each GPU node in the region and adopts a dynamic allocation strategy to accurately distribute each sub-task to the most idle GPU node. Step S3: Data interaction of the hierarchical and differentiated strategy model, specifically including (1) intra-node interaction (memory exchange), (2) cross-node interaction (data bus based on DDS). Step S4: Simulation-as-analysis streaming data processing, which includes the intermediate and final results generated by the model calculation at each node during the large-scale execution of the simulation task, continuously flowing to the central distributed stream processing engine through the DDS bus or local regional data aggregator, on which analysis jobs are pre-deployed. Step S5: Result aggregation and persistence.

[0033] Optionally, the method further includes step S6: constructing the aforementioned pre-computation system, calculating the optimal computation path for the task based on the task's computational characteristics using a pre-built heterogeneous graph neural network, and pre-calculating the path for use by the central layer data analysis job; when the central layer performs distributed computation, it first searches the memory of the pre-computation system for the existence of the required pre-computation results. If they exist, they are adopted for the analysis job; otherwise, the node data reception and analysis job proceeds normally; and the results adopted by the pre-computation are used as the current proportion of the final computation results to set the current confidence level for each mapping in the mapping relationship, and the current confidence level is continuously and dynamically updated based on the accumulation of historical tasks; If the confidence level corresponding to the calculated feature map does not meet the preset range, the required pre-calculated result will not be adopted in the execution of the new task, and the normal steps of receiving and analyzing node data will be performed; otherwise, the required pre-calculated result will be adopted.

[0034] Beneficial effects 1. Significantly Improved System Scalability and Task Throughput: Compared to centralized architectures (Solutions 1 and 4), which are limited by the performance of the central node, and HPC scheduling (Solution 2), which suffers from coarse-grained overhead, this invention distributes management pressure through a three-tiered "center-region-node" architecture and two-tiered intelligent scheduling, achieving horizontal scalability of scheduling capabilities. Real-world testing shows that at a scale of thousands of nodes, the subtask distribution speed of this invention is more than an order of magnitude faster than traditional workflow engines, supporting the initiation of tens of thousands of subtasks per second.

[0035] 2. Significantly Optimized Heterogeneous Resource Utilization: Compared to existing technologies (Solutions 1, 2, and 3) which lack task feature awareness and precise matching capabilities, this invention efficiently guides CFD tasks to GPU clusters and logical tasks to CPU clusters through a two-level matching process of "task finding region" and "task finding node." Combined with dynamic load awareness, the overall utilization rate of heterogeneous computing clusters can be increased by 30%-50%, significantly reducing resource waste and task queuing time.

[0036] 3. Significantly shortened simulation analysis cycle: Compared to the traditional serial mode of "simulation first, processing later" (Schemes 2 and 3), the "simulation-analysis-while-analyzing" streaming processing mode of this invention realizes the parallelization of simulation and analysis. For large-sample simulations containing 100,000 operating conditions, if some results can reveal the core laws, the analysis feedback time can be shortened from several days to several hours, achieving "instant feedback" and greatly accelerating design iteration.

[0037] 4. Significantly Enhanced System Robustness and Simulation Success Rate: Compared to the single-point failure risk of centralized architectures (Solutions 1 and 4) and the vulnerability of simple distributed solutions (Solution 3), the regional autonomous fault tolerance mechanism of this invention limits the impact of faults to the region and achieves local self-healing through automated task migration. Combined with breakpoint continuation and refined retry strategies, the success rate of ultra-long-duration, ultra-large-scale simulations can be increased to over 99.9%.

[0038] 5. Optimal Balance Between Model Interaction Performance and Flexibility: Compared to all existing solutions (1 to 4), the adaptive data interaction method of this invention is unique. It provides shared memory communication performance (microsecond level) comparable to single-machine multi-process for high-frequency coupled models within the same node, while retaining the standardized, loosely coupled, and highly reliable network communication capabilities based on DDS for cross-node models. This "win-win" design is something that pure DDS or pure HLA solutions cannot achieve.

[0039] 6. By adopting a pre-computation system, the computational pressure on nodes is further reduced. The system can selectively adopt the results by querying the pre-computation system to see if the calculation results of the corresponding optimal path node exist. Overall, the system improves the overall operational efficiency of the new system without abandoning the traditional architecture and improving its utilization rate. Attached Figure Description

[0040] Figure 1 This is a three-layer system architecture diagram of "center-region-node" according to Embodiment 1 of the present invention. Figure 2 This is a flowchart illustrating the lifecycle management of a large-sample simulation task performed by the system in Embodiment 1 of the present invention. Figure 3 This is a flowchart of the streaming data processing for "simulation-as-analysis". Figure 4 Flowchart of the method for establishing a mapping relationship between the optimal computation path and the corresponding task computation features. Figure 5 This is a flowchart illustrating the adoption of pre-calculation results by the system including the pre-calculation system in Embodiment 3 of the present invention. Figure 6 This is a schematic diagram of the system architecture based on large-scale cluster simulation at the regional layer. Detailed Implementation

[0041] Example 1 The system is further described in this embodiment as follows: This embodiment provides a method such as Figure 1 A hierarchical distributed task scheduling and data interaction system for large-sample simulation is proposed, comprising a three-tier architecture of "center-region-node," including, for example... Figure 1 The topmost central layer includes: a global task manager, a global resource and status monitor, a global time base generator, and a global results database interface. The global task manager provides a Web UI or API interface, receiving large-sample tasks defined by users through a "parameterized configuration" interface. For example, users can define variables such as "host power (50%-100%, step size 10%)" and "speed (10-20 knots, step size 2 knots)" and select the associated basic simulation scenario template. The system background automatically performs Cartesian product operations to generate a list of all parameter combinations (e.g., 6 power types × 6 speed types = 36 sub-tasks). The global task manager is responsible for generating a unique ID and a sub-task description file containing specific parameter values ​​for each sub-task, and storing them in a distributed task queue. The global resource and status monitor continuously monitors the health status and real-time load information (CPU utilization, GPU utilization, memory usage, network bandwidth utilization) of all registered region and node layers through a heartbeat mechanism (such as once every 1 second), and builds and maintains a global dynamic resource view. For simulation scenarios with strong cross-region timing consistency requirements (generally, more than half of the data interaction requires cross-region interaction), the global time base generator is responsible for generating a globally unique, high-precision time synchronization signal and distributing it to each region layer and node layer through NTP or its enhanced protocol to ensure the uniformity of the simulation time of the entire system. The global results database interface provides a standard interface for receiving the final simulation results aggregated from various regions and persistently storing them in a distributed file system or time series database for global analysis and visualization.

[0042] In this embodiment, the intermediate region layer is exemplified by two regions, Region 1 and Region 2, and further includes: a region task scheduler, which pulls a batch of subtasks allocated to its region from the global task queue. It maintains a real-time local dynamic resource view of all nodes within its region and performs secondary fine-grained scheduling. The scheduling algorithm not only considers the idle CPU / memory of nodes, but more importantly, it combines the computational feature labels of tasks (such as "CFD-high precision", "CSM-fast", "control-real-time") to accurately deliver tasks to the most suitable nodes (e.g., assigning CFD tasks to nodes with idle GPUs). The regional fault handler is responsible for ensuring the reliability of its designated region. It continuously monitors the heartbeat of each node within the region (e.g., every 500ms). If a node fails to respond to its heartbeat three times consecutively, it is considered to have failed. The regional fault handler immediately executes a local fault recovery process: it re-adds the running and waiting subtasks on that node to the regional task queue and schedules them for execution on other healthy nodes within the same region. The entire process is completely transparent to the upper-level central layer, achieving local "immunity" to faults.

[0043] The regional data aggregator is responsible for the initial processing and aggregation of simulation data within its region. It receives real-time result data streams from nodes within the region, performs data format standardization, outlier cleanup, and data compression (e.g., using LZ4 or Snappy algorithms to compress non-real-time batch data). The compressed data is then packaged according to time windows or data volume thresholds and uploaded in batches to the global result database at the central layer, effectively reducing cross-regional network bandwidth pressure and the write pressure at the central layer. Inter-regional communication gateways are responsible for handling cross-domain data interaction requests between their own region and other regions, and ensuring the security of cross-regional control commands and data transmission through encrypted channels (such as TLS).

[0044] The node layer described below further includes: a model access adapter that constitutes a pluggable component library, providing adaptation capabilities to various mainstream simulation model interface specifications, including but not limited to: a) FMI / FMU Standard: Supports importing models that conform to the functional model interface standard; b) Simulink Model: Supports encapsulating Simulink-generated code or models into executable units; c) Modelica models: Supports importing models based on the Modelica language; d) Self-developed code model: Provides APIs to support users in integrating models written in languages ​​such as C++, Fortran, and Python after encapsulation; The local scheduler, which constitutes a lightweight scheduling component, is responsible for receiving instructions (such as start, pause, stop, and step) from the regional scheduler and controlling the execution of the local model according to the instructions. It is responsible for managing the process or thread lifecycle of the local model.

[0045] The data caching module is a component used to achieve high-performance communication within a node. It opens a high-speed shared buffer in the node's memory and adopts a "publish-subscribe" local memory version mode. When a model running on the same node needs to output data, it only needs to write the data to a specific area of ​​the corresponding "topic" in the shared buffer. Another model that has subscribed to the "topic" can directly read the data through memory mapping.

[0046] The entire process is completed entirely in user space, without going through any network protocol stack or system calls, thus achieving microsecond-level communication latency.

[0047] The local resource monitoring agent is responsible for collecting real-time resource usage data (CPU, GPU, memory, network I / O) of this node and periodically reporting it to the regional fault handler via heartbeat packets.

[0048] Figure 1 In this process, the central layer allocates tasks in batches to the most suitable regions based on their computational characteristics. The regional layer then performs fine-grained allocation based on the real-time resource status of nodes in the node layer. Computational results generated on different nodes flow to the distributed stream processing engine in the central layer to complete the pre-deployed analysis job (i.e., a distributed computing job). Data interaction between different regions in the regional layer uses the DDS data exchange bus, and the flow of computational results to the distributed stream processing engine in the central layer also uses the DDS data exchange bus. The node layer can automatically and seamlessly switch between microsecond-level "shared memory communication" and real-time "network communication" based on DDS, depending on whether the models involved in the interactive data are located on the same physical node.

[0049] The operating mechanism of the system described in this embodiment is as follows: 1. Parametric Task Generation Mechanism: Unlike manually creating individual tasks, this invention allows users to define variables, their value ranges, and step sizes, enabling the system to automatically complete full-factor combination design and generate massive amounts of working conditions with a single click. This is the starting point for large-sample simulation, greatly improving productivity in the task definition phase.

[0050] This mechanism pertains to central-level task processing operations.

[0051] 2. Two-level intelligent scheduling mechanism based on task characteristics and resource profiles: This is the core scheduling logic of this invention.

[0052] (1) Task feature extraction: The system assigns a “computation feature label” to each task by analyzing the model metadata associated with the task (such as solver type, whether GPU is required) or by predicting through machine learning models.

[0053] (2) Resource profile construction: The system continuously collects the resource composition of each region (such as GPU model and quantity, CPU architecture) and the real-time load of each node to form dynamic global and regional resource profiles.

[0054] (3) Two-level decision-making: The first-level decision is "task finds region", which realizes macro-level matching between computing type and region capability; the second-level decision is "task finds node", which realizes micro-level load balancing. This hierarchical decision-making mechanism avoids the performance bottleneck of the global scheduler in handling massive fine-grained tasks.

[0055] This mechanism is a logical description of a two-level intelligent scheduling mechanism at the region and node levels.

[0056] This logical description specifically corresponds to a two-level intelligent scheduling method based on task characteristics and resource profiles. This method includes: First, the central layer allocates tasks in batches to the most suitable "regions" based on the task's "computational feature tags"; second, the regional layer performs fine-grained allocation based on the "real-time resource status" of nodes within each region. This method is the core scheduling mechanism for achieving precise matching from "task type" to "heterogeneous hardware resources," significantly improving resource utilization.

[0057] 3. Adaptive switching mechanism for memory swapping and DDS network communication: This is the key to achieving a balance between ultimate performance and flexible deployment.

[0058] (1) Location awareness: When assigning tasks, the local scheduler records the physical node ID of each model.

[0059] (2) Communication strategy decision: When model A needs to send data to model B, the communication agent will determine the communication type based on the node IDs of both models. If they are the same, the "Memory Swap" API will be called; if they are different, the "DDS Network Communication" API will be called.

[0060] (3) Transparency: For model developers, they do not need to care about where the model is ultimately deployed; they only need to call the unified "sendData()" and "receiveData()" interfaces. The complex underlying routing and communication protocol switching are completely handled automatically by the platform.

[0061] This mechanism describes the data interaction mechanism between regions and nodes, and the reason for the seamless switching at the microsecond level.

[0062] 4. A streaming data processing method for large-sample simulation: This method integrates simulation data streams with a distributed stream processing engine, performing real-time analysis on the data streams generated during simulation task execution. This changes the traditional batch processing model of "simulate first, then process," and is a core data processing innovation that significantly shortens the simulation-analysis-decision cycle. This mechanism is described as a streaming data processing method.

[0063] 5. A local fault tolerance mechanism based on regional autonomy: This mechanism autonomously monitors node status at the regional layer and automatically performs task migration and retry within the region when a node fault is detected, without reporting to the central layer. This is a key reliability design for achieving local fault "immunity" and rapid recovery, enhancing the system's robustness in large-scale simulation scenarios.

[0064] The "center-region-node" three-tier system architecture, by introducing a "regional layer" with local scheduling, data aggregation, and fault autonomy capabilities, achieves an organic combination of global centralized control and regional local autonomy. It is a fundamental architecture for solving the problems of excessive global center load and high communication latency in large-scale, cross-regional collaborative simulation.

[0065] Example 2 This embodiment will describe the hierarchical distributed task scheduling and data interaction method for large-sample simulation of the system described in Embodiment 1, taking a ship propulsion system as an example, including the following steps: Step S1: Parameterization and automatic splitting of large sample tasks.

[0066] The marine propulsion system engineer logs into the system's web interface and selects the "Propulsion System Simulation" scenario template. In the parameter configuration panel, he defines the two variables to be scanned: Variable 1: Host power (percentage), ranging from 50% to 100%, with a step size of 10%, for a total of 6 values.

[0067] Variable 2: Speed ​​(knots), ranging from 10 to 20 knots, with a step size of 2 knots, for a total of 6 values.

[0068] Click the "Generate Task" button and set the task properties, such as "Maximum Parallelism = 100" and "Number of Retry Failures = 3". The system's background global task manager automatically performs Cartesian product operations to generate 36 independent subtasks. Each subtask description file (e.g., in JSON format) contains a unique ID for that operating condition, the main engine power value, the speed value, and a path to the basic simulation model.

[0069] Step S2: Secondary intelligent scheduling based on task characteristics and resource profiles.

[0070] (1) First-level scheduling (central layer -> regional layer): The global task manager analyzes these 36 subtasks. It finds that the "100% power + 20 knots speed" condition involves fluid cavitation and fluid-structure interaction under high load, with high computational complexity, belonging to the "CFD-high precision" characteristic, and requires GPU acceleration. On the other hand, the "50% power + 10 knots speed" condition belongs to the conventional propulsion system performance calculation, belonging to the "control-conventional" characteristic, which can be handled by the CPU. At this time, the global resource profile provided by the global resource monitor shows that "Area A" in the Shanghai data center has a large number of idle NVIDIA A100 GPU clusters, while "Area B" in the Wuhan data center is mainly composed of high-frequency CPU nodes. Therefore, the global task manager batches subtasks with the "CFD-high precision" characteristic to Area A, and subtasks with the "control-conventional" characteristic to Area B.

[0071] (2) Second-level scheduling (region layer -> node layer): After receiving a batch of CFD subtasks, the region task scheduler in region A combines the real-time resource status of each GPU node in the region (such as low GPU utilization and sufficient video memory of a certain node) and adopts a dynamic allocation strategy to accurately distribute each subtask to the most idle GPU node. The scheduler in region B also allocates the subtasks it is responsible for to idle CPU nodes in a similar way.

[0072] Step S3: Data interaction of hierarchical and differentiated policies.

[0073] Suppose that on a GPU node in region A, to compute a certain CFD scenario, two highly coupled models need to be run: a high-precision fluid dynamics model (Model A) and a real-time cavitation model (Model B), such as... Figure 3 As shown.

[0074] (1) Intra-node interaction (memory swapping): Since Model A and Model B are scheduled to the same physical node, when the local scheduler starts them, it automatically notifies the data caching module to create shared topics in shared memory for the output parameter "flow field pressure distribution" of Model A and the output parameter "cavitation volume fraction" of Model B, respectively. After Model A completes the calculation of each time step, it writes the "flow field pressure distribution" data to the shared memory. When M2 needs this data for the next time step calculation, it reads it directly from the shared memory. The whole process is completed in microseconds, as if the two models are communicating directly in the same memory space.

[0075] (2) Cross-node interaction (DDS-based data bus): Simultaneously, the fluid model A on this node also needs to send the calculated "thrust and torque" data to the control system model C located on the CPU node in area B in real time to simulate the controller's response. The system automatically enables the DDS-based data exchange bus. Model A, as the publisher, publishes the "thrust and torque" data to the " / propulsion / output" topic on the DDS bus. Model C on the area B node, as the subscriber to this topic, receives the data in real time through the DDS bus. The DDS QoS policy ensures reliable transmission (Reliable mode) and low latency of this critical data.

[0076] Step S4: Simulation-based analysis of streaming data processing.

[0077] During large-scale simulation execution, intermediate and final results generated by model computations at various nodes in regions A and B continuously flow to the central distributed stream processing engine, Apache Flink, via the DDS bus or local regional data aggregators. Analysis jobs are pre-deployed on this engine, for example: (1) Real-time monitoring: Real-time monitoring of "thrust" and "fuel consumption rate" under all operating conditions. Once an abnormal decrease in thrust or an abnormal increase in fuel consumption is detected under a certain operating condition, an alarm will be immediately displayed on the instrument panel.

[0078] (2) Online aggregation: For the completed working conditions, calculate statistical information such as "average thrust under different power" and "optimal fuel consumption point under different speeds" in real time, and dynamically update the charts.

[0079] Once a portion (e.g., 20) of the 36 operating conditions has been completed, engineers can already see preliminary trend analysis results on the dashboard, such as a "thrust versus power and speed curve." If data in a certain area is found to be inconsistent with expectations, they can immediately check and adjust parameters, or even stop subsequent tasks, avoiding meaningless full calculations.

[0080] Step S5: Result aggregation and persistence.

[0081] Once all 36 subtasks are completed, the regional data aggregator uploads the compressed results for its region in batches to the central layer. The central global results database interface is responsible for decompression, verification, and storing the final results in a distributed file system (such as HDFS) according to the "operating condition ID-parameter" rule. Engineers can then initiate a deep analysis task to read the entire results set for more complex post-processing, such as drawing design space maps or training surrogate models.

[0082] The above steps can be corresponded to Figure 2The task lifecycle management process occurs within the three-tier architecture. Users directly operate the central layer to define parameterized tasks, while secondary task allocation and data scheduling are performed at the regional and node layers. Data interaction is conducted through memory sharing between nodes and cross-regional DDS bus data exchange, and the data flows uniformly to the central layer stream processing engine, which in this embodiment is Apache Flink, to complete the pre-deployed analysis jobs on this engine.

[0083] Example 3 This embodiment adds a pre-calculation system to the basis of embodiment 1, as detailed below: The optimal computation path for completing a task is calculated using a pre-built heterogeneous graph neural network based on the task's computational characteristics, and is pre-calculated for use in central layer data analysis tasks. The physical layer of the heterogeneous graph neural network consists of multiple memory nodes distributed in different regional layers. Its logical layer uses these memory nodes as network nodes. By using the computational characteristics of historical tasks and the data interaction paths between nodes, it achieves node aggregation and path aggregation. Different node importances are used to obtain the optimal computational path through path aggregation. A mapping relationship is established between the optimal computational path and the corresponding task computational characteristics. When a new task is received, the computational characteristics of the new task are used to obtain the current optimal computational path. The analysis jobs on the path nodes are calculated using the pre-computation system to form pre-computation results. When the central layer performs distributed computing jobs, it first searches in the memory of the pre-computation system to see if the required pre-computation results exist. If they do, they are adopted; otherwise, the node data reception and analysis jobs are performed normally.

[0084] like Figure 4 The diagram illustrates how network node aggregation and path aggregation are achieved through the computational characteristics of historical tasks and the data interaction paths between nodes. Different node importances are assessed through path aggregation to obtain the optimal computational path. The method for establishing a mapping relationship between the optimal computational path and the corresponding task computational characteristics is as follows: The first step is to obtain the computational characteristics of all computational tasks within a single historical task. The second step is to obtain the nodes in the node layer that are finally assigned based on computational characteristics, as well as the data interaction flow between nodes, forming multiple paths. The third step involves performing node aggregation based on the acquired nodes to obtain the importance features of the nodes corresponding to the calculated features. Furthermore, based on the acquired multiple paths, path aggregation is performed to obtain the weight of each path corresponding to the calculated features. Figure 4The example given is two paths α (abcd) and β (abcef) formed by nodes af. Path α exists within region A of the region layer, while nodes abc in path β are secondary-level nodes assigned to regions A of the region layer, and nodes e and f are secondary-level nodes assigned to regions B of the region layer. These physical nodes are the network nodes in a heterogeneous graph neural network, and their physical layers are memory nodes in the node layers.

[0085] The fourth step is to select the path with the highest weight as the optimal computation path, and construct a mapping from computational features to the optimal computation path. Figure 4 The process involves calculating the weights of path α and path β, and selecting the path with the larger weight as the optimal calculation path. Thus, the mapping expression is: Calculate feature → Optimal calculation path.

[0086] The fifth step involves constructing multiple mappings from computational features to optimal computational paths for multiple historical tasks, following all the steps outlined above. This mapping set serves as the mapping relationship between the optimal computational path and the corresponding task computational features.

[0087] like Figure 5 As shown, without using the pre-computation system, a new task is input, and the system (i.e., the new system) is run independently (i.e., without using the pre-computation system) to obtain independent running results; using the pre-computation system, the actual pre-computation adoption results of the system are statistically analyzed, and the independent running results are compared with the pre-computation adoption results. If the difference between the two is within a preset threshold range, the pre-computation adoption result is used as the final calculation result; otherwise, the independent running result is adopted as the final calculation result.

[0088] The pre-calculated adoption results are used as the current proportion of the final calculation results. A current confidence level is set for each mapping in the mapping relationship, and the current confidence level is continuously and dynamically updated based on the accumulation of historical tasks.

[0089] If the confidence level corresponding to the calculated feature map does not meet the preset range, the required pre-calculated result will not be adopted in the execution of the new task, and the normal steps of receiving and analyzing node data will be performed; otherwise, the required pre-calculated result will be adopted.

[0090] The specific pre-computation system can adopt solutions in the existing technology, such as Scheme 2 and Scheme 3 in the background technology.

[0091] Accordingly, the method also includes step S6, which constructs the aforementioned pre-computation system. Based on the task's computational characteristics, the system utilizes a pre-built heterogeneous graph neural network to calculate the optimal computational path for the task, and pre-computes this path for use in the central layer's data analysis work. When the central layer performs distributed computational work, it first searches the memory of the pre-computation system to see if the required pre-computation results exist. If they do, they are adopted for the analysis work; otherwise, the node data reception and analysis work proceeds normally. The results adopted by the pre-computation system are used as the current proportion of the final computational results. A current confidence level is set for each mapping in the mapping relationship, and the current confidence level is continuously and dynamically updated based on the accumulation of historical tasks.

[0092] If the confidence level of the calculated feature-corresponding mapping does not meet the preset range, the required pre-calculated results will not be adopted in the execution of the new task, and the normal steps of receiving and analyzing node data will be performed; otherwise, the required pre-calculated results will be adopted.

[0093] Example 4 This embodiment is an alternative to the system in Embodiment 1.

[0094] 1. Implementation alternatives for the regional layer: The physical form of the regional coordination node can be diversified.

[0095] (1) Cluster configuration: A small cluster can be formed by a group of servers. The master node election and failover can be achieved through consensus algorithms (such as Raft and Paxos), providing a more available regional service.

[0096] like Figure 6 The diagram shows a large-scale simulation cluster architecture. A single-region layer enables data interaction between multiple nodes, while a central layer distributed stream processing engine processes jobs. The processing results are stored in a results database and displayed on a real-time monitoring dashboard.

[0097] (2) Virtual form: In a cloud-native environment, the region layer can be a namespace in a Kubernetes cluster. Its "region scheduler" can be an independent Deployment, and the function of the "region fault handler" can be partially replaced or enhanced by the K8s controller (such as the StatefulSet controller).

[0098] (3) Function Decentralization: For deployment scenarios with extremely small scale (such as a single region), the functions of the regional layer can be proxied by the central layer or decentralized to the "super node" with strong control capabilities, forming a flexible shrinkage architecture.

[0099] 2. Alternatives to task feature matching algorithms: (1) Rule-based matching: The current solution can be based on predefined rules (such as "solver type=CFD" -> "assign to GPU area"). This is a simple and effective implementation method.

[0100] (2) Machine learning-based prediction matching: historical task resource consumption data (CPU duration, GPU memory, peak memory) and task metadata (grid size, time step) can be collected to train a regression model to predict the resource requirements of new tasks, thereby achieving more accurate matching.

[0101] (3) Matching based on simulation performance model: For simulation tasks with clear physical meaning, a simplified proxy model can be established to estimate its running time on different hardware, which can be used as the basis for scheduling.

[0102] 3. Alternatives to data interaction methods: (1) Alternatives for intra-node communication: In addition to shared memory, the following technologies can also be used to achieve high-speed intra-node communication: a) Unix Domain Socket (UDS): An efficient way to communicate between processes on the same host with much lower latency than TCP / IP.

[0103] b) Local communication based on RDMA: If the node is equipped with hardware that supports RDMA (such as InfiniBand), it can achieve direct memory access, achieving performance similar to or even better than shared memory.

[0104] c) Direct method calls between processes: If the model is running in different threads within the same process, method calls can be made directly (thread safety must be ensured).

[0105] (2) Cross-node communication alternatives: DDS is currently the preferred choice, but other real-time communication protocols with publish-subscribe mode and QoS guarantees can also be used in the future, such as: a) ZeroMQ (ØMQ): A high-performance asynchronous messaging library that supports multiple communication modes and can build flexible publish-subscribe networks.

[0106] b) MQTT (Message Queuing Telemetry Transport): A lightweight publish-subscribe protocol, particularly suitable for the Internet of Things and bandwidth-constrained environments.

[0107] c) ROS 2's underlying communication stack: ROS 2 itself is based on DDS, but its upper-level abstraction provides a unified interface, which is also an optional implementation method.

[0108] 4. Alternatives to streaming engines: (1) Open source frameworks: Apache Flink, Apache Spark Streaming, Kafka Streams, Apache Storm, etc. can all be used as streaming processing engines.

[0109] (2) Commercial products: such as cloud-native stream processing services like Google Cloud Dataflow and AWS Kinesis Data Analytics.

[0110] (3) Self-developed lightweight engine: For specific scenarios, a targeted, lightweight streaming computing framework can be developed, focusing on specific types of real-time aggregation and analysis.

Claims

1. A hierarchical distributed task scheduling and data interaction system for large-sample simulation, characterized in that, The system comprises a three-tier architecture: a "center-region-node" structure. The center tier, deployed in the core data center, is responsible for overall data management and task decision-making, and contains the system's operating software. The region layer is used as a logical control unit divided according to physical location, network topology, or logical organization; The node layer includes a simulation model execution unit; The central layer allocates tasks in batches to the most suitable regions based on their computational characteristics. The regional layer then performs fine-grained allocation based on the real-time resource status of nodes in the node layer. Computational results generated on different nodes flow to the distributed stream processing engine in the central layer to complete the pre-deployed analysis jobs. Data interaction between different regions in the regional layer uses the DDS data exchange bus, while the flow of computational results to the distributed stream processing engine in the central layer uses either the DDS data exchange bus or a regional data aggregator. The node layer can automatically and seamlessly switch between microsecond-level "shared memory communication" and DDS-based real-time "network communication" depending on whether the models involved in the interactive data are located on the same physical node.

2. The system of claim 1, wherein, The physical locations include R&D centers in different geographical locations and different computer rooms in the same data center; the network topology includes local area networks; the logical organization includes power system subsystems and avionics system subsystems. The central layer further includes: a global task manager, a global resource and status monitor, a global time base generator, and a global result database interface. The global task manager provides a Web UI or API interface to receive large sample tasks defined by users through the "parameterized configuration" interface. The global task manager is responsible for generating a unique ID and a subtask description file containing specific parameter values ​​for each subtask and storing them in a distributed task queue. The global resource and status monitor continuously monitors the health status and real-time load information of all registered region and node layers through a heartbeat mechanism, and builds and maintains a global dynamic resource view. For simulation scenarios with strong cross-region timing consistency requirements, the global time base generator is responsible for generating a globally unique, high-precision time synchronization signal and distributing it to each region layer and node layer through NTP or its enhanced protocol to ensure the uniformity of simulation time across the entire system. The global results database interface provides a standard interface for receiving the final simulation results aggregated from various regions and persistently storing them in a distributed file system or time series database for global analysis and visualization.

3. The system of claim 1 or 2, wherein, The regional layer further includes: a regional task scheduler, which pulls a batch of subtasks allocated to the region from the global task queue, maintains a local dynamic resource view of all nodes in the region in real time, and performs secondary fine-grained scheduling; The regional fault processor is responsible for ensuring the reliability of its region. It continuously monitors the heartbeat of each node in the region. If a node fails to respond to the heartbeat three times in a row, it is determined that the node is faulty. The regional fault processor immediately executes the following local fault recovery process: the subtasks that are running and waiting to run on the node are added back to the regional task queue and scheduled to be executed on other healthy nodes in the same region. The regional data aggregator is responsible for the initial processing and aggregation of simulation data in the region. It receives real-time result data streams generated by nodes in the region, performs data format unification, outlier cleaning, and data compression. The compressed data is packaged according to time windows or data volume thresholds and then uploaded in batches to the global result database of the central layer, thereby effectively reducing the network bandwidth pressure across regions and the write pressure of the central layer. Inter-regional communication gateways are responsible for handling cross-domain data interaction requests between their own region and other regions, and ensuring the security of cross-regional control commands and data transmission through encrypted channels.

4. The system according to claim 3, characterized in that, The node layer further includes: a model access adapter constituting a pluggable component library, providing adaptation capabilities to various mainstream simulation model interface specifications, including but not limited to: a) FMI / FMU Standard: Supports importing models that conform to the functional model interface standard; b) Simulink Model: Supports encapsulating Simulink-generated code or models into executable units; c) Modelica models: Supports importing models based on the Modelica language; d) Self-developed code model: Provides APIs to support users in integrating models written in C++, Fortran, and Python after encapsulation; The local scheduler, which constitutes a lightweight scheduling component, is responsible for receiving instructions from the regional scheduler and controlling the execution of the local model according to the instructions. The data caching module is a component used to achieve high-performance communication within a node. It opens a high-speed shared buffer in the node's memory and adopts a "publish-subscribe" local memory version mode. When a model running on the same node needs to output data, it only needs to write the data to a specific area of ​​the corresponding "topic" in the shared buffer. Another model that has subscribed to the "topic" can directly read the data through memory mapping. The local resource monitoring agent is responsible for collecting real-time resource usage data for this node and periodically reporting it to the regional fault handler via heartbeat packets.

5. The system according to any one of claims 1, 2, and 4, characterized in that, The system also includes a pre-computation system, which calculates the optimal computation path for completing the task using a pre-built heterogeneous graph neural network based on the task's computational characteristics, and pre-computes the path for use in the central layer data analysis operation. The physical layer of the heterogeneous graph neural network consists of multiple memory nodes distributed in different regional layers. Its logical layer uses these memory nodes as network nodes. By using the computational characteristics of historical tasks and the data interaction paths between nodes, it achieves node aggregation and path aggregation. Different node importances are used to obtain the optimal computational path through path aggregation. A mapping relationship is established between the optimal computational path and the corresponding task computational characteristics. When a new task is received, the computational characteristics of the new task are used to obtain the current optimal computational path. The analysis jobs on the path nodes are calculated using the pre-computation system to form pre-computation results. When the central layer performs distributed computing jobs, it first searches in the memory of the pre-computation system to see if the required pre-computation results exist. If they do, they are adopted; otherwise, the node data reception and analysis jobs are performed normally.

6. The system according to claim 5, characterized in that, Network node aggregation and path aggregation are achieved by leveraging the computational characteristics of historical tasks and the data interaction paths between nodes. The optimal computational path is obtained through path aggregation based on the importance of different nodes. The method for establishing a mapping relationship between the optimal computational path and the corresponding task computational characteristics is as follows: The first step is to obtain the computational characteristics of all computational tasks within a single historical task. The second step is to obtain the nodes in the node layer that are finally assigned based on computational characteristics, as well as the data interaction flow between nodes, forming multiple paths. The third step involves performing node aggregation based on the acquired nodes to obtain the importance features of the nodes corresponding to the calculated features. Furthermore, based on the acquired multiple paths, path aggregation is performed to obtain the weight of each path corresponding to the calculated features. The fourth step is to select the path with the highest weight as the optimal computation path and construct a mapping from computational features to the optimal computation path. The fifth step involves constructing multiple mappings from computational features to optimal computational paths for multiple historical tasks, following all the steps outlined above. This mapping set serves as the mapping relationship between the optimal computational path and the corresponding task computational features.

7. The system according to claim 6, characterized in that, When the pre-computation system is not used, the system is run independently to obtain independent running results. When the pre-computation system is used, the pre-computation adoption results that are actually adopted by the system are statistically analyzed. The independent running results are compared with the pre-computation adoption results. If the difference between the two is within a preset threshold range, the pre-computation adoption results are used as the final calculation results. Otherwise, the independent running results are adopted as the final calculation results.

8. The system according to claim 7, characterized in that, The current percentage of the pre-calculated adoption results is used as the current proportion of the final calculation results. A current confidence level is set for each mapping in the mapping relationship, and the current confidence level is continuously and dynamically updated based on the accumulation of historical tasks. If the confidence level of the calculated feature-corresponding mapping does not meet the preset range, the required pre-calculated results will not be adopted in the execution of the new task, and the normal steps of receiving and analyzing node data will be performed; otherwise, the required pre-calculated results will be adopted.

9. A hierarchical distributed task scheduling and data interaction method for large-sample simulation based on any one of the systems described in claims 1-4, characterized in that, Includes the following steps: Step S1: Parameterized definition and automatic splitting of large sample tasks. The system engineer logs into the web interface of the system software, selects the "System Simulation" scenario template, and defines at least one variable to be scanned in the parameter configuration panel; Click the "Generate Task" button and set the task properties. The global task manager in the system background will automatically perform the Cartesian product operation to generate multiple independent subtasks. Each subtask description file contains a unique ID for that working condition, variable values, and a path pointing to the basic simulation model. Step S2: Secondary intelligent scheduling based on task characteristics and resource profiles, including (1) first-level scheduling from the central layer to the regional layer, and second-level scheduling from the regional layer to the node layer; (2) after the regional task scheduler receives a batch of sub-tasks, it combines the real-time resource status of each GPU node in the region and adopts a dynamic allocation strategy to accurately distribute each sub-task to the most idle GPU node. Step S3: Data interaction of hierarchical and differentiated strategies, specifically including (1) intra-node interaction and (2) cross-node interaction; Step S4: Simulation-as-analysis streaming data processing, which includes the intermediate and final results generated by the model calculation at each node during the large-scale execution of the simulation task, continuously flowing to the central distributed stream processing engine through the DDS bus or local regional data aggregator, on which analysis jobs are pre-deployed. Step S5: Result aggregation and persistence.

10. A hierarchical distributed task scheduling and data interaction method for large-sample simulation based on any one of the systems described in claims 5-8, characterized in that, Includes the following steps: Step S1: Parameterized definition and automatic splitting of large sample tasks. The system engineer logs into the web interface of the system software, selects the "System Simulation" scenario template, and defines at least one variable to be scanned in the parameter configuration panel; Click the "Generate Task" button and set the task properties. The global task manager in the system background will automatically perform the Cartesian product operation to generate multiple independent subtasks. Each subtask description file contains a unique ID for that working condition, variable values, and a path pointing to the basic simulation model. Step S2: Secondary intelligent scheduling based on task characteristics and resource profiles, including (1) first-level scheduling from the central layer to the regional layer, and second-level scheduling from the regional layer to the node layer; (2) after the regional task scheduler receives a batch of sub-tasks, it combines the real-time resource status of each GPU node in the region and adopts a dynamic allocation strategy to accurately distribute each sub-task to the most idle GPU node. Step S3: Data interaction of hierarchical and differentiated strategies, specifically including (1) intra-node interaction and (2) cross-node interaction; Step S4: Simulation-as-analysis streaming data processing, which includes the intermediate and final results generated by the model calculation at each node during the large-scale execution of the simulation task, continuously flowing to the central distributed stream processing engine through the DDS bus or local regional data aggregator, on which analysis jobs are pre-deployed. Step S5: Result aggregation and persistence; The system includes step S6, which constructs a pre-computation system as described in any one of claims 5-8. Based on the task's computational characteristics, it utilizes a pre-built heterogeneous graph neural network to calculate the optimal computational path for the task completion, and pre-calculates this path for use in the central layer's data analysis operations. When the central layer performs distributed computation, it first searches the pre-computation system's memory for the required pre-computation results. If found, these results are adopted for the analysis operation; otherwise, node data reception and analysis proceed normally. The pre-computation results are used as the current proportion of the final computational results to set a current confidence level for each mapping in the mapping relationship, and the current confidence level is dynamically updated based on the accumulation of historical tasks. If the confidence level corresponding to the calculated feature map does not meet the preset range, the required pre-calculated result will not be adopted in the execution of the new task, and the normal steps of receiving and analyzing node data will be performed; otherwise, the required pre-calculated result will be adopted.

Images