Super-computing network communication and task scheduling collaborative optimization method for large-scale electromagnetic transient simulation

Abstract

The invention discloses a super-computing network communication and task scheduling collaborative optimization method for large-scale electromagnetic transient simulation, and the method comprises the steps: constructing a cross-sub-cloud communication channel, a sub-cloud internal node communication method, and an electronic transient simulation data flow control technical scheme. Based on this, a super-computing network node communication technical scheme based on large-scale electromagnetic transient simulation is constructed; constructing an electromagnetic transient simulation single task cross-contact scheduling scheme, an electromagnetic transient simulation task and computing resource coordinated adaptive sub-network scheduling scheme and a super-computing network task priority scheduling scheme, and constructing a large-scale electromagnetic transient simulation computing task scheduling strategy scheme based on the schemes; large-scale electromagnetic transient simulation is realized through cooperative linkage of a supercomputing network node communication technical scheme and a large-scale electromagnetic transient simulation calculation task scheduling strategy scheme.

Description

Technical Field

[0001] This invention relates to the field of high-performance computing and distributed systems technology, and more specifically, to a method for collaborative optimization of communication and task scheduling in supercomputing networks for large-scale electromagnetic transient simulation. Background Technology

[0002] I. Distributed Parallel Computing Communication Technology System

[0003] In a supercomputing network environment, data exchange between nodes is the foundation of parallel computing, and its performance directly determines the scalability and efficiency of the entire system. After years of development, the industry has formed a mature and well-structured distributed parallel computing communication technology system, covering a full-stack solution from underlying hardware to upper-layer applications.

[0004] The Message Passing Interface (MPI) and its communication modes define a set of library specifications for inter-process communication. MPI not only provides rich interfaces such as point-to-point communication (e.g., MPI_Send / MPI_Recv) and collective communication (e.g., MPI_Bcast, MPI_Allreduce), but more importantly, it supports multiple communication modes to adapt to different scenarios. For example, synchronous communication requires the sender to wait for the receiver's confirmation, while asynchronous communication allows the sender to continue execution even when data transmission is incomplete. Mechanisms such as MPI_Isend / MPI_Irecv and MPI_Wait enable the overlap of computation and communication, which is key to improving parallel efficiency. Furthermore, blocking and non-blocking modes provide programmers with finer control to optimize program performance. In electromagnetic transient simulations, MPI is widely used to implement boundary data exchange and global synchronization between computing nodes after each simulation step; its efficient communication library is the core of achieving parallel efficiency.

[0005] The communication performance of supercomputing networks relies heavily on the support of underlying hardware. Currently, the mainstream high-performance network technologies include InfiniBand and high-speed Ethernet (such as 100Gbps and 200Gbps). InfiniBand is renowned for its extremely low latency (sub-microsecond level) and high bandwidth, and natively supports **Remote Direct Memory Access (RDMA)** technology. RDMA allows one node's memory to directly read and write to another node's memory without CPU intervention, significantly reducing data transmission latency and CPU overhead. High-speed Ethernet also implements RDMA capabilities through technologies such as RoCE (RDMA over Converged Ethernet) or iWARP, providing a more cost-effective solution. These hardware technologies provide a solid performance foundation for upper-layer communication software.

[0006] With the increasing prevalence of microservice architectures in HPC management systems, Remote Procedure Call (RPC) and Representational State Transfer (REST) ​​have become the mainstream choices for inter-service communication. gRPC, based on HTTP / 2, supports streaming transmission, offers excellent performance, and is suitable for low-latency scenarios such as management nodes issuing control commands to compute nodes. RESTful APIs, due to their simplicity and flexibility, are commonly used for system status queries, configuration management, and monitoring services, providing standardized interfaces for upper-layer applications. For scenarios requiring asynchronous decoupling and peak shaving, message queue (MQ) middleware (such as Kafka and RabbitMQ) is a mature solution. It can be used for compute nodes to report non-real-time progress information, logs, or final massive amounts of result data, improving the system's robustness and scalability.

[0007] While pursuing ultimate performance, reliability is equally crucial. The TCP protocol at the transport layer has a mature built-in congestion control algorithm. The application layer often uses a heartbeat mechanism to detect node liveness and implements reliable data transmission and selective retransmission mechanisms through sequence numbers and acknowledgment (ACK) mechanisms. These are all fundamental technologies that ensure the stable operation of large-scale distributed systems.

[0008] To fully utilize hardware performance, a range of communication optimization techniques are widely employed. Zero-copy technology reduces CPU and memory overhead by avoiding multiple copies of data between kernel space and user space. Kernel bypass technology allows applications to interact directly with the network card, bypassing the operating system's network protocol stack, further reducing latency. Furthermore, communication topology-aware routing algorithms can intelligently select the optimal path based on the application's actual communication pattern (such as the MPI communication graph) and the physical network topology, avoiding network congestion and ensuring the smooth flow of critical data.

[0009] II. Characteristics of Electromagnetic Transient Parallel Simulation Tasks

[0010] The all-electromagnetic transient parallel simulation task is characterized by a large number of subnets, computational intensity, large demand for CPU and storage resources, a large number of fragmented files, and long computation time. To address these characteristics, a partitioned distributed task scheduling algorithm is proposed to realize the parallel computing of massive electromagnetic transient simulation tasks.

[0011] By establishing a central control gateway scheduling server with multiple sub-servers, the upper limit of the number of IP addresses in a single network segment is broken, enabling the hierarchical reporting and coordinated management of computing and storage resources. By applying a layered distributed consensus algorithm, the supercomputing platform resources and task information are synchronized and highly available at the millisecond level, achieving unified management and scheduling of platform resources, improving resource utilization, and greatly enhancing computing efficiency.

[0012] 1. Large number of subnets

[0013] Electromagnetic transients involve a large number of buses and a wide variety of fault types. Users often spend several days calculating electromagnetic transient tasks on a single node. Therefore, it is necessary to divide the electromagnetic transient task into multiple subnets and perform the calculation in a many-core, multi-subnet mode. This way, a large electromagnetic transient task is divided into smaller subtasks based on parameters such as bus type and fault type, and distributed across multiple computers for concurrent calculation. Finally, the results are aggregated, thereby shortening the calculation time and greatly improving the calculation efficiency.

[0014] 2. Computationally intensive

[0015] Computationally intensive electromagnetic transient tasks refer to situations where users may submit a large number of electromagnetic transient simulation tasks at a specific point in time. This requires the supercomputer to have sufficient available resources to provide reliable computing services to users, such as a stable and reliable system, sufficient disk space and CPU resources, and smooth coordination among distributed clusters in each partition, so as to ensure that the user's electromagnetic transient tasks can be completed within the specified time.

[0016] 3. High demand for CPU and storage resources

[0017] Electromagnetic transient subtasks often require approximately 4 to 24 CPU cores, thus requiring the distributed scheduling platform to have sufficient CPU resources to meet user needs.

[0018] In addition, electromagnetic transient tasks require a large amount of disk space. Often, a single subtask file and directory can reach hundreds of megabytes or even gigabytes in size. To address the issue of insufficient disk space on computing nodes, the distributed cluster adopts a scheme of using disk arrays to form a shared public disk. All computing files are stored in the public disk space, and when needed, the computing nodes copy the allocated task files to their local computing locations.

[0019] 4. Numerous fragmented files

[0020] Electromagnetic transient fragment files refer to various types of files with small individual file sizes. An electromagnetic transient overall project task contains multiple subtasks, each of which contains its own subdirectories. The entire project contains a public directory, multiple subtask directories, and a task configuration file TaskList. When computing nodes compute subtasks, they copy all files from the public directory to the subtask directories for computation. Therefore, the entire project file is relatively large.

[0021] 5. Long calculation time

[0022] Due to their unique characteristics, electromagnetic transient tasks, even when broken down into multiple subnets, require a considerable amount of time to complete, typically several hours. To ensure users' tasks are reliably completed within the specified time, the distributed computing platform's shell program periodically sends progress updates to users, allowing them to monitor the real-time progress of their submitted tasks. Furthermore, given the long computation time of electromagnetic transient tasks, to ensure the supercomputing platform provides stable computing services and prevent resource shortages due to unlimited resource consumption, the computing node application sets a maximum computation time for each electromagnetic transient task. If a task fails to complete within the specified time, the platform forcibly terminates the task, releases computing resources, and informs the user of the timeout, suggesting possible reasons such as an excessively short time limit or inappropriate subnetting, prompting the user to modify and resubmit.

[0023] III. Supercomputing Platform Resource Scheduling and Task Distribution Technology

[0024] In today's era of rapid technological advancement, supercomputing platforms, as the core infrastructure for handling large-scale and complex computing tasks, play an indispensable role in numerous fields such as scientific research, engineering simulation, weather forecasting, and bioinformatics. The resource scheduling and task distribution technologies of supercomputing platforms are crucial for ensuring efficient platform operation, optimizing the allocation of computing resources, and enabling rapid task processing.

[0025] 1. The Importance and Mature Solutions of Resource Scheduling Technology

[0026] Supercomputing platforms typically integrate a large number of computing nodes, storage devices, and network resources. These resources need to be rationally allocated and scheduled to meet the computing needs of different users and tasks. The core objective of resource scheduling technology is to dynamically allocate tasks to the most suitable computing nodes for execution based on task characteristics (such as computational load, data volume, and priority) and the current state of resources (such as load and availability), thereby achieving efficient resource utilization and efficient task completion.

[0027] Currently, the industry has developed a variety of mature resource scheduling algorithms and technologies. For example, priority-based scheduling algorithms can allocate resources according to the urgency and importance of tasks, ensuring that critical tasks can be processed first; load-balancing-based scheduling algorithms can monitor the load of each computing node in real time and distribute tasks evenly to nodes with lighter loads, avoiding situations where some nodes are overloaded while others are idle; in addition, there are intelligent scheduling algorithms based on machine learning, which can automatically optimize scheduling strategies by learning from historical data and task execution, further improving resource utilization efficiency and task completion speed.

[0028] 2. Key Role and Implementation Methods of Task Distribution Technology

[0029] Task distribution technology serves as the bridge connecting users and computing resources in a supercomputing platform. It is responsible for breaking down user-submitted tasks into multiple subtasks (if necessary) and distributing these subtasks to the appropriate computing nodes for execution. The efficiency of task distribution directly impacts the overall platform's computing performance and user experience.

[0030] To achieve efficient task distribution, supercomputing platforms typically employ a variety of technologies. On one hand, by building an efficient task queue management system, tasks can be queued, prioritized, and their status tracked, ensuring that tasks are processed according to a predetermined order and priority. On the other hand, utilizing advanced communication protocols and network technologies, such as high-speed network interconnection and RDMA (Remote Direct Memory Access), can significantly reduce communication latency during task distribution, increase data transmission speed, and thus accelerate task execution.

[0031] In addition, some supercomputing platforms have introduced containerization and virtualization technologies, packaging tasks and their dependent environments into independent containers or virtual machines to achieve rapid deployment and isolated execution of tasks. This approach not only improves task portability and security but also simplifies the complexity of task distribution, enabling the platform to more flexibly respond to the computing needs of different users and different tasks.

[0032] 3. Coordinated optimization of resource scheduling and task distribution technologies

[0033] In practical applications, resource scheduling and task distribution technologies often need to work together to achieve overall optimization of the supercomputing platform. On the one hand, resource scheduling technology needs to allocate the most suitable computing nodes to tasks based on the characteristics of the tasks and the current state of the resources; on the other hand, task distribution technology needs to ensure that tasks can reach these nodes efficiently and accurately and execute smoothly.

[0034] To achieve this collaborative optimization, supercomputing platforms typically employ integrated resource management and task scheduling systems. These systems can monitor the platform's resource status and task execution in real time, automatically allocating resources and distributing tasks according to preset scheduling strategies and algorithms. They also provide rich user interfaces and management tools, enabling users and administrators to easily submit tasks, monitor task execution status, and adjust scheduling strategies.

[0035] 4. Challenges and Future Development Trends

[0036] Despite significant progress in resource scheduling and task distribution technologies for supercomputing platforms, several challenges remain. For example, as platform scale increases and task complexity grows, further improvements in resource scheduling and task distribution efficiency, reduced communication latency, and enhanced system reliability and scalability are all pressing issues that need to be addressed. Summary of the Invention

[0037] To address the shortcomings of existing technologies, this invention provides a collaborative optimization method for communication and task scheduling in supercomputing networks for large-scale electromagnetic transient simulation.

[0038] According to one aspect of the present invention, a method for collaborative optimization of communication and task scheduling in supercomputing networks for large-scale electromagnetic transient simulation is provided, comprising:

[0039] Construct a cross-sub-cloud communication channel, a sub-cloud internal node communication method, and an electronic transient simulation data flow control technology solution;

[0040] Based on cross-sub-cloud communication channels, sub-cloud internal communication methods, and electronic transient simulation data flow control technology, a supercomputing network node communication technology scheme based on large-scale electromagnetic transient simulation is constructed.

[0041] We construct a single-task cross-node scheduling scheme for electromagnetic transient simulation, an adaptive sub-network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and a task priority scheduling scheme for supercomputing networks.

[0042] Based on the single-task cross-connection scheduling scheme for electromagnetic transient simulation, the adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and the task priority scheduling scheme for supercomputing networks, a scheduling strategy scheme for large-scale electromagnetic transient simulation computing tasks is constructed.

[0043] Large-scale electromagnetic transient simulation is achieved through the coordinated operation of supercomputing network node communication technology and large-scale electromagnetic transient simulation task scheduling strategy.

[0044] Optionally, a cross-sub-cloud communication channel can be established, including:

[0045] Research and determine the server communication methods between different sub-clouds in the supercomputing network, and establish communication channels based on high-speed dedicated lines or optimized wide area network acceleration technologies;

[0046] Achieve load balancing across sub-clouds in communication channels through intelligent routing and traffic scheduling;

[0047] Based on the communication needs of different service scenarios, the optimal communication protocol in the communication channels is dynamically selected.

[0048] Optionally, the optimal communication protocol includes:

[0049] For status query and configuration management scenarios, the REST protocol is adopted to achieve lightweight, stateless communication;

[0050] For remote procedure calls and inter-service synchronization operations, an RPC framework is used to achieve efficient and low-latency calls;

[0051] For asynchronous event notification, log collection, and result reporting scenarios, MQ message queues are used to achieve decoupling, peak shaving, and reliable transmission.

[0052] Optionally, a method for communication between nodes within the sub-cloud is constructed, including:

[0053] For the different types of computing nodes, management nodes, and storage nodes contained within the sub-cloud, we analyze their communication volume, reliability, and real-time requirements, and obtain the analysis results.

[0054] Based on the analysis results, a heartbeat mechanism was designed between the computing nodes and the management nodes. Lightweight, high-frequency probe packets were used to monitor the node's liveness status in real time and quickly detect faulty nodes.

[0055] Based on the analysis results, a result information transmission mechanism between the computing nodes and the management nodes is designed, and a reliable transmission protocol is adopted to ensure that key data such as simulation results and status reports are delivered completely.

[0056] Based on the analysis results, a packet loss retransmission mechanism was designed, and acknowledgment and timeout retransmission strategies were introduced in the result transmission to ensure the reliability of data transmission.

[0057] Optionally, an electronic transient simulation data flow control technology solution is constructed, including:

[0058] Based on the different data flow characteristics generated by the electromagnetic transient simulation task at the start of the calculation, during the calculation, and after the calculation is completed, an adaptive flow control strategy is designed.

[0059] By controlling the sender's transmission rate, the data flow is smoothed, preventing sudden traffic surges from impacting the network.

[0060] Based on the network's maximum transmission unit and application characteristics, optimize the size of a single packet to balance transmission efficiency and network overhead.

[0061] Congestion avoidance and control are implemented through program design, which proactively reduces the sending rate when the network load is high to avoid network congestion.

[0062] Optionally, a single-task cross-node scheduling scheme for electromagnetic transient simulation is constructed, including:

[0063] Based on the number of subnets, the number of computing nodes required, and the CPU / memory resource configuration information required for each task in a single large-scale electromagnetic transient simulation task, and combined with the current resource availability of the supercomputing network, a refined resource allocation algorithm is designed.

[0064] A single large-scale electromagnetic transient simulation task is broken down into multiple sub-tasks and distributed to multiple computing nodes for parallel execution.

[0065] The task is organized into cross-node computation using MPI network architecture, and the efficient communication primitives of MPI are used to realize state synchronization and data exchange between computing nodes.

[0066] Optionally, an adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources is constructed, including:

[0067] Establish a dynamic feedback mechanism for task scheduling and resource status;

[0068] When the supercomputing network has sufficient available resources, the scheduling node service program automatically increases the number of subnets and the amount of resources used by a single task;

[0069] When the supercomputing network is short of available resources, the scheduling node service program automatically reduces the number of subnets and the amount of resources used by a single task.

[0070] Dynamic allocation and optimization are performed based on real-time resource load and task characteristics.

[0071] Optionally, a supercomputing network task priority scheduling scheme is constructed, including:

[0072] Priority tags are assigned to electromagnetic transient tasks at the task submission end, which are divided into general tasks and urgent tasks.

[0073] Based on priority tags, the supercomputing platform gateway scheduling service module manages the task queue according to task priority and task characteristics. Newly submitted high-priority tasks are directly inserted at the front of the queue, otherwise they are inserted at the back of the queue. When resources are scarce, the tasks in the existing queue can be dynamically reordered to advance high-priority tasks.

[0074] When available computing resources, the scheduling service module strictly selects tasks from the task queue according to priority for distribution and computation, ensuring that high-priority tasks receive priority response and processing. According to another aspect of the present invention, a supercomputing network communication and task scheduling collaborative optimization device for large-scale electromagnetic transient simulation is provided, comprising:

[0075] The first building module is used to build cross-sub-cloud communication channels, sub-cloud internal node communication methods, and electronic transient simulation data flow control technology solutions.

[0076] The second construction module is used to construct a supercomputing network node communication technology scheme based on large-scale electromagnetic transient simulation, according to cross-sub-cloud communication channels, sub-cloud internal communication methods, and electronic transient simulation data flow control technology scheme.

[0077] The third construction module is used to construct a single-task cross-connection scheduling scheme for electromagnetic transient simulation, an adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and a task priority scheduling scheme for the supercomputing network.

[0078] The fourth module is used to construct a large-scale electromagnetic transient simulation computing task scheduling strategy based on the single-task cross-connection scheduling scheme for electromagnetic transient simulation, the adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and the supercomputing network task priority scheduling scheme.

[0079] The simulation module is used to achieve large-scale electromagnetic transient simulation through the coordinated operation of supercomputing network node communication technology and large-scale electromagnetic transient simulation task scheduling strategy.

[0080] Optionally, the first building module constructs a cross-sub-cloud communication channel, including:

[0081] Research and determine the server communication methods between different sub-clouds in the supercomputing network, and establish communication channels based on high-speed dedicated lines or optimized wide area network acceleration technologies;

[0082] Achieve load balancing across sub-clouds in communication channels through intelligent routing and traffic scheduling;

[0083] Based on the communication needs of different service scenarios, the optimal communication protocol in the communication channels is dynamically selected.

[0084] Optionally, the optimal communication protocol includes:

[0085] For status query and configuration management scenarios, the REST protocol is adopted to achieve lightweight, stateless communication;

[0086] For remote procedure calls and inter-service synchronization operations, an RPC framework is used to achieve efficient and low-latency calls;

[0087] For asynchronous event notification, log collection, and result reporting scenarios, MQ message queues are used to achieve decoupling, peak shaving, and reliable transmission.

[0088] Optionally, the first building module constructs a method for communication between nodes within the sub-cloud, including:

[0089] For the different types of computing nodes, management nodes, and storage nodes contained within the sub-cloud, we analyze their communication volume, reliability, and real-time requirements, and obtain the analysis results.

[0090] Based on the analysis results, a heartbeat mechanism was designed between the computing nodes and the management nodes. Lightweight, high-frequency probe packets were used to monitor the node's liveness status in real time and quickly detect faulty nodes.

[0091] Based on the analysis results, a result information transmission mechanism between the computing nodes and the management nodes is designed, and a reliable transmission protocol is adopted to ensure that key data such as simulation results and status reports are delivered completely.

[0092] Based on the analysis results, a packet loss retransmission mechanism was designed, and acknowledgment and timeout retransmission strategies were introduced in the result transmission to ensure the reliability of data transmission.

[0093] Optionally, the first building module constructs an electronic transient simulation data flow control technology solution, including:

[0094] Based on the different data flow characteristics generated by the electromagnetic transient simulation task at the start of the calculation, during the calculation, and after the calculation is completed, an adaptive flow control strategy is designed.

[0095] By controlling the sender's transmission rate, the data flow is smoothed, preventing sudden traffic surges from impacting the network.

[0096] Based on the network's maximum transmission unit and application characteristics, optimize the size of a single packet to balance transmission efficiency and network overhead.

[0097] Congestion avoidance and control are implemented through program design, which proactively reduces the sending rate when the network load is high to avoid network congestion.

[0098] Optionally, the third building module constructs a single-task cross-node scheduling scheme for electromagnetic transient simulation, including:

[0099] Based on the number of subnets, the number of computing nodes required, and the CPU / memory resource configuration information required for each task in a single large-scale electromagnetic transient simulation task, and combined with the current resource availability of the supercomputing network, a refined resource allocation algorithm is designed.

[0100] A single large-scale electromagnetic transient simulation task is broken down into multiple sub-tasks and distributed to multiple computing nodes for parallel execution.

[0101] The task is organized into cross-node computation using MPI network architecture, and the efficient communication primitives of MPI are used to realize state synchronization and data exchange between computing nodes.

[0102] Optionally, the third building module constructs an adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, including:

[0103] Establish a dynamic feedback mechanism for task scheduling and resource status;

[0104] When the supercomputing network has sufficient available resources, the scheduling node service program automatically increases the number of subnets and the amount of resources used by a single task;

[0105] When the supercomputing network is short of available resources, the scheduling node service program automatically reduces the number of subnets and the amount of resources used by a single task.

[0106] Dynamic allocation and optimization are performed based on real-time resource load and task characteristics.

[0107] Optionally, the third building module constructs a supercomputing network task priority scheduling scheme, including:

[0108] Priority tags are assigned to electromagnetic transient tasks at the task submission end, which are divided into general tasks and urgent tasks.

[0109] Based on priority tags, the supercomputing platform gateway scheduling service module manages the task queue according to task priority and task characteristics. Newly submitted high-priority tasks are directly inserted at the front of the queue, otherwise they are inserted at the back of the queue. When resources are scarce, the tasks in the existing queue can be dynamically reordered to advance high-priority tasks.

[0110] When there are available computing resources, the scheduling service module strictly selects tasks from the task queue according to priority order for distribution and computation, ensuring that high-priority tasks receive priority response and processing.

[0111] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the methods described in any of the above aspects of the present invention.

[0112] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method described in any of the preceding aspects of the present invention.

[0113] Therefore, by establishing a dynamic feedback mechanism between task scheduling and network communication, this invention enables adaptive selection of communication methods, intelligent decision-making for task scheduling, and collaborative linkage between the two, thereby significantly improving the execution efficiency, reliability, and resource utilization of large-scale electromagnetic transient simulation on supercomputing platforms. Attached Figure Description

[0114] Exemplary embodiments of the present invention can be more fully understood by referring to the following figures:

[0115] Figure 1This is a flowchart illustrating a method for collaborative optimization of communication and task scheduling in supercomputing networks for large-scale electromagnetic transient simulation, provided by an exemplary embodiment of the present invention.

[0116] Figure 2 This is a schematic diagram of the overall architecture of a supercomputing network communication and task scheduling collaborative optimization system for large-scale electromagnetic transient simulation provided by an exemplary embodiment of the present invention;

[0117] Figure 3 This is a diagram of a cross-sub-cloud and sub-cloud intra-cloud communication technology architecture provided by an exemplary embodiment of the present invention;

[0118] Figure 4 This is a flowchart of adaptive flow control based on electromagnetic transient simulation data provided in an exemplary embodiment of the present invention;

[0119] Figure 5 This is a schematic diagram of single-task cross-node scheduling based on MPI network segmentation provided by an exemplary embodiment of the present invention;

[0120] Figure 6 This is a flowchart of the adaptive network partitioning and priority scheduling collaborative workflow provided by an exemplary embodiment of the present invention;

[0121] Figure 7 This is a schematic diagram of the structure of a supercomputing network communication and task scheduling collaborative optimization device for large-scale electromagnetic transient simulation provided by an exemplary embodiment of the present invention;

[0122] Figure 8 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0123] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0124] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0125] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of the present invention are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0126] It should also be understood that in the embodiments of the present invention, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0127] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more unless explicitly defined or given contrary instructions in the context.

[0128] Furthermore, the term "and / or" in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.

[0129] It should also be understood that the description of the various embodiments in this invention emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0130] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0131] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0132] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0133] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0134] The embodiments of this invention can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Well-known examples of terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0135] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0136] Exemplary methods

[0137] Figure 1 This is a flowchart illustrating a method for collaborative optimization of communication and task scheduling in supercomputing networks for large-scale electromagnetic transient simulation, provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as… Figure 1 As shown, the supercomputing network communication and task scheduling collaborative optimization method 100 for large-scale electromagnetic transient simulation includes the following steps:

[0138] Step 101: Construct cross-sub-cloud communication channels, sub-cloud internal node communication methods, and electronic transient simulation data flow control technology solutions;

[0139] Step 102: Based on the cross-sub-cloud communication channels, sub-cloud internal communication methods, and electronic transient simulation data flow control technology scheme, construct a supercomputing network node communication technology scheme based on large-scale electromagnetic transient simulation.

[0140] Step 103: Construct a single-task cross-node scheduling scheme for electromagnetic transient simulation, an adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and a task priority scheduling scheme for the supercomputing network.

[0141] Step 104: Based on the single-task cross-connection scheduling scheme for electromagnetic transient simulation, the adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and the task priority scheduling scheme for supercomputing networks, a scheduling strategy scheme for large-scale electromagnetic transient simulation computing tasks is constructed.

[0142] Step 105: Through the coordinated operation of the supercomputing network node communication technology scheme and the large-scale electromagnetic transient simulation computing task scheduling strategy scheme, large-scale electromagnetic transient simulation is realized.

[0143] Specifically, this invention aims to study communication technologies between and within sub-clouds of a supercomputing network. By determining suitable communication methods, a stable and efficient communication channel is established to achieve load balancing across sub-clouds, ensuring that data and tasks can be transmitted quickly and reliably between different sub-clouds and between nodes within sub-clouds. Simultaneously, by designing reasonable heartbeat mechanisms, result information transmission mechanisms, and packet loss retransmission mechanisms, the real-time performance and reliability of communication are guaranteed, improving the stability and efficiency of supercomputing network communication.

[0144] By studying the scheduling technology for massive computing tasks in supercomputing networks, we aim to achieve the rational allocation and efficient utilization of computing resources. Based on the characteristics of electromagnetic transient simulation tasks and the actual situation of computing resources, we employ single-task cross-node scheduling technology, adaptive network scheduling technology, and task priority scheduling technology to achieve rapid task splitting, distribution, and execution. This improves task execution efficiency, ensures that tasks of different priorities are processed in a timely manner, and meets the diverse task scheduling needs of large-scale electromagnetic transient simulation.

[0145] This paper presents a collaborative optimization method for supercomputing network communication and task scheduling for large-scale electromagnetic transient simulation. The method aims to significantly improve the execution efficiency, reliability, and resource utilization of large-scale electromagnetic transient simulation on supercomputing platforms by establishing a dynamic feedback mechanism between task scheduling and network communication. This mechanism enables adaptive selection of communication methods, intelligent decision-making in task scheduling, and collaborative linkage between the two.

[0146] Specifically, refer to Figure 2 As shown, the present invention provides a supercomputing network communication and task scheduling collaborative optimization method for large-scale electromagnetic transient simulation, which consists of the following two mutually supportive and collaborative technical modules.

[0147] I. Design of Supercomputing Network Node Communication Technology Scheme Based on Large-Scale Electromagnetic Transient Simulation

[0148] 1. Establishing cross-sub-cloud communication channels, such as Figure 3 As shown:

[0149] 1) Research and determine the server communication methods between different sub-clouds in the supercomputing network, and establish a stable and efficient communication channel based on high-speed dedicated lines or optimized wide area network (WAN) acceleration technology.

[0150] 2) Achieve load balancing across sub-clouds by using intelligent routing and traffic scheduling to ensure that data and tasks are not affected by network bottlenecks when being transmitted across sub-clouds.

[0151] 3) Dynamically select the optimal communication protocol based on the communication requirements of different service scenarios:

[0152] (1) For scenarios such as status query and configuration management, the REST (Representational State Transfer) protocol is adopted to achieve lightweight and stateless communication.

[0153] (2) For remote procedure calls and inter-service synchronous operations, the RPC (Remote Procedure Call) framework (such as gRPC) is used to achieve efficient and low-latency calls.

[0154] (3) For scenarios such as asynchronous event notification, log collection, and result reporting, MQ (Message Queue, such as Kafka, RabbitMQ) message queues are used to achieve decoupling, peak shaving and valley filling and reliable transmission.

[0155] Thus, communication methods are dynamically selected based on the supercomputing service scenario to achieve load balancing and efficient transmission across sub-clouds.

[0156] 2. Design of communication method for internal nodes in sub-cloud

[0157] 1) Analyze the communication volume, reliability, and real-time requirements of different types of computing nodes, management nodes, and storage nodes contained within the sub-cloud.

[0158] 2) Design a heartbeat mechanism between computing nodes and management nodes, using lightweight, high-frequency probe packets (such as those based on UDP or a dedicated heartbeat protocol) to monitor the node's liveness status in real time and quickly detect faulty nodes.

[0159] 3) Design a result information transmission mechanism between computing nodes and management nodes, and adopt a reliable transmission protocol (such as TCP or TCP-based encapsulation protocol) to ensure that key data such as simulation results and status reports are delivered completely.

[0160] 4) Design a packet loss retransmission mechanism, and introduce acknowledgment (ACK) and timeout retransmission strategies in the result transmission to ensure the reliability of data transmission.

[0161] Therefore, a high-frequency, lightweight heartbeat mechanism, a reliable transmission protocol, and a packet loss retransmission mechanism are designed for computing nodes and management nodes respectively to meet different communication needs.

[0162] 3. Electromagnetic transient simulation data flow control technology solutions, such as... Figure 4 As shown:

[0163] 1) Design an adaptive flow control strategy to address the different data flow characteristics generated during the electromagnetic transient simulation task at the start of the calculation (task submission), during the calculation (progress information), and after the calculation (result return).

[0164] 2) By controlling the sender's transmission rate, the data flow is smoothed, preventing sudden traffic surges from impacting the network.

[0165] 3) Design data packet size reasonably. Optimize the size of a single packet based on the network MTU (Maximum Transmission Unit) and application characteristics to balance transmission efficiency and network overhead.

[0166] 4) Congestion avoidance and control are achieved through program design. When the network load is high, the sending rate is actively reduced to avoid network congestion and achieve the best match between communication efficiency and stability.

[0167] Therefore, based on the traffic characteristics at different stages of the task, network congestion can be avoided by controlling the sending rate, optimizing the data packet size, and implementing congestion control.

[0168] II. Design of Scheduling Strategy for Large-Scale Electromagnetic Transient Simulation Computation Tasks

[0169] 1. Electromagnetic transient simulation single-task cross-node scheduling scheme, such as... Figure 5 As shown:

[0170] 1) For a single large-scale electromagnetic transient simulation task, based on the configuration information such as the number of subnets, the number of computing nodes required, and the CPU / memory resources required for each task, and combined with the current resource availability of the supercomputing network, a refined resource allocation algorithm is designed.

[0171] 2) Break down a single task into multiple sub-tasks and distribute them to multiple computing nodes for parallel execution.

[0172] 3) The task is organized into cross-node computation using MPI network, and the efficient communication primitives of MPI (such as MPI_Send, MPI_Recv, MPI_Allreduce) are used to realize state synchronization and data exchange between computing nodes, which significantly improves the parallel computing efficiency of a single task.

[0173] 2. Design of an adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources

[0174] 1) First, establish a dynamic feedback mechanism for task scheduling and resource status.

[0175] 2) When the supercomputing network has sufficient available resources, the scheduling node service program automatically increases the number of subnets and the number of resources occupied by a single task, thereby accelerating the completion of batch tasks through finer-grained parallelization.

[0176] 3) When the supercomputing network has limited available resources, the scheduling node service program automatically reduces the number of subnets and the amount of resources used by individual tasks to ensure that tasks can start and compute normally, and avoid tasks waiting for a long time or failing due to insufficient resources.

[0177] 4) Implement adaptive scheduling, dynamically allocate and optimize tasks based on real-time resource load and task characteristics (such as compute-intensive and communication-intensive tasks) to ensure efficient task processing and efficient resource utilization, thereby improving the flexibility of task scheduling and the maintainability of the system.

[0178] Therefore, the single-task cross-node scheduling method based on MPI network splits a single electromagnetic transient simulation task into network subtasks, allocates them according to resource requirements and current state, and uses MPI to achieve cross-node parallel computing.

[0179] 3. Design of supercomputing network task priority scheduling scheme, such as... Figure 6 As shown:

[0180] 1) First, set priority tags for electromagnetic transient tasks on the task submission end, dividing them into general tasks and urgent tasks.

[0181] 2) The supercomputing platform gateway scheduling service module manages the task queue based on task priority and characteristics. Newly submitted high-priority tasks are directly inserted at the front of the queue; otherwise, they are inserted at the back. When resources are scarce, tasks in the existing queue can be dynamically reordered, bringing high-priority tasks forward.

[0182] 3) When there are available computing resources, the scheduling service module strictly selects tasks from the task queue according to priority order for distribution and computation, ensuring that high-priority tasks receive priority response and processing.

[0183] Therefore, the priority scheduling and queue management strategy, through task priority tags and queue sorting rules, ensures that high-priority tasks are executed first when resources are idle, and supports multi-task parallel scheduling.

[0184] Therefore, by deeply integrating the two major technical modules of communication and scheduling, this invention achieves systematic collaborative optimization, resulting in the following significant beneficial effects:

[0185] 1. Communication efficiency and reliability are significantly improved.

[0186] By constructing optimized communication channels across sub-clouds, communication latency caused by geographical or administrative isolation is effectively reduced, and bandwidth utilization for cross-domain data transmission is improved. Differentiated communication protocols (REST, RPC, MQ) and customized heartbeat and retransmission mechanisms are employed within the sub-clouds to meet the specific needs of different service scenarios (real-time performance, reliability, decoupling), significantly improving the efficiency and stability of inter-node communication. Adaptive flow control technology performs fine-grained management based on the dynamic characteristics of simulation data streams, effectively smoothing data bursts, avoiding network congestion, and ensuring the smooth and reliable exchange of massive amounts of data in large-scale simulation tasks, fundamentally solving the communication bottleneck problem that restricts simulation.

[0187] 2. Task execution efficiency and resource utilization have been greatly improved.

[0188] MPI-based cross-node network scheduling technology enables the efficient parallel execution of single large-scale simulation tasks, significantly reducing the computation time of a single task. The adaptive network scheduling strategy achieves dynamic optimization of computing resources, intelligently adjusting the parallel granularity of tasks based on system load. This maximizes computational acceleration when resources are abundant while ensuring normal task operation when resources are scarce, avoiding resource waste or task delays and maximizing the utilization of supercomputing network resources. This dynamic adjustment capability allows the system to adapt to constantly changing computing demands and resource conditions.

[0189] 3. System reliability and task continuity have been fundamentally enhanced.

[0190] Traditional supercomputing platforms often fail when faced with node failures or network jitter, leading to the complete failure of simulation tasks and the loss of hours or even days of computational results. This invention, through a built-in heartbeat mechanism, selective retransmission protocol, and dynamic task migration technology, can detect anomalies within seconds and automatically migrate affected tasks to healthy nodes for continued execution. This reduces the risk of task interruption due to single points of failure by nearly an order of magnitude, ensuring the integrity and success rate of long-running, high-value simulation tasks.

[0191] 4. The system's flexibility and maintainability are significantly improved.

[0192] The modular design allows the communication and scheduling modules to be optimized and upgraded independently, while also working closely together. Adaptive scheduling mechanisms and priority queue management enable the system to flexibly handle simulation tasks of varying scales, priorities, and timeliness, achieving optimal resource allocation without manual intervention. This intelligent and automated nature significantly reduces the complexity and labor costs of system operation and maintenance, while enhancing the system's maintainability and scalability.

[0193] 5. The utilization mode of computing resources has shifted from static allocation to dynamic optimization.

[0194] The adaptive scheduling engine of this invention completely changes the rigid "first-come, first-served, resource-reserved" model of the past. It can perceive the global resource load and task characteristics in real time and dynamically adjust the task scale and resource allocation. In multi-user, high-concurrency simulation scenarios, this mechanism effectively avoids resource idleness and overload, significantly improves the overall resource utilization and task throughput of the supercomputing platform, and enables expensive supercomputing investments to generate greater benefits.

[0195] Exemplary device

[0196] Figure 7 This is a schematic diagram of the structure of a supercomputing network communication and task scheduling collaborative optimization device for large-scale electromagnetic transient simulation provided by an exemplary embodiment of the present invention. Figure 7 As shown, the device 700 includes:

[0197] The first construction module 710 is used to construct cross-sub-cloud communication channels, sub-cloud internal node communication methods, and electronic transient simulation data flow control technology solutions.

[0198] The second construction module 720 is used to construct a supercomputing network node communication technology scheme based on large-scale electromagnetic transient simulation according to the cross-sub-cloud communication channel, the sub-cloud internal communication method and the electronic transient simulation data flow control technology scheme.

[0199] The third construction module 730 is used to construct a single-task cross-connection scheduling scheme for electromagnetic transient simulation, an adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and a task priority scheduling scheme for the supercomputing network.

[0200] The fourth construction module 740 is used to construct a large-scale electromagnetic transient simulation computing task scheduling strategy based on the electromagnetic transient simulation single-task cross-connection scheduling scheme, the adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and the supercomputing network task priority scheduling scheme.

[0201] The simulation module 750 is used to achieve large-scale electromagnetic transient simulation through the coordinated operation of the supercomputing network node communication technology scheme and the large-scale electromagnetic transient simulation computing task scheduling strategy scheme.

[0202] Optionally, the first building module 710 constructs a cross-sub-cloud communication channel, including:

[0203] Research and determine the server communication methods between different sub-clouds in the supercomputing network, and establish communication channels based on high-speed dedicated lines or optimized wide area network acceleration technologies;

[0204] Achieve load balancing across sub-clouds in communication channels through intelligent routing and traffic scheduling;

[0205] Based on the communication needs of different service scenarios, the optimal communication protocol in the communication channels is dynamically selected.

[0206] Optionally, the optimal communication protocol includes:

[0207] For status query and configuration management scenarios, the REST protocol is adopted to achieve lightweight, stateless communication;

[0208] For remote procedure calls and inter-service synchronization operations, an RPC framework is used to achieve efficient and low-latency calls;

[0209] For asynchronous event notification, log collection, and result reporting scenarios, MQ message queues are used to achieve decoupling, peak shaving, and reliable transmission.

[0210] Optionally, the first building module 710 constructs a method for building communication between nodes within the sub-cloud, including:

[0211] For the different types of computing nodes, management nodes, and storage nodes contained within the sub-cloud, we analyze their communication volume, reliability, and real-time requirements, and obtain the analysis results.

[0212] Based on the analysis results, a heartbeat mechanism was designed between the computing nodes and the management nodes. Lightweight, high-frequency probe packets were used to monitor the node's liveness status in real time and quickly detect faulty nodes.

[0213] Based on the analysis results, a result information transmission mechanism between the computing nodes and the management nodes is designed, and a reliable transmission protocol is adopted to ensure that key data such as simulation results and status reports are delivered completely.

[0214] Based on the analysis results, a packet loss retransmission mechanism was designed, and acknowledgment and timeout retransmission strategies were introduced in the result transmission to ensure the reliability of data transmission.

[0215] Optionally, the first building module 710 constructs an electronic transient simulation data flow control technology solution, including:

[0216] Based on the different data flow characteristics generated by the electromagnetic transient simulation task at the start of the calculation, during the calculation, and after the calculation is completed, an adaptive flow control strategy is designed.

[0217] By controlling the sender's transmission rate, the data flow is smoothed, preventing sudden traffic surges from impacting the network.

[0218] Based on the network's maximum transmission unit and application characteristics, optimize the size of a single packet to balance transmission efficiency and network overhead.

[0219] Congestion avoidance and control are implemented through program design, which proactively reduces the sending rate when the network load is high to avoid network congestion.

[0220] Optionally, the third building module 730 constructs a single-task cross-node scheduling scheme for electromagnetic transient simulation, including:

[0221] Based on the number of subnets, the number of computing nodes required, and the CPU / memory resource configuration information required for each task in a single large-scale electromagnetic transient simulation task, and combined with the current resource availability of the supercomputing network, a refined resource allocation algorithm is designed.

[0222] A single large-scale electromagnetic transient simulation task is broken down into multiple sub-tasks and distributed to multiple computing nodes for parallel execution.

[0223] The task is organized into cross-node computation using MPI network architecture, and the efficient communication primitives of MPI are used to realize state synchronization and data exchange between computing nodes.

[0224] Optionally, the third building module 730 constructs an adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, including:

[0225] Establish a dynamic feedback mechanism for task scheduling and resource status;

[0226] When the supercomputing network has sufficient available resources, the scheduling node service program automatically increases the number of subnets and the amount of resources used by a single task;

[0227] When the supercomputing network is short of available resources, the scheduling node service program automatically reduces the number of subnets and the amount of resources used by a single task.

[0228] Dynamic allocation and optimization are performed based on real-time resource load and task characteristics.

[0229] Optionally, the third building module 730 constructs a supercomputing network task priority scheduling scheme, including:

[0230] Priority tags are assigned to electromagnetic transient tasks at the task submission end, which are divided into general tasks and urgent tasks.

[0231] Based on priority tags, the supercomputing platform gateway scheduling service module manages the task queue according to task priority and task characteristics. Newly submitted high-priority tasks are directly inserted at the front of the queue, otherwise they are inserted at the back of the queue. When resources are scarce, the tasks in the existing queue can be dynamically reordered to advance high-priority tasks.

[0232] When there are available computing resources, the scheduling service module strictly selects tasks from the task queue according to priority order for distribution and computation, ensuring that high-priority tasks receive priority response and processing.

[0233] Exemplary electronic devices

[0234] Figure 8 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. For example... Figure 8 As shown, the electronic device 80 includes one or more processors 81 and memory 82.

[0235] The processor 81 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0236] The memory 82 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 81 may execute the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above, and / or other desired functions. In one example, the electronic device may also include an input device 83 and an output device 84, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0237] In addition, the input device 83 may also include, for example, a keyboard, a mouse, etc.

[0238] The output device 84 can output various information to the outside. The output device 84 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0239] Of course, for the sake of simplicity, Figure 8 Only some of the components of this electronic device relevant to the present invention are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0240] Exemplary computer program products and computer-readable storage media

[0241] In addition to the methods and apparatus described above, embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.

[0242] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of the present invention. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0243] Furthermore, embodiments of the present invention may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.

[0244] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0245] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.

[0246] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0247] The block diagrams of devices, systems, devices, and systems involved in this invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, systems, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0248] The methods and systems of the present invention may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of the present invention are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the methods according to the present invention. Thus, the present invention also covers recording media storing programs for performing the methods according to the present invention.

[0249] It should also be noted that in the systems, apparatus, and methods of the present invention, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered equivalents of the present invention. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0250] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A collaborative optimization method for communication and task scheduling in supercomputing networks for large-scale electromagnetic transient simulation, characterized in that, include: Construct a cross-sub-cloud communication channel, a sub-cloud internal node communication method, and an electronic transient simulation data flow control technology solution; Based on the cross-sub-cloud communication channel, the sub-cloud internal communication method, and the electronic transient simulation data flow control technology scheme, a supercomputing network node communication technology scheme based on large-scale electromagnetic transient simulation is constructed. We construct a single-task cross-node scheduling scheme for electromagnetic transient simulation, an adaptive sub-network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and a task priority scheduling scheme for supercomputing networks. Based on the aforementioned single-task cross-connection scheduling scheme for electromagnetic transient simulation, the adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and the supercomputing network task priority scheduling scheme, a scheduling strategy scheme for large-scale electromagnetic transient simulation computing tasks is constructed. The supercomputing network node communication technology scheme and the large-scale electromagnetic transient simulation computing task scheduling strategy scheme work together to realize large-scale electromagnetic transient simulation.

2. The method according to claim 1, characterized in that, Building cross-sub-cloud communication channels includes: Research and determine the server communication methods between different sub-clouds in the supercomputing network, and establish communication channels based on high-speed dedicated lines or optimized wide area network acceleration technologies; Through intelligent routing and traffic scheduling, load balancing across sub-clouds is achieved in the communication channels; The optimal communication protocol among the communication channels is dynamically selected based on the communication requirements of different service scenarios.

3. The method according to claim 2, characterized in that, The optimal communication protocol includes: For status query and configuration management scenarios, the REST protocol is adopted to achieve lightweight, stateless communication; For remote procedure calls and inter-service synchronization operations, an RPC framework is used to achieve efficient and low-latency calls; For asynchronous event notification, log collection, and result reporting scenarios, MQ message queues are used to achieve decoupling, peak shaving, and reliable transmission.

4. The method according to claim 1, characterized in that, Methods for constructing internal node communication within a sub-cloud include: For the different types of computing nodes, management nodes, and storage nodes contained within the sub-cloud, we analyze their communication volume, reliability, and real-time requirements, and obtain the analysis results. Based on the analysis results, a heartbeat mechanism between computing nodes and management nodes is designed, using lightweight, high-frequency detection packets to monitor the node's liveness status in real time and quickly detect faulty nodes. Based on the analysis results, a result information transmission mechanism between the computing node and the management node is designed, and a reliable transmission protocol is adopted to ensure that key data such as simulation results and status reports are delivered completely. Based on the analysis results, a packet loss retransmission mechanism is designed, and acknowledgment and timeout retransmission strategies are introduced in the result transmission to ensure the reliability of data transmission.

5. The method according to claim 1, characterized in that, A technical solution for electronic transient simulation data flow control is constructed, including: Based on the different data flow characteristics generated by the electromagnetic transient simulation task at the start of the calculation, during the calculation, and after the calculation is completed, an adaptive flow control strategy is designed. By controlling the sender's transmission rate, the data flow is smoothed, preventing sudden traffic surges from impacting the network. Based on the network's maximum transmission unit and application characteristics, optimize the size of a single packet to balance transmission efficiency and network overhead. Congestion avoidance and control are implemented through program design, which proactively reduces the sending rate when the network load is high to avoid network congestion.

6. The method according to claim 1, characterized in that, Construct a single-task cross-node scheduling scheme for electromagnetic transient simulation, including: Based on the number of subnets, the number of computing nodes required, and the CPU / memory resource configuration information required for each task in a single large-scale electromagnetic transient simulation task, and combined with the current resource availability of the supercomputing network, a refined resource allocation algorithm is designed. A single large-scale electromagnetic transient simulation task is broken down into multiple sub-tasks and distributed to multiple computing nodes for parallel execution. The task is organized into cross-node computation using MPI network architecture, and the efficient communication primitives of MPI are used to realize state synchronization and data exchange between computing nodes.

7. The method according to claim 1, characterized in that, An adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources is constructed, including: Establish a dynamic feedback mechanism for task scheduling and resource status; When the supercomputing network has sufficient available resources, the scheduling node service program automatically increases the number of subnets and the amount of resources used by a single task; When the supercomputing network is short of available resources, the scheduling node service program automatically reduces the number of subnets and the amount of resources used by a single task. Dynamic allocation and optimization are performed based on real-time resource load and task characteristics.

8. The method according to claim 1, characterized in that, Construct a task priority scheduling scheme for the supercomputing network, including: Priority tags are assigned to electromagnetic transient tasks at the task submission end, which are divided into general tasks and urgent tasks. Based on priority tags, the supercomputing platform gateway scheduling service module manages the task queue according to task priority and task characteristics. Newly submitted high-priority tasks are directly inserted at the front of the queue, otherwise they are inserted at the back of the queue. When resources are scarce, the tasks in the existing queue can be dynamically reordered to advance high-priority tasks. When there are available computing resources, the scheduling service module strictly selects tasks from the task queue according to priority order for distribution and computation, ensuring that high-priority tasks receive priority response and processing.

9. A supercomputing network communication and task scheduling collaborative optimization device for large-scale electromagnetic transient simulation, characterized in that, include: The first building module is used to build cross-sub-cloud communication channels, sub-cloud internal node communication methods, and electronic transient simulation data flow control technology solutions. The second construction module is used to construct a supercomputing network node communication technology scheme based on large-scale electromagnetic transient simulation, according to the cross-sub-cloud communication channel, the sub-cloud internal communication method, and the electronic transient simulation data flow control technology scheme. The third construction module is used to construct a single-task cross-connection scheduling scheme for electromagnetic transient simulation, an adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and a task priority scheduling scheme for the supercomputing network. The fourth construction module is used to construct a large-scale electromagnetic transient simulation computing task scheduling strategy based on the electromagnetic transient simulation single-task cross-connection scheduling scheme, the adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, and the supercomputing network task priority scheduling scheme. The simulation module is used to achieve large-scale electromagnetic transient simulation through the coordinated operation of the supercomputing network node communication technology scheme and the large-scale electromagnetic transient simulation computing task scheduling strategy scheme.

10. The apparatus according to claim 9, characterized in that, The first building module constructs cross-sub-cloud communication channels, including: Research and determine the server communication methods between different sub-clouds in the supercomputing network, and establish communication channels based on high-speed dedicated lines or optimized wide area network acceleration technologies; Through intelligent routing and traffic scheduling, load balancing across sub-clouds is achieved in the communication channels; The optimal communication protocol among the communication channels is dynamically selected based on the communication requirements of different service scenarios.

11. The apparatus according to claim 10, characterized in that, The optimal communication protocol includes: For status query and configuration management scenarios, the REST protocol is adopted to achieve lightweight, stateless communication; For remote procedure calls and inter-service synchronization operations, an RPC framework is used to achieve efficient and low-latency calls; For asynchronous event notification, log collection, and result reporting scenarios, MQ message queues are used to achieve decoupling, peak shaving, and reliable transmission.

12. The apparatus according to claim 9, characterized in that, The first construction module constructs methods for communication between nodes within the sub-cloud, including: For the different types of computing nodes, management nodes, and storage nodes contained within the sub-cloud, we analyze their communication volume, reliability, and real-time requirements, and obtain the analysis results. Based on the analysis results, a heartbeat mechanism between computing nodes and management nodes is designed, using lightweight, high-frequency detection packets to monitor the node's liveness status in real time and quickly detect faulty nodes. Based on the analysis results, a result information transmission mechanism between the computing node and the management node is designed, and a reliable transmission protocol is adopted to ensure that key data such as simulation results and status reports are delivered completely. Based on the analysis results, a packet loss retransmission mechanism is designed, and acknowledgment and timeout retransmission strategies are introduced in the result transmission to ensure the reliability of data transmission.

13. The apparatus according to claim 9, characterized in that, The first construction module constructs an electronic transient simulation data flow control technology solution, including: Based on the different data flow characteristics generated by the electromagnetic transient simulation task at the start of the calculation, during the calculation, and after the calculation is completed, an adaptive flow control strategy is designed. By controlling the sender's transmission rate, the data flow is smoothed, preventing sudden traffic surges from impacting the network. Based on the network's maximum transmission unit and application characteristics, optimize the size of a single packet to balance transmission efficiency and network overhead. Congestion avoidance and control are implemented through program design, which proactively reduces the sending rate when the network load is high to avoid network congestion.

14. The apparatus according to claim 9, characterized in that, The third construction module constructs a single-task cross-node scheduling scheme for electromagnetic transient simulation, including: Based on the number of subnets, the number of computing nodes required, and the CPU / memory resource configuration information required for each task in a single large-scale electromagnetic transient simulation task, and combined with the current resource availability of the supercomputing network, a refined resource allocation algorithm is designed. A single large-scale electromagnetic transient simulation task is broken down into multiple sub-tasks and distributed to multiple computing nodes for parallel execution. The task is organized into cross-node computation using MPI network architecture, and the efficient communication primitives of MPI are used to realize state synchronization and data exchange between computing nodes.

15. The apparatus according to claim 9, characterized in that, The third building module constructs an adaptive network scheduling scheme for coordinating electromagnetic transient simulation tasks and computing resources, including: Establish a dynamic feedback mechanism for task scheduling and resource status; When the supercomputing network has sufficient available resources, the scheduling node service program automatically increases the number of subnets and the amount of resources used by a single task; When the supercomputing network is short of available resources, the scheduling node service program automatically reduces the number of subnets and the amount of resources used by a single task. Dynamic allocation and optimization are performed based on real-time resource load and task characteristics.

16. The apparatus according to claim 9, characterized in that, The third building module constructs a supercomputing network task priority scheduling scheme, including: Priority tags are assigned to electromagnetic transient tasks at the task submission end, which are divided into general tasks and urgent tasks. Based on priority tags, the supercomputing platform gateway scheduling service module manages the task queue according to task priority and task characteristics. Newly submitted high-priority tasks are directly inserted at the front of the queue, otherwise they are inserted at the back of the queue. When resources are scarce, the tasks in the existing queue can be dynamically reordered to advance high-priority tasks. When there are available computing resources, the scheduling service module strictly selects tasks from the task queue according to priority order for distribution and computation, ensuring that high-priority tasks receive priority response and processing.

17. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1-8.

18. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-8.

Images