Supercomputing cluster implementation method and device, electronic equipment and storage medium
By partitioning and unifying the logical scheduling of supercomputing software within a supercomputing cluster, the use of multiple operating system software within a single supercomputing cluster is realized, solving the problem that traditional supercomputing clusters cannot support different operating systems and reducing user costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2025-11-27
- Publication Date
- 2026-04-21
AI Technical Summary
Traditional supercomputing clusters only support a single operating system, which cannot meet the needs of using software under different operating systems within a single supercomputing cluster.
Within the target supercomputing cluster, the supercomputing software is partitioned to obtain a unified processing logic across operating systems. Upon receiving a job script, the target computing nodes in the target queue are scheduled to load the target supercomputing software and execute the job according to the unified processing logic.
This enables the simultaneous deployment of computing nodes with different operating systems within a single supercomputing cluster, solving the software usage problem under different operating systems and saving user costs.
Smart Images
Figure CN121900831A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of computer technology, and specifically relates to a method, apparatus, electronic device and storage medium for implementing a supercomputing cluster. Background Technology
[0002] Traditional supercomputing clusters involve cross-platform scenarios such as multiple operating systems. They are often designed for unified management of multiple clusters and emphasize cross-platform compatibility between clusters, including clusters with different resource configurations, clusters with different schedulers (such as Slurm clusters and PBS clusters), and clusters with different operating systems (such as CentOS clusters and Windows clusters). The target cluster is selected based on the amount of resources requested by the user, the type of scheduler requested, and the type of operating system requested, and then the job is executed on the target cluster (the operating system and other components within the target cluster are usually consistent).
[0003] In existing technologies, a single cluster typically supports only a single operating system, and cross-operating system support is usually limited to between clusters, which cannot meet the user's needs for using business software under different operating systems within a single supercomputing cluster. Summary of the Invention
[0004] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for implementing a supercomputing cluster, which can solve the problem that software under different operating systems cannot be used within a single supercomputing cluster.
[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a method for implementing a supercomputing cluster, the method comprising: Within the target supercomputing cluster, the supercomputing software is partitioned, and a unified processing logic for loading software across operating systems is obtained. The target supercomputing cluster includes multiple computing nodes with different operating systems. Upon receiving the job script, the target computing node in the scheduling target queue loads the target supercomputing software and executes the job according to the unified processing logic. The target supercomputing software is the supercomputing software corresponding to the target operating system in the target computing node.
[0006] Secondly, embodiments of this application provide an apparatus for implementing a supercomputing cluster, comprising: The software processing module is used to partition the supercomputing software within the target supercomputing cluster and obtain unified processing logic for loading software across operating systems. The target supercomputing cluster includes multiple computing nodes with different operating systems. The scheduling module is used to schedule the target computing nodes under the target queue to load the target supercomputing software and execute the job according to the unified processing logic when a job script is received. The target supercomputing software is the supercomputing software corresponding to the target operating system in the target computing node.
[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.
[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.
[0009] Fifthly, embodiments of this application provide a computer program product, including a program or instructions, which, when executed by a processor, implement the steps of the supercomputing cluster implementation method as described in the first aspect.
[0010] In a sixth aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.
[0011] In this embodiment, by partitioning the supercomputing software within the target supercomputing cluster, a unified processing logic for loading software across operating systems is obtained. The target supercomputing cluster includes multiple computing nodes with different operating systems. Upon receiving a job script, the target computing nodes in the target queue are scheduled to load the target supercomputing software and execute the job according to the unified processing logic. This enables the simultaneous deployment of computing nodes with different operating systems within a single supercomputing cluster. Furthermore, based on software partitioning and the unified processing logic, the required software can be loaded and jobs executed across operating systems, achieving the goal of using software under different operating systems within a single supercomputing cluster. This solves the problem of not being able to use software under different operating systems within a single supercomputing cluster. Users do not need to deploy multiple supercomputing clusters with different operating systems; they can use a single cluster to use software under different operating systems, saving user costs. Attached Figure Description
[0012] Figure 1 This is a flowchart illustrating an implementation method for a supercomputing cluster provided in an embodiment of this application; Figure 2 This is a summary flowchart of an implementation method for a supercomputing cluster provided in an embodiment of this application; Figure 3This is a flowchart illustrating the implementation method of a supercomputing cluster provided in an embodiment of this application. Figure 4 This is an example diagram of the software architecture in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of a supercomputing cluster implementation device provided in an embodiment of this application; Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. Detailed Implementation
[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0015] Before detailing the technical solutions of the embodiments of this application, the terms mentioned in the embodiments of this application are explained as follows: Supercomputing clusters, also known as supercomputing clusters, are distributed computing systems composed of a large number of high-performance computing nodes interconnected by a high-speed network. They are designed to solve complex scientific, engineering, or commercial problems that traditional computers cannot handle. Through parallel computing technology, tasks are broken down and executed simultaneously across multiple nodes, significantly improving computational efficiency. They are suitable for scenarios requiring massive computing resources, ultra-high precision, or large-scale data processing. Typically, they are used for computer modeling, simulation, and analysis to solve advanced problems and conduct research activities, leveraging aggregated computing power to handle computational tasks that standard workstations cannot complete.
[0016] Job scheduling system: A manager used for resource allocation, management, and job submission for supercomputing jobs, such as Slurm and PBS.
[0017] Slurm (Simple Linux Utility for Resource Management) is a mainstream open-source resource allocation and job scheduling system for resource management. Slurm is an open-source, fault-tolerant, and highly scalable cluster management and job scheduling system suitable for both large and small Linux clusters. Slurm operates without kernel modifications and is relatively independent. As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and / or non-exclusive access to resources (compute nodes) to users over a period of time so they can execute jobs. Second, it provides a framework for starting, executing, and monitoring jobs (typically parallel jobs) on the allocated set of nodes. Finally, it arbitrates resource contention by managing queues of pending jobs.
[0018] PBS (Portable Batch System) is a distributed job management system widely used in high-performance computing environments to manage and monitor computing workloads. PBS consists of daemons and commands used to manage jobs on one or more computers. PBS can be used to perform tasks such as submitting, querying, modifying, monitoring, moving, and deleting jobs.
[0019] OpenMPI (Open Source Message Passing Interface implementation): The Open Message Passing Interface Project is an open-source message passing interface implementation developed and maintained by a consortium of academic, research, and industry partners.
[0020] HPFS (CT-HPFS, High Performance File Storage): is a high-performance parallel file storage system characterized by high performance, high reliability, and high scalability.
[0021] OS (Operation System): Operating system, such as CentOS, Kylin, Ubuntu, etc.
[0022] The implementation method of the supercomputing cluster provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.
[0023] Figure 1 This is a flowchart illustrating an implementation method for a supercomputing cluster provided in an embodiment of this application. This implementation method can be executed by the target supercomputing cluster, such as... Figure 1 As shown, the implementation method of this supercomputing cluster includes steps 110 to 120.
[0024] Step 110: Within the target supercomputing cluster, the supercomputing software is partitioned, and a unified processing logic for loading software across operating systems is obtained. The target supercomputing cluster includes multiple computing nodes with different operating systems.
[0025] The supercomputing software may include at least one of the following: Slurm scheduler, OpenMPI, uCX communication library, and supercomputing application software.
[0026] Within the target supercomputing cluster, the supercomputing software required by the target supercomputing cluster is obtained, and the supercomputing software is partitioned in the shared storage of the target supercomputing cluster according to the hierarchical architecture of each supercomputing software.
[0027] The unified processing logic is operating system independent. Within the same supercomputing cluster, scheduler configuration and software loading / unloading can be handled uniformly across the cluster, regardless of the operating system. This unified processing logic abstracts the scheduler configuration and software loading / unloading logic into a unified processing method, enabling globally consistent processing within the cluster, independent of the operating system.
[0028] The target supercomputing cluster can deploy multiple operating systems, such as two or more, with each operating system corresponding to at least one computing node. For example, based on the existing deployment of at least one operating system within a single supercomputing cluster, it is possible to expand to a hybrid deployment and unified scheduling of more operating systems to further meet future needs for scheduling and adaptation of industrial software across multiple operating systems within a single supercomputing cluster. For instance, based on the dual-operating system hybrid deployment already implemented in a certain supercomputing platform project, it can be horizontally scaled to a hybrid deployment and unified scheduling of three or even more operating systems.
[0029] In one exemplary embodiment, after the supercomputing software is built within the target supercomputing cluster, it can be partitioned during storage, and at the same time, a unified processing logic for loading software across operating systems can be obtained and stored in the shared storage of the target supercomputing cluster.
[0030] In another exemplary embodiment, after obtaining the supercomputing software from the supercomputing software distribution platform, the supercomputing software can be partitioned when stored in the target supercomputing cluster. At the same time, a unified processing logic for software loading across operating systems can be obtained from the supercomputing software distribution platform and stored in the shared storage of the target supercomputing cluster.
[0031] Step 120: Upon receiving the job script, the target computing node under the scheduling target queue loads the target supercomputing software and executes the job according to the unified processing logic. The target supercomputing software is the supercomputing software corresponding to the target operating system in the target computing node.
[0032] The job script includes information about the target queue (such as queue identifier), job information, and the required number of computing nodes. The target queue is the queue to which the computing nodes of the job to be executed belong.
[0033] Users can submit jobs to the target supercomputing cluster using job scripts. One implementation allows users to directly write the job scripts. Another implementation provides an interface to obtain key job information; the user inputs this information into the interface, and the system generates a job script that includes this key information.
[0034] When the target supercomputing cluster receives a job script, it can obtain the target queue information and the required number of computing nodes from the script. Based on the required number of computing nodes, it determines the corresponding number of target computing nodes from the target queue to execute the job, and schedules the job information to these nodes. The target computing nodes then load the target supercomputing software and execute the job according to a unified processing logic. This scheduling process can be implemented using a scheduler. Within the target supercomputing cluster, a single scheduler and multiple operating systems enable unified job submission and scheduling without the operating systems being aware of each other, and allow jobs to run independently in their respective computing queues on different operating systems.
[0035] The supercomputing cluster implementation method provided in this application partitions the supercomputing software within the target supercomputing cluster to obtain a unified processing logic for loading software across operating systems. The target supercomputing cluster includes multiple computing nodes with different operating systems. When a job script is received, the target computing nodes in the target queue are scheduled to load the target supercomputing software and execute the job according to the unified processing logic. This enables the simultaneous deployment of computing nodes with different operating systems within a single supercomputing cluster. Furthermore, based on software partitioning and the unified processing logic, the required software can be loaded and jobs executed across operating systems, achieving the goal of using software under different operating systems within a single supercomputing cluster. This solves the problem of not being able to use software under different operating systems within a single supercomputing cluster. Users do not need to deploy multiple supercomputing clusters with different operating systems; they can use a single cluster to use software under different operating systems, saving user costs.
[0036] Based on the above technical solution, the partitioning of the supercomputing software includes: storing each of the supercomputing software programs in the shared storage of the target supercomputing cluster according to the software architecture of the supercomputing software.
[0037] The relevant software in a supercomputing cluster can include schedulers (such as the Slurm scheduler), OpenMPI, uCx communication libraries, and supercomputing application software. By implementing a reasonable software architecture and management for the supercomputing software, platform-independent configuration and software loading capabilities can be provided for subsequent cross-operating system job submission and scheduling within a single supercomputing cluster.
[0038] Software architecture is the hierarchical structure of software, which is the storage directory information of software. It can include instruction set (arch) information, vendor information, operating system (OS) information, and software information. In other words, the software organizational structure can be classified according to arch / vendor / os / software / to achieve differentiated deployment on different OSes such as CentOS 7.9 and Kylin 10. For example, a software architecture can be amd64 / intel / centos7.9 / scheduler / slurm.
[0039] Once the supercomputing software is built, the required operating environment—the software architecture—is determined. Based on this architecture, the supercomputing software can be stored in the shared storage of the target supercomputing cluster. Shared storage is a shared storage space within the target supercomputing cluster, accessible to all nodes within the cluster. Shared storage can be, for example, HPFS, or other storage systems.
[0040] By storing each supercomputing software in the shared storage of the target supercomputing cluster according to the software architecture, it is convenient for computing nodes within the target supercomputing cluster to load the corresponding software from the shared storage based on the software architecture, thus providing a foundation for cross-operating system software scheduling.
[0041] Based on the above technical solution, the step of scheduling the target computing nodes under the target queue to load the target supercomputing software and execute the job according to the unified processing logic when the job script is received includes: when the job script is received, scheduling the target computing nodes under the target queue to load the target supercomputing software and execute the job according to the unified processing logic through the target scheduler.
[0042] The target scheduler can be any scheduler capable of performing intra-cluster scheduling; for example, the target scheduler may include the Slurm scheduler or the PBS scheduler.
[0043] You can choose one of the hybrid operating systems deployed in the target supercomputing cluster as the operating system for the management node. For example, if the hybrid operating systems deployed in the target supercomputing cluster are CentOS 7.9 and Kylin 10, then either CentOS 7.9 or Kylin 10 can be selected as the management node.
[0044] Within the target supercomputing cluster, users can submit jobs on any operating system, and jobs on different operating systems can run simultaneously. For example, if the target supercomputing cluster has both operating system A and operating system B deployed, users can submit jobs on either operating system A or operating system B, and jobs on each operating system can run concurrently.
[0045] The control node can obtain the target scheduler's globally unified supercomputing configuration file, such as slurm.conf, slurmdbd.conf, and cgroup.conf, from the target supercomputing cluster's shared storage (such as HPFS), and load the configuration file.
[0046] The `slurm.conf` file is the core configuration file for the Slurm scheduler. It describes general Slurm configuration information, the nodes to be managed, how these nodes are grouped into partitions, and various scheduling parameters associated with these partitions. This configuration file is kept consistent across all nodes in the cluster. The file location can be modified at runtime using the environment variable `SLURM_CONF`.
[0047] `slurmdbd.conf` is an ASCII file that describes the configuration information for the Slurm Database Daemon (SlurmDBD), including parameters related to database interaction, such as database type, connection information, and authentication method. This configuration file is always located in the same directory as `slurm.conf`. The file content is case-insensitive, except for node names and nodes.
[0048] cgroup.conf is the configuration file for Slurm's cgroup mechanism. It limits a job's resource usage by setting cgroup-related parameters. Proper cgroup configuration can effectively prevent jobs from excessively consuming resources, ensuring fair allocation of cluster resources and stable operation.
[0049] When the control node receives the job script, the target scheduler can obtain the target queue information and the required number of computing nodes from the job script. Based on the required number of computing nodes, it determines the target computing nodes that need to execute the job from the target queue and schedules the job information to the target computing nodes. The target computing nodes then load the target supercomputing software and execute the job according to the unified processing logic.
[0050] By scheduling target computing nodes under the target queue to load target supercomputing software and execute jobs through the target scheduler, the nodes within the target supercomputing cluster can be effectively scheduled. Moreover, different schedulers are supported, and the appropriate scheduler can be selected based on the requirements to meet more diverse needs.
[0051] Based on the above technical solution, before performing partitioning processing on the supercomputing software within the target supercomputing cluster, the method further includes: obtaining the supercomputing software from the distribution platform through a target distribution method within the target supercomputing cluster.
[0052] The target distribution method may include object storage, static image provisioning, or a software repository, and other distribution methods may also be used. Object storage is a storage architecture that processes, stores, and retrieves data on an object-by-object basis. Unlike traditional file storage and block storage, object storage manages data as objects, each containing the data itself, metadata, and a globally unique identifier. This unique storage method gives object storage a significant advantage when handling large amounts of unstructured data. Object storage reads and writes data through API (Application Programming Interface) calls, typically based on HTTP or HTTPS protocols. Object storage systems distribute data across multiple hardware devices and can automatically handle data redundancy backups and scalability issues, making them excellent for large-scale data storage. Static image provisioning refers to pre-configuring static IP addresses, usernames, passwords, etc., when creating a cloud server or virtual machine, avoiding manual configuration later. A software repository is a centralized storage location for collecting, managing, and distributing software packages, acting as a database for software retrieval and installation on various systems and devices. Software repositories play a vital role in the software development and distribution process, providing a centralized platform for developers and publishers to upload software packages, enabling users to access these packages, and then download and install them onto their devices.
[0053] A distribution platform can be a distribution platform for supercomputing software. After the software stack corresponding to each operating system is built, it is distributed. Software distribution can be carried out through various channels such as object storage, static image provisioning, and software repositories. Target supercomputing clusters can obtain supercomputing software from the distribution platform through targeted distribution methods.
[0054] A distribution platform can be a platform provided by a supercomputing platform for distributing supercomputing software. By obtaining the supercomputing software from the distribution platform within the target supercomputing cluster through the target distribution method, the required supercomputing software can be easily obtained, providing a foundation for subsequent job execution.
[0055] Based on the above technical solution, before obtaining the supercomputing software from the distribution platform through the target distribution method within the target supercomputing cluster, the method further includes: constructing supercomputing software corresponding to each operating system through the target platform.
[0056] The target platform includes at least one of Docker, physical machines, and virtual machines. Docker is a platform-as-a-service product that packages software and its dependencies into containers based on operating system-level virtualization technology. The operating systems can be various operating systems that can be used by the supercomputing cluster, such as CentOS 7.9 and Kylin 10, or other versions of CentOS or Kylin, or other operating systems.
[0057] Before distributing supercomputing software, it needs to be built. During the building process, corresponding software stacks can be built for each operating system. Various platforms such as Docker, physical machines, and virtual machines can be selected to build corresponding supercomputing software for different operating systems. The software stack is a set of independent, cooperating components that collectively support application execution. These components are stacked hierarchically to form a stack-like structure. The software stack consists of: operating system, architecture layer, protocols, runtime environment, database, and function calls. The operating system, as the foundation of the software stack, manages the computer's hardware and software resources. The architecture layer provides support for the software architecture, ensuring that components can interact in a predetermined manner. Protocols define the rules and standards for communication between components. The runtime environment provides the support required for application execution, such as the Java Virtual Machine and Python interpreter. The database stores and manages the data required by the application. Function calls implement the various functions required by the application.
[0058] By building supercomputing software for each operating system on the target platform, the supercomputing software can be adapted to different operating systems, making it easier for users to select the supercomputing software for the appropriate operating system.
[0059] Based on the above technical solution, the target supercomputing cluster also includes a management and control node; Upon receiving the job script, the target computing nodes in the scheduling target queue load the target supercomputing software and execute the job according to the unified processing logic, including: Upon receiving a job script, at least one target computing node is determined through the control node, and the job script is sent to the first computing node among the at least one target computing node to schedule the job to a computing node under the target operating system. The target supercomputing software is loaded by the first computing node according to the unified processing logic, the target supercomputing software is synchronized to other target computing nodes outside the first computing node, and the job is executed by each of the target computing nodes.
[0060] The first computing node can be any one of the at least one target computing nodes.
[0061] Upon receiving a user-submitted job script, the control node determines the corresponding number of target computing nodes based on the number of computing nodes in the job script. From these at least one target computing node, a primary computing node is selected, and the job script and the identifiers of the other target computing nodes are sent to the primary computing node. The primary computing node executes the job script, retrieves the target supercomputing software and its corresponding runtime environment information from the shared storage of the target supercomputing cluster, based on its own target operating system (instruction set information, vendor information, operating system information), and loads the target supercomputing software and environment information. Simultaneously, the primary computing node sends the job information, the storage location information of the target supercomputing software (target operating system information), and the environment information to the other target computing nodes. The other target computing nodes load the target supercomputing software and environment information. Once each target computing node has completed loading, the corresponding target supercomputing software is activated, and each can execute the job.
[0062] Upon receiving the job script, the control node identifies at least one target computing node and sends the job script to the primary computing node. The primary computing node loads the target supercomputing software and synchronizes it to other target computing nodes. This eliminates the need for the control node to send the job script to each target computing node, thus saving the control node's resources.
[0063] In some embodiments of this application, loading the target supercomputing software through the first computing node according to the unified processing logic includes: obtaining the target operating system in the first computing node through the first computing node according to the unified processing logic, and loading the target supercomputing software according to the target operating system.
[0064] The first computing node probes its own operating system information, obtaining its own instruction set information, vendor information, and operating system information. This information represents the target operating system information. Based on this information and the software information in the job script, it retrieves the target supercomputing software and corresponding operating environment information under its own target operating system (instruction set information, vendor information, and operating system information) from the shared storage of the target supercomputing cluster, and loads the target supercomputing software and environment information.
[0065] By obtaining the target operating system from the first compute node and loading the target supercomputing software under the target operating system, it is not necessary to specify the operating system in the job script. This achieves seamless unified job submission and scheduling, making it convenient for users.
[0066] In some embodiments of this application, loading the target supercomputing software according to the target operating system includes: querying the target supercomputing software corresponding to the target operating system in shared storage, and loading the target supercomputing software.
[0067] Based on the target operating system, including instruction set information, vendor information, and operating system information, the target operating system is queried in the shared storage of the target supercomputing cluster, and the target supercomputing software is queried among all the software under the target operating system, and then the target supercomputing software is loaded.
[0068] The embodiments of this application aim to address the limitations of traditional solutions by organizing scheduling software across operating systems within a supercomputing cluster and by establishing a unified submission and scheduling mechanism for supercomputing jobs across multiple operating systems. These limitations include the fact that a single cluster typically only supports a single operating system, and cross-operating system functionality is usually limited to between clusters, lacking a standardized and effective cross-operating system design within a single cluster.
[0069] This application proposes a hybrid deployment design scheme for multiple operating systems within a single supercomputing cluster. Through a reasonable architecture design under multiple operating systems within a single supercomputing cluster, this scheme enables unified job submission, scheduling, and independent operation, effectively meeting users' needs for simultaneous simulation and adaptation of industrial software under multiple operating systems within a single supercomputing cluster. It can effectively reduce user costs and has high practicality and usability.
[0070] Figure 2 This is a summary flowchart of an implementation method for a supercomputing cluster provided in an embodiment of this application, as shown below. Figure 2 As shown, the implementation method of this supercomputing cluster may include: Step 210, Supercomputing software construction. This can be done using platforms such as Docker, physical machines, and virtual machines.
[0071] Step 220, Supercomputing Software Distribution. Software distribution can be carried out through object storage, image provisioning, software repositories, etc.
[0072] Step 230: Organize related software across operating systems within the supercomputing cluster to achieve unified partitioning of the software stack and supercomputing configuration and software loading.
[0073] The specific process of this step can be referred to step 110 in the above embodiment, and will not be repeated here.
[0074] Step 240: Submission and scheduling of cross-operating system jobs within a single supercomputing cluster. This involves the unified submission and scheduling of supercomputing application jobs across multiple operating systems within a single supercomputing cluster, as well as the independent execution of each operating system.
[0075] The specific process of this step can be referred to step 120 in the above embodiment, and will not be repeated here.
[0076] Step 250: Simulation task completed. Steps 240-250 can be repeated as needed.
[0077] The embodiments of this application have achieved good technical results, realizing the hybrid deployment of a single scheduler and multiple operating systems within a single supercomputing cluster. Jobs can be submitted and scheduled uniformly, and can run independently under the computing queues and stacks of their respective target systems. This effectively meets the needs of cost-sensitive industrial simulation users, enabling them to achieve synchronous scheduling, simulation, and adaptation of industrial software under multiple operating systems using only one supercomputing cluster.
[0078] Figure 3 This is a flowchart illustrating the implementation method of a supercomputing cluster provided in an embodiment of this application. Figure 3 This example uses the Slurm scheduler and a hybrid deployment of operating systems A and B within a supercomputing cluster as illustrations. Figure 3 As shown, the implementation method of this supercomputing cluster may include: Step 310: Building the software stack for each operating system. Various platforms such as Docker, physical machines, and virtual machines can be selected to build the corresponding software for different operating systems (OS).
[0079] Step 320, Software Distribution. After the software stack corresponding to each operating system is built, software distribution is carried out. Distribution can be carried out through various channels such as object storage, static image provisioning, and software repositories.
[0080] Step 330: Organizing software across operating systems within a single supercomputing cluster.
[0081] Software partitioning within a single supercomputing cluster, as well as unified processing of supercomputing configuration and software loading. Related supercomputing software includes the Slurm scheduler, OpenMPI, the uCx communication library, and supercomputing applications. Through a well-designed software architecture and management, platform-independent configuration and software loading capabilities can be provided for subsequent cross-operating system job submission and scheduling within the single supercomputing cluster.
[0082] Software partitioning within a single supercomputing cluster allows for software organization and architecture classification based on arch / vendor / os / software / , especially for differentiated deployment on different operating systems such as CentOS 7.9 and Kylin 10, such as amd64 / intel / centos7.9 / scheduler / slurm. Figure 4 This is an example diagram of the software architecture in the embodiments of this application, such as... Figure 4 As shown, the software architecture can be as follows: the instruction set is amd64, the vendor is Intel, the operating system is CentOS 7.9, and the software can include base software, environment information, MPI (Message Passing Interface, parallel computing interface), and scheduler. Among them, the base software can be uC++, version 1.14.1, the MPI is OpenMPI, version 4.1.4, and the scheduler is Slurm, version 22.05.9.1.
[0083] Unified supercomputing configuration and software loading within a single supercomputing cluster. Building upon software partitioning, the scheduler configuration and software loading / unloading logic are further abstracted into a unified processing logic, achieving global consistency and independence from the operating system.
[0084] Step 340: Unified submission and scheduling of cross-operating system jobs within a single supercomputing cluster.
[0085] Through the unified processing logic described above, a unified job submission and scheduling process is achieved without the operating systems being aware of each other, and independent execution within their respective computing queues, all within a single scheduler and multiple operating systems. The following example, using the Slurm scheduler and a hybrid deployment of operating systems A and B within a supercomputing cluster, illustrates the detailed process of unified job submission and scheduling.
[0086] First, select the control node. You can choose one of the operating systems deployed in the supercomputing cluster as the control node's operating system. For example, if the cluster uses a mix of CentOS 7.9 and Kylin 10, then either CentOS 7.9 or Kylin 10 can be selected as the control node. Secondly, the Slurm job script is built and submitted. This job script is operating system-independent; within this supercomputing cluster, jobs can be submitted for both operating system A and operating system B, and jobs on different operating systems can run simultaneously. Below is an example job script: #! / bin / bash #SBATCH-N 2 ### Submit assignments for either Operating System A or Operating System B as needed. #SBATCH--partition=partition A #SBATCH-J test_job ... #module # Seamless software loading across operating systems module load openmpi / 4.1.4 ucx / 1.14.1 #config openmp export OMP_NUM_THREADs=14 export OMP_PROC_BIND=true export OMP_PLACEs=threads #config mpi export UCX_NET_DEVICEs=ens4 export SLURM_MPI_TYPE=pmix=v3 #run application srun --ntasks-node=4--cpus-per-task=14--cpu-bind=cores-m plane=4 / share / saas_app In the above job script, SBATCH-N 2 indicates that the number of computing nodes is 2, SBATCH--partition=partition A indicates that the target queue is queue A, and module load openmpi / 4.1.4 ucx / 1.14.1 indicates that the supercomputing software to be loaded is openmpi / 4.1.4 and ucx / 1.14.1. There is no distinction between the operating system here. Instead, during the execution of the job script, the unified processing logic can first detect its own operating system information (instruction set, vendor, and operating system), and then obtain the corresponding supercomputing software based on the operating system information. The last line of the above job script indicates running the supercomputing software and executing the job.
[0087] The script that loads the supercomputing software openmpi / 4.1.4 in the above job script is as follows: #%Module set name openmpi Set version 4.1.4 set type mpi conflict $name #compute installation path based on selected toolchain and flavors set PREFIX / opt / galaxy / $CTHPC_PROCESSOR_ARCHITECTURE / $CTHPC_PROCESSOR_MAKER / $CTHPC_OS_VERSION / $type / $name / $version setenv MPL_HOME / opt / galaxy / $CTHPC_PROCESSOR_ARCHITECTURE / $CTHPC_PROCESSOR_MAKER / $CTHPC_OS_VERSION / $type / $name / $version # Set software environment Prepend-path PATH $PREFIX / bin Prepend-path LD_LIBRARY_PATH $PREFIX / lib In the script above, `set PREFIX / opt / galaxy / $CTHPC_PROCESSOR_ARCHITECTURE / $CTHPC_PROCESSOR_MAKER / $CTHPC_OS_VERSION / $type / $name / $version` means retrieving operating system information (instruction set, vendor, operating system).
[0088] The operating system information obtained through cross-OS macros is represented as follows: export CTHPC_PROCESSOR_ARCHITECTURE=amd64 export CTHPC_PROCESSOR_MAKER=intel export CTHPC_OS_VERSION=centos7.9 The script above indicates that the instruction set obtained is amd64, the manufacturer is Intel, and the operating system is CentOS 7.9.
[0089] Next, configuration reading. Abstracted to the global unified supercomputing configuration of HPFS, such as slurm.conf, slurmdbd.conf, and cgroup.conf.
[0090] Then, the jobs are scheduled uniformly. Job A is scheduled to queue (partition) A, which corresponds to operating system A, and job B is scheduled to queue (partition) B, which corresponds to operating system B.
[0091] Next, the first compute nodes perform software loading. M compute nodes (A-1, A-2, ..., A-3) are determined to execute job A, and N compute nodes (B-1, B-2, ..., B-3) are determined to execute job B. First compute nodes A and B correspond to jobs A and B respectively. Using the same logic, both first compute nodes dynamically load the job software and environment within their respective software stack frameworks, such as OpenMPI and uCx, without any cross-operating system awareness.
[0092] Afterward, the jobs are executed independently. Once the first compute node has completed software loading, it can pass the relevant stack and environment information to the remaining scheduled compute node instances under its partition, enabling the parallel and independent execution of each job.
[0093] Finally, the job ends. After job A or job B is completed, the same or different jobs can be submitted again as needed. The steps in the above Slurm job script from construction and submission to job completion can be executed repeatedly as needed.
[0094] This application embodiment realizes the unified submission, scheduling, and independent execution of supercomputing jobs under a mixed deployment of multiple operating systems within a single supercomputing cluster. Compared to the limitations of traditional solutions, where a single cluster typically only supports a single operating system and cross-operating system support is usually limited to between clusters, this application embodiment allows users to activate only one cluster and simultaneously perform simulation verification of industrial software under multiple operating systems within the cluster, reducing user costs and workload and improving user experience.
[0095] In this embodiment of the application, under the mixed deployment of multiple operating systems within a single supercomputing cluster, the global unification of custom software stack partitioning, supercomputing configuration, and software loading and releasing is achieved, providing a foundation for the unified submission, scheduling, and independent operation of supercomputing jobs across operating systems within the cluster.
[0096] It should be noted that the supercomputing cluster implementation method provided in this application embodiment can be executed by a supercomputing cluster implementation device, or a control module within that device for executing the supercomputing cluster loading implementation method. This application embodiment uses the execution of the supercomputing cluster loading implementation method by a supercomputing cluster implementation device as an example to illustrate the supercomputing cluster implementation method provided in this application embodiment.
[0097] Figure 5 This is a schematic diagram of the structure of a supercomputing cluster implementation device provided in an embodiment of this application, as shown below. Figure 5 As shown, the implementation device of this supercomputing cluster includes: The software processing module 510 is used to partition the supercomputing software within the target supercomputing cluster and obtain unified processing logic for loading software across operating systems. The target supercomputing cluster includes multiple computing nodes with different operating systems. The scheduling module 520 is used to schedule the target computing nodes under the target queue to load the target supercomputing software and execute the job according to the unified processing logic when a job script is received. The target supercomputing software is the supercomputing software corresponding to the target operating system in the target computing node.
[0098] Optionally, the software processing module includes: A software partitioning unit is used to store each of the supercomputing software programs in the shared storage of the target supercomputing cluster according to the software architecture of the supercomputing software.
[0099] Optionally, the scheduling module is specifically used for: Upon receiving the job script, the target computing nodes under the target queue are scheduled by the target scheduler to load the target supercomputing software and execute the job according to the unified processing logic.
[0100] Optionally, the target scheduler may include a Slurm scheduler or a PBS scheduler.
[0101] Optionally, the device further includes: The software acquisition module is used to acquire the supercomputing software from the distribution platform through a target distribution method within the target supercomputing cluster.
[0102] Optionally, the target distribution method includes object storage, static image provisioning, or software repository.
[0103] Optionally, the device further includes: The software building module is used to build supercomputing software for various operating systems on the target platform.
[0104] Optionally, the target platform includes at least one of Docker, physical machines, and virtual machines.
[0105] Optionally, the target supercomputing cluster may also include a management and control node; The scheduling module includes: The scheduling unit is used to determine at least one of the target computing nodes through the control node when a job script is received, and to send the job script to the first computing node among the at least one target computing node; The job execution unit is used to load the target supercomputing software through the first computing node according to the unified processing logic, synchronize the target supercomputing software to other target computing nodes outside the first computing node, and execute the job through each of the target computing nodes.
[0106] Optionally, the job execution unit includes: The software loading subunit is used to obtain the target operating system in the first computing node according to the unified processing logic through the first computing node, and load the target supercomputing software according to the target operating system.
[0107] Optionally, loading the target supercomputing software according to the target operating system includes: The target supercomputing software corresponding to the target operating system is queried in the shared storage, and the target supercomputing software is loaded.
[0108] The supercomputing cluster implementation device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.
[0109] The supercomputing cluster implementation device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific implementation.
[0110] The supercomputing cluster implementation device provided in this application embodiment can achieve Figures 1 to 4 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.
[0111] The supercomputing cluster implementation method provided in this application partitions the supercomputing software within the target supercomputing cluster to obtain a unified processing logic for loading software across operating systems. The target supercomputing cluster includes multiple computing nodes with different operating systems. When a job script is received, the target computing nodes in the target queue are scheduled to load the target supercomputing software and execute the job according to the unified processing logic. This enables the simultaneous deployment of computing nodes with different operating systems within a single supercomputing cluster. Furthermore, based on software partitioning and the unified processing logic, the required software can be loaded and jobs executed across operating systems, achieving the goal of using software under different operating systems within a single supercomputing cluster. This solves the problem of not being able to use software under different operating systems within a single supercomputing cluster. Users do not need to deploy multiple supercomputing clusters with different operating systems; they can use a single cluster to use software under different operating systems, saving user costs.
[0112] Optionally, this application embodiment also provides an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described supercomputing cluster implementation method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.
[0113] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.
[0114] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. The electronic device 600 includes, but is not limited to, components such as: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610. The input unit 604 may include an image processor 6041 and a microphone 6042; the display unit 606 may include a display panel 6061; and the user input unit 607 may include a touch panel 6071 and other input devices (such as a keyboard) 6072.
[0115] Those skilled in the art will understand that the electronic device 600 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 610 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here. The memory 609 stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor 610, it implements the various processes of the above-described supercomputing cluster implementation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0116] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described supercomputing cluster implementation method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.
[0117] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0118] This application also provides a computer program product, including a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described supercomputing cluster implementation method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.
[0119] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described supercomputing cluster implementation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0120] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0121] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0123] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A method for implementing a supercomputing cluster, characterized in that, include: Within the target supercomputing cluster, the supercomputing software is partitioned, and a unified processing logic for loading software across operating systems is obtained. The target supercomputing cluster includes multiple computing nodes with different operating systems. Upon receiving the job script, the target computing node in the scheduling target queue loads the target supercomputing software and executes the job according to the unified processing logic. The target supercomputing software is the supercomputing software corresponding to the target operating system in the target computing node.
2. The method according to claim 1, characterized in that, The partitioning process for the supercomputing software includes: According to the software architecture of the supercomputing software, each of the supercomputing software programs is stored in the shared storage of the target supercomputing cluster.
3. The method according to claim 1, characterized in that, Upon receiving the job script, the target computing nodes in the scheduling target queue load the target supercomputing software and execute the job according to the unified processing logic, including: Upon receiving the job script, the target computing nodes under the target queue are scheduled by the target scheduler to load the target supercomputing software and execute the job according to the unified processing logic.
4. The method according to claim 3, characterized in that, The target scheduler includes the Slurm scheduler or the PBS scheduler.
5. The method according to claim 1, characterized in that, Before performing partitioning processing on the supercomputing software within the target supercomputing cluster, the method further includes: Within the target supercomputing cluster, the supercomputing software is obtained from the distribution platform through a target distribution method.
6. The method according to claim 5, characterized in that, The target distribution methods include object storage, static image pre-provisioning, or software repository.
7. The method according to claim 5, characterized in that, Within the target supercomputing cluster, before obtaining the supercomputing software from the distribution platform via the target distribution method, the process further includes: Build supercomputing software for each operating system using the target platform.
8. The method according to claim 7, characterized in that, The target platform includes at least one of Docker, physical machines, and virtual machines.
9. The method according to any one of claims 1-8, characterized in that, The target supercomputing cluster also includes a control node; Upon receiving the job script, the target computing nodes in the scheduling target queue load the target supercomputing software and execute the job according to the unified processing logic, including: Upon receiving the job script, at least one of the target computing nodes is determined through the control node, and the job script is sent to the first computing node among the at least one target computing node; The target supercomputing software is loaded by the first computing node according to the unified processing logic, and the target supercomputing software is synchronized to other target computing nodes outside the first computing node, and the job is executed by each of the target computing nodes.
10. The method according to claim 9, characterized in that, The loading of the target supercomputing software through the first computing node according to the unified processing logic includes: The target operating system in the first computing node is obtained according to the unified processing logic, and the target supercomputing software is loaded according to the target operating system.
11. The method according to claim 10, characterized in that, The step of loading the target supercomputing software according to the target operating system includes: The target supercomputing software corresponding to the target operating system is queried in the shared storage, and the target supercomputing software is loaded.
12. An implementation device for a supercomputing cluster, characterized in that, include: The software processing module is used to partition the supercomputing software within the target supercomputing cluster and obtain unified processing logic for loading software across operating systems. The target supercomputing cluster includes multiple computing nodes with different operating systems. The scheduling module is used to schedule the target computing nodes under the target queue to load the target supercomputing software and execute the job according to the unified processing logic when a job script is received. The target supercomputing software is the supercomputing software corresponding to the target operating system in the target computing node.
13. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method for implementing a supercomputing cluster as described in any one of claims 1-11.
14. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when the program or instructions are executed by the processor, they implement the steps of the supercomputing cluster implementation method as described in any one of claims 1-11.
15. A computer program product, characterized in that, Includes a program or instructions that, when executed by a processor, implement the steps of the method for implementing a supercomputing cluster as described in any one of claims 1-11.