Polymorphic heterogeneous intelligent computing resource fusion scheduling method and system compatible with meteorological supercomputing architecture

Abstract

The invention discloses a polymorphic heterogeneous intelligent computing resource fusion scheduling method and system compatible with a meteorological supercomputing architecture. The method comprises the following steps: creating a service cluster and a computing cluster; constructing a computing cluster computing power aggregation application service; creating a meteorological supercomputing architecture intelligent computing cluster fusion scheduling support environment; an elastic supercomputing service is constructed in the meteorological supercomputing architecture intelligent computing cluster; and constructing a software stack support environment adaptive to the polymorphic heterogeneous intelligent computing resources. According to the method, a service cluster is established, a computing cluster is scheduled based on a Kubernetes scheduling component of a fusion scheduler, adaptive heterogeneous resources are scheduled by adopting a container technology, a software interface is managed based on an intelligent computing cluster queue of a meteorological super computing architecture, scheduling based on a Slurm scheduling component of the fusion scheduler is realized, and native job scheduling of the intelligent computing cluster of the super computing architecture is realized; an elastic super-computing container technology is adopted, and an adaptive super-computing architecture intelligent computing power and software stack distributed environment is constructed; a fusion unified scheduling technology is formed, unified efficient scheduling utilization is achieved, and the resource utilization efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of fusion scheduling of polymorphic heterogeneous intelligent computing resources, and more specifically to a method and system for fusion scheduling of polymorphic heterogeneous intelligent computing resources compatible with a meteorological supercomputer architecture. Background Art

[0002] With the deep integration and development of artificial intelligence and the meteorological industry, especially the rapid development of large meteorological models, there has been a spurt in demand for intelligent computing resources. The meteorological department has built imported and domestically produced polymorphic heterogeneous intelligent computing resources in phases, including supercomputing architecture intelligent computing GPU clusters, GPU servers, and domestic Haiguang and Ascend acceleration cards. How to coordinate management and flexibly schedule the application of these intelligent computing resources built in different periods, with different card types and different architectures has become a major problem.

[0003] Intelligent computing resource scheduling technology involves resource pooling technology, supercomputing Internet technology, and cloud-native resource scheduling technology based on Kubernetes. Resource pooling scheduling is a method of virtualizing, aggregating multiple cards, and dynamically releasing physical accelerator cards through software definition. Specific models of GPU accelerator cards support resource division at the hardware and driver levels to form isolated virtualized resources. In China, resource elastic scheduling is mainly achieved through hijacking and controlling CUDA calls. Supercomputing Internet technology uses Internet thinking to set up intelligent computing resources in supercomputing centers and provide intelligent computing resource services through the Internet based on the use of supercomputing. Cloud-native resource scheduling technology based on Kubernetes mainly uses container technology to achieve scheduling of different server resources. The core technology is also to achieve resource elastic scheduling through hijacking and controlling CUDA calls. Based on DevicePlugin, the device resources on the node are incorporated into Kubernetes resource management, so that the container can schedule resources.

[0004] The technical method of accelerator card resource pooling is limited by the accelerator card type. Only specific card types support virtualization, and different accelerator cards have different virtualization technologies, which makes it difficult to solve the problem of integrated scheduling of polymorphic heterogeneous intelligent computing resources in the meteorological department. The intelligent computing resources of the supercomputing architecture are suitable for the scheduling system of supercomputing. They are generally shared and scheduled by professional cluster software, which makes it difficult to pool and schedule. At the same time, there are certain challenges in the adaptation and shielding of heterogeneous intelligent computing resources by the artificial intelligence framework. The scheduling of supercomputing Internet intelligent computing resources relies on the scheduling technology of the integrated computing power service platform. It is necessary to transform the original scheduling ecology of heterogeneous intelligent computing power and change the original construction design of computing power. It is difficult to solve the problem of integrated scheduling of polymorphic heterogeneous intelligent computing resources in the meteorological department. The cloud-native resource scheduling technology based on Kubernetes is suitable for the unified management of server-type intelligent computing resources, but it is difficult to be compatible with the scheduling and adaptation of supercomputing architecture intelligent computing resources. Summary of the invention

[0005] To this end, the present invention aims to solve the problems of unified scheduling of polymorphic heterogeneous resources compatible with supercomputing architecture and adaptation of heterogeneous computing power software stacks, and provide a polymorphic heterogeneous intelligent computing resource fusion scheduling method compatible with meteorological supercomputing architecture, to realize a heterogeneous resource scheduling method compatible with computing power native architecture and ecology, which can support unified scheduling of polymorphic heterogeneous intelligent computing resources, adapt and be compatible with different software ecological environments, shield the complexity of the underlying details of heterogeneous resources, meet the unified and efficient scheduling and utilization of various heterogeneous intelligent computing resources in different construction periods, and provide support for the innovation and development of meteorological science and technology driven by artificial intelligence technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A method for fusion scheduling of polymorphic heterogeneous intelligent computing resources compatible with a meteorological supercomputer architecture comprises the following steps:

[0008] Step P1: Create a service cluster and a computing cluster;

[0009] Step P2: Build computing cluster computing power aggregation application service;

[0010] Step P3: Create a scheduling support environment for the meteorological supercomputing architecture intelligent computing cluster;

[0011] Step P4: Build elastic supercomputing services in the meteorological supercomputing architecture intelligent computing cluster;

[0012] Step P5: Build a software stack support environment that is adapted to polymorphic and heterogeneous intelligent computing resources;

[0013] Step P6: Release the meteorological artificial intelligence basic support technology platform in the service cluster and build computing power scheduling tasks.

[0014] In the above-mentioned multi-state heterogeneous intelligent computing resource fusion scheduling system compatible with the meteorological supercomputer architecture, in step P1,

[0015] Computing cluster: Create a Kubernetes computing cluster from scattered and polymorphic intelligent computing server resources; the Kubernetes computing cluster mainly carries the computing tasks of various artificial intelligence applications of users;

[0016] Service cluster: includes the deployed meteorological artificial intelligence basic support technology platform, storage management, image warehouse and fusion scheduler. The fusion scheduler includes K8S Scheduler and Slurm Scheduler components. The fusion scheduler makes a task request to create K8S computing to the computing cluster, and the fusion scheduler makes a task request to create Slurm job to the meteorological supercomputing architecture intelligent computing cluster.

[0017] The above-mentioned polymorphic heterogeneous intelligent computing resource fusion scheduling system compatible with the meteorological supercomputer architecture, in step P2, builds a Kubernetes computing cluster computing power aggregation application service within the computing cluster; that is, builds a Horovod-based distributed training framework in the Kubernetes computing cluster image, realizes the support of multi-machine and multi-card computing power aggregation communication of the Kubernetes computing cluster, and forms the ability to aggregate scattered computing power on multiple machines.

[0018] The above-mentioned multi-state heterogeneous intelligent computing resource fusion scheduling system compatible with the meteorological supercomputing architecture, in step P3, creates a scheduling support environment for the meteorological supercomputing architecture intelligent computing cluster, and the constructed scheduling support environment includes a supercomputing architecture intelligent computing cluster login node, a supercomputing architecture intelligent computing cluster management node, a supercomputing architecture intelligent computing cluster computing node, and a high-performance distributed shared storage;

[0019] On the login node of the supercomputing architecture intelligent computing cluster: deploy log collection service, log storage query service and proxy service; the log collection service is used to collect the logs of slurm job computing tasks, the log storage query service is used to store the computing task logs collected in the log collection service and provide query services, and the proxy service is used to forward the user's computing task request to the supercomputing management node;

[0020] On the management node of the supercomputing architecture intelligent computing cluster: deploy the SlurmRestd service and the Slurmctld service; the SlurmRestd service provides the interaction between the login node of the supercomputing architecture intelligent computing cluster and the management node of the supercomputing architecture intelligent computing cluster, accepts the request to create a Slurm Job from the proxy service, and provides the Slurm Rest API for creating, deleting, and querying computing tasks; the Slurmctld service is used to monitor resources and jobs, perform user token authentication, and propose to create a Slurm Job to the computing node of the supercomputing architecture intelligent computing cluster;

[0021] Build a computing task job script file on the high-performance distributed shared storage of the supercomputing architecture intelligent computing cluster, and run the computing task on N supercomputing nodes according to the number of nodes N requested by the computing task.

[0022] The above-mentioned polymorphic heterogeneous intelligent computing resource fusion scheduling system compatible with the meteorological supercomputing architecture, in step P4, deploys an elastic supercomputing container service in the computing node of the supercomputing architecture intelligent computing cluster, builds a Sigularity distributed operating environment based on the shared storage of the computing nodes of the supercomputing architecture intelligent computing cluster, supports distributed training communication, and adapts to the non-super-privileged user access characteristics of the supercomputing architecture, starts the container based on the Sigularity distributed operating environment inheriting the user permissions initiated by the computing task, and the user context remains unchanged, keeping the user permissions the same inside and outside the container; the construction of the elastic supercomputing container service supports the specific software environment runtime of the user's artificial intelligence computing task, builds the data environment set by the task, starts the computing job and schedules the computing node accelerator device on the computing node of the specific scale of the task allocated by Slurm, and realizes cross-node communication access.

[0023] The above-mentioned polymorphic heterogeneous intelligent computing resource fusion scheduling system compatible with the meteorological supercomputing architecture, in step P5, builds a software stack support environment adapted to polymorphic heterogeneous intelligent computing resources in the image repository of the service cluster for the computing cluster and the meteorological supercomputing architecture intelligent computing cluster: encapsulates the CUDA / Pytorch image adapted to the GPU accelerator card of the meteorological supercomputing architecture intelligent computing cluster, the DTK / Pytorch image adapted to the Haiguang DCU accelerator card of the meteorological supercomputing architecture intelligent computing cluster, the CUDA / Pytorch image adapted to the computing power resources of the Kubernetes computing cluster, and encapsulates the large model training AI framework, acceleration framework, and meteorological application software environment adapted to the application, forming a software stack support environment adapted to heterogeneous resources that can be used out of the box with one click.

[0024] The above-mentioned polymorphic heterogeneous intelligent computing resource fusion scheduling system compatible with the meteorological supercomputer architecture, in step P6, builds options for the specifications of multiple heterogeneous intelligent computing resources on the user interaction layer of the meteorological artificial intelligence basic support technology platform of the service cluster. The specifications of the multiple heterogeneous intelligent computing resources include the computing power specifications of the meteorological supercomputer architecture, the resource specifications of different types of Kubernetes acceleration cards, and a variety of adapted software image environments; users interactively build artificial intelligence application computing tasks, select resource specifications and adapted software environments with one click, shield the underlying differences of polymorphic heterogeneous resources, and conveniently submit computing tasks and obtain computing feedback through a unified interactive interface.

[0025] The above-mentioned multi-state heterogeneous intelligent computing resource fusion scheduling system compatible with the meteorological supercomputing architecture includes a service cluster, a computing cluster, a meteorological supercomputing architecture intelligent computing cluster and a software stack support environment;

[0026] The service cluster includes a meteorological artificial intelligence basic support technology platform, storage management and control, a mirror warehouse and a fusion scheduler; the meteorological artificial intelligence basic support technology platform is used to support user interaction and obtain user computing task requests;

[0027] The computing cluster is created by scattered and polymorphic intelligent computing server resources, which is used to receive computing task requests from the service cluster; in the computing cluster, a Kubernetes computing cluster computing power aggregation application service is built; that is, a distributed training framework based on Horovod is built in the Kubernetes computing cluster image to support the computing power aggregation communication of multiple machines and multiple cards in the Kubernetes computing cluster, forming the ability to aggregate scattered computing power on multiple machines;

[0028] The meteorological supercomputing architecture intelligent computing cluster includes a supercomputing architecture intelligent computing cluster login node, a supercomputing architecture intelligent computing cluster management node, a supercomputing architecture intelligent computing cluster computing node, and a supercomputing architecture intelligent computing cluster high-performance distributed shared storage; on the supercomputing architecture intelligent computing cluster login node: deploy a log collection service, a log storage query service, and a proxy service; the log collection service is used to collect logs of computing tasks, the log storage query service is used to store and provide query services for the computing task logs collected in the log collection service, and the proxy service is used to forward the user's computing task request to the supercomputing architecture intelligent computing cluster management node;

[0029] Deploy SlurmRestd service and Slurmctld service on the management node of the supercomputing architecture intelligent computing cluster; SlurmRestd service provides interaction between the login node of the supercomputing architecture intelligent computing cluster and the management node of the supercomputing architecture intelligent computing cluster, accepts the request to create Slurm Job from the proxy service, and provides Slurm Rest API for creating, deleting and querying computing tasks; Slurmctld service is used to monitor resources and jobs, perform user Token authentication, and propose to create Slurm Job to the computing node of the supercomputing architecture intelligent computing cluster; construct computing task job script files on the high-performance distributed shared storage of the supercomputing architecture intelligent computing cluster, and run computing tasks on N supercomputing architecture intelligent computing cluster computing nodes according to the number of nodes N requested by the computing task;

[0030] The software stack supports the following environments: CUDA / Pytorch images adapted to the GPU accelerator cards of the meteorological supercomputing architecture intelligent computing cluster, DTK / Pytorch images adapted to the Haiguang DCU accelerator cards of the meteorological supercomputing architecture intelligent computing cluster, CUDA / Pytorch images adapted to the computing resources of the Kubernetes computing cluster, and encapsulated large-model training AI frameworks, acceleration frameworks, and meteorological application software environments adapted to the applications, forming a one-click out-of-the-box software stack support environment.

[0031] The above-mentioned polymorphic heterogeneous intelligent computing resource fusion scheduling system compatible with the meteorological supercomputing architecture deploys elastic supercomputing container services in the computing nodes of the supercomputing architecture intelligent computing cluster, builds a Sigularity distributed operating environment based on the shared storage of the computing nodes of the supercomputing architecture intelligent computing cluster, supports distributed training communication, and adapts to the non-super-privileged user access characteristics of the supercomputing architecture. The container is started based on the Sigularity distributed operating environment inheriting the user permissions initiated by the computing task, and the user context remains unchanged, keeping the user permissions the same inside and outside the container; the construction of the elastic supercomputing container service supports the specific software environment operation of the user's artificial intelligence computing task, builds the data environment set by the task, starts the computing job and schedules the computing node accelerator equipment on the computing nodes of the task-specific scale allocated by Slurm, and realizes cross-node communication access.

[0032] The technical solution of the present invention achieves the following beneficial technical effects:

[0033] This method proposes a method and system for the fusion scheduling of polymorphic heterogeneous intelligent computing resources compatible with the meteorological supercomputing architecture, which organizes scattered intelligent computing servers into a Kubernetes computing cluster, schedules the computing cluster based on the fusion scheduler K8S scheduling component of the service cluster, and uses container technology to schedule and adapt heterogeneous resources. Based on the meteorological supercomputing architecture intelligent computing cluster queue management software interface, the fusion scheduler Slurm scheduling component scheduling is used to achieve supercomputing architecture cluster native job scheduling, and elastic supercomputing container technology is used to build a software stack distributed environment that adapts to supercomputing computing power and application characteristics. Through the integration and development of multiple technologies, a polymorphic heterogeneous intelligent computing resource fusion unified scheduling technology is formed, which shields the complexity of the underlying details of heterogeneous resources for meteorological users, realizes the unified and efficient scheduling and utilization of multiple heterogeneous intelligent computing resources, improves resource utilization efficiency, and provides timely and effective support for the rapid development of meteorological scientific and technological innovation driven by artificial intelligence technology.

[0034] Realize unified dispatching services for polymorphic and heterogeneous intelligent computing resources in the meteorological field: compatible with native computing architecture, meet the unified and efficient dispatching and utilization of various heterogeneous intelligent computing resources constructed in different construction periods, effectively solve the application complexity of scattered and heterogeneous intelligent computing resources in meteorological departments, and conveniently provide a variety of flexible unified resource services for different application scenarios.

[0035] Shielding the differences in the underlying software and hardware of intelligent computing resources: We have built an environmental adaptation technology for polymorphic heterogeneous intelligent computing resources. This technology automatically adapts the software environment ecology of AI chips for a variety of computing powers, shields the differences in the underlying software and hardware of intelligent computing resources for users, provides a unified application service model, simplifies user operations, and provides convenient, flexible, and elastic resource usage.

[0036] Distributed computing power aggregation technology: Based on the distributed framework, a single physical GPU server is logically aggregated to form a distributed computing power cluster with multiple machines and multiple cards, and computing power support for larger-scale scenarios is provided based on the Horovod distributed framework.

[0037] Online / offline hybrid scheduling improves resource efficiency: It realizes optimized use of application scenario characteristics and resource architecture characteristics. For R&D, the code development process avoids GPU card resource occupation and waste, and provides CPU resources and single-card GPU resources: For small model training, the Horovod distributed training framework can provide multi-card resource aggregation and use multi-machine multi-card resources; for large model training, the resources of the supercomputing architecture can be called for high-performance and efficient distributed training calculations.

[0038] High scalability of intelligent computing resource scheduling: Through cloud-native scheduling and scheduling interface adaptation technology, it is convenient to realize the expansion, access, scheduling and exit of various new computing powers, while conveniently supporting the scaling of different intelligent computing nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The overall technical method diagram of the polymorphic heterogeneous intelligent computing resource fusion scheduling method compatible with the meteorological supercomputer architecture of the present invention. DETAILED DESCRIPTION

[0040] Example 1: A multi-state heterogeneous intelligent computing resource fusion scheduling method compatible with a meteorological supercomputing architecture.

[0041] Step P1: Create a service cluster and a computing cluster;

[0042] Computing cluster: Create a Kubernetes computing cluster from scattered and polymorphic intelligent computing server resources; the Kubernetes computing cluster mainly carries the computing tasks of various artificial intelligence applications of users;

[0043] Service cluster: includes the deployed meteorological artificial intelligence basic support technology platform, storage management, image warehouse and fusion scheduler. The fusion scheduler includes the scheduling system K8S Scheduler and Slurm Scheduler components. The fusion scheduler makes a task request to create K8S computing to the computing cluster, and the fusion scheduler makes a request to create a Slurm job to the meteorological supercomputing architecture intelligent computing cluster.

[0044] Step P2: Build computing cluster computing power aggregation application service;

[0045] In the computing cluster, build a Kubernetes computing cluster computing power aggregation application service; that is, build a Horovod-based distributed training framework in the Kubernetes computing cluster image to support the computing power aggregation communication of multiple machines and multiple cards in the Kubernetes computing cluster, and form the ability to aggregate scattered computing power on multiple machines.

[0046] Based on the distributed framework, a single physical GPU server is logically aggregated to form a distributed computing cluster with multiple machines and multiple cards, and based on the Horovod distributed framework, computing support for larger-scale scenarios is provided.

[0047] Step P3: Create a scheduling support environment for the meteorological supercomputing architecture intelligent computing cluster;

[0048] Create a scheduling support environment for multiple meteorological supercomputing architecture intelligent computing clusters. The constructed scheduling support environment includes supercomputing architecture intelligent computing cluster login nodes, supercomputing architecture intelligent computing cluster management nodes, supercomputing architecture intelligent computing cluster computing nodes, and supercomputing architecture intelligent computing cluster high-performance distributed shared storage;

[0049] On the login node of the supercomputing architecture intelligent computing cluster: deploy the log collection service, log storage query service and proxy service; the log collection service is used to collect the logs of computing tasks and push them to Grafrana loki, the log storage query service is used to store the computing task logs collected in the log collection service and provide query services, as well as query the running computing tasks and job status, and the proxy service is used to forward the user's computing task request to the supercomputing architecture intelligent computing cluster management node;

[0050] Deploy SlurmRestd service and Slurmctld service on the management node of the supercomputing architecture intelligent computing cluster; SlurmRestd service provides interaction between the login node of the supercomputing architecture intelligent computing cluster and the management node of the supercomputing architecture intelligent computing cluster, accepts the request to create Slurm Job from the proxy service, and provides Slurm Rest API for creating, deleting and querying computing tasks; Slurmctld service is used to monitor resources and jobs, and uses SlurmRestd service to call the relevant interface of SlurmRestd service to propose the creation of Slurm Job to the computing node of the supercomputing architecture intelligent computing cluster;

[0051] Build a computing task job script file on the high-performance distributed shared storage of the supercomputing architecture intelligent computing cluster, and run the computing task on N supercomputing architecture intelligent computing cluster computing nodes according to the number of nodes N requested by the computing task. The high-performance distributed shared storage of the supercomputing architecture intelligent computing cluster can enable N supercomputing architecture intelligent computing cluster computing nodes to share storage content, with faster and more intensive performance.

[0052] By setting up computing clusters and meteorological supercomputing architecture intelligent computing clusters, the scattered and heterogeneous intelligent computing resources of the meteorological department can be integrated, and by utilizing service clusters, unified and efficient scheduling and utilization for different application scenarios can be achieved, thereby providing a variety of flexible resource specification services.

[0053] Step P4: Build elastic supercomputing services in the meteorological supercomputing architecture intelligent computing cluster;

[0054] Elastic supercomputing container services are deployed in the computing nodes of the supercomputing architecture intelligent computing cluster. The elastic supercomputing container services are used to build a hardware and software environment and a distributed computing environment that adapts to resources and computing tasks. Based on the shared storage of the computing nodes of the supercomputing architecture intelligent computing cluster, a Sigularity distributed operating environment is built to support distributed training communication and adapt to the non-super-privileged user access characteristics of the supercomputing architecture. The Sigularity distributed operating environment inherits the user permissions initiated by the computing task to start the container. The user context remains unchanged, and the user permissions are kept the same inside and outside the container. The construction of the elastic supercomputing container service supports the specific software environment runtime of the user's artificial intelligence computing task, builds the data environment set by the task, starts the computing job and schedules the computing node accelerator equipment on the computing nodes of the task-specific scale allocated by Slurm, and realizes cross-node communication access.

[0055] The elastic supercomputing container service is used to build a hardware and software environment and a distributed computing environment that adapts resources and computing tasks. The image in Singularity reads data through shared storage or accesses Minio in the local area network through Nginx to pull data sets, project codes, and upload training results.

[0056] It supports full utilization of the software and hardware resources of the meteorological supercomputing architecture intelligent computing cluster, including the supercomputing architecture intelligent computing cluster scheduler Slurm, cross-node communication libraries (IntelMPI, OpenMPI), network interconnection (Infiniband), file systems, and acceleration devices (GPU / DCU, etc.).

[0057] In the meteorological supercomputing architecture intelligent computing cluster, by building a proxy service in the supercomputing architecture intelligent computing cluster login node and building SlurmRestd service and Slurmctld service in the supercomputing architecture intelligent computing cluster management node, high scalability of supercomputing architecture intelligent computing cluster architecture intelligent computing resource scheduling can be achieved: through the scheduling interface adaptation technology, the expansion access scheduling and exit of multiple sets of Slurm-based scheduling supercomputing architecture intelligent computing cluster systems can be conveniently realized. At the same time, building an elastic supercomputing container service on the supercomputing architecture intelligent computing cluster computing node can conveniently support computing tasks of different scales.

[0058] Step P5: Build a software stack support environment that is adapted to polymorphic and heterogeneous intelligent computing resources;

[0059] For computing clusters and meteorological supercomputing architecture intelligent computing clusters, a software stack support environment that is adapted to polymorphic heterogeneous intelligent computing resources is built: encapsulating CUDA / Pytorch images that are adapted to GPU accelerator cards in meteorological supercomputing architecture intelligent computing clusters, DTK / Pytorch images that are adapted to Haiguang DCU accelerator cards in meteorological supercomputing architecture intelligent computing clusters, CUDA / Pytorch images that are adapted to computing resources in Kubernetes computing clusters, as well as encapsulating large-model training AI frameworks, acceleration frameworks, and meteorological application software environments that are adapted to applications, forming a software stack support environment that is adapted to heterogeneous resources and can be used out of the box with one click.

[0060] The one-click out-of-the-box software stack supports the environment to shield the differences in the underlying software and hardware of intelligent computing resources: an environmental adaptation technology for polymorphic heterogeneous intelligent computing resources is built to automatically adapt the software environment ecology of its AI chip for various computing powers, shield the differences in the underlying software and hardware of intelligent computing resources for users, provide a unified application service model, simplify user operations, and provide convenient, flexible and elastic resource usage.

[0061] Step P6: Release the meteorological artificial intelligence basic support technology platform in the service cluster and build computing power scheduling tasks.

[0062] On the user interaction layer of the meteorological artificial intelligence basic support technology platform of the service cluster, options for the specifications of various heterogeneous intelligent computing resources are constructed (drop-down options for various resource specifications in the interactive page). The specifications of various heterogeneous intelligent computing resources include meteorological supercomputing architecture computing power specifications, Kubernetes different types of acceleration card resource specifications, and a variety of adapted software image environments. Users interact to build artificial intelligence application computing tasks, select resource specifications and adapted software environments with one click, shield the underlying differences of polymorphic heterogeneous resources, and conveniently submit computing tasks and obtain computing feedback through a unified interactive interface.

[0063] The polymorphic heterogeneous intelligent computing resource fusion scheduling method compatible with the meteorological supercomputing architecture of this application realizes online / offline hybrid scheduling to improve resource efficiency: it realizes the optimized use of application scenario characteristics and resource architecture characteristics, and is oriented to R&D. The code development process avoids GPU card resource occupation and waste, and provides CPU resources and single-card GPU resources.

[0064] Users submit task requests through the meteorological artificial intelligence basic support technology platform of the service cluster, and the fusion scheduler schedules and allocates task resources according to the scale of model training in the task request.

[0065] For task requests for small model training, the K8S Scheduler component of the scheduling system submits task requests for creating K8S computing and querying jobs to the Kubernetes computing cluster; the Horovod distributed training framework provides aggregation of multi-card resources and uses multi-machine and multi-card resources.

[0066] For task requests for large model training, a task request for creating a Slurm job is submitted to the meteorological supercomputing architecture intelligent computing cluster through the Slurm Scheduler component. After the proxy service of the supercomputing architecture intelligent computing cluster login node receives the task request for creating a Slurmjob, it is forwarded to the SlurmRestd service, the management node of the supercomputing architecture intelligent computing cluster, and then transmitted to the Slurmctld service through the SlurmRestd service, thereby creating a Slurm job task and building a computing task job script file on the high-performance distributed shared storage of the supercomputing architecture intelligent computing cluster. According to the number of nodes N requested by the computing task, Srun runs the computing task on the N supercomputing architecture intelligent computing cluster computing nodes. By deploying elastic supercomputing container services, distributed training communication support is provided, and the resources of the meteorological supercomputing architecture intelligent computing cluster architecture can be called to perform high-performance and efficient distributed training calculations.

[0067] Based on the polymorphic heterogeneous intelligent computing resource fusion scheduling method compatible with the meteorological supercomputer architecture, the multi-dimensional intelligent computing resource scheduling of the meteorological artificial intelligence basic support technology platform is realized, and the intelligent computing power of 525 polymorphic and heterogeneous acceleration cards is uniformly scheduled to support the full AI process of artificial intelligence meteorological applications from data processing, model development, model training, and model services.

[0068] Table 1. Unified scheduling computing power list

[0069]

[0070] Example 2: A polymorphic heterogeneous intelligent computing resource fusion scheduling system compatible with a meteorological supercomputing architecture.

[0071] A multi-state heterogeneous intelligent computing resource fusion scheduling system, including service clusters, computing clusters, meteorological supercomputing architecture intelligent computing clusters, and software stack support environment;

[0072] The service cluster includes a meteorological artificial intelligence basic support technology platform, storage management, image warehouse and joint scheduler; the meteorological artificial intelligence basic support technology platform is used to support user interaction and obtain user computing task requests;

[0073] The computing cluster is created by scattered and polymorphic intelligent computing server resources, which is used to receive computing task requests from the service cluster; within the computing cluster, a Kubernetes computing cluster computing power aggregation application service is built; that is, a Horovod-based distributed training framework is built in the Kubernetes computing cluster image to support multi-machine and multi-card computing power aggregation communication;

[0074] The meteorological supercomputing architecture intelligent computing cluster constructs a scheduling support environment, including a supercomputing architecture intelligent computing cluster login node, a supercomputing architecture intelligent computing cluster management node, a supercomputing architecture intelligent computing cluster computing node, and a supercomputing architecture intelligent computing cluster high-performance distributed shared storage; on the supercomputing architecture intelligent computing cluster login node: deploy log collection services, log storage query services, and proxy services;

[0075] Log collection service: set up log collection components to collect logs of computing tasks (slurm jobs); log storage query service: used to provide external exposure of slurm log interface, that is, used to store and provide query services for computing task logs collected in log collection service; proxy service: here, it is used as a proxy to access Minio deployed in the local area network, and is used to forward computing task requests of meteorological artificial intelligence basic support technology platform to the supercomputing architecture intelligent computing cluster management node;

[0076] Deploy SlurmRestd service and Slurmctld service on the management node of the supercomputing architecture intelligent computing cluster; SlurmRestd service provides interaction between the supercomputing architecture intelligent computing cluster login node and the supercomputing architecture intelligent computing cluster management node, accepts requests for creating Slurm Jobs from the proxy service, and provides Slurm Rest API for creating, deleting, and querying computing tasks; Slurmctld service is used to monitor resources and jobs, perform user Token authentication, and propose to create Slurm Jobs to the computing nodes of the supercomputing architecture intelligent computing cluster; build computing task job script files on the high-performance distributed shared storage of the supercomputing architecture intelligent computing cluster, and run computing tasks on N supercomputing architecture intelligent computing cluster computing nodes through Srun according to the number of nodes N requested by the computing task.

[0077] Elastic supercomputing container services are deployed in the computing nodes of the supercomputing architecture intelligent computing cluster. The Sigularity distributed operating environment is built based on the shared storage of the computing nodes of the supercomputing architecture intelligent computing cluster to support distributed training communication and adapt to the non-super-privileged user access characteristics of the supercomputing architecture. The container is started based on the Sigularity distributed operating environment inheriting the user permissions initiated by the computing task. The user context remains unchanged and the user permissions are kept the same inside and outside the container. The construction of the elastic supercomputing container service supports the specific software environment runtime of the user's artificial intelligence computing task, builds the data environment set by the task, starts the computing job and schedules the computing node accelerator equipment on the computing nodes of the task-specific scale allocated by Slurm, and realizes cross-node communication access.

[0078] The software stack support environment includes: CUDA / Pytorch images adapted to the GPU accelerator card of the meteorological supercomputing architecture intelligent computing cluster, DTK / Pytorch images adapted to the Haiguang DCU accelerator card of the supercomputing architecture intelligent computing cluster, CUDA / Pytorch images adapted to the computing resources of the Kubernetes computing cluster, and encapsulated large-model training AI frameworks, acceleration frameworks, and meteorological application software environments adapted to the applications, forming a one-click out-of-the-box software stack support environment.

[0079] This application organizes scattered intelligent computing servers into a Kubernetes computing cluster, schedules the computing cluster based on the fusion scheduler K8S scheduling component of the service cluster, uses container technology to schedule and adapt heterogeneous resources, and implements scheduling based on the fusion scheduler Slurm scheduling component based on the meteorological supercomputing architecture intelligent computing cluster queue management software interface to achieve supercomputing architecture cluster native job scheduling, and uses elastic supercomputing container technology to build a software stack distributed environment that adapts to supercomputing computing power and application characteristics.

[0080] Through the integration and development of multiple technologies, a unified scheduling technology for the integration of polymorphic heterogeneous intelligent computing resources has been formed, which shields meteorological users from the complexity of the underlying details of heterogeneous resources, realizes the unified and efficient scheduling and utilization of multiple heterogeneous intelligent computing resources, improves resource utilization efficiency, and provides timely and effective support for the rapid development of meteorological scientific and technological innovation driven by artificial intelligence technology.

[0081] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the claims of this patent application.

Claims

1. A method for fusion scheduling of polymorphic heterogeneous intelligent computing resources compatible with meteorological supercomputing architecture, characterized in that: The steps include: Step P1: Create a service cluster and a computing cluster; Step P2: Build computing cluster computing power aggregation application service; Step P3: Create a scheduling support environment for the meteorological supercomputing architecture intelligent computing cluster; Step P4: Build elastic supercomputing services in the meteorological supercomputing architecture intelligent computing cluster; Step P5: Build a software stack support environment that is adapted to polymorphic and heterogeneous intelligent computing resources; Step P6: Release the meteorological artificial intelligence basic support technology platform in the service cluster and build computing power scheduling tasks.

2. According to the method for fusion scheduling of multi-state heterogeneous intelligent computing resources compatible with meteorological supercomputing architecture in claim 1, it is characterized in that: In step P1, Computing cluster: Create a Kubernetes computing cluster from scattered and polymorphic intelligent computing server resources; the Kubernetes computing cluster mainly carries the computing tasks of various artificial intelligence applications of users; Service cluster: includes the deployed meteorological artificial intelligence basic support technology platform, storage management, image warehouse and fusion scheduler. The fusion scheduler includes K8S Scheduler and Slurm Scheduler components. The fusion scheduler submits a task request to the computing cluster to create a K8S computing task, and the fusion scheduler submits a task request to the meteorological supercomputing architecture intelligent computing cluster to create a Slurm job task.

3. According to the method of claim 1, a multi-state heterogeneous intelligent computing resource fusion scheduling method compatible with a meteorological supercomputer architecture is characterized in that: In step P2, a Kubernetes computing cluster computing power aggregation application service is built in the computing cluster; that is, a Horovod-based distributed training framework is built in the Kubernetes computing cluster image to support the computing power aggregation communication of multiple machines and multiple cards in the Kubernetes computing cluster, thus forming the ability to aggregate scattered computing power on multiple machines.

4. According to the method of claim 3, a multi-state heterogeneous intelligent computing resource fusion scheduling method compatible with a meteorological supercomputer architecture is characterized in that: In step P3, a scheduling support environment for the meteorological supercomputing architecture intelligent computing cluster is created. The constructed scheduling support environment includes a supercomputing architecture intelligent computing cluster login node, a supercomputing architecture intelligent computing cluster management node, a supercomputing architecture intelligent computing cluster computing node, and a high-performance distributed shared storage; On the login node of the supercomputing architecture intelligent computing cluster: deploy log collection service, log storage query service and proxy service; The log collection service is used to collect the logs of slurm job computing tasks. The log storage query service is used to store the computing task logs collected by the log collection service and provide query services. The proxy service is used to forward the user's computing task request to the supercomputing management node. On the management node of the supercomputing architecture intelligent computing cluster: deploy the SlurmRestd service and the Slurmctld service; the SlurmRestd service provides the interaction between the login node of the supercomputing architecture intelligent computing cluster and the management node of the supercomputing architecture intelligent computing cluster, accepts the request to create a Slurm Job from the proxy service, and provides the Slurm Rest API for creating, deleting, and querying computing tasks; the Slurmctld service is used to monitor resources and jobs, perform user token authentication, and propose to create a Slurm Job to the computing node of the supercomputing architecture intelligent computing cluster; Build a computing task job script file on the high-performance distributed shared storage of the supercomputing architecture intelligent computing cluster, and run the computing task on N supercomputing nodes according to the number of nodes N requested by the computing task.

5. According to the method for fusion scheduling of multi-state heterogeneous intelligent computing resources compatible with meteorological supercomputing architecture of claim 4, it is characterized in that: In step P4, an elastic supercomputing container service is deployed in the computing nodes of the supercomputing architecture intelligent computing cluster, and a Sigularity distributed operating environment is built based on the shared storage of the computing nodes of the supercomputing architecture intelligent computing cluster to support distributed training communication and adapt to the non-super-privileged user access characteristics of the supercomputing architecture. The container is started based on the Sigularity distributed operating environment inheriting the user permissions initiated by the computing task, and the user context remains unchanged, keeping the user permissions the same inside and outside the container. The construction of the elastic supercomputing container service supports the specific software environment runtime of the user's artificial intelligence computing task, builds the data environment set by the task, starts the computing job and schedules the computing node accelerator device on the computing node of the specific scale of the task allocated by Slurm, and realizes cross-node communication access.

6. According to the method of claim 4, a multi-state heterogeneous intelligent computing resource fusion scheduling method compatible with a meteorological supercomputer architecture is characterized in that: In step P5, for the computing cluster and the meteorological supercomputing architecture intelligent computing cluster, a software stack support environment that is adapted to polymorphic heterogeneous intelligent computing resources is built in the image repository of the service cluster: CUDA / Pytorch images that are adapted to the GPU accelerator card of the meteorological supercomputing architecture intelligent computing cluster, DTK / Pytorch images that are adapted to the Haiguang DCU accelerator card of the meteorological supercomputing architecture intelligent computing cluster, CUDA / Pytorch images that are adapted to the computing power resources of the Kubernetes computing cluster, as well as encapsulation of large-model training AI frameworks, acceleration frameworks, and meteorological application software environments that are adapted to applications, are encapsulated to form a software stack support environment that is adapted to heterogeneous resources and can be used out of the box with one click.

7. According to the method for fusion scheduling of polymorphic heterogeneous intelligent computing resources compatible with meteorological supercomputing architecture in claim 1, it is characterized in that: In step P6, on the user interaction layer of the meteorological artificial intelligence basic support technology platform of the service cluster, options for specifications of various heterogeneous intelligent computing resources are constructed. The specifications of various heterogeneous intelligent computing resources include meteorological supercomputing architecture computing power specifications, Kubernetes different types of acceleration card resource specifications, and various adapted software image environments; users interactively construct artificial intelligence application computing tasks, select resource specifications and adapted software environments with one click, shield the underlying differences of polymorphic heterogeneous resources, and conveniently submit computing tasks and obtain computing feedback through a unified interactive interface.

8. A multi-state heterogeneous intelligent computing resource fusion scheduling system compatible with meteorological supercomputer architecture, characterized in that: Including service clusters, computing clusters, meteorological supercomputing architecture intelligent computing clusters and software stack support environment; The service cluster includes a meteorological artificial intelligence basic support technology platform, storage management and control, image warehouse and fusion scheduler; The meteorological artificial intelligence basic support technology platform is used to support user interaction and obtain user computing task requests; The computing cluster is created by scattered and polymorphic intelligent computing server resources, which is used to receive computing task requests from the service cluster; in the computing cluster, a Kubernetes computing cluster computing power aggregation application service is built; that is, a distributed training framework based on Horovod is built in the Kubernetes computing cluster image to support the computing power aggregation communication of multiple machines and multiple cards in the Kubernetes computing cluster, forming the ability to aggregate scattered computing power on multiple machines; The meteorological supercomputing architecture intelligent computing cluster includes a supercomputing architecture intelligent computing cluster login node, a supercomputing architecture intelligent computing cluster management node, a supercomputing architecture intelligent computing cluster computing node and a supercomputing architecture intelligent computing cluster high-performance distributed shared storage; On the login node of the supercomputing architecture intelligent computing cluster: deploy log collection service, log storage query service and proxy service; The log collection service is used to collect computing task logs. The log storage query service is used to store and provide query services for computing task logs collected by the log collection service. The proxy service is used to forward the user's computing task request to the supercomputing architecture intelligent computing cluster management node. Deploy SlurmRestd service and Slurmctld service on the management node of the supercomputing architecture intelligent computing cluster; SlurmRestd service provides interaction between the login node of the supercomputing architecture intelligent computing cluster and the management node of the supercomputing architecture intelligent computing cluster, accepts the request to create Slurm Job from the proxy service, and provides Slurm Rest API for creating, deleting, and querying computing tasks; Slurmctld service is used to monitor resources and jobs, perform user Token authentication, and propose to create Slurm Job to the computing node of the supercomputing architecture intelligent computing cluster; Build a computing task job script file on the high-performance distributed shared storage of the supercomputing architecture intelligent computing cluster, and run the computing task on N computing nodes of the supercomputing architecture intelligent computing cluster according to the number of nodes N requested by the computing task. The software stack supports the following environments: CUDA / Pytorch images adapted to the GPU accelerator cards of the meteorological supercomputing architecture intelligent computing cluster, DTK / Pytorch images adapted to the Haiguang DCU accelerator cards of the meteorological supercomputing architecture intelligent computing cluster, CUDA / Pytorch images adapted to the computing resources of the Kubernetes computing cluster, and encapsulated large-model training AI frameworks, acceleration frameworks, and meteorological application software environments adapted to the applications, forming a one-click out-of-the-box software stack support environment.

9. A multi-state heterogeneous intelligent computing resource fusion scheduling system compatible with a meteorological supercomputer architecture according to claim 8, characterized in that: Elastic supercomputing container services are deployed in the computing nodes of the supercomputing architecture intelligent computing cluster. The Sigularity distributed operating environment is built based on the shared storage of the computing nodes of the supercomputing architecture intelligent computing cluster to support distributed training communication and adapt to the non-super-privileged user access characteristics of the supercomputing architecture. The container is started based on the Sigularity distributed operating environment inheriting the user permissions initiated by the computing task. The user context remains unchanged and the user permissions are kept the same inside and outside the container. The construction of the elastic supercomputing container service supports the specific software environment runtime of the user's artificial intelligence computing task, builds the data environment set by the task, starts the computing job and schedules the computing node accelerator equipment on the computing nodes of the task-specific scale allocated by Slurm, and realizes cross-node communication access.

Images