Method, device and medium for automatic configuration of an accelerator card communication library

Abstract

The application relates to the computer technical field, in particular to an automatic configuration method and device of an acceleration card communication library, equipment and medium. The method comprises the following steps: determining a to-be-identified acceleration card; executing a preset hardware identification script by using an initialization container to identify the to-be-identified acceleration card, and obtaining an acceleration card type of the to-be-identified acceleration card; determining a target communication library matched with the acceleration card type, generating a communication configuration file based on the target communication library, determining configuration information based on a preset container group creation process, and injecting the initialization container and the communication configuration file based on the configuration information, so as to automatically configure the acceleration card communication library. Therefore, the problems of low configuration efficiency, errors and lack of fine communication control of heterogeneous acceleration card communication libraries in the Kubernetes environment in the prior art are solved, automatic identification, dynamic configuration and network connectivity optimization are realized, and the deployment efficiency and communication performance of a distributed computing task are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an automatic configuration method and device of an acceleration card communication library, equipment and medium. BACKGROUND

[0002] Under the background of increasingly complex artificial intelligence model training and inference tasks, efficient configuration and management of hardware acceleration cards such as GPUs (Graphics Processing Unit) have become a key link in distributed computing platforms. Especially in container orchestration environments such as Kubernetes (container orchestration platform), the configuration of communication libraries (such as NCCL (NVIDIA Collective Communications Library), RCCL (Radeon Collective Communications Library), etc.) for heterogeneous computing resources usually relies on manual arrangement or static script processing, which is not only inefficient, but also prone to cause training task failure or performance degradation due to configuration errors. With the increasing deployment of products from multiple acceleration card manufacturers (such as NVIDIA and AMD) in clusters, how to automatically identify the type of acceleration card on the host node and dynamically match the corresponding communication library configuration has become a problem that needs to be solved in actual deployment.

[0003] Traditional methods lack intelligent identification mechanism for acceleration card types and fail to fully utilize the dynamic injection capabilities of Kubernetes, making it difficult to meet the needs of high-performance computing tasks in flexibility, automation and maintainability. In addition, existing solutions usually ignore the fine control of network communication paths and communication configuration files, making it difficult to provide consistency and stability guarantees in real scenarios with multiple nodes, multiple tenants and high concurrency, which needs to be solved urgently. SUMMARY

[0004] The present application provides an automatic configuration method, device, equipment and medium of an acceleration card communication library to solve the problems of low efficiency, error-prone and lack of fine communication control of heterogeneous acceleration card communication library configuration in Kubernetes environment in the prior art, realizing automatic identification, dynamic configuration and network connectivity optimization, thereby improving the deployment efficiency and communication performance of distributed computing tasks.

[0005] To achieve the above-mentioned purpose, the first aspect embodiment of the present application provides an automatic configuration method of an acceleration card communication library, comprising the following steps:

[0006] In the case of the need for automatic configuration of the acceleration card communication library, the acceleration card to be identified is determined;

[0007] The initialization container is used to execute a preset hardware identification script to identify the to-be-identified accelerator card, and obtain an accelerator card type of the to-be-identified accelerator card.

[0008] A target communication library matched with the accelerator card type is determined, a communication configuration file is generated based on the target communication library, configuration information is determined based on a preset pod (a minimum scheduling unit in Kubernetes), and the initialization container and the communication configuration file are injected based on the configuration information, so as to automatically configure the accelerator card communication library.

[0009] By using the above technical means, the problems of low configuration efficiency, error-prone and lack of fine communication control of the heterogeneous accelerator card communication library in the Kubernetes environment in the prior art are solved, automatic identification, dynamic configuration and network connectivity optimization are realized, and therefore the deployment efficiency and communication performance of the distributed computing task are improved.

[0010] According to the automatic configuration method of the accelerator card communication library provided in the embodiments of the present application, the initialization container is used to execute a preset hardware identification script to identify the to-be-identified accelerator card, and obtain an accelerator card type of the to-be-identified accelerator card; a target communication library matched with the accelerator card type is determined, a communication configuration file is generated based on the target communication library, configuration information is determined based on a preset pod creation process, and the initialization container and the communication configuration file are injected based on the configuration information, so as to automatically configure the accelerator card communication library. Therefore, the problems of low configuration efficiency, error-prone and lack of fine communication control of the heterogeneous accelerator card communication library in the Kubernetes environment in the prior art are solved, automatic identification, dynamic configuration and network connectivity optimization are realized, and therefore the deployment efficiency and communication performance of the distributed computing task are improved.

[0011] To achieve the above object, the second aspect of the present application provides an automatic configuration device of an accelerator card communication library, comprising:

[0012] A determination module is configured to determine a to-be-identified accelerator card when an automatic configuration demand of the accelerator card communication library exists.

[0013] An identification module is configured to use an initialization container to execute a preset hardware identification script to identify the to-be-identified accelerator card, and obtain an accelerator card type of the to-be-identified accelerator card.

[0014] A configuration module is configured to determine a target communication library matched with the accelerator card type, generate a communication configuration file based on the target communication library, determine configuration information based on a preset pod creation process, and inject the initialization container and the communication configuration file based on the configuration information, so as to automatically configure the accelerator card communication library.

[0015] According to the automatic configuration device of the acceleration card communication library provided in the embodiments of the present application, a preset hardware identification script is executed by using an initialization container to identify a to-be-identified acceleration card, and an acceleration card type of the to-be-identified acceleration card is obtained; a target communication library matched with the acceleration card type is determined, and a communication configuration file is generated based on the target communication library; configuration information is determined based on a preset container group creation process, and the initialization container and the communication configuration file are injected based on the configuration information, so as to automatically configure the acceleration card communication library. In this way, the problems of low configuration efficiency, easy errors and lack of fine communication control of heterogeneous acceleration card communication library in the Kubernetes environment in the prior art are solved, automatic identification, dynamic configuration and network connectivity optimization are achieved, and therefore the deployment efficiency and communication performance of the distributed computing task are improved.

[0016] To achieve the above object, the third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the automatic configuration method of the acceleration card communication library as described in the above embodiments.

[0017] To achieve the above object, the fourth aspect of the present application provides a non-volatile computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the automatic configuration method of the acceleration card communication library as described in the above embodiments.

[0018] To achieve the above object, the fifth aspect of the present application provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the automatic configuration method of the acceleration card communication library as described in the above embodiments.

[0019] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 Flowchart of the automatic configuration method of the acceleration card communication library according to the embodiments of the present application;

[0022] Figure 2 Schematic diagram of the initialization container identifying the acceleration card type according to an embodiment of the present application;

[0023] Figure 3 A schematic diagram of initializing a container for a webhook configuration according to an embodiment of the present application;

[0024] Figure 4 A schematic diagram of RDMA (Remote Direct Memory Access) network test tuning according to an embodiment of the present application;

[0025] Figure 5 A block schematic diagram of an automated configuration of an accelerator communication library according to an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, any other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0028] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0029] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0030] In view of the low efficiency, error-prone and lack of fine communication control of heterogeneous accelerator communication library configuration in the Kubernetes environment mentioned in the above background art, therefore, there is an urgent need for a solution that supports heterogeneous GPU automatic identification, communication configuration automatic generation, and has high scalability and system integration capability, to improve the deployment efficiency and running performance of AI tasks under the containerized platform.

[0031] The application provides an automatic configuration method of an accelerator card communication library. The automatic configuration method combines an InitContainer to automatically detect a host GPU type, dynamically injects communication configuration based on a Mutating Admission Webhook, uses an NCCL / RCCL self-provided tool to perform network connectivity pre-detection, and feeds back test results for optimizing communication parameters, thereby forming a closed-loop mechanism with automatic identification, dynamic configuration, real-time detection and adaptive optimization capabilities, and significantly improving the deployment efficiency, communication performance and system robustness of distributed computing under a heterogeneous GPU cluster.

[0032] The application provides an automatic configuration method of an accelerator card communication library, a device, an apparatus and a medium.

[0033] Specifically, Figure 1 is a flowchart of an automatic configuration method of an accelerator card communication library according to an embodiment of the application.

[0034] As shown in Figure 1 , the automatic configuration method of the accelerator card communication library comprises the following steps:

[0035] S101, in the case of an accelerator card communication library automatic configuration requirement, determining a to-be-identified accelerator card.

[0036] Specifically, the application embodiment can determine whether there is an accelerator card communication library automatic configuration requirement by checking the running environment of the container, the hardware configuration of the host node and the configuration file of the container, and if there is an accelerator card communication library automatic configuration requirement, determine the to-be-identified accelerator card.

[0037] S102, using an initialization container to execute a preset hardware identification script to identify the to-be-identified accelerator card, and obtaining an accelerator card type of the to-be-identified accelerator card.

[0038] Further, in some embodiments, using the initialization container to execute the preset hardware identification script to identify the to-be-identified accelerator card and obtain the accelerator card type of the to-be-identified accelerator card comprises: using the initialization container to execute the preset hardware identification script to obtain kernel module information and peripheral component interconnect (PCI) device information of the target host node; and obtaining the accelerator card type of the to-be-identified accelerator card based on the kernel module information and the peripheral component interconnect device information.

[0039] Specifically, the embodiment of the present application uses InitContainer (initialization container) to automatically execute the hardware identification script during the container startup phase, and parses the kernel module information and PCI device information of the host node by executing the lsmod and lspci commands to identify the GPU type of the host machine (such as NVIDIA or AMD / ROCm).

[0040] Furthermore, an InitContainer is injected into the target container group in the Mutating Admission Webhook.

[0041] The InitContainer image must include tools such as bash, lsmod, lspci, and jq, and must mount necessary directories such as / proc, / sys, / dev, and / etc as read-only.

[0042] For example, the script logic of the embodiment of the present application may be as follows:

[0043] a) Check kernel modules:

[0044] - lsmod | grep nvidia to check NVIDIA;

[0045] - lsmod | grep amdgpu or lsmod | grep roc to check AMD / ROCm;

[0046] b) Check peripheral interconnection devices:

[0047] - lspci | grep -i nvidia;

[0048] - lspci | grep -i amd.

[0049] Furthermore, in some embodiments, after using the initialization container to execute a preset hardware identification script to obtain the kernel module information and peripheral interconnection device information of the target host node, it also includes: outputting the kernel module information and peripheral interconnection device information as structured data in a preset data exchange format; and writing the structured data in the preset data exchange format to a shared mounted volume.

[0050] Among them, the preset data exchange format of the embodiment of the present application can be JSON format.

[0051] Specifically, after the preset hardware identification script is executed by using the initialized container to obtain the kernel module information and the peripheral interconnection device information of the target host node, the kernel module information and the peripheral interconnection device information are combined, and a JSON structure is output, such as: {"type": "nvidia", "model": "Tesla V100", "count": 4}, and the result is written into a shared mounting volume, such as / etc / accelerator-info / accelerator.json.

[0052] Further, in some embodiments, after obtaining the accelerator card type of the to-be-identified accelerator card based on the kernel module information and the peripheral interconnection device information, the method further includes: in a case where the accelerator card type of the to-be-identified accelerator card is not identified, writing a record identifying a missing device, and returning a non-zero exit code to terminate execution.

[0053] Specifically, when the type of the graphics processing unit (GPU) carried by the host computer cannot be detected, a record identifying a missing device is written in the script, such as {"type": "none"}, and a non-zero exit code is returned to terminate.

[0054] Further, after the record identifying the missing device is written, the system returns a non-zero exit code. In a computer system, an exit code is a status code used to represent the result of program execution. Generally, an exit code of 0 indicates that the program is executed normally and no error is encountered, and a non-zero exit code indicates that the program encounters an error or an abnormal situation during execution. By returning a non-zero exit code, the caller (such as a container orchestration system or an automated script) can be explicitly informed that the current execution flow cannot continue due to the failure to identify the type of the graphics processing unit, preventing misuse of the main container. In addition, the detection process and results can be recorded in detail through the log, facilitating troubleshooting.

[0055] To facilitate those skilled in the art to more clearly and intuitively understand the process of automatically identifying the type of the accelerator card in the Kubernetes environment according to the embodiments of the present application, the following will be described in detail in combination with Figure 2 .

[0056] First, the container group using NVIDIA accelerator cards is introduced. In the Pod startup phase, the initial container (InitContainer) shares the same resource specification and mounted volume with the main container, runs the device detection engine, and identifies the GPU type as nvidia by querying the kernel module and PCI information. The identification result triggers the network configuration engine to generate the RDMA network configuration file nccl.conf adapted to NVIDIA GPU, which contains key parameters such as RDMA card binding and topology. Subsequently, the main container loads this configuration file and realizes efficient communication based on NCCL with the help of NVIDIA driver.

[0057] Second, the container group using AMD accelerator cards is introduced, which is similar to the above container group using NVIDIA accelerator cards. The initial container identifies the GPU type as "amd", and the network configuration engine generates rccl.conf accordingly. Finally, the main container completes RCCL communication initialization in cooperation with the AMD ROCm driver based on the configuration.

[0058] S103, determining a target communication library matched with the accelerator card type, and generating a communication configuration file based on the target communication library, determining configuration information based on a preset container group creation process, and injecting an initialization container and the communication configuration file based on the configuration information to automatically configure the accelerator card communication library.

[0059] In some embodiments, the target communication library is at least one of a first preset type collective communication library and a second preset type collective communication library.

[0060] Optionally, the first preset type collective communication library can be at least one of an NCCL collective communication library (NVIDIA collective communication library) and an RCCL collective communication library (ROCm collective communication library).

[0061] Further, in some embodiments, determining a target communication library matched with the accelerator card type includes: if the accelerator card type is a first preset type accelerator card, determining the target communication library as the first preset type collective communication library; if the accelerator card type is a second preset type accelerator card, determining the target communication library as the second preset type collective communication library; and if the accelerator card type is none, triggering a preset error handling mechanism or enabling a degraded communication strategy.

[0062] For example, if the accelerator card type uses an NCCL collective communication library, the target communication library is determined as the NCCL collective communication library; if the accelerator card type uses an RCCL collective communication library, the target communication library is determined as the RCCL collective communication library; and if the accelerator card type is none, a preset error handling mechanism is triggered or a degraded communication strategy is enabled.

[0063] Specifically, a corresponding communication library is automatically selected according to the identified accelerator type (for example, NCCL corresponds to NVIDIA, and RCCL corresponds to AMD).

[0064] In the main container startup script or entrypoint, / etc / accelerator-info / accelerator.json is first read. By parsing the JSON, the type field is determined:

[0065] If the type field is nvidia, the environment variable COMM_LIB=NCCL is set.

[0066] If the type field is amd, the environment variable COMM_LIB=RCCL is set.

[0067] If the type field is none, an error is reported and the program exits or is downgraded.

[0068] Further, according to COMM_LIB, the target communication library path is automatically exported:

[0069] export LD_LIBRARY_PATH= / opt / nccl / lib:$LD_LIBRARY_PATH.

[0070] Or export LD_LIBRARY_PATH= / opt / rccl / lib:$LD_LIBRARY_PATH.

[0071] It should be noted that if the image does not have a built-in communication library, it can be provided by mounting a HostPath or using a sidecar container. If multiple communication libraries need to coexist, the script needs to support priority configuration, such as controlling through the container group annotation comm-lib-prefer: nccl, and writing the selection result to the log for subsequent tracing.

[0072] Further, in some embodiments, a communication configuration file is generated based on the target communication library, including: detecting remote direct memory access (RDMA) device information of a host node, and obtaining a cluster node topology structure; and generating a communication configuration file according to the RDMA device information and the cluster node topology structure.

[0073] In some embodiments, the communication configuration file includes at least one of RDMA card information, inter-node communication topology structure, node ordering, and RDMA card interface.

[0074] Specifically, the configuration file is generated according to the communication library and node information, and the content includes RDMA network card information, inter-node communication topology, related parameters, etc. The communication configuration file is mounted to the main container specified path through ConfigMap, Secret or shared volume.

[0075] Further, in the InitContainer or main container startup script, the following operations are performed:

[0076] a) Detect the local RDMA device:

[0077] - ibdev2netdev gets the IB device name;

[0078] - lspci | grep -i mellanox gets the network card model.

[0079] b) Get node information:

[0080] - Inject the node name and IP through the Downward API;

[0081] - Query the IP and hostname of all container groups under the Job / Container Group Group through the Kubernetes API.

[0082] c) Generate communication topology:

[0083] - Sort by node IP / hostname to generate ring / tree / mesh topology.

[0084] d) Generate configuration file content.

[0085] Further, the following takes NCCL as an example to show the process of generating configuration file content.

[0086] # Enable debug log

[0087] NCCL_DEBUG=INFO;

[0088] NCCL_DEBUG_SUBSYS=INIT,NET,IB;

[0089] # Enable IB

[0090] NCCL_IB_DISABLE=0;

[0091] # RoCE usually uses GID Index 3;

[0092] NCCL_IB_GID_INDEX=3;

[0093] QoS related tuning

[0094] NCCL_IB_SL=5;

[0095] NCCL_IB_TC=136;

[0096] # Network card name (check with ibv_devinfo)

[0097] NCCL_IB_HCA=mlx5_0;

[0098] # Enable GPUDirect RDMA

[0099] NCCL_IB_CUDA_SUPPORT=1;

[0100] NCCL_IB_GDR_LEVEL=SYS;

[0101] # Optimize Queue Pair concurrency

[0102] NCCL_IB_QPS_PER_CONNECTION=4;

[0103] e) If RCCL, generate corresponding parameters

[0104] f) Write the configuration file to / etc / nccl.conf or / etc / rccl.conf.

[0105] In addition, if the node information acquisition fails, it needs to be retried and the log is recorded, and if it times out, it terminates the startup. The configuration file permission needs to be 0644 to ensure that the main container can be read.

[0106] Thus, the Mutating Admission Webhook identifies the tenant to which the container belongs according to the label, annotation or namespace in the container group, and dynamically generates an isolated communication configuration file for each tenant.

[0107] Further, in some embodiments, the configuration information is determined based on a preset container group creation process, and the initialization container and the communication configuration file are injected based on the configuration information, to automatically configure the accelerator communication library, including: utilizing a Mutating Admission Webhook (Mutating Admission Webhook) to listen to a container group creation request; screening a target container group that needs to be injected according to a container group annotation or label; determining configuration information of the target container group, and injecting an initialization container and a communication configuration file into the configuration of the target container group based on the configuration information.

[0108] In some embodiments, the communication configuration file adopts a data serialization format (such as YAML format) and contains configuration information such as communication group topology, node ordering, RDMA network card interface, etc. The communication configuration file is mounted to a preset designated path of the host container through configuration mapping, keys or shared volumes.

[0109] Specifically, as shown in Figure 3 Fig. 1, an InitContainer and a communication configuration file are automatically injected during the container group creation process using the Mutating Admission Webhook of Kubernetes, to realize transparent deployment.

[0110] Further, the Mutating Admission Webhook is deployed and registered to the Kubernetes API Server. The Webhook listens to the container group creation and screens the targets to be injected (such as container groups with the comm-inject: "true" annotation).

[0111] Further, the Webhook automatically modifies the container group Spec: first, an InitContainer for detecting the acceleration card is injected, second, a mounting volume (such as emptyDir or PVC) of the communication configuration file is injected, and a communication library and configuration related environment variable (such as COMM_LIB and COMM_CONFIG_PATH) are injected. If necessary, a sidecar container is injected to provide the communication library.

[0112] Further, the Webhook returns the modified container group Spec. The user does not need to be aware. The Webhook needs to give a detailed AdmissionReview response and log for abnormal situations (such as injection failure and field conflict). The embodiments of the present application can also configure the injection strategy through CRD (Custom Resource Definition, custom resource definition) to realize flexible management.

[0113] Further, in some embodiments, after injecting the initialization container and the communication configuration file based on the configuration information, the method further includes: calling a test tool of a target communication library to pre-detect the network connectivity between nodes, obtaining communication performance data, and obtaining a communication test result according to the communication performance data; and feeding back the communication test result to the configuration generation logic to dynamically adjust the target parameters in the communication configuration file according to the communication test result.

[0114] In some embodiments, the target parameters include at least one of communication topology mapping, port selection, link binding strategy and communication channel quantity.

[0115] Further, in some embodiments, the test tool invoking the target communication library pre-detects the inter-node network connectivity, including: reading the hostnames of all nodes and performing connectivity tests on each node; testing the bandwidth and delay between nodes based on a first preset communication performance test tool or a second preset communication performance test tool; verifying the connectivity of the remote direct memory access port based on a preset hardware information query command or a preset network connectivity test command.

[0116] For example, the hostnames of all nodes are read and ping tests are performed on each node; the bandwidth and delay indicators between pairs of nodes are measured based on the nccl-test tool or the rccl-test tool; the connectivity of the RDMA port is verified based on the ibstat command or the ibping command.

[0117] Specifically, the nccl-tests or rccl-tests tool package is integrated in the InitContainer.

[0118] The script flow for pre-detecting the inter-node network connectivity includes:

[0119] a) Read the IP / hostnames of all nodes, perform ping tests one by one, and record the packet loss and delay;

[0120] b) Use the all_reduce_perf of the nccl-tests to start in parallel among multiple nodes to test the bandwidth and delay;

[0121] c) If it is RCCL, use the rccl-tests to test the bandwidth and delay between nodes;

[0122] d) Check the connectivity of the RDMA port (such as ibstat, ibping).

[0123] Further, the detection results are formatted into JSON and written into the / etc / comm-test / result.json file.

[0124] In this way, by running the nccl-test or rccl-test tool before starting the container, the network connectivity, bandwidth, delay, and other indicators between nodes are checked, so that communication bottlenecks or failures can be discovered in advance.

[0125] Further, in some embodiments, after obtaining the communication test results according to the communication performance data, further including: if the communication test results do not satisfy a preset passing condition, terminating the starting of the container group, and / or, after automatically degrading the communication parameters, re-invoking the test tool of the target communication library to pre-detect the inter-node network connectivity.

[0126] Specifically, if the communication test result does not meet the preset passing condition, it means that the detection fails, and the script should output detailed logs and can be configured to directly terminate the container group startup or retry after degrading the parameters, that is, the node pair with a communication test result lower than the preset threshold will be marked as unavailable, and the system will automatically exclude or degrade the related link from the communication configuration. The detection log and result file need to be persisted for subsequent analysis.

[0127] Further, in some embodiments, the target parameters in the communication configuration file are dynamically adjusted according to the communication test result, including: analyzing the bandwidth index, delay index and packet loss rate index in the communication test result; if any of the bandwidth index, delay index and packet loss rate index is not in the corresponding preset range, adjust the communication topology mapping and / or link binding strategy; regenerate a new communication configuration file and remount the new communication configuration file to the container.

[0128] Specifically, the RDMA tuning engine periodically reads / etc / comm-test / result.json and analyzes the bandwidth, delay, packet loss and other indicators to determine whether the current configuration is optimal. If a bottleneck or anomaly is found, the configuration parameters are automatically adjusted, for example:

[0129] a) Enable GPUDirectRDMA and optimize NUMA (Non-Uniform Memory Access) topology (such as NCCL_IB_CUDA_SUPPORT=1);

[0130] b) Set NCCL network interface reasonably (ensure that fallback uses high-performance interface, not the default eth0);

[0131] c) Control channel concurrency and topology (NCCL_MAX_NRINGS=8).

[0132] Further, after automatically adjusting the configuration parameters, a new communication configuration file is generated, hot updated to the mounted volume, and the steps of analyzing the bandwidth, delay, packet loss and other indicators are repeated based on the preset number of executions to the step of regenerating the communication configuration file, hot updating to the mounted volume, finding the optimal configuration, and fixing the found optimal configuration to the communication configuration file nccl.conf or rccl.conf.

[0133] Thus, by feeding back the test data to the configuration generation logic, the communication parameters (such as link binding, port selection, topology structure) are dynamically adjusted, improving the robustness and overall communication performance in the communication initialization stage.

[0134] It should be noted that if the configuration fails, an alarm needs to be automatically reported (such as writing K8s Event, Prometheus index, email notification, etc.), and all optimization operations need to be recorded in detail to change history, so as to trace and roll back.

[0135] In order to make the RDMA network test and optimization process of the embodiments of the present application more clear and more intuitive for those skilled in the art, the following will be described in detail in combination with Figure 4 .

[0136] Specifically, as shown in Figure 4 , the upper layer is the RDMA network detection and optimization stage. All container groups run the initial container when starting. The nccl-test or rccl-test is executed in the initial container, the bandwidth, delay and other indicators of the inter-node RDMA network are measured, and the results are reported to the RDMA optimization engine. The optimization engine dynamically updates the network parameters in nccl.conf (NVIDIA scenario) or rccl.conf (AMD scenario) according to these test data, thereby completing a closed loop of "detection-feedback-optimization". The initial container exits immediately after running, leaving only the optimized configuration file.

[0137] The lower layer is the main container main business running stage. When the initial container exits, the main container starts to read the latest nccl.conf / rccl.conf written by the optimization engine from the same mounted volume, initializes the NCCL / RCCL communication environment according to the configuration file, and then enters the normal distributed training or inference task, and communicates efficiently through the optimized RDMA network, which ensures that the network configuration is always based on real detection results, and the optimization is completed before the main business starts, realizing the dual improvement of performance and stability.

[0138] The application provides a Kubernetes environment-oriented automatic acceleration card identification and communication library configuration method, aiming to simplify the system configuration process in the heterogeneous GPU deployment process and significantly improve the execution efficiency of distributed computing tasks. The method combines the InitContainer mechanism, Mutating Admission Webhook component, and node-level hardware detection script to build an intelligent closed-loop process that can automatically identify acceleration card types, dynamically generate and inject communication configurations. In actual implementation, the scheme automatically runs hardware identification scripts at the container startup stage using InitContainer, accurately determines the GPU type (such as NVIDIA or AMD / ROCm) carried by the host node by parsing kernel modules (such as lsmod) and PCI device information (such as lspci). After identification, the system will automatically select a matching communication library (such as NCCL or RCCL) according to the acceleration card type, and generate a communication configuration file based on the preset logic, which covers RDMA card binding information and inter-node communication topology structure. Subsequently, with the help of Mutating Admission Webhook, InitContainer and its required configurations are automatically injected into the container group creation process, realizing transparent deployment of the communication environment without user intervention. To further ensure communication stability and deployment success rate, the scheme integrates the test tools (such as nccl-test or rccl-test) provided by the communication library, and performs pre-detection on the network connectivity between cluster nodes before container deployment. By running these test programs, the system can identify potential communication failures or bandwidth bottlenecks in time, avoiding distributed task failures caused by node abnormalities. By running the test tools (such as nccl-test or rccl-test) of the communication library at the container group startup stage, network connectivity, bandwidth, and delay performance data are collected and fed back to the configuration generation logic to dynamically modify key parameters (such as topology mapping, port selection, link binding, etc.) in the communication configuration file, ultimately optimizing the communication initialization process and improving robustness.

[0139] Therefore, the application automatically identifies acceleration card types, intelligently selects communication libraries, and dynamically generates communication configuration files in the Kubernetes environment, and combines communication library test tools for network connectivity pre-detection and feedback optimization, realizing full-process automation and high-reliability deployment of the communication environment. The scheme effectively reduces the labor cost and error risk of heterogeneous GPU system configuration, improves the communication efficiency and system robustness of distributed training tasks, and is particularly suitable for multi-node, large-scale, high-performance AI computing cluster deployment, with significant practical value and promotion prospects.

[0140] According to the automatic configuration method of the accelerator communication library proposed in the embodiment of the present application, the preset hardware identification script is executed by using the initialization container to identify the to-be-identified accelerator card, and the accelerator card type of the to-be-identified accelerator card is obtained; the target communication library matched with the accelerator card type is determined, and the communication configuration file is generated based on the target communication library; the configuration information is determined based on the preset container group creation process, and the initialization container and the communication configuration file are injected based on the configuration information, so as to automatically configure the accelerator communication library. In this way, the problems of low configuration efficiency, easy error and lack of fine communication control of the heterogeneous accelerator communication library in the Kubernetes environment in the prior art are solved, the performance and robustness of the communication initialization process are optimized, and thus the deployment efficiency and communication performance of the distributed computing task are improved.

[0141] Secondly, the automatic configuration device of the accelerator communication library proposed in the embodiment of the present application is described with reference to the accompanying drawings.

[0142] Figure 5 is a block schematic diagram of the automatic configuration device of the accelerator communication library in the embodiment of the present application.

[0143] As shown in Figure 5 , the automatic configuration device 10 of the accelerator communication library includes a determination module 100, an identification module 200 and a configuration module 300.

[0144] The determination module 100 is configured to determine the to-be-identified accelerator card in the case that the accelerator communication library automatic configuration demand exists; the identification module 200 is configured to execute the preset hardware identification script by using the initialization container to identify the to-be-identified accelerator card, and obtain the accelerator card type of the to-be-identified accelerator card; and the configuration module 300 is configured to determine the target communication library matched with the accelerator card type, generate the communication configuration file based on the target communication library, determine the configuration information based on the preset container group creation process, and inject the initialization container and the communication configuration file based on the configuration information, so as to automatically configure the accelerator communication library.

[0145] Further, in some embodiments, after the initialization container and the communication configuration file are injected based on the configuration information, the configuration module 300 is further configured to: call the test tool of the target communication library to pre-detect the network connectivity between nodes, obtain the communication performance data, and obtain the communication test result according to the communication performance data; feed back the communication test result to the configuration generation logic, and dynamically adjust the target parameter in the communication configuration file according to the communication test result.

[0146] Furthermore, in some embodiments, the identification module 200 is used to: use the initialization container to execute a preset hardware identification script to obtain the kernel module information and peripheral interconnection device information of the target host node; based on the kernel module information and peripheral interconnection device information, determine whether the type of graphics processor installed in the host machine is identified; if the type of graphics processor is identified, determine the type of accelerator card according to the type of graphics processor.

[0147] Furthermore, in some embodiments, after using the initialization container to execute a preset hardware identification script to obtain the kernel module information and peripheral interconnection device information of the target host node, the identification module 200 is also used to: output the kernel module information and peripheral interconnection device information as structured data in a preset data exchange format; and write the structured data in the preset data exchange format to a shared mounted volume.

[0148] Furthermore, in some embodiments, after determining whether the type of the graphics processor installed in the host machine is identified based on the kernel module information and the peripheral interconnection device information, the identification module 200 is also used to: if the type of the graphics processor is not identified, write a record indicating that the device is missing and return a non-zero exit code to terminate execution.

[0149] Further, in some embodiments, the target communication library is at least one of the NVIDIA collective communication library and the ROCm collective communication library.

[0150] Furthermore, in some embodiments, the configuration module 300 is used to: if the accelerator card type is NVIDIA, determine that the target communication library is the NVIDIA collective communication library; if the accelerator card type is AMD, determine that the target communication library is the ROCm collective communication library; if the accelerator card type is none, trigger the preset error handling mechanism or enable the degraded communication strategy.

[0151] Furthermore, in some embodiments, the configuration module 300 is configured to: detect the RDMA device information of the host node and obtain the cluster node topology; and generate a communication configuration file according to the RDMA device information and the cluster node topology.

[0152] Furthermore, in some embodiments, the configuration module 300 is configured to: monitor container group creation requests using a mutating admission webhook; filter target container groups to be injected based on container group annotations or tags; determine target container group configuration information, and inject the initialization container and communication configuration file into the target container group configuration based on the configuration information.

[0153] Further, in some embodiments, the configuration module 300 is configured to read the hostnames of all nodes and perform a ping test on each node, test the bandwidth and latency between nodes based on an nccl-test tool or an rccl-test tool, and verify the connectivity of the RDMA port based on an ibstat command or an ibping command.

[0154] Further, in some embodiments, after obtaining the communication test result according to the communication performance data, the configuration module 300 is further configured to terminate the container group startup if the communication test result does not satisfy a preset passing condition, and / or re-call a test tool of the target communication library to pre-detect the network connectivity between nodes after automatically downgrading the communication parameters.

[0155] Further, in some embodiments, the configuration module 300 is configured to analyze the bandwidth indicators, the latency indicators, and the packet loss rate indicators in the communication test result, adjust the communication topology mapping and / or the link binding strategy if any of the bandwidth indicators, the latency indicators, and the packet loss rate indicators is not within a corresponding preset range, regenerate a new communication configuration file, and remount the new communication configuration file to the container.

[0156] Further, in some embodiments, the communication configuration file is mounted to a preset designated path of the host container through configuration mapping, a key, or a shared volume.

[0157] Further, in some embodiments, the communication configuration file adopts a data serialization format.

[0158] Further, in some embodiments, the communication configuration file includes at least one of RDMA network card information, an inter-node communication topology structure, node ordering, and an RDMA network card interface.

[0159] Further, in some embodiments, the target parameters include at least one of a communication topology mapping, a port configuration, a link binding strategy, and a communication channel quantity.

[0160] It should be noted that the foregoing explanation and description of the embodiment of the automatic configuration method of the accelerator communication library also apply to the embodiment of the automatic configuration device of the accelerator communication library, which will not be described here again.

[0161] According to the automatic configuration device of the acceleration card communication library provided in the embodiment of the present application, according to the automatic configuration method of the acceleration card communication library provided in the embodiment of the present application, a preset hardware identification script is executed by using an initialization container to identify a to-be-identified acceleration card, and an acceleration card type of the to-be-identified acceleration card is obtained; a target communication library matched with the acceleration card type is determined, a communication configuration file is generated based on the target communication library, configuration information is determined based on a preset container group creation process, and the initialization container and the communication configuration file are injected based on the configuration information, so as to automatically configure the acceleration card communication library. Therefore, the problems of low configuration efficiency, easy error and lack of fine communication control of the heterogeneous acceleration card communication library in the Kubernetes environment in the prior art are solved, automatic identification, dynamic configuration and network connectivity optimization are realized, and therefore the deployment efficiency and communication performance of the distributed computing task are improved.

[0162] Figure 6 The structure schematic diagram of the electronic device provided in the embodiment of the present application is provided. The electronic device can include:

[0163] The memory 601, the processor 602 and the computer program stored in the memory 601 and executable on the processor 602.

[0164] The processor 602 implements the timing management method provided in the above embodiment when executing the program.

[0165] Further, the electronic device further includes:

[0166] The communication interface 603 is used for communication between the memory 601 and the processor 602.

[0167] The memory 601 is used for storing the computer program executable on the processor 602.

[0168] The memory 601 can include a high-speed RAM (Random Access Memory, random access memory) memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0169] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected with each other through a bus and complete communication between each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 6 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0170] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can complete communication between each other through an internal interface.

[0171] The processor 602 can be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0172] The embodiment of the present application further provides a non-volatile computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the timing management method.

[0173] The embodiment of the present application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the timing management method.

[0174] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0175] The above describes in detail an integrated circuit board, a timing management method, a server, a device, a medium and a product provided by the present application. The principles and implementation modes of the present application are described by applying specific examples, and the above description of the examples is only used to help understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for automated configuration of an accelerator card communication library, the method comprising: The method comprises the following steps: In the case of the need for automatic configuration of the accelerator card communication library, determining an accelerator card to be identified; Executing a preset hardware identification script by using an initialization container to identify the accelerator card to be identified, and obtaining an accelerator card type of the accelerator card to be identified; Determining a target communication library matched with the accelerator card type, generating a communication configuration file based on the target communication library, determining configuration information based on a preset container group creation process, and injecting the initialization container and the communication configuration file based on the configuration information to automatically configure the accelerator card communication library; The method further comprises the following steps after injecting the initialization container and the communication configuration file based on the configuration information:

2. The method of claim 1, wherein, Calling a test tool of the target communication library to pre-detect network connectivity between nodes, obtaining communication performance data, and obtaining a communication test result based on the communication performance data; Feeding back the communication test result to a configuration generation logic, and dynamically adjusting target parameters in the communication configuration file based on the communication test result. The method further comprises the following steps of executing the preset hardware identification script by using the initialization container to obtain kernel module information and peripheral interconnection device information of a target host node:

3. The method according to claim 1, characterized in that Obtaining the accelerator card type of the accelerator card to be identified based on the kernel module information and the peripheral interconnection device information. The method further comprises the following steps after obtaining the kernel module information and the peripheral interconnection device information of the target host node by executing the preset hardware identification script by using the initialization container: Outputting the kernel module information and the peripheral interconnection device information as structured data in a preset data exchange format; 4. The method of claim 3, wherein, Writing the structured data in the preset data exchange format into a shared mounting volume. The method further comprises the following steps of obtaining the accelerator card type of the accelerator card to be identified based on the kernel module information and the peripheral interconnection device information: In the case where the accelerator card type of the accelerator card to be identified is not identified, writing a record indicating that a device is missing, and returning a non-zero exit code to terminate execution.

5. The method of claim 3, wherein, The target communication library is at least one of a first preset type collective communication library and a second preset type collective communication library. The method further comprises the following steps of determining the target communication library matched with the accelerator card type:

6. The method of claim 1, wherein, If the accelerator card type is a first preset type accelerator card, determining that the target communication library is the first preset type collective communication library; 7. The method of claim 6, wherein, If the accelerator card type is a second preset type accelerator card, determining that the target communication library is the second preset type collective communication library; ​ ​ If the accelerator card type is empty, a preset error handling mechanism is triggered or a degraded communication strategy is enabled.

8. The method of claim 1, wherein, The communication configuration file is generated based on the target communication library, including: Detecting remote direct memory access device information of the host node and obtaining cluster node topology structure; According to the remote direct memory access device information and the cluster node topology structure, the communication configuration file is generated.

9. The method of claim 2, wherein, The test tool of the target communication library is called to pre-detect network connectivity between nodes, including: Reading the hostnames of all nodes and performing connectivity testing on each node; Testing the bandwidth and delay between nodes based on a first preset communication performance test tool or a second preset communication performance test tool; Verifying the connectivity of the remote direct memory access port based on a preset hardware information query command or a preset network connectivity test command.

10. The method of claim 2, wherein, After obtaining the communication test result according to the communication performance data, the method further includes: If the communication test result does not meet a preset passing condition, the container group startup is terminated, and / or, after automatically degrading the communication parameters, the test tool of the target communication library is called again to pre-detect network connectivity between nodes.

11. The method of claim 2, wherein, The target parameters in the communication configuration file are dynamically adjusted according to the communication test result, including: Analyzing the bandwidth index, delay index and packet loss rate index in the communication test result; If any of the bandwidth index, delay index and packet loss rate index is not in the corresponding preset range, adjusting the communication topology mapping and / or link binding strategy; A new communication configuration file is regenerated, and the new communication configuration file is remounted to the container.

12. The method of claim 1, wherein, The communication configuration file is mounted to a preset designated path of the host container through configuration mapping, keys or shared volumes.

13. The method of claim 1, wherein, The communication configuration file adopts a data serialization format.

14. The method of claim 1, wherein, The communication configuration file includes at least one of remote direct memory access network card information, inter-node communication topology structure, node ordering and remote direct memory access network card interface.

15. The method of claim 2, wherein, The target parameters include at least one of communication topology mapping, port configuration, link binding strategy and communication channel number.

16. An apparatus for automated configuration of an accelerator card communication library, the apparatus comprising: The method includes the following steps: A determination module is configured to determine a to-be-identified accelerator card when there is a need for automatic configuration of an accelerator card communication library; An identification module is configured to execute a preset hardware identification script using an initialization container to identify the to-be-identified accelerator card, and obtain an accelerator card type of the to-be-identified accelerator card; A configuration module is configured to determine a target communication library matched with the accelerator card type, generate a communication configuration file based on the target communication library, determine configuration information based on a preset container group creation process, and inject an initialization container and a communication configuration file based on the configuration information to automatically configure the accelerator card communication library. The configuration module is configured to listen to a container group creation request using a variable admission control hook, filter a target container group that needs to be injected according to a container group annotation or a label, determine configuration information of the target container group, and inject the initialization container and the communication configuration file into the target container group based on the configuration information to automatically configure the accelerator card communication library.

17. An electronic device, comprising: The method includes: A memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the computer program to implement the method of claim 1-15 for automated configuration of an acceleration card communication library.

18. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor for implementing the method of claim 1-15 for automated configuration of an acceleration card communication library.

19. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method of claim 1-15 for automated configuration of an acceleration card communication library.

Images