Automatic configuration method and device of accelerator card communication library, equipment and medium

Abstract

The invention relates to the technical field of computers, in particular to an automatic configuration method and device for an accelerator card communication library, equipment and a medium. Executing a preset hardware identification script by using the initialization container, and identifying a to-be-identified accelerator card to obtain an accelerator card type of the to-be-identified accelerator card; and determining a target communication library matched with the accelerator card type, generating a communication configuration file based on the target communication library, establishing a process based on a preset container group to determine configuration information, and injecting the initialization container and the communication configuration file based on the configuration information so as to automatically configure the accelerator card communication library. Therefore, the problems that in the prior art, in the Kubernetes environment, the heterogeneous accelerator card communication library configuration efficiency is low, errors are prone to occurring, and fine communication control is lacked are solved, automatic identification, dynamic configuration and network connectivity optimization are achieved, and therefore the deployment efficiency and communication performance of distributed computing tasks are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for automatically configuring an accelerator card communication library. Background Art

[0002] Given the increasing complexity of AI model training and inference tasks, efficient configuration and management of hardware accelerators such as GPUs (Graphics Processing Units) have become critical for distributed computing platforms. In particular, in container orchestration environments like Kubernetes, the configuration of communication libraries for heterogeneous computing resources, such as NCCL (NVIDIA Collective Communications Library) and RCCL (Radeon Collective Communications Library), often relies on manual orchestration or static scripting. This is not only inefficient but also prone to training failures or performance degradation due to configuration errors. With the increasing prevalence of mixed deployments of accelerator card products from multiple vendors (such as NVIDIA and AMD) in clusters, the ability to automatically identify the accelerator card type on the host node and dynamically match the corresponding communication library configuration has become a pressing issue in practical deployments.

[0003] Traditional approaches lack intelligent recognition of accelerator card types and fail to fully leverage Kubernetes' dynamic injection capabilities, making them difficult to meet the flexibility, automation, and maintainability requirements of high-performance computing tasks. Furthermore, existing solutions often neglect the fine-grained control of network communication paths and communication profiles, making it difficult to ensure consistency and stability in multi-node, multi-tenant, and high-concurrency scenarios. This urgently needs to be addressed. Summary of the Invention

[0004] The present application provides a method, apparatus, device, and medium for automatically configuring an accelerator card communication library to address the existing problems of inefficient, error-prone, and lack of fine-grained communication control in heterogeneous accelerator card communication library configuration in a Kubernetes environment. This method implements automated identification, dynamic configuration, and network connectivity optimization, thereby improving the deployment efficiency and communication performance of distributed computing tasks.

[0005] To achieve the above objectives, a first embodiment of the present application provides an automated configuration method for an accelerator card communication library, comprising the following steps: If there is a need for automatic configuration of the accelerator card communication library, determine the accelerator card to be identified; Utilizing the initialization container to execute a preset hardware identification script to identify the accelerator card to be identified, and obtaining the accelerator card type of the accelerator card to be identified; Determine the target communication library that matches the accelerator card type, generate a communication configuration file based on the target communication library, determine the configuration information based on the preset container group (pod, the smallest scheduling unit in Kubernetes) creation process, and inject the initialization container and communication configuration file based on the configuration information to automatically configure the accelerator card communication library.

[0006] Through the above technical means, the problems of inefficient and error-prone configuration of heterogeneous accelerator card communication libraries in the Kubernetes environment in the existing technology are solved, and automatic identification, dynamic configuration and network connectivity optimization are achieved, thereby improving the deployment efficiency and communication performance of distributed computing tasks.

[0007] According to the automated configuration method for the accelerator card communication library proposed in the embodiment of the present application, the preset hardware identification script is executed by using the initialization container to identify the accelerator card to be identified and obtain the accelerator card type of the accelerator card to be identified; the target communication library that matches the accelerator card type is determined, and a communication configuration file is generated based on the target communication library. The configuration information is determined based on the preset container group creation process, and the initialization container and communication configuration file are injected based on the configuration information to automatically configure the accelerator card communication library. This solves the problems of low efficiency, easy errors and lack of fine communication control in the configuration of heterogeneous accelerator card communication libraries in the Kubernetes environment in the prior art, realizes automated identification, dynamic configuration and network connectivity optimization, thereby improving the deployment efficiency and communication performance of distributed computing tasks.

[0008] To achieve the above objectives, a second embodiment of the present application provides an automatic configuration device for an accelerator card communication library, comprising: A determination module, used to determine the accelerator card to be identified when there is a demand for automatic configuration of the accelerator card communication library; an identification module, configured to execute a preset hardware identification script using an initialization container to identify the accelerator card to be identified and obtain the accelerator card type of the accelerator card to be identified; A configuration module is used to determine a target communication library that matches the accelerator card type, generate a communication configuration file based on the target communication library, determine configuration information based on a preset container group creation process, and inject an initialization container and communication configuration file based on the configuration information to automatically configure the accelerator card communication library.

[0009] According to the automated configuration device for the accelerator card communication library proposed in the embodiment of the present application, the initialization container is used to execute a preset hardware identification script to identify the accelerator card to be identified and obtain the accelerator card type of the accelerator card to be identified; a target communication library that matches the accelerator card type is determined, and a communication configuration file is generated based on the target communication library. Configuration information is determined based on a preset container group creation process, and the initialization container and communication configuration file are injected based on the configuration information to automatically configure the accelerator card communication library. This solves the problems of low efficiency, easy errors, and lack of fine communication control in the configuration of heterogeneous accelerator card communication libraries in the Kubernetes environment in the prior art, realizes automated identification, dynamic configuration, and network connectivity optimization, thereby improving the deployment efficiency and communication performance of distributed computing tasks.

[0010] To achieve the above-mentioned objectives, the third aspect of the present application proposes an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein the processor executes the program to implement the automatic configuration method of the acceleration card communication library as described in the above embodiment.

[0011] To achieve the above-mentioned objectives, the fourth embodiment of the present application proposes a non-volatile computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement the automated configuration method of the acceleration card communication library as described in the above embodiment.

[0012] To achieve the above-mentioned objectives, the fifth embodiment of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the automatic configuration method of the acceleration card communication library as described in the above embodiment.

[0013] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0015] Figure 1 This is a flowchart of a method for automatically configuring an accelerator card communication library according to an embodiment of the present application; Figure 2 A schematic diagram illustrating how an initialization container identifies an accelerator card type according to an embodiment of the present application; Figure 3A schematic diagram of configuring an initialization container for a webhook according to an embodiment of the present application; Figure 4 A schematic diagram of RDMA (Remote Direct Memory Access) network testing and tuning according to an embodiment of the present application; Figure 5 A block diagram illustrating automated configuration of an accelerator card communication library according to an embodiment of the present application; Figure 6 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0017] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0018] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0019] In response to the problems mentioned in the above background technology, such as inefficient and error-prone configuration of heterogeneous accelerator card communication libraries in the Kubernetes environment, and the lack of fine-grained communication control, there is an urgent need for a solution that supports automatic identification of heterogeneous GPUs, automatic generation of communication configurations, and has high scalability and system integration capabilities to improve the deployment efficiency and operational performance of AI tasks under containerized platforms.

[0020] This application provides an automated configuration method for an accelerator card communication library. By combining InitContainer to automatically detect the host GPU type, dynamically injecting communication configuration based on Mutating Admission Webhook, and using NCCL / RCCL's own tools to pre-detect network connectivity, and using the test results as feedback to optimize communication parameters, a closed-loop mechanism with automatic identification, dynamic configuration, real-time detection, and adaptive optimization capabilities is formed, significantly improving the deployment efficiency, communication performance, and system robustness of distributed computing in heterogeneous GPU clusters.

[0021] The following describes, with reference to the accompanying drawings, the method, apparatus, device, and medium for automatically configuring the accelerator card communication library according to the embodiments of the present application.

[0022] Specifically, Figure 1 This is a flowchart of an automatic configuration method for an accelerator card communication library according to an embodiment of the present application.

[0023] like Figure 1 As shown, the automatic configuration method of the accelerator card communication library includes the following steps: S101: When there is a requirement for automatic configuration of an accelerator card communication library, determine an accelerator card to be identified.

[0024] Specifically, the embodiment of the present application can determine whether there is a need for automatic configuration of the accelerator card communication library by checking the operating environment of the container, the hardware configuration of the host node, and the configuration file of the container. If there is a need for automatic configuration of the accelerator card communication library, the accelerator card to be identified is determined.

[0025] S102: Execute a preset hardware identification script using the initialization container to identify the accelerator card to be identified and obtain the accelerator card type of the accelerator card to be identified.

[0026] Furthermore, in some embodiments, a preset hardware identification script is executed by using an initialization container to identify the accelerator card to be identified and obtain the accelerator card type of the accelerator card to be identified, including: using the initialization container to execute the preset hardware identification script to obtain kernel module information and peripheral component interconnect (PCI) device information of the target host node; and obtaining the accelerator card type of the accelerator card to be identified based on the kernel module information and the peripheral component interconnect device information.

[0027] Specifically, the embodiment of the present application uses InitContainer (initialization container) to automatically execute the hardware identification script during the container startup phase, and parses the kernel module information and PCI device information of the host node by executing the lsmod and lspci commands to identify the GPU type of the host machine (such as NVIDIA or AMD / ROCm).

[0028] Furthermore, an InitContainer is injected into the target container group in the Mutating Admission Webhook.

[0029] The InitContainer image must include tools such as bash, lsmod, lspci, and jq, and must mount necessary directories such as / proc, / sys, / dev, and / etc as read-only.

[0030] For example, the script logic of the embodiment of the present application may be as follows: a) Check kernel modules: - lsmod | grep nvidia to check NVIDIA; - lsmod | grep amdgpu or lsmod | grep roc to check AMD / ROCm; b) Check peripheral interconnection devices: - lspci | grep -i nvidia; - lspci | grep -i amd.

[0031] Furthermore, in some embodiments, after using the initialization container to execute a preset hardware identification script to obtain the kernel module information and peripheral interconnection device information of the target host node, it also includes: outputting the kernel module information and peripheral interconnection device information as structured data in a preset data exchange format; and writing the structured data in the preset data exchange format to a shared mounted volume.

[0032] Among them, the preset data exchange format of the embodiment of the present application can be JSON format.

[0033] Specifically, after using the initialization container to execute the preset hardware identification script to obtain the kernel module information and peripheral interconnection device information of the target host node, the embodiment of the present application combines the kernel module information and peripheral interconnection device information to output a JSON structure, such as: {"type": "nvidia", "model": "Tesla V100", "count": 4}, and writes the result to a shared mounted volume, such as / etc / accelerator-info / accelerator.json.

[0034] Furthermore, in some embodiments, after obtaining the accelerator card type of the accelerator card to be identified based on the kernel module information and the peripheral interconnection device information, it also includes: if the accelerator card type of the accelerator card to be identified is not identified, writing a record indicating that the device is missing, and returning a non-zero exit code to terminate execution.

[0035] Specifically, when the type of the graphics processing unit (GPU) installed on the host machine cannot be detected, a record indicating the missing device is written into the script, such as {"type": "none"}, and the script terminates with a non-zero exit code. Furthermore, after writing the record identifying the missing device, the system returns a non-zero exit code. In a computer system, an exit code is a status code used to indicate the result of program execution. Typically, an exit code of 0 indicates that the program executed normally and did not encounter any errors, while a non-zero exit code indicates that the program encountered an error or exception during execution. By returning a non-zero exit code, the caller (such as a container orchestration system or automated script) can be clearly informed that the current execution process cannot continue due to failure to identify the graphics processor type, thereby preventing misuse of the main container. In addition, the detection process and results can be recorded in detail through logs for easy troubleshooting.

[0036] In order to facilitate those skilled in the art to more clearly and intuitively understand the process of automatically identifying the accelerator card type in the Kubernetes environment in the embodiment of the present application, the following is combined with Figure 2 Provide detailed explanation.

[0037] First, let's describe a container group using an NVIDIA accelerator card. During pod startup, the initial container (InitContainer) shares the same resource specifications and mounted volumes as the main container. It then runs the device detection engine, which identifies the GPU type as NVIDIA by querying kernel modules and PCI information. This identification triggers the network configuration engine to generate the RDMA network configuration file nccl.conf, which is optimized for NVIDIA GPUs. This file contains key parameters such as RDMA network card binding and topology. The main container then loads this configuration file, enabling efficient NCCL-based communication with the NVIDIA driver.

[0038] Next, we'll introduce container groups using AMD accelerator cards. This process is similar to the container group using NVIDIA accelerator cards. The initial container identifies the GPU type as "amd," and the network configuration engine generates the rccl.conf file based on this information. Finally, the main container collaborates with the AMD ROCm driver based on this configuration to initialize RCCL communication.

[0039] S103: Determine a target communication library that matches the accelerator card type, generate a communication configuration file based on the target communication library, determine configuration information based on a preset container group creation process, and inject an initialization container and communication configuration file based on the configuration information to automatically configure the accelerator card communication library.

[0040] In some embodiments, the target communication library is at least one of a first preset type collective communication library and a second preset type collective communication library.

[0041] Optionally, the first preset type collective communication library may be an NCCL collective communication library (NVIDIA collective communication library), and the second preset type collective communication library may be at least one of an RCCL collective communication library (ROCm collective communication library).

[0042] Further, in some embodiments, determining the target communication library that matches the accelerator card type includes: if the accelerator card type is a first preset type accelerator card, determining the target communication library as a first preset type collective communication library; if the accelerator card type is a second preset type accelerator card, determining the target communication library as a second preset type collective communication library; if the accelerator card type is empty, triggering a preset error handling mechanism or enabling a degraded communication strategy.

[0043] Exemplarily, if the accelerator card type uses the NCCL collective communication library, the target communication library is determined to be the NCCL collective communication library; if the accelerator card type uses the RCCL collective communication library, the target communication library is determined to be the RCCL collective communication library; if the accelerator card type is none, the preset error handling mechanism is triggered or the degraded communication strategy is enabled.

[0044] Specifically, the corresponding communication library (such as NCCL for NVIDIA and RCCL for AMD) is automatically selected based on the identified accelerator card type.

[0045] In the main container startup script or entrypoint, first read / etc / accelerator-info / accelerator.json. By parsing the JSON, determine the type field: -If the type field is nvidia, set the environment variable COMM_LIB=NCCL; -If the type field is amd, set the environment variable COMM_LIB=RCCL; -If the type field is none, exit with an error or downgrade.

[0046] Furthermore, based on COMM_LIB, the target communication library path is automatically exported: - export LD_LIBRARY_PATH= / opt / nccl / lib:$LD_LIBRARY_PATH; - Or export LD_LIBRARY_PATH= / opt / rccl / lib:$LD_LIBRARY_PATH.

[0047] It should be noted that if the image does not have a built-in communication library, it can be provided by mounting the HostPath or using a sidecar container. If multiple communication libraries need to coexist, the script must support priority configuration, such as controlling it through the container group annotation comm-lib-prefer: nccl, and writing the selection results to the log for subsequent tracing.

[0048] Furthermore, in some embodiments, generating a communication profile based on a target communication library includes: detecting remote direct memory access (RDMA) device information of a host node and obtaining a cluster node topology; and generating a communication profile based on the RDMA device information and the cluster node topology.

[0049] In some embodiments, the communication configuration file includes at least one of RDMA network card information, inter-node communication topology, node sorting, and RDMA network card interface.

[0050] Specifically, a configuration file is generated based on the communication library and node information, including RDMA network card information, inter-node communication topology, related parameters, etc. The communication configuration file is mounted to the specified path of the main container through ConfigMap, Secret or shared volume.

[0051] Furthermore, in the InitContainer or main container startup script, perform the following operations: a) Detect local RDMA devices: - ibdev2netdev gets the IB device name; - lspci | grep -i mellanox obtains the network card model.

[0052] b) Get node information: - Inject the node name and IP through Downward API; - Use the Kubernetes API to query the IP and host names of all container groups under the same Job / container group.

[0053] c) Generate communication topology: - Sort by node IP / hostname and generate ring / tree / mesh topology.

[0054] d) Generate configuration file content.

[0055] Furthermore, the following takes NCCL as an example to illustrate the process of generating configuration file content.

[0056] # Enable debug logging NCCL_DEBUG=INFO; NCCL_DEBUG_SUBSYS=INIT,NET,IB; # Enable IB NCCL_IB_DISABLE=0; # RoCE usually uses GID Index 3; NCCL_IB_GID_INDEX=3; # QoS related tuning NCCL_IB_SL=5; NCCL_IB_TC=136; # Network card name (viewed via ibv_devinfo) NCCL_IB_HCA=mlx5_0; # Enable GPUDirect RDMA NCCL_IB_CUDA_SUPPORT=1; NCCL_IB_GDR_LEVEL=SYS; # Optimizing Queue Pair Concurrency NCCL_IB_QPS_PER_CONNECTION=4; e) If it is RCCL, generate the corresponding parameters f) Write the configuration file to / etc / nccl.conf or / etc / rccl.conf.

[0057] In addition, if node information acquisition fails, retry and log it. If it times out, the startup is terminated. The configuration file permissions must be 0644 to ensure that the main container can read it.

[0058] Therefore, the Mutating Admission Webhook identifies the tenant to which the container belongs based on the label, annotation, or namespace in the container group, and dynamically generates an isolated communication profile for each tenant.

[0059] Furthermore, in some embodiments, configuration information is determined based on a preset container group creation process, and an initialization container and a communication configuration file are injected based on the configuration information to automatically configure the accelerator card communication library, including: using a MutatingAdmission Webhook (variable admission control hook) to listen to container group creation requests; filtering the target container group to be injected based on the container group annotation or label; determining the target container group configuration information, and injecting the initialization container and the communication configuration file into the configuration of the target container group based on the configuration information.

[0060] In some embodiments, the communication configuration file uses a data serialization format (e.g., YAML format) and includes configuration information such as the communication group topology, node ordering, and RDMA network interface. The communication configuration file is mounted to a preset designated path in the main container via a configuration map, key, or shared volume.

[0061] Specifically, if Figure 3 As shown in the figure, using Kubernetes's Mutating Admission Webhook, the InitContainer and communication configuration files are automatically injected during the container group creation process to achieve transparent deployment.

[0062] Furthermore, deploy a Mutating Admission Webhook and register it with the Kubernetes API Server. The Webhook listens for container group creation and filters targets to be injected (such as container groups annotated with comm-inject: "true").

[0063] Furthermore, the webhook automatically modifies the container group Spec: first, it injects the InitContainer that detects the accelerator card, then injects the mounted volume (such as emptyDir or PVC) of the communication configuration file, and injects the communication library and configuration-related environment variables (such as COMM_LIB and COMM_CONFIG_PATH). If necessary, it also injects the communication library provided by the sidecar container.

[0064] Furthermore, the webhook returns the modified container group spec, without the user's knowledge. The webhook provides a detailed AdmissionReview response and logs for exceptions (such as injection failures and field conflicts). This embodiment of the application also allows for flexible management of injection policies through custom resource definitions (CRDs).

[0065] Furthermore, in some embodiments, after injecting the initialization container and the communication configuration file based on the configuration information, it also includes: calling the test tool of the target communication library to pre-detect the network connectivity between nodes, obtaining communication performance data, and obtaining communication test results based on the communication performance data; feeding back the communication test results to the configuration generation logic, and dynamically adjusting the target parameters in the communication configuration file according to the communication test results.

[0066] In some embodiments, the target parameters include at least one of communication topology mapping, port selection, link binding strategy, and number of communication channels.

[0067] Furthermore, in some embodiments, the test tool of the target communication library is called to perform preliminary detection of network connectivity between nodes, including: reading the host names of all nodes and performing connectivity tests on each node; testing the bandwidth and delay between nodes based on the first preset communication performance test tool or the second preset communication performance test tool; and verifying the connectivity of the remote direct memory access port based on a preset hardware information query command or a preset network connectivity test command.

[0068] Exemplarily, the host names of all nodes are read, and a ping test is performed on each node; the bandwidth and latency indicators between each node pair are measured based on the nccl-test tool or the rccl-test tool; and the connectivity of the RDMA port is verified based on the ibstat command or the ibping command.

[0069] Specifically, integrate the nccl-tests or rccl-tests toolkit into InitContainer.

[0070] The script process for pre-checking network connectivity between nodes includes: a) Read all node IPs / host names, perform ping tests on each node, and record packet loss and latency. b) Use all_reduce_perf in nccl-tests to launch the test in parallel on multiple nodes to test bandwidth and latency; c) If it is RCCL, use rccl-tests to test the bandwidth and latency between nodes; d) Check the connectivity of the RDMA port (for example, ibstat or ibping).

[0071] Furthermore, the test results are formatted as JSON and written to the / etc / comm-test / result.json file.

[0072] Therefore, by running the nccl-test or rccl-test tool before starting the container, you can check network connectivity, bandwidth, latency, and other indicators between nodes, thereby discovering communication bottlenecks or failures in advance.

[0073] Furthermore, in some embodiments, after obtaining the communication test results based on the communication performance data, it also includes: if the communication test results do not meet the preset pass conditions, terminating the container group startup, and / or, after automatically downgrading the communication parameters, re-calling the test tool of the target communication library to pre-check the network connectivity between nodes.

[0074] Specifically, if the communication test results don't meet the preset pass criteria, the test fails. The script should output detailed logs and be configurable to terminate the container group startup or retry after downgrading parameters. In other words, node pairs with communication test results below the preset threshold will be marked as unavailable, and the system will automatically remove or downgrade the relevant links from the communication configuration. The test logs and result files must be persisted to facilitate subsequent analysis.

[0075] Furthermore, in some embodiments, the target parameters in the communication configuration file are dynamically adjusted according to the communication test results, including: analyzing the bandwidth indicator, delay indicator and packet loss rate indicator in the communication test results; if any of the bandwidth indicator, delay indicator and packet loss rate indicator is not within the corresponding preset range, adjusting the communication topology mapping and / or link binding strategy; regenerating a new communication configuration file, and re-mounting the new communication configuration file to the container.

[0076] Specifically, the RDMA tuning engine periodically reads / etc / comm-test / result.json and analyzes metrics such as bandwidth, latency, and packet loss to determine whether the current configuration is optimal. If bottlenecks or anomalies are found, the configuration parameters are automatically adjusted. For example: a) Enable GPU DirectRDMA and optimize NUMA (Non-Uniform Memory Access) topology (e.g., NCCL_IB_CUDA_SUPPORT=1); b) Set up the NCCL network interface appropriately (make sure the fallback uses a high-performance interface, not the default eth0); c) Control channel concurrency and topology (NCCL_MAX_NRINGS=8).

[0077] Furthermore, after automatically adjusting the configuration parameters, the communication configuration file is regenerated and hot-updated to the mounted volume, and the steps of analyzing bandwidth, latency, packet loss and other indicators are repeated based on a preset number of executions until the communication configuration file is regenerated and hot-updated to the mounted volume, the optimal configuration is found, and the found optimal configuration is fixed to the communication configuration file nccl.conf or rccl.conf.

[0078] Therefore, by feeding the test data back to the configuration generation logic, communication parameters (such as link binding, port selection, and topology) are dynamically adjusted to improve the robustness of the communication initialization phase and the overall communication performance.

[0079] It should be noted that if the configuration fails, an alarm must be automatically reported (such as writing to K8s Event, Prometheus indicators, email notification, etc.), and all optimization operations must record the change history in detail to facilitate tracing and rollback.

[0080] In order to facilitate those skilled in the art to more clearly and intuitively understand the RDMA network test and tuning process of the embodiment of the present application, the following Figure 4 Provide detailed explanation.

[0081] Specifically, if Figure 4 As shown in the figure, the upper layer represents the RDMA network detection and tuning phase. All container groups first run the initial container upon startup. Within the initial container, nccl-test or rccl-test is executed to measure inter-node RDMA network bandwidth, latency, and other metrics, and report the results to the RDMA tuning engine. The tuning engine dynamically updates the network parameters in nccl.conf (NVIDIA scenario) or rccl.conf (AMD scenario) based on this test data, completing a closed loop of "detection → feedback → optimization." Upon completion, the initial container exits immediately, leaving only the optimized configuration file.

[0082] The lower layer is the main container's main business operation phase. After all initial containers exit, the main container starts up, reads the latest nccl.conf / rccl.conf written by the tuning engine from the same mounted volume, and initializes its respective NCCL / RCCL communication environment. It then begins normal distributed training or inference tasks, efficiently communicating over a tuned RDMA network. This ensures that network configuration is always based on real-world test results and is optimized before the main business starts, achieving both performance and stability improvements.

[0083] This paper proposes an automated accelerator card identification and communication library configuration method for Kubernetes environments, aiming to simplify the system configuration process during heterogeneous GPU deployment and significantly improve the execution efficiency of distributed computing tasks. This method combines the InitContainer mechanism, the Mutating Admission Webhook component, and a node-level hardware detection script to construct an intelligent closed-loop process that automatically identifies accelerator card types and dynamically generates and injects communication configurations. In practical implementation, this solution utilizes the InitContainer to automatically run the hardware identification script during container startup. By parsing kernel module information (such as lsmod) and PCI device information (such as lspci), it accurately determines the host node's GPU type (e.g., NVIDIA or AMD / ROCm). After identification, the system automatically selects a matching communication library (such as NCCL or RCCL) based on the accelerator card type and generates a communication configuration file based on pre-defined logic. This configuration includes RDMA network card binding information and inter-node communication topology. Subsequently, using the Mutating Admission Webhook, the InitContainer and its required configuration are automatically injected into the container group creation process, achieving transparent deployment of the communication environment without user intervention. To further ensure communication stability and deployment success, this solution integrates the communication library's built-in testing tools (such as nccl-test or rccl-test) to pre-check the network connectivity between cluster nodes before container deployment. By running these test programs, the system can promptly identify potential communication failures or bandwidth bottlenecks, avoiding distributed task failures due to node anomalies. By running the communication library's testing tools (such as nccl-test or rccl-test) during the container group startup phase, performance data such as network connectivity, bandwidth, and latency are collected, and the results are used as input and fed back to the configuration generation logic to dynamically modify key parameters in the communication configuration file (such as topology mapping, port selection, link binding, etc.), ultimately achieving optimization and improved robustness of the communication initialization process.

[0084] This approach achieves full automation and highly reliable deployment of the communication environment by automatically identifying accelerator card types, intelligently selecting communication libraries, and dynamically generating communication configuration files within a Kubernetes environment. This approach, combined with communication library testing tools for network connectivity pre-testing and feedback optimization, effectively reduces the labor cost and error risk of heterogeneous GPU system configuration, improving the communication efficiency and system robustness of distributed training tasks. It is particularly suitable for the deployment of multi-node, large-scale, high-performance AI computing clusters, and has significant practical value and potential for promotion.

[0085] According to the automated configuration method of the accelerator card communication library proposed in the embodiment of the present application, the automated configuration method of the accelerator card communication library proposed in the embodiment of the present application utilizes the initialization container to execute the preset hardware identification script to identify the accelerator card to be identified and obtain the accelerator card type of the accelerator card to be identified; determines the target communication library that matches the accelerator card type, and generates a communication configuration file based on the target communication library; determines the configuration information based on the preset container group creation process, and injects the initialization container and the communication configuration file based on the configuration information to automatically configure the accelerator card communication library. Thus, the problems of low efficiency, easy errors and lack of fine communication control in the configuration of heterogeneous accelerator card communication libraries in the Kubernetes environment in the prior art are solved, the performance and robustness of the communication initialization process are optimized, and the deployment efficiency and communication performance of distributed computing tasks are improved.

[0086] Next, the automatic configuration device of the accelerator card communication library proposed in the embodiment of the present application is described with reference to the accompanying drawings.

[0087] Figure 5 4 is a block diagram of an automatic configuration device for an accelerator card communication library according to an embodiment of the present application.

[0088] like Figure 5 As shown, the automatic configuration device 10 of the acceleration card communication library includes: a determination module 100, an identification module 200 and a configuration module 300.

[0089] Among them, the determination module 100 is used to determine the accelerator card to be identified when there is a need for automatic configuration of the accelerator card communication library; the identification module 200 is used to use the initialization container to execute the preset hardware identification script to identify the accelerator card to be identified and obtain the accelerator card type of the accelerator card to be identified; the configuration module 300 is used to determine the target communication library that matches the accelerator card type, and generate a communication configuration file based on the target communication library, determine the configuration information based on the preset container group creation process, and inject the initialization container and communication configuration file based on the configuration information to automatically configure the accelerator card communication library.

[0090] Furthermore, in some embodiments, after injecting the initialization container and the communication configuration file based on the configuration information, the configuration module 300 is also used to: call the test tool of the target communication library to pre-detect the network connectivity between nodes, obtain communication performance data, and obtain communication test results based on the communication performance data; feed back the communication test results to the configuration generation logic, and dynamically adjust the target parameters in the communication configuration file based on the communication test results.

[0091] Furthermore, in some embodiments, the identification module 200 is used to: use the initialization container to execute a preset hardware identification script to obtain the kernel module information and peripheral interconnection device information of the target host node; based on the kernel module information and peripheral interconnection device information, determine whether the type of graphics processor installed in the host machine is identified; if the type of graphics processor is identified, determine the type of accelerator card according to the type of graphics processor.

[0092] Furthermore, in some embodiments, after using the initialization container to execute a preset hardware identification script to obtain the kernel module information and peripheral interconnection device information of the target host node, the identification module 200 is also used to: output the kernel module information and peripheral interconnection device information as structured data in a preset data exchange format; and write the structured data in the preset data exchange format to a shared mounted volume.

[0093] Furthermore, in some embodiments, after determining whether the type of the graphics processor installed in the host machine is identified based on the kernel module information and the peripheral interconnection device information, the identification module 200 is also used to: if the type of the graphics processor is not identified, write a record indicating that the device is missing and return a non-zero exit code to terminate execution.

[0094] Further, in some embodiments, the target communication library is at least one of the NVIDIA collective communication library and the ROCm collective communication library.

[0095] Furthermore, in some embodiments, the configuration module 300 is used to: if the accelerator card type is NVIDIA, determine that the target communication library is the NVIDIA collective communication library; if the accelerator card type is AMD, determine that the target communication library is the ROCm collective communication library; if the accelerator card type is none, trigger the preset error handling mechanism or enable the degraded communication strategy.

[0096] Furthermore, in some embodiments, the configuration module 300 is configured to: detect the RDMA device information of the host node and obtain the cluster node topology; and generate a communication configuration file according to the RDMA device information and the cluster node topology.

[0097] Furthermore, in some embodiments, the configuration module 300 is configured to: monitor container group creation requests using a mutating admission webhook; filter target container groups to be injected based on container group annotations or tags; determine target container group configuration information, and inject the initialization container and communication configuration file into the target container group configuration based on the configuration information.

[0098] Furthermore, in some embodiments, the configuration module 300 is used to: read the host names of all nodes and perform a ping test on each node; test the bandwidth and latency between nodes based on the nccl-test tool or the rccl-test tool; and verify the connectivity of the RDMA port based on the ibstat command or the ibping command.

[0099] Furthermore, in some embodiments, after obtaining the communication test results based on the communication performance data, the configuration module 300 is also used to: if the communication test results do not meet the preset pass conditions, terminate the container group startup, and / or, after automatically downgrading the communication parameters, re-call the test tool of the target communication library to pre-check the network connectivity between nodes.

[0100] Furthermore, in some embodiments, the configuration module 300 is used to: analyze the bandwidth indicator, delay indicator and packet loss rate indicator in the communication test results; if any of the bandwidth indicator, delay indicator and packet loss rate indicator is not within the corresponding preset range, adjust the communication topology mapping and / or link binding strategy; regenerate a new communication configuration file, and re-mount the new communication configuration file to the container.

[0101] Furthermore, in some embodiments, the communication configuration file is mounted to a preset designated path of the main container via configuration mapping, keys, or shared volumes.

[0102] Furthermore, in some embodiments, the communication configuration file adopts a data serialization format.

[0103] Furthermore, in some embodiments, the communication configuration file includes at least one of RDMA network card information, inter-node communication topology, node sorting, and RDMA network card interface.

[0104] Further, in some embodiments, the target parameters include at least one of a communication topology mapping, a port configuration, a link binding policy, and a number of communication channels.

[0105] It should be noted that the above explanation of the embodiment of the automatic configuration method of the accelerator card communication library is also applicable to the automatic configuration device of the accelerator card communication library of this embodiment, and will not be repeated here.

[0106] According to the automated configuration device of the accelerator card communication library proposed in the embodiment of the present application, and according to the automated configuration method of the accelerator card communication library proposed in the embodiment of the present application, the preset hardware identification script is executed by using the initialization container to identify the accelerator card to be identified, and the accelerator card type of the accelerator card to be identified is obtained; the target communication library that matches the accelerator card type is determined, and a communication configuration file is generated based on the target communication library, the configuration information is determined based on the preset container group creation process, and the initialization container and the communication configuration file are injected based on the configuration information to automatically configure the accelerator card communication library. As a result, the problems of low efficiency, easy errors and lack of fine communication control in the configuration of heterogeneous accelerator card communication libraries in the Kubernetes environment in the prior art are solved, and automated identification, dynamic configuration and network connectivity optimization are achieved, thereby improving the deployment efficiency and communication performance of distributed computing tasks.

[0107] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include: A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0108] When the processor 602 executes the program, the timing management method provided in the above embodiment is implemented.

[0109] Furthermore, the electronic device further includes: The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0110] The memory 601 is used to store computer programs that can be run on the processor 602 .

[0111] The memory 601 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0112] If the memory 601, processor 602, and communication interface 603 are implemented independently, the communication interface 603, memory 601, and processor 602 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0113] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0114] The processor 602 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0115] An embodiment of the present application further provides a non-volatile computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned timing management method when executed by a processor.

[0116] An embodiment of the present application further provides a computer program product, including a computer program, which implements the above timing management method when executed by a processor.

[0117] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] The above is a detailed introduction to an integrated circuit board, timing management method, server, device, medium and product provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. An automated configuration method for an accelerator card communication library, characterized in that: The following steps are involved: If there is a need for automatic configuration of the accelerator card communication library, determine the accelerator card to be identified; Utilizing the initialization container to execute a preset hardware identification script to identify the accelerator card to be identified, and obtaining the accelerator card type of the accelerator card to be identified; Determine a target communication library that matches the accelerator card type, generate a communication configuration file based on the target communication library, determine configuration information based on a preset container group creation process, and inject an initialization container and communication configuration file based on the configuration information to automatically configure the accelerator card communication library.

2. The method according to claim 1, characterized in that After injecting the initialization container and the communication configuration file based on the configuration information, the method further includes: Calling a test tool of the target communication library to pre-detect network connectivity between nodes, obtaining communication performance data, and obtaining a communication test result based on the communication performance data; The communication test results are fed back to the configuration generation logic, and the target parameters in the communication configuration file are dynamically adjusted according to the communication test results.

3. The method according to claim 1, characterized in that The step of using the initialization container to execute a preset hardware identification script to identify the accelerator card to be identified and obtaining the accelerator card type of the accelerator card to be identified includes: Utilize the initialization container to execute the preset hardware identification script to obtain kernel module information and peripheral interconnection device information of the target host node; Based on the kernel module information and the peripheral interconnection device information, the accelerator card type of the accelerator card to be identified is obtained.

4. The method according to claim 3, characterized in that After executing the preset hardware identification script using the initialization container to obtain kernel module information and peripheral interconnection device information of the target host node, the method further includes: Outputting the kernel module information and the peripheral interconnection device information as structured data in a preset data exchange format; The structured data in the preset data exchange format is written to the shared mounted volume.

5. The method according to claim 3, characterized in that After obtaining the accelerator card type of the accelerator card to be identified based on the kernel module information and the peripheral interconnection device information, the method further includes: If the accelerator card type of the accelerator card to be identified is not identified, a record indicating that the device is missing is written, and a non-zero exit code is returned to terminate execution.

6. The method according to claim 1, characterized in that The target communication library is at least one of a first preset type collective communication library and a second preset type collective communication library.

7. The method according to claim 6, characterized in that Determining a target communication library that matches the accelerator card type includes: If the accelerator card type is a first preset type accelerator card, determining that the target communication library is the first preset type collective communication library; If the accelerator card type is a second preset type accelerator card, determining that the target communication library is the second preset type collective communication library; If the accelerator card type is empty, a preset error handling mechanism is triggered or a degraded communication strategy is enabled.

8. The method according to claim 1, characterized in that The generating of a communication configuration file based on the target communication library includes: Detect the remote direct memory access device information of the host node and obtain the cluster node topology; The communication configuration file is generated according to the remote direct memory access device information and the cluster node topology structure.

9. The method according to claim 1, characterized in that The process of determining configuration information based on a preset container group creation process, and injecting an initialization container and a communication configuration file based on the configuration information to automatically configure the accelerator card communication library includes: Use variable admission control hooks to listen for container group creation requests; Filter the target container group to be injected based on the container group annotation or label; Determine the target container group configuration information, and based on the configuration information, inject the initialization container and the communication configuration file into the configuration of the target container group.

10. The method according to claim 2, characterized in that The test tool calling the target communication library to perform a preliminary check on the network connectivity between nodes includes: Read the host names of all nodes and perform connectivity tests on each node; Testing the bandwidth and delay between nodes based on the first preset communication performance testing tool or the second preset communication performance testing tool; The connectivity of the remote direct memory access port is verified based on a preset hardware information query command or a preset network connectivity test command.

11. The method according to claim 2, characterized in that After obtaining the communication test result according to the communication performance data, the method further includes: If the communication test result does not meet the preset pass condition, the container group startup is terminated, and / or, after automatically downgrading the communication parameters, the test tool of the target communication library is re-called to pre-check the network connectivity between nodes.

12. The method according to claim 2, characterized in that The dynamically adjusting the target parameters in the communication configuration file according to the communication test result includes: Analyze the bandwidth index, delay index and packet loss rate index in the communication test results; If any one of the bandwidth indicator, the delay indicator, and the packet loss rate indicator is not within the corresponding preset range, adjusting the communication topology mapping and / or the link binding strategy; A new communication configuration file is regenerated, and the new communication configuration file is remounted to the container.

13. The method according to claim 1, wherein The communication configuration file is mounted to a preset designated path of the main container through configuration mapping, keys or shared volumes.

14. The method according to claim 1, wherein The communication configuration file adopts a data serialization format.

15. The method according to claim 1, wherein The communication configuration file includes at least one of remote direct memory access network card information, inter-node communication topology, node sorting, and remote direct memory access network card interface.

16. The method according to claim 2, characterized in that The target parameters include at least one of a communication topology mapping, a port configuration, a link binding strategy, and a number of communication channels.

17. An automatic configuration device for an accelerator card communication library, characterized in that: The following steps are involved: A determination module, used to determine the accelerator card to be identified when there is a demand for automatic configuration of the accelerator card communication library; an identification module, configured to execute a preset hardware identification script using an initialization container to identify the accelerator card to be identified and obtain the accelerator card type of the accelerator card to be identified; A configuration module is used to determine a target communication library that matches the accelerator card type, generate a communication configuration file based on the target communication library, determine configuration information based on a preset container group creation process, and inject an initialization container and communication configuration file based on the configuration information to automatically configure the accelerator card communication library.

18. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the automatic configuration method for the accelerator card communication library according to any one of claims 1 to 16.

19. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the automatic configuration method of the accelerator card communication library as described in any one of claims 1 to 16.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the automatic configuration method of the accelerator card communication library according to any one of claims 1 to 16.

Images