A PyTorch adaptation method based on a domestic Muxi AI accelerator card

Abstract

The present application relates to a PyTorch adaptation method based on a domestic Muxi AI accelerator card, which belongs to the field of artificial intelligence, the method comprising: MACA software stack environment variable configuration, installing Muxi MACA software stack and configuring the first system environment variable; cu-bridge compilation tool escape encapsulation layer environment variable configuration, obtaining cu-bridge compilation tool containing cmake_maca script, cucc script and escape header file set, creating a simulated CUDA toolkit installation directory and setting the second system environment variable; PyTorch source code compilation adaptation, using cmake_maca script to replace the system cmake command, based on the configured environment, source code compilation, analyzing the compilation error piece by piece and using the corresponding adaptation modification scheme until the Python package is completely compiled and passed. The present application also contains a verification test step, which verifies the correctness of the adaptation through Python module import test and single / multi-card large model inference test. The present application realizes the correct adaptation and efficient operation of the original PyTorch in the domestic Muxi AI accelerator card environment, significantly reducing the code migration workload.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a PyTorch adaptation method based on the domestically produced Muxi AI accelerator card. Background Technology

[0002] In recent years, large-scale models have sparked a surge in the field of artificial intelligence. Deep learning frameworks, due to their powerful functionality, ease of use, high scalability, and ease of debugging, have become the mainstream choice. PyTorch has become the industry's most widely chosen underlying deep learning framework for training and inference of large models. However, native PyTorch is only deeply optimized for AI accelerator cards from NVIDIA, AMD, and other manufacturers. In many domestic AI accelerator card environments, native PyTorch cannot be used directly.

[0003] Because native PyTorch is only deeply bound and optimized for the software stacks of mainstream foreign AI accelerator cards such as NVIDIA and AMD, it cannot be used directly in many domestic AI accelerator card environments. Many mature upper-layer AI applications rely on PyTorch at their core, which leads to a large amount of work in migrating program code in domestic AI accelerator card environments, thus affecting the ecosystem of domestic AI accelerator cards.

[0004] Existing technical solutions typically require large-scale, invasive modifications to the PyTorch source code, resulting in long adaptation cycles, high maintenance costs, and difficulty in keeping up with PyTorch version iterations. Therefore, there is an urgent need for a systematic, low-intrusion, and reproducible PyTorch adaptation method. This method should fully utilize the cu-bridge escaping and encapsulation capabilities, minimize source code modifications through conditional compilation mechanisms, and achieve efficient adaptation and stable operation of native PyTorch in the domestic Muxi AI accelerator card environment. Summary of the Invention

[0005] The purpose of this invention is to provide a PyTorch adaptation method based on the domestically produced Muxi AI accelerator card, so as to solve the problem that the native PyTorch cannot be directly compiled and run in the Muxi AI accelerator card environment, and to achieve correct adaptation and efficient operation of the native PyTorch in the domestic AI accelerator card environment.

[0006] This invention provides a PyTorch adaptation method based on the domestically produced Muxi AI accelerator card. The method includes: MACA Software Stack Environment Variable Configuration: Install the Muxi MACA software stack, which includes the compiler mxcc for the Muxi AI accelerator card and high-performance computing libraries, communication libraries, and runtime libraries compatible with the CUDA ecosystem. Configure the first system environment variables to ensure that the compiler, high-performance computing libraries, communication libraries, and runtime libraries can be correctly retrieved. The first system environment variables include PATH and LD_LIBRARY_PATH. cu-bridge compiler tool escape layer environment variable configuration: Obtain the Muxi cu-bridge compiler tool, which includes the cmake_maca script, the cucc script, and a set of header files that escape the CUDA Toolkit identifier to the Muxi MACA software stack identifier through macro definitions. The cu-bridge compiler tool creates a simulated CUDA toolkit installation directory and sets the second system environment variables. The cucc script is used to redirect nvcc calls to the compiler mxcc. The cu-bridge compiler tool maps CUDA dependency libraries to the corresponding libraries in the Muxi MACA software stack through dynamic library soft links. The second system environment variables include PATH, LD_LIBRARY_PATH, CUDA_PATH, CMAKE_CUDA_COMPILER, NCCL_INCLUDE_DIR, and CUDNN_INCLUDE_DIR. PyTorch source code compilation and adaptation: Obtain the official PyTorch source code, use the cmake_maca script to replace the system cmake command to perform compilation and build, compile the source code based on the configured MACA software stack environment and cu-bridge escape wrapper layer environment, analyze the compilation error information one by one during the compilation process, determine the root cause of each compilation error and adopt the corresponding source code adaptation and modification scheme until the compilation is completely successful, generating a Python program package that can run in the Muxi AI accelerator card environment.

[0007] Optionally, in the MACA software stack environment configuration, the PATH is appended with the / opt / maca / bin path and the / opt / maca / mxgpu_llvm / bin path of the MACA software stack; The LD_LIBRARY_PATH includes the / opt / maca / lib64 and / opt / maca / mxgpu_llvm / lib paths of the MACA software stack.

[0008] Optionally, in the cu-bridge compiler escape wrapper layer environment variable configuration, the PATH is appended with the cu-bridge compiler's / opt / maca / tools / cu-bridge / tools path; The CUDA_PATH is set to simulate the installation directory path of the CUDA toolkit. The bin subdirectory under this directory contains nvcc symbolic links pointing to the cucc script, the lib64 subdirectory contains symbolic links from the CUDA library files to the corresponding dynamic libraries of the Muxi MACA software stack, and the include subdirectory links to the header file directory of the cu-bridge compilation tool. The CMAKE_CUDA_COMPILER is set to the path / root / cu-bridge / CUDA_DIR / bin / nvcc of the cu-bridge compilation tool; The NCCL_INCLUDE_DIR and CUDNN_INCLUDE_DIR are set to the / root / cu-bridge / include path of the cu-bridge compilation tool.

[0009] Optional adaptation and modification schemes in PyTorch source code compilation adaptation include: Add the USE_MACA build option to the CMakeLists.txt file in the root directory of the PyTorch source code and disable incompatible build options; Add the USE_MACA compiler macro definition to the CMakeLists.txt files in multiple subdirectories of the PyTorch source code. This macro is used to conditionally compile incompatible code branches or enable compatible code branches in the corresponding source code files.

[0010] Optionally, the options for disabling incompatible compilation include: Disable the USE_CUFILE build option in the CMakeLists.txt file and / or disable the USE_FLASH_ATTENTION build option in the CMakeLists.txt file and / or disable the USE_MEM_EFF_ATTENTION build option in the CMakeLists.txt file.

[0011] Optionally, the step of shielding incompatible code branches through conditional compilation includes: In the CuDNN version detection module, the FindCUDNN.cmake file is modified to directly return the version number that meets the PyTorch version requirements, so as to adapt to the CuDNN version identifier escaped in the header file of the cu-bridge compiler. In the cuda_private.cu file of the third-party library gloo, the USE_MACA compiler macro definition masks the __host__ __device__ prefix declaration branch that contains the __assert_fail macro in c10 / macros / Macros.h; In the CuptiRangeProfilerApi.cpp file of the third-party library kineto, the code logic that includes the nvperf_host.h header file is masked by the USE_MACA compiler macro definition. In the c10 / cuda / driver_api.h and CUDACachingAllocator.cpp files, the USE_MACA compilation macro definition adapts the nvml interface name to the macro mapping name defined in the cu-bridge header file; In the c10 / cuda / driver_api.cpp file, the number of input parameters to the cudaGetDriverEntryPoint function is adjusted by the USE_MACA compiler macro definition to adapt to the function interface parameters defined in the cu-bridge header file.

[0012] Optionally, the method of shielding incompatible code branches through conditional compilation further includes: In source files containing asm inline assembly instructions, the asm instruction branches are replaced with C++ implementation branches using the USE_MACA compiler macro definition; In source code files that include cutlass-related header files, the USE_MACA compiler macro is used to disable the compilation of the cutlass branch code.

[0013] Optionally, the source code files include at least one of the following: aten / src / ATen / native / cuda / MemoryAccess.cuh, aten / src / ATen / native / cuda / TensorShape.cu, aten / src / ATen / native / quantized / cuda / EmbeddingBag.cu, aten / src / ATen / cuda / AsmUtils.cuh, torch / csrc / distributed / c10d / symm_mem / CUDASymmetricMemory-inl.h, aten / src / ATen / native / cuda / MixedDtypesLinear.cu, and aten / src / ATen / native / sparse / cuda / SparseSemiStructuredApplyDense.cu.

[0014] Optionally, the method of shielding incompatible code branches through conditional compilation further includes: In the file torch / csrc / distributed / c10d / ProcessGroupNCCL.hpp, the USE_MACA compiler macro definition ensures that the ncclConfig_t configuration code compiles successfully; In the torch / csrc / cuda / nccl.h file, enable the HAS_NCCL_BF16_DATATYPE macro definition via the USE_MACA compilation macro definition; In the code files under the torch / _inductor directory, the USE_MACA compiler macro definition is used for adaptation.

[0015] Optional, verification tests are also included: Install the compiled Python package and perform a Python module import test to verify the usability of the torch module and the Muxi AI accelerator card; Large model inference tests were performed to verify the correctness of the compiled and adapted PyTorch in the Muxi AI accelerator card environment. The large model inference tests included single-card model inference tests and multi-card model inference tests.

[0016] In summary, the present invention has at least one of the following beneficial technical effects: 1. The invention proposes a complete three-step process from configuring the MACA software stack environment variables, configuring the cu-bridge escape encapsulation environment variables, to compiling and adapting the PyTorch source code, forming a reproducible and scalable PyTorch adaptation methodology, which significantly improves adaptation efficiency and success rate.

[0017] 2. By fully utilizing the escaping and encapsulation capabilities of the cu-bridge compiler and combining it with the USE_MACA conditional compilation macro definition system, the amount of modification required to the PyTorch source code is minimized, maintaining the maintainability and upgradability of the source code.

[0018] 3. Systematic adaptation solutions are provided for various incompatible scenarios such as CuDNN version detection, nvml interface mapping, inline assembly instructions, cutlass template library, NCCL configuration, and BF16 data type, ensuring the complete availability of PyTorch core functions.

[0019] 4. Through Python module import testing, single-card model inference testing, and multi-card model inference testing, the functional correctness and operational stability of the adapted PyTorch in the Muxi AI accelerator card environment were fully verified. Attached Figure Description

[0020] Figure 1 This is a flowchart of a PyTorch adaptation method based on the domestically produced Muxi AI accelerator card, according to an embodiment of this application. Detailed Implementation

[0021] The present invention will be further described in detail below with reference to the accompanying drawings.

[0022] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of the present invention.

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The embodiments of the present invention will now be described in further detail with reference to the accompanying drawings.

[0025] like Figure 1As shown, this invention provides a flowchart of a PyTorch adaptation method based on the domestically produced Muxi AI accelerator card. Specifically, it includes the configuration of MACA software stack environment variables, the configuration of cu-bridge compilation tool escape wrapper layer environment variables, PyTorch source code compilation adaptation, and verification testing. The specific process is as follows: Step S1: Configure the environment variables for the MACA software stack.

[0026] The official MACA software stack from Muxi is installed by default in the system's / opt / maca directory. The MACA software stack provides the compiler mxcc for Muxi AI accelerator cards. The mxcc compiler is used to compile GPU code on the Muxi platform. The MACA software stack also provides high-performance computing libraries (such as mcblas, mcdnn, etc.), communication libraries (such as mccl, etc.), and runtime libraries compatible with the CUDA ecosystem. The corresponding primary system environment variables are set to ensure that the correct compiler mxcc, high-performance computing libraries, communication libraries, and runtime libraries can be found during subsequent compilation. The high-performance computing libraries, communication libraries, and runtime libraries are collectively referred to as library files. The primary system environment variables include PATH and LD_LIBRARY_PATH.

[0027] After the MACA software stack is installed, correctly set the first system environment variable: PATH is appended with the bin directory path of the MACA software stack, and LD_LIBRARY_PATH is appended with the lib directory path of the MACA software stack.

[0028] The bin directory path includes the / opt / maca / bin path (which contains executable files such as __metaxgpu_arch and macainfo) and the / opt / maca / mxgpu_llvm / bin path (which contains related Muxi LLVM compiler components such as mxcc).

[0029] The lib directory path includes the / opt / maca / lib64 path (which contains various runtime libraries, computing libraries, communication libraries, etc. of the MACA software stack) and the / opt / maca / mxgpu_llvm / lib path (which contains library files related to the compiler mxcc, etc.).

[0030] Step S2: Configure the cu-bridge compiler to escape the encapsulation layer environment variables.

[0031] The cu-bridge compiler tool, officially released by Muxi, is installed by default in the / opt / maca / tools directory of the system. The cu-bridge compiler tool includes a set of compilation scripts and a set of header files that convert CUDA identifiers to MACA software stack identifiers. It converts the identifiers in the CUDA Toolkit to the identifiers of the Muxi MACA software stack through macro definitions. Then, it symbolically links the corresponding dynamic library files of CUDA that the upper-level framework depends on to the corresponding dynamic libraries of the Muxi MACA software stack (such as linking libcublas.so to libmcblas.so). This can minimize the modification of the source code of the upper-level deep learning framework and ensure successful compilation. Configure the second system environment variables to establish a compilation environment that simulates the CUDA Toolkit installation directory.

[0032] The compilation script set includes cmake_maca, make_maca, ninja_maca, cucc, etc., among which: The cmake_maca script is used to prepare the linking of the MACA software stack in a specified directory before calling the system's cmake command. The purpose of the make_maca script is to prepare the linking of the MACA software stack in the specified directory before calling the system's make command. The purpose of the ninja_maca script is to prepare the links to the MACA software stack in the specified directory before calling the system's ninja command; The purpose of the cucc script is to create a symbolic link to the nvcc script that the system needs to call. cucc will then perform the compilation process internally and call the Muxi compiler mxcc.

[0033] The second system environment variables include PATH and LD_LIBRARY_PATH, as well as related environment variables for compiling PyTorch such as NCCL_INCLUDE_DIR, CUDNN_INCLUDE_DIR, CMAKE_CUDA_COMPILER, and CUDA_PATH, among which: PATH is appended with the bin directory path of the cu-bridge compilation tools ( / opt / maca / tools / cu-bridge / tools). LD_LIBRARY_PATH is appended with the lib directory path of the cu-bridge compilation tools; CUDA_PATH is set to the / root / cu-bridge / CUDA_DIR compilation environment directory path of the cu-bridge compilation tool. This path is used to simulate the installation directory path of the CUDA toolkit. Within this installation directory, the bin subdirectory contains the nvcc symbolic link (which actually links to the cucc script in the cu-bridge directory), the lib64 directory contains symbolic links to various CUDA-related library files (such as libcudnn.so linking to libmcdnn.so), and the include directory links to the cu-bridge include directory (containing various identifiers in the CUDA libraries that are escaped and encapsulated into the MACA software stack).

[0034] NCCL_INCLUDE_DIR and CUDNN_INCLUDE_DIR are set to the include directory path of the cu-bridge compilation tool ( / root / cu-bridge / include), which is an environment variable required by the cmake stage of compiling PyTorch; CMAKE_CUDA_COMPILER is set to the directory path of the cu-bridge compilation tool's / root / cu-bridge / CUDA_DIR compilation environment ( / root / cu-bridge / CUDA_DIR / bin / nvcc), which is an environment variable required by the cmake stage of compiling PyTorch.

[0035] Step S3: Compile and adapt the PyTorch source code.

[0036] Obtain the official PyTorch source code, use the cmake_maca script to replace the system cmake command to perform compilation and build, compile the source code based on the configured MACA software stack environment and cu-bridge escape wrapper layer environment, analyze the compilation error information one by one during the compilation process, determine the root cause of each compilation error and adopt the corresponding source code adaptation and modification scheme until the compilation is completely successful, and generate a Python program package that can run in the Muxi AI accelerator card environment.

[0037] The adaptation and modification schemes for compiling and adapting PyTorch source code include: Add the USE_MACA build option to the CMakeLists.txt file in the root directory of the PyTorch source code and disable incompatible build options; Add the USE_MACA compiler macro definition to the CMakeLists.txt files in multiple subdirectories of the PyTorch source code. This macro is used to conditionally compile incompatible code branches or enable compatible code branches in the corresponding source code files.

[0038] Disabling incompatible build options includes disabling the USE_CUFILE build option in the CMakeLists.txt file and / or disabling the USE_FLASH_ATTENTION build option in the CMakeLists.txt file and / or disabling the USE_MEM_EFF_ATTENTION build option in the CMakeLists.txt file.

[0039] Conditional compilation can be used to suppress incompatible code branches, including: In the CuDNN version detection module, the FindCUDNN.cmake file is modified to directly return the version number that meets the PyTorch version requirements, so as to adapt to the CuDNN version identifier escaped in the header file of the cu-bridge compiler. In the cuda_private.cu file of the third-party library gloo, the USE_MACA compiler macro definition masks the __host__ __device__ prefix declaration branch that contains the __assert_fail macro in c10 / macros / Macros.h; In the CuptiRangeProfilerApi.cpp file of the third-party library kineto, the code logic that includes the nvperf_host.h header file is masked by the USE_MACA compiler macro definition. In the c10 / cuda / driver_api.h and CUDACachingAllocator.cpp files, the USE_MACA compilation macro definition adapts the nvml interface name to the macro mapping name defined in the cu-bridge header file; In the c10 / cuda / driver_api.cpp file, the number of input parameters to the cudaGetDriverEntryPoint function is adjusted by the USE_MACA compiler macro definition to adapt to the function interface parameters defined in the cu-bridge header file.

[0040] In source files containing asm inline assembly instructions, the asm instruction branches are replaced with C++ implementation branches using the USE_MACA compiler macro definition; In source code files that include cutlass-related header files, the USE_MACA compiler macro is used to disable the compilation of the cutlass branch code.

[0041] In the file torch / csrc / distributed / c10d / ProcessGroupNCCL.hpp, the USE_MACA compiler macro definition ensures that the ncclConfig_t configuration code compiles successfully; In the torch / csrc / cuda / nccl.h file, enable the HAS_NCCL_BF16_DATATYPE macro definition via the USE_MACA compilation macro definition; In the code files under the torch / _inductor directory, the USE_MACA compiler macro definition is used for adaptation.

[0042] Step S4: Verification test.

[0043] Install the compiled Python package and perform a Python module import test to verify the usability of the torch module and the Muxi AI accelerator card; Large model inference tests were performed to verify the correctness of the compiled and adapted PyTorch in the Muxi AI accelerator card environment. The large model inference tests included single-card model inference tests and multi-card model inference tests.

[0044] To facilitate a better understanding of this invention, we will take the installation of the MACA software stack v3.5.3.18 in the Muxi C500 accelerator card environment as an example. The default installation path is / opt / maca. The Muxi MACA software stack provides a set of header files and runtime libraries that are compatible with the CUDA ecosystem. It also provides Muxi's cu-bridge compilation tools, compiler mxcc, etc.

[0045] Step S1: Configure the environment variables for the MACA software stack.

[0046] export PATH= / opt / maca / bin: / opt / maca / mxgpu_llvm / bin:${PATH}; export LD_LIBRARY_PATH= / opt / maca / lib64: / opt / maca / mxgpu_llvm / lib:${LD_LIBRARY_PATH}.

[0047] This application example uses the code of the master branch of the open-source cu-bridge compilation tool by Muxi as an example. The default installation path is / opt / maca / tools. The cu-bridge compilation tool includes a series of script tools such as cmake_maca, make_maca, and ninja_maca, as well as a set of CUDA interfaces, type definitions and escape mappings between the MACA software stack interface and type definitions. The compilation process relies on the cu-bridge compilation tool.

[0048] Step S2: Configure the cu-bridge compiler to escape the encapsulation layer environment variables.

[0049] export CUCC_PATH= / opt / maca / tools / cu-bridge; export PATH=${CUCC_PATH} / tools:${PATH}; export LD_LIBRARY_PATH=${CUCC_PATH} / lib:${LD_LIBRARY_PATH}; export NCCL_INCLUDE_DIR=${CUCC_PATH} / include; export CUDNN_INCLUDE_DIR=${CUCC_PATH} / include; export CMAKE_CUDA_COMPILER= / root / cu-bridge / CUDA_DIR / bin / nvcc; export CUDA_PATH= / root / cu-bridge / CUDA_DIR.

[0050] Step S3: Compile and adapt the PyTorch source code.

[0051] Specifically, the cmake_maca script is used to replace the system cmake command for compilation and building.

[0052] During the compilation process, each compilation error message is analyzed and corresponding adaptation solutions are adopted (taking adaptation to native PyTorch v2.8.0 as an example): 1. CuDNN version detection error.

[0053] Error message: "PyTorch needs CuDNN-8.5 or above, but found 8.4.1".

[0054] Root cause analysis: The error message is located in cmake / Dependencies.cmake. The reason is that the version of CuDNN parsed by cmake is lower than the requirement of PyTorch v2.8.0. The root cause is that the version number of the escaped CuDNN defined in the cu-bridge compilation tool is low, which causes this error. However, the actual internal interface of libmcdnn.so is compatible.

[0055] Solution: By modifying the cmake / Modules_CUDA_fix / FindCUDNN.cmake file, a version number greater than 8.5 can be directly returned.

[0056] 2. libcufile.so dependency error.

[0057] Error message: "libcufile.so needed by 'lib / libtorch_cuda.so', missing and no known rule to make it".

[0058] Root cause analysis: By searching the cufile-related code in the PyTorch source code, the root cause was found to be that the USE_CUFILE option was enabled in the CMakeLists.txt file in the PyTorch root directory. This caused the compiled libtorch_cuda.so to depend on libcufile.so, while the Muxi MACA software stack did not provide the corresponding dynamic library file.

[0059] Solution: Disable the USE_CUFILE build option in the CMakeLists.txt file.

[0060] 3. The __assert_fail prefix declaration is incorrect.

[0061] Error message: Error related to __assert_fail occurred when compiling third_party / gloo / cuda_private.cu.

[0062] Root cause analysis: The file third_party / gloo / cuda_private.cu contains the header file c10 / macros / Macros.h, and the definition of __assert_fail is located in c10 / macros / Macros.h. Further analysis reveals that __assert_fail is a macro definition that supports the __host__ and __device__ prefix declarations under the USE_CUDA code branch. The root cause of the error is that the Muxi platform does not support the __host__ and __device__ prefix declarations.

[0063] Solution: Add the USE_MACA compilation option (enabled by default) to the CMakeLists.txt file in the PyTorch root directory. Also, add the compilation macro definition USE_MACA to third_party / gloo / gloo / CMakeLists.txt. Finally, disable the compilation of this branch of code in c10 / macros / Macros.h using the macro USE_MACA.

[0064] 4. The nvperf_host.h header file is missing.

[0065] Error message: "nvperf_host.h: No such file or directory".

[0066] Root cause analysis: The issue stems from the compilation of the `third_party / kineto / libkineto / src / CuptiRangeProfilerApi.cpp` file within the `third_party / kineto` library, which PyTorch depends on. The `CuptiRangeProfilerApi.cpp` source file includes `nvperf_host.h`, and the root cause is that this header file is a CUDA library header file, and there is no corresponding replacement header file in the MuXi MACA software stack.

[0067] Solution: Add the compilation macro definition USE_MACA to third_party / kineto / libkineto / CMakeLists.txt, and then use the USE_MACA macro definition in the CuptiRangeProfilerApi.cpp source file to comment out the code logic that includes the nvperf_host.h header file. It should be noted that when continuing to compile the third_party / kineto library, several similar compilation errors will occur, all of which can be resolved using similar solutions. These will not be listed here individually. 5. NVML interface name mismatch error.

[0068] Error message: "error: 'nvmlSystemGetCudaDriverVersion_v2' was notdeclared".

[0069] Root cause analysis: The error message is located in c10 / cuda / driver_api.h. The root cause is the macro definition mapping relationship in / opt / maca / tools / cu-bridge / include / bridge / sml / nvml_wrapper.h, which is #define nvmlSystemGetDriverVersion mxSmlExSystemGetDriverVersion.

[0070] Solution: Modify the c10 / cuda / driver_api.h file and change nvmlSystemGetCudaDriverVersion_v2 to nvmlSystemGetDriverVersion using the USE_MACA compiler macro definition.

[0071] 6. nvml structure type mismatch error.

[0072] Error message: "error: 'nvmlProcessInfo_v1_t' was not declared".

[0073] Root cause analysis: The error message is located in c10 / cuda / CUDACachingAllocator.cpp. The root cause is also found to be the macro definition mapping relationship in / opt / maca / tools / cu-bridge / include / bridge / sml / nvml_wrapper.h, which is #define nvmlProcessInfo_t mxsmlExProcessInfo_t.

[0074] Solution: Modify the c10 / cuda / CUDACachingAllocator.cpp file and change nvmlProcessInfo_v1_t to nvmlProcessInfo_t using the USE_MACA compiler macro definition.

[0075] 7. cudaGetDriverEntryPoint parameter mismatch error.

[0076] Error message: "error: too many arguments to function 'mcError_twcudaGetDriverEntryPoint(const char void , long long unsigned int)".

[0077] Root cause analysis: The error message is located in c10 / cuda / driver_api.cpp. The root cause is that the macro definition mapping in / opt / maca / tools / cu-bridge / include / bridge / runtime / cuda_to_maca_mcr_adaptor.h is #define cudaGetDriverEntryPoint wcudaGetDriverEntryPoint. The wcudaGetDriverEntryPoint() function in / opt / maca / tools / cu-bridge / include / bridge / runtime / cuda_runtime_wrapper.h has 3 interface parameters, while the actual interface call in driver_api.cpp passed 4 parameters.

[0078] Solution: Modify the c10 / cuda / driver_api.cpp file and use the USE_MACA compiler macro to modify the number of parameters passed to the cudaGetDriverEntryPoint function.

[0079] 8. Flash-attention / cutlass dependency error.

[0080] Error message: Compilation error related to cutlass.

[0081] Root cause analysis: The CMakeLists.txt file in the PyTorch root directory has the USE_FLASH_ATTENTION compilation option enabled by default, which will compile the third_party / flash_attention folder. The .cu file and the flash-attention library depend on NVIDIA's cutlass library, which Muxi does not directly support, leading to the error.

[0082] Solution: Disable the USE_FLASH_ATTENTION compilation option in CMakeLists.txt, thereby preventing the compilation of flash-attention related code.

[0083] 9. NCCL configuration code compilation error.

[0084] Error message: "error: 'struct c10d::ProcessGroupNCCL::Options' has no member named 'config'".

[0085] Root cause analysis: The error message is located in csrc / distributed / c10d / ProcessGroupNCCL.cpp. Analysis revealed that the cause is that the NCCL_VERSION_CODE version in torch / csrc / distributed / c10d / NCCLUtils.hpp is lower than the specified version, causing the NCCL_HAS_CONFIG macro to fail to be defined. This, in turn, caused the ncclConfig_t config = NCCL_CONFIG_INITIALIZER code in torch / csrc / distributed / c10d / ProcessGroupNCCL.hpp to fail to compile.

[0086] Solution: Modify the file torch / csrc / distributed / c10d / ProcessGroupNCCL.hpp and use the USE_MACA compiler macro definition to ensure that the code that failed to compile compile compiles successfully.

[0087] 10. The asm inline assembly instruction is not supported.

[0088] Error message: Compilation errors related to ASM in source files such as aten / src / ATen / native / cuda / MemoryAccess.cuh.

[0089] Root cause analysis: The error message is located in aten / src / ATen / native / cuda / MemoryAccess.cuh. It is caused by the inclusion of the above header file when compiling the file torch / csrc / distributed / c10d / symm_mem / CUDASymmetricMemory.cu, which leads to compilation errors of the ASM-related code. The reason is that the Muxi MACA software stack does not support NVIDIA's ASM code.

[0090] Solution: Add the compiler macro definition USE_MACA for torch_cuda to caffe2 / CMakeLists.txt, and then use the USE_MACA compiler macro definition in the MemoryAccess.cuh code to replace the asm code with the code that executes the C++ branch.

[0091] Source code files include: aten / src / ATen / native / cuda / MemoryAccess.cuh, aten / src / ATen / native / cuda / TensorShape.cu, aten / src / ATen / native / quantized / cuda / EmbeddingBag.cu, aten / src / ATen / cuda / AsmUtils.cuh, torc At least one of h / csrc / distributed / c10d / symm_mem / CUDASymmetricMemory-inl.h, aten / src / ATen / native / cuda / MixedDtypesLinear.cu and aten / src / ATen / native / sparse / cuda / SparseSemiStructuredApplyDense.cu.

[0092] 11. ASSERT_HOST_DEVICE_LAMBDA macro error.

[0093] Error message: "error: 'func_t' does not refer to a value."

[0094] Root cause analysis: The error message is located in aten / src / ATen / native / cuda / Loops.cuh. The reason is that ASSERT_HOST_DEVICE_LAMBDA(func_t) is wrong. The definition of the ASSERT_HOST_DEVICE_LAMBDA macro is located in aten / src / ATen / native / cuda / CUDALoops.cuh. The reason for the error is that the code branch that is only supported by CUDA is compiled by default.

[0095] Solution: Define the USE_MACA compiler macro to disable the compilation of this branch of code.

[0096] 12. A cutlass error occurred in MixedDtypesLinear.cu.

[0097] Error message: Compilation error in aten / src / ATen / native / cuda / MixedDtypesLinear.cu.

[0098] Root cause analysis: The reason is that the MixedDtypesLinear.cu file contains header files related to cutlass, while the Muxi MACA software stack does not support cutlass.

[0099] Solution: Use the USE_MACA compiler macro definition to disable the compilation of the cutlass branch code.

[0100] 13. A cutlass error occurs in SparseSemiStructuredApplyDense.cu.

[0101] Error message: Multiple .cu files in the aten / src / ATen / native / sparse / cuda / directory are experiencing compilation errors.

[0102] Root cause analysis: The above file contains cutlass-related header files.

[0103] Solution: Use the USE_MACA compiler macro to disable the compilation of cutlass branch code in these source files.

[0104] 14. Cutlass error in mem_eff_attention.

[0105] Error message: Multiple .cu files in the aten / src / ATen / native / transformers / cuda / mem_eff_attention / directory are experiencing compilation errors.

[0106] Root cause analysis: The above file contains cutlass-related header files.

[0107] Solution: Modify the CMakeLists.txt file in the PyTorch root directory and disable the USE_MEM_EFF_ATTENTION compilation option.

[0108] 15. An asm error occurred in cuda_vectorized_test.cu.

[0109] Error message: Compilation error reported for aten / src / ATen / test / cuda_vectorized_test.cu.

[0110] Root cause analysis: The error message is located in aten / src / ATen / native / cuda / MemoryAccess.cuh, and was caused during the compilation of the file aten / src / ATen / test / cuda_vectorized_test.cu.

[0111] Solution: Modify caffe2 / CMakeLists.txt and add the USE_MACA compiler macro definition to ${test_name} in the file.

[0112] Step S4: Verification test.

[0113] Specifically, after adapting and modifying the above compilation error messages, you can successfully compile native PyTorch v2.8.0 in the Muxi C500 environment and generate a Python wheel package for installation.

[0114] After successfully installing the PyTorch v2.8.0 package, perform basic Python module import tests. For example, execute the `python` command in the terminal first, and then execute the `import torch` command to check if the `torch` module is imported normally. If the `torch` module is imported normally, you can continue to execute the `torch.cuda.is_available()` command to check if the obtained Muxi AI accelerator card device is normal. You can also execute some test cases to verify whether the compiled and adapted PyTorch functional modules are normal. They will not be listed here.

[0115] After completing the basic functional tests of the successfully adapted PyTorch v2.8.0 package, we can test whether the single-card model inference of vllm is normal. Here, we take the inference of the Qwen3-8B model as an example. The startup command is vllm server / path / to / Qwen3-8B. The test found that the vllm model service startup process failed. The error message involved the torch / _inductor related files. After adapting and modifying the code of the torch / _inductor related files, the vllm model service started normally, and the model inference output results were normal after sending a request using the curl command.

[0116] After successfully completing the single-card model inference test of the PyTorch v2.8.0 package, we can test whether the multi-card model inference of vllm works properly. Here, we take the inference test of the Qwen3-32B model in a 2-card environment as an example. The startup command is vllm serve / path / to / Qwen3-32B -tp 2 --dtype bfloat16. The test found that the vllm model service startup failed, and the error message was that NCCL does not support bfloat16. After analysis, the reason was that the default macro definition in torch / csrc / cuda / nccl.h failed to compile the code #define HAS_NCCL_BF16_DATATYPE 1. The solution is to modify the torch / csrc / cuda / nccl.h file and use the USE_MACA macro to enable the code #define HAS_NCCL_BF16_DATATYPE1 to compile successfully, thus resolving the above error. After that, the vllm model service started normally in the 2-card environment, and the inference results were normal.

[0117] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A PyTorch adaptation method based on the domestically produced Muxi AI accelerator card, characterized in that, include: MACA Software Stack Environment Variable Configuration: Install the Muxi MACA software stack, which includes the compiler mxcc for the Muxi AI accelerator card and high-performance computing libraries, communication libraries, and runtime libraries compatible with the CUDA ecosystem. Configure the first system environment variables to ensure that the compiler, high-performance computing libraries, communication libraries, and runtime libraries can be correctly retrieved. The first system environment variables include PATH and LD_LIBRARY_PATH. cu-bridge compiler tool escape layer environment variable configuration: Obtain the Muxi cu-bridge compiler tool, which includes the cmake_maca script, the cucc script, and a set of header files that escape the CUDA Toolkit identifier to the Muxi MACA software stack identifier through macro definitions. The cu-bridge compiler tool creates a simulated CUDA toolkit installation directory and sets the second system environment variables. The cucc script is used to redirect nvcc calls to the compiler mxcc. The cu-bridge compiler tool maps CUDA dependency libraries to the corresponding libraries in the Muxi MACA software stack through dynamic library soft links. The second system environment variables include PATH, LD_LIBRARY_PATH, CUDA_PATH, CMAKE_CUDA_COMPILER, NCCL_INCLUDE_DIR, and CUDNN_INCLUDE_DIR. PyTorch source code compilation and adaptation: Obtain the official PyTorch source code, use the cmake_maca script to replace the system cmake command to perform compilation and build, compile the source code based on the configured MACA software stack environment and cu-bridge escape wrapper layer environment, analyze the compilation error information one by one during the compilation process, determine the root cause of each compilation error and adopt the corresponding source code adaptation and modification scheme until the compilation is completely successful, generating a Python program package that can run in the Muxi AI accelerator card environment.

2. The PyTorch adaptation method based on the domestically produced Muxi AI accelerator card according to claim 1, characterized in that, In the MACA software stack environment configuration, the PATH includes the / opt / maca / bin path and the / opt / maca / mxgpu_llvm / bin path of the MACA software stack; The LD_LIBRARY_PATH includes the / opt / maca / lib64 and / opt / maca / mxgpu_llvm / lib paths of the MACA software stack.

3. The PyTorch adaptation method based on the domestically produced Muxi AI accelerator card according to claim 1, characterized in that, In the cu-bridge compiler's escape wrapper layer environment variable configuration, the PATH is appended to the cu-bridge compiler's path / opt / maca / tools / cu-bridge / tools; The CUDA_PATH is set to simulate the installation directory path of the CUDA toolkit. The bin subdirectory under this directory contains nvcc symbolic links pointing to the cucc script, the lib64 subdirectory contains symbolic links from the CUDA library files to the corresponding dynamic libraries of the Muxi MACA software stack, and the include subdirectory links to the header file directory of the cu-bridge compilation tool. The CMAKE_CUDA_COMPILER is set to the path / root / cu-bridge / CUDA_DIR / bin / nvcc of the cu-bridge compilation tool; The NCCL_INCLUDE_DIR and CUDNN_INCLUDE_DIR are set to the / root / cu-bridge / include path of the cu-bridge compilation tool.

4. The PyTorch adaptation method based on the domestically produced Muxi AI accelerator card according to claim 1, characterized in that, The adaptation and modification schemes for compiling and adapting PyTorch source code include: Add the USE_MACA build option to the CMakeLists.txt file in the root directory of the PyTorch source code and disable incompatible build options; Add the USE_MACA compiler macro definition to the CMakeLists.txt files in multiple subdirectories of the PyTorch source code. This macro is used to conditionally compile incompatible code branches or enable compatible code branches in the corresponding source code files.

5. The PyTorch adaptation method based on the domestically produced Muxi AI accelerator card according to claim 4, characterized in that, The options for disabling incompatible compilations include: Disable the USE_CUFILE build option in the CMakeLists.txt file and / or disable the USE_FLASH_ATTENTION build option in the CMakeLists.txt file and / or disable the USE_MEM_EFF_ATTENTION build option in the CMakeLists.txt file.

6. The PyTorch adaptation method based on the domestically produced Muxi AI accelerator card according to claim 4, characterized in that, The method of shielding incompatible code branches through conditional compilation includes: In the CuDNN version detection module, the FindCUDNN.cmake file is modified to directly return the version number that meets the PyTorch version requirements, so as to adapt to the CuDNN version identifier escaped in the header file of the cu-bridge compiler. In the cuda_private.cu file of the third-party library gloo, the USE_MACA compiler macro definition masks the __host__ __device__ prefix declaration branch that contains the __assert_fail macro in c10 / macros / Macros.h; In the CuptiRangeProfilerApi.cpp file of the third-party library kineto, the code logic that includes the nvperf_host.h header file is masked by the USE_MACA compiler macro definition. In the c10 / cuda / driver_api.h and CUDACachingAllocator.cpp files, the USE_MACA compilation macro definition adapts the nvml interface name to the macro mapping name defined in the cu-bridge header file; In the c10 / cuda / driver_api.cpp file, the number of input parameters to the cudaGetDriverEntryPoint function is adjusted by the USE_MACA compiler macro definition to adapt to the function interface parameters defined in the cu-bridge header file.

7. The PyTorch adaptation method based on the domestically produced Muxi AI accelerator card according to claim 4, characterized in that, The method of shielding incompatible code branches through conditional compilation also includes: In source files containing asm inline assembly instructions, the asm instruction branches are replaced with C++ implementation branches using the USE_MACA compiler macro definition; In source code files that include cutlass-related header files, the USE_MACA compiler macro is used to disable the compilation of the cutlass branch code.

8. The PyTorch adaptation method based on the domestically produced Muxi AI accelerator card according to claim 7, characterized in that, The source code files include at least one of the following: aten / src / ATen / native / cuda / MemoryAccess.cuh, aten / src / ATen / native / cuda / TensorShape.cu, aten / src / ATen / native / quantized / cuda / EmbeddingBag.cu, aten / src / ATen / cuda / AsmUtils.cuh, torch / csrc / distributed / c10d / symm_mem / CUDASymmetricMemory-inl.h, aten / src / ATen / native / cuda / MixedDtypesLinear.cu, and aten / src / ATen / native / sparse / cuda / SparseSemiStructuredApplyDense.cu.

9. A PyTorch adaptation method based on the domestically produced Muxi AI accelerator card according to claim 4, characterized in that, The method of shielding incompatible code branches through conditional compilation also includes: In the file torch / csrc / distributed / c10d / ProcessGroupNCCL.hpp, the USE_MACA compiler macro definition ensures that the ncclConfig_t configuration code compiles successfully; In the torch / csrc / cuda / nccl.h file, enable the HAS_NCCL_BF16_DATATYPE macro definition via the USE_MACA compilation macro definition; In the code files under the torch / _inductor directory, the USE_MACA compiler macro definition is used for adaptation.

10. A PyTorch adaptation method based on the domestically produced Muxi AI accelerator card according to claim 1, characterized in that, It also includes verification testing: Install the compiled Python package and perform a Python module import test to verify the usability of the torch module and the Muxi AI accelerator card; Large model inference tests were performed to verify the correctness of the compiled and adapted PyTorch in the Muxi AI accelerator card environment. The large model inference tests included single-card model inference tests and multi-card model inference tests.

Images