Artificial intelligence accelerator card heterogeneous cluster adaptation method and system and storage medium

Abstract

The invention relates to an artificial intelligence accelerator card heterogeneous cluster adaptation method and system and a storage medium, and the method comprises the steps: obtaining a unified standardized adaptation module registered according to a private keyword provided by a training framework; the training framework is a training framework of a heterogeneous cluster model; accessing a hardware layer software stack of each heterogeneous acceleration card through a factory mode in the unified standardized adaptation module; a calculation task distribution mechanism of the training framework is expanded in a plug-in form, so that the training framework autonomously optimizes a target heterogeneous accelerator card; performing condition compiling according to an operation environment where the unified standardized adaptation module is located so as to access a target heterogeneous accelerator card and be used for training framework autonomous optimization; and according to the unified test module, the adaptation state of the unified standardized adaptation module and each target heterogeneous acceleration card is verified, and the problems that the artificial intelligence acceleration card heterogeneous cluster adaptation development cost is high and autonomous optimization cannot be achieved are solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, system and storage medium for adapting a heterogeneous cluster of artificial intelligence accelerator cards. Background Art

[0002] With the rapid development of graphics card technology, many manufacturers have launched high-performance computing solutions. However, these manufacturers face many challenges when connecting to PyTorch (deep learning framework), a mainstream deep learning framework. At present, each manufacturer needs to independently develop an adaptation layer, which not only leads to a lot of duplication of work, but also due to the lack of unified standards, the adaptation methods vary, which greatly increases the difficulty of management and coordination between heterogeneous frameworks. At the same time, due to the coupling of manufacturer code and framework code, whether the code version of the manufacturer or the framework changes, a new round of adaptation and verification work will be introduced.

[0003] In addition, the scale of heterogeneous clusters required for large models is getting larger and larger. Clusters composed of a single AI accelerator card are limited by production capacity and cost, and heterogeneous clusters composed of different AI accelerator cards are attracting more and more attention. At present, the training of large models on heterogeneous clusters is mainly achieved by developing a new training framework for different AI accelerator card manufacturers, or intrusively modifying frameworks such as PyTorch and TVM (TensorVirtual Machine), resulting in high development and maintenance costs. In addition, in a heterogeneous environment, it is still impossible to autonomously optimize which AI accelerator card to use for the same training task. Currently, it is mainly through manual configuration and tuning, while Microsoft's Copilot+PCs (Microsoft artificial intelligence personal computers) can only select one AI accelerator card for one task through power mode switching or manual selection.

[0004] There is currently no effective solution to the problems of high development costs for adapting heterogeneous clusters of AI accelerator cards and the inability to find optimal solutions independently in related technologies. Summary of the invention

[0005] In this embodiment, a method, system and storage medium for adapting a heterogeneous cluster of artificial intelligence accelerator cards are provided to solve the problems of high development cost and inability to independently optimize the adaptation of heterogeneous clusters of artificial intelligence accelerator cards in related technologies.

[0006] In the first aspect, a method for adapting a heterogeneous cluster of artificial intelligence accelerator cards is provided in this embodiment, including: Obtain a unified standardized adaptation module registered according to a private keyword provided by a training framework; the training framework is a training framework for a heterogeneous cluster model; the unified standardized adaptation module is registered to the training framework through the private keyword provided by the training framework, and is used to connect the hardware layer software stack of each heterogeneous acceleration card to the training framework; In a unified standardized adapter module, the hardware layer software stack of each heterogeneous accelerator card is accessed through the factory mode; The computing task distribution mechanism of the training framework is expanded through plug-ins, so that the training framework can autonomously optimize the target heterogeneous accelerator card; Conditional compilation is performed according to the operating environment of the unified standardized adapter module to access the target heterogeneous accelerator card and use it for autonomous optimization of the training framework; The unified test module verifies the adaptation status of the unified standardized adaptation module and each target heterogeneous accelerator card.

[0007] In some of the embodiments, obtaining a unified standardized adaptation module registered according to a private keyword provided by a training framework includes: Obtain a unified standardized adaptation module registered according to private keyword 1, private keyword 2, and private keyword 3 provided by the training framework.

[0008] In some of the embodiments, obtaining a unified standardized adaptation module registered according to a private keyword provided by a training framework includes: At least unify the interface name, parameters, implementation method and calling logic of the adapter modules provided by the heterogeneous accelerator cards to obtain a unified standardized adapter module; According to the private keywords provided by the training framework, the unified standardized adaptation module is registered into the training framework.

[0009] In some of the embodiments, in a unified standardized adapter module, the hardware layer software stack of each heterogeneous accelerator card is accessed through a factory mode, including: Define abstract classes corresponding to the functions and sub-functions of the heterogeneous accelerator cards, and encapsulate the common underlying implementation of the heterogeneous accelerator cards through the abstract classes; wherein the abstract classes contain multiple virtual functions, and the multiple virtual functions are used to encapsulate the functional interfaces of the accelerator cards; Create a factory class to manage the abstract class. Each heterogeneous accelerator card implements a specific factory, which inherits from the abstract factory. Rewrite the virtual functions in the abstract class to access the hardware layer software stack of each heterogeneous accelerator card; The specific factories of each heterogeneous accelerator card are registered into a unified standardized adapter module through the registration mechanism.

[0010] In some of the embodiments, the computing task distribution mechanism of the training framework is extended in the form of a plug-in, including: The operator adaptation interface of the training framework is accessed and expanded in the form of plug-ins, and the operator adaptation interfaces of various heterogeneous accelerator cards are unified to obtain a unified operator adaptation interface. Expand the training framework through plug-ins to support the integration of self-developed operators and compilers into the training framework; The operator registration and distribution mechanism is extended to the target heterogeneous accelerator card, and a search recommendation algorithm is added to the operator call module based on the limitations of operator performance, video memory occupancy, and communication time, so that the training framework can autonomously optimize the target heterogeneous accelerator card.

[0011] In some of the embodiments, conditional compilation is performed according to the operating environment where the unified standardized adaptation module is located to access the target heterogeneous accelerator card and use it for autonomous optimization of the training framework, including: Based on the hardware environment and compilation requirements of the unified standardized adapter module, the target heterogeneous accelerator card is polled and connected to the training framework.

[0012] In some of the embodiments, obtaining a unified test module and verifying the adaptation status of the unified standardized adaptation module and each target heterogeneous accelerator card according to the unified test module include: Obtain a unified test module, perform unit tests on various heterogeneous accelerator cards at the same time according to the unified test module, or perform large model training tests, obtain test results, and adjust and optimize the unified standardized adaptation module based on the test results.

[0013] In a second aspect, in this embodiment, a heterogeneous cluster adaptation system for artificial intelligence accelerator cards is provided, and the system includes: An acquisition module is used to acquire a unified standardized adaptation module registered according to a private keyword provided by a training framework; the training framework is a training framework for a heterogeneous cluster model; the unified standardized adaptation module is registered to the training framework through the private keyword provided by the training framework, and is used to connect the hardware layer software stack of each heterogeneous accelerator card to the training framework; The accelerator card access module is used to access the hardware layer software stack of various heterogeneous accelerator cards through the factory mode in a unified standardized adapter module; Extension module, which is used to extend the computing task distribution mechanism of the training framework in the form of plug-ins, so that the training framework can autonomously optimize the target heterogeneous accelerator card; The compilation module is used to perform conditional compilation according to the operating environment of the unified standardized adapter module, so as to access the target heterogeneous accelerator card and be used for autonomous optimization of the training framework; And a test module is used to verify the adaptation status of the unified standardized adaptation module and each target heterogeneous accelerator card.

[0014] In a third aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the method for heterogeneous cluster adaptation of artificial intelligence acceleration cards described in the first aspect is implemented.

[0015] Compared with the related art, the artificial intelligence acceleration card heterogeneous cluster adaptation method provided in this embodiment includes: obtaining a unified standardized adaptation module registered according to a private keyword provided by a training framework; the training framework is a training framework for a heterogeneous cluster model; the unified standardized adaptation module is registered to the training framework through the private keyword provided by the training framework, and is used to connect the hardware layer software stack of each heterogeneous acceleration card to the training framework; in the unified standardized adaptation module, the hardware layer software stack of each heterogeneous acceleration card is connected through the factory mode; the computing task distribution mechanism of the training framework is extended in the form of a plug-in, so that the training framework can autonomously optimize the target heterogeneous acceleration card; conditional compilation is performed according to the operating environment where the unified standardized adaptation module is located, so as to connect to the target heterogeneous acceleration card and use it for autonomous optimization of the training framework; the adaptation status of the unified standardized adaptation module and each target heterogeneous acceleration card is verified according to the unified test module, which solves the problems of high development cost and inability to autonomously optimize the adaptation of the heterogeneous cluster of artificial intelligence acceleration cards.

[0016] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 This is a hardware structure block diagram of a terminal of the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment; Figure 2 is a flow chart of the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment; Figure 3 1 is an architectural diagram of the artificial intelligence accelerator card heterogeneous cluster adaptation method of this embodiment; Figure 4 This is a schematic diagram of the adaptation process of the heterogeneous cluster of artificial intelligence acceleration cards in this embodiment; Figure 5 is a compilation flow chart of the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment; Figure 6a This is a schematic diagram of a single-node homogeneous cluster and a multi-node heterogeneous cluster corresponding to the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment; Figure 6bThis is a schematic diagram of a single-node heterogeneous cluster corresponding to the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment; Figure 7 This is a schematic diagram of the framework of the artificial intelligence acceleration card heterogeneous cluster adaptation system of this embodiment; Figure 8 It is a timing diagram of the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment. DETAILED DESCRIPTION

[0018] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0019] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0020] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 : is a hardware structure block diagram of a terminal of the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0021] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the artificial intelligence acceleration card heterogeneous cluster adaptation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0022] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0023] In this embodiment, a method for adapting a heterogeneous cluster of artificial intelligence accelerator cards is provided. Figure 2 : is a flow chart of the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment. Figure 2 As shown, the process includes the following steps: Step S201, obtain a unified standardized adaptation module registered according to the private keywords provided by the training framework; the training framework is a training framework for a heterogeneous cluster model; the unified standardized adaptation module is registered to the training framework through the private keywords provided by the training framework, and is used to connect the hardware layer software stack of each heterogeneous acceleration card to the training framework.

[0024] Specifically, the PyTorch framework recently proposed identifiers such as PrivateUse1 (private keyword 1), PrivateUse2 (private keyword 2), and PrivateUse3 (private keyword 3) as universal keywords for identifying the backend of hardware devices, aiming to promote platform access and avoid intrusive modification of the PyTorch framework code by manufacturers. Among them, the accelerator cards provided by various manufacturers have been successfully connected to the PyTorch framework through the PrivateUse1 mechanism, but due to the diversity of adaptation codes and architectures, there are still certain fragmentation problems in actual operations. For example, although the upper-level framework and underlying registration of hardware devices use a unified PrivateUse1 identifier, in the adaptation layer code, each manufacturer still uses its own naming rules, introduces personalized implementation logic and software call stacks, and maps back to the manufacturer name through the renaming mechanism provided by the PyTorch framework, thereby realizing the access of the accelerator card. This process increases the complexity of the access process. Therefore, in this embodiment, a unified standardized adaptation module is separately constructed in the adaptation layer to achieve the unification of different implementation logics and software call stacks of various manufacturers.

[0025] Take, for example, running a large model training task on a heterogeneous cluster consisting of an AI accelerator card MLU (Machine Learning Unit), an AI accelerator card MUSA (Moore Threads Unified System Architecture), and an AI accelerator card HPU (Homemade Processing Unit). Figure 3 is an architectural diagram of the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment, such as Figure 3 As shown, from top to bottom, it is divided into: application layer, training framework, adaptation layer and heterogeneous cluster. Among them, the application layer is mainly the mainstream large models, such as GPT, LLAMA and Qwen, etc., and the specific large models are not specifically limited in this embodiment. The parallel strategies, mixed precision, recalculation and other technologies of these mainstream large model training are mainly based on the PyTorch framework. Therefore, in this embodiment, the training framework adopts the PyTorch framework to reduce the migration cost of the technology.

[0026] At the adaptation layer, since the adaptation of the three AI accelerator cards (MLU, MUSA and HPU) to the PyTorch framework is not consistent, the AI ​​accelerator card MLU and the AI ​​accelerator card MUSA both support the PyTorch framework, but the adaptation code of the AI ​​accelerator card MLU only supports PyTorch versions 2.1, 2.3 and 2.4, while the adaptation code of the AI ​​accelerator card MUSA supports PyTorch versions no greater than version 2.2. There are also considerable differences in the interface names, parameters, implementation methods and calling logic in the adaptation codes corresponding to the AI ​​accelerator cards MLU and MUSA, and intrusive modifications have been introduced to the PyTorch framework. In addition, the accelerator card HPU has not yet adapted to the PyTorch framework. Starting from PyTorch 2.1, PyTorch officially announced backend keywords such as PrivateUse1 (private keyword 1) to provide non-invasive access to AI accelerator card manufacturers. Although some manufacturers have already adopted this method to access PyTorch, their adaptation code interface names and call stacks are also very different, and they are only applicable to their own products. Therefore, this embodiment develops a set of unified standardized adaptation codes registered to the PyTorch framework based on the private keywords PrivateUse1 (keyword 1), PrivateUse2 (keyword 2) and PrivateUse3 (keyword 3) provided by the PyTorch framework in the adaptation layer. On the basis of not making invasive modifications to PyTorch, the interface names and code call stack logic of the adaptation codes of various manufacturers are unified, and the PyTorch version switching and the access of multiple AI accelerator cards are automatically followed, reducing the workload of AI accelerator card manufacturers to adapt the PyTorch framework, while facilitating the maintenance of large model training of heterogeneous clusters.

[0027] The unified standardized adaptation module includes three parts: standardized device interface, extension module and compilation module. Figure 3 As shown in the figure, the standardized device interface module includes the adaptation of functional modules such as device, event, stream, tensor, storage, memory, log, error handling, context, etc. The extension module includes the unification of operators and communication interfaces; the extension of operator distribution mechanism; the custom operator and compiler access functional modules; the compilation module includes the automatic environment detection, automatic selection of compilation, and polling access to the underlying hardware and software stack functional modules.

[0028] A heterogeneous cluster is a cluster of heterogeneous acceleration cards, including the three types of AI acceleration cards mentioned above, such as MLU, MUSA, and HPU.

[0029] Step S202, in a unified standardized adaptation module, accessing the hardware layer software stack of each heterogeneous accelerator card through a factory mode; Specifically, the standardized device interface is the first part of the unified standardized adaptation module of the adaptation layer, and is connected to the hardware layer software stack of each AI accelerator card through registration and factory mode. The hardware layer of the AI ​​accelerator card is abstracted, and corresponding abstract classes are defined for each function and sub-function to encapsulate the common underlying implementation of different AI accelerator cards. Each abstract class contains multiple virtual functions, which are used to encapsulate the functional interfaces specific to each accelerator card. A factory class is created to manage the abstract class. Each AI accelerator card creates a specific factory instance that inherits the abstract class. When implementing unique functions, the virtual functions in the abstract class are rewritten to access the hardware layer software stack of the AI ​​accelerator card. A registration mechanism is introduced to dynamically access the specific implementations of different types of AI accelerator cards, allowing developers to expand support for more types of AI accelerator cards by simply registering new hardware implementations without modifying existing code.

[0030] Step S203, expanding the computing task distribution mechanism of the training framework in the form of a plug-in, so that the training framework can autonomously optimize the target heterogeneous accelerator card.

[0031] like Figure 3 As shown in the figure, the extension module includes (1) unification of interfaces such as operators, communication, and graph fusion; (2) extension of operator adaptation and distribution mechanism; and (3) custom operator and compiler adaptation. Figure 4 Schematic diagram of the AI ​​accelerator card heterogeneous cluster adaptation process in this embodiment. Figure 4 As shown in the figure, the extension module is connected to the PyTorch framework through private keywords and C++ extensions. The extension module unifies the names and input and output of the heterogeneous AI accelerator card operator interfaces, and reserves the adaptation of custom operator interfaces. The computing tasks can be distributed to any AI accelerator card in the node where the framework is located through the extended distribution mechanism. The search recommendation algorithm is introduced to automatically optimize the applicable heterogeneous AI accelerator card operators according to the requirements of operator performance, video memory occupancy, communication time, etc., or customized AI accelerator universal operators such as triton operators. Specifically, the schematic code of one implementation method is given below.

[0032] The extension module accesses the PyTorch framework code through C++ extension as follows: from setuptools import setup from torch.utils.cpp_extension import BuildExtension, CppExtension # Initialize the extension module list ext_modules=[] # Add C++ extension module # Define an extension module named "privateuseone_ops", the source file is "privateuseone_ops.cpp" ext_modules.append(CppExtension('privateuseone_ops', ['privateuseone_ops.cpp'])) # Call the setup function to configure and build the extension module code as follows: setup( name='torch', # package name ext_modules=ext_modules, # List of extension modules that need to be compiled cmdclass={ 'build_ext': BuildExtension # Use BuildExtension to build C++ extension }) The code for registering to the training framework through private keywords is as follows: TORCH_LIBRARY_IMPL(ops, PrivateUse1, m){ # Register the implementation of the "native_ops_pu1" operation for the PrivateUse1 device type m.impl("native_ops_pu1",&native_ops_pu1); } TORCH_LIBRARY_IMPL(ops, PrivateUse2, m){ # Register the implementation of the "native_ops_pu2" operation for the PrivateUse2 device type m.impl("native_ops_pu2",&native_ops_pu2); } TORCH_LIBRARY_IMPL(ops, PrivateUse3, m){ # Register the implementation of the "native_ops_pu3" operation for the PrivateUse3 device type m.impl("native_ops_pu3",&native_ops_pu3); } The unified operator adaptation interface and operator adaptation extension code are as follows: at::Tensor&native_ops_pu1(at::Tensor&self){ return mlu_ops(self); # native_ops_pu1 corresponds to the MLU operator implementation } at::Tensor&native_ops_pu2(at::Tensor&self){ return musa_ops(self); # native_ops_pu2 corresponds to the MUSA operator implementation } at::Tensor&native_ops_pu3(at::Tensor&self){ return hpu_ops(self); # native_ops_pu3 corresponds to the MUSA operator implementation } The extended distribution code is as follows: # Define a function called "native_ops" that takes a tensor as input and returns a tensor # variants: function means this is a normal function # dispatch: defines the specific implementation of different device types that can be distributed -func: native_ops(Tensor input)->(Tensor) variants: function dispatch: # When you need to distribute to the CPU device, call the "native_ops_cpu" function CPU: native_ops_cpu # When you need to distribute to the device corresponding to privateuse1, call the "native_ops_pu1" function PRIVATEUSE1: native_ops_pu1 # When you need to distribute to the device corresponding to privateuse2, call the "native_ops_pu2" function PRIVATEUSE2: native_ops_pu2 # When you need to distribute to the device corresponding to privateuse3, call the "native_ops_pu3" function PRIVATEUSE3: native_ops_pu3 Operator calling code, the automatic tuning algorithm is implemented in torch.jit.script in this example: @torch.jit.script def native_function(x: torch.Tensor)->torch.Tensor: return native_ops(x) The content following the “#” symbol indicates a code comment.

[0033] Step S204, conditional compilation is performed according to the operating environment where the unified standardized adaptation module is located, so as to access the target heterogeneous acceleration card and be used for autonomous optimization of the training framework.

[0034] Specifically, the compilation module compiles the model into a format that can run on an AI accelerator card, detects the current hardware environment to determine which type of AI accelerator card needs to be connected, and polls the corresponding AI accelerator card based on the detection results. Figure 5 is a compilation flow chart of the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment, such as Figure 5 As shown, the compilation process includes: Step S501, start; Step S502, automatically detect the heterogeneous environment; if the user specifies the version, execute step S503, otherwise, execute step S504; Step S503, compile the user specified version; Specifically, if the user specifies, for example, to compile only the CPU version, the compilation script will only compile the CPU version of the PyTorch framework.

[0035] Step S504, use lspci to check the AI ​​accelerator card; Specifically, if the user does not specify, the compilation script will use lspci to check the AI ​​accelerator card connected to the current node, and scan whether there is a toolkit corresponding to the AI ​​accelerator card in the / usr / local path. If so, the AI ​​accelerator card environment of the current node can be determined, and the corresponding hardware layer software stack code can be adapted to the PyTorch framework based on the detected hardware type. If there is no toolkit corresponding to the AI ​​accelerator card, only the CPU version will be compiled.

[0036] Step S505, end.

[0037] In this way, based on step S501 to step S505, conditional compilation according to the operating environment where the unified standardized adaptation module is located is completed.

[0038] Step S205: verifying the adaptation status of the unified standardized adaptation module and each target heterogeneous accelerator card according to the unified test module.

[0039] Specifically, in order to ensure the correctness and stability of the adaptation code, this embodiment also provides a unified test module, which is a set of unified test codes. The test code covers all key functions and interfaces, including device initialization, data transmission, operator execution, memory management, etc. By running these test codes, the performance of the adaptation code on different AI accelerator cards is verified, so as to discover and fix potential problems of the AI ​​accelerator cards, and provide conditions for performance comparison of AI accelerator cards. Among them, the test code includes unit testing, integration testing, and performance testing to comprehensively evaluate the functionality and performance of the adaptation code. The sample test code is as follows (test_native_ops.py): import torch import unittest def get_available_devices(): devices = [] # Define available accelerator cards, pu is the abbreviation of private use available_devices = ['pu1', 'pu2', 'pu3'] # Poll to get the list of available accelerator cards for device in available_devices: if getattr(torch, device).is_available() and getattr(torch,device).device_count()>0: devices.append(device) return devices def test_native_op(device): def test(self): # Create a random tensor and move it to the specified device x = torch.randn(10, 10).to(device) # Call native operations result = native_op(x) # Assert that the result is not None self.assertIsNotNone(result) return test class TestNativeOps(unittest.TestCase): self.available_devices = get_available_devices() # Dynamically create test methods for device in get_available_devices(): locals()[f'test_{device}_ops'] = test_native_op(device) if __name__ == '__main__': # Returns unit test execution status, success rate and execution time unittest.main() The artificial intelligence acceleration card heterogeneous cluster adaptation method provided in this embodiment is not only applicable to single-node homogeneous and multi-node heterogeneous clusters, but also to single-node heterogeneous clusters. Figure 6a : is a schematic diagram of a single-node homogeneous and multi-node heterogeneous cluster corresponding to the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment, such as Figure 6a As shown in the figure, an AI accelerator card (MLU or MUSA or HPU) is connected to a central processing unit (CPU, here representing a server) through PCIE (high-speed serial computer expansion bus) to form an AI accelerator card homogeneous node, and nodes connected to different accelerator cards are interconnected through MPI (message passing interface) to form a multi-node heterogeneous cluster; Figure 6b : is a schematic diagram of a single-node heterogeneous cluster corresponding to the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment, such as Figure 6b As shown in the figure, multiple different AI accelerator cards are connected to the same server through PCIE to form a single-node heterogeneous cluster.

[0040] Through the above steps S201 to S205, a unified standardized adaptation module registered according to the private keywords provided by the training framework is obtained; the training framework is a training framework for a heterogeneous cluster model; the unified standardized adaptation module is registered to the training framework through the private keywords provided by the training framework, and is used to connect the hardware layer software stack of each heterogeneous acceleration card to the training framework; in the unified standardized adaptation module, the hardware layer software stack of each heterogeneous acceleration card is connected through the factory mode; the computing task distribution mechanism of the training framework is extended in the form of a plug-in, so that the training framework can autonomously optimize the target heterogeneous acceleration card; conditional compilation is performed according to the operating environment where the unified standardized adaptation module is located, so as to connect to the target heterogeneous acceleration card and use it for autonomous optimization of the training framework; and the adaptation status of the unified standardized adaptation module and each target heterogeneous acceleration card is verified according to the unified test module. Compared with the prior art that requires separate development of different adaptation layers for different AI accelerator cards or intrusive modification of the training framework to adapt the training framework, this embodiment provides a unified and standardized adaptation code for heterogeneous cluster model training based on the unified and standardized adaptation module registered with private keywords provided by the training framework, which can achieve the adaptation of various heterogeneous accelerator cards without intrusive modification of the training framework, and expand the computing task distribution mechanism of the training framework in the form of plug-ins, thereby achieving the expansion of accessible AI accelerator cards and automatic optimization of AI accelerator cards, reducing the development cost of heterogeneous cluster adaptation of artificial intelligence accelerator cards, and improving the flexibility of large model training. Adaptation verification is performed through a unified test module to ensure the consistency and repeatability of adaptation, and automatically detect compatibility issues for timely adjustment and optimization.

[0041] In one embodiment, obtaining a unified standardized adaptation module registered according to a private keyword provided by a training framework includes: Obtain a unified standardized adaptation module registered according to private keyword 1, private keyword 2, and private keyword 3 provided by the training framework.

[0042] Specifically, the PyTorch framework recently proposed identifiers such as PrivateUse1 (private keyword 1), PrivateUse2 (private keyword 2), and PrivateUse3 (private keyword 3) as common keywords for identifying the backend of hardware devices, aiming to promote platform-based access and avoid intrusive modifications of the PyTorch framework code by manufacturers. Among them, the accelerator cards provided by various manufacturers have been successfully connected to the PyTorch framework through the PrivateUse1 mechanism, but due to the diversity of adaptation codes and architectures, there are still certain fragmentation problems in actual operations. For example, although the upper-level framework and underlying registration of hardware devices use a unified PrivateUse1 identifier, in the adaptation layer code, each manufacturer still uses its own naming rules, introduces personalized implementation logic and software call stacks, and maps back to the manufacturer name through the renaming mechanism provided by the PyTorch framework, thereby achieving access to the accelerator card. This process increases the complexity of the access process. Therefore, this embodiment develops a set of unified standardized adaptation codes registered to the PyTorch framework according to the private keywords PrivateUse1 (keyword 1), PrivateUse2 (keyword 2) and PrivateUse3 (keyword 3) provided by the PyTorch framework in the adaptation layer. On the basis of not making invasive modifications to PyTorch, the interface names and code call stack logic of the adaptation codes of various manufacturers are unified, and the switching of PyTorch versions and the access of multiple AI accelerator cards are automatically followed, reducing the workload of AI accelerator card manufacturers in adapting to the PyTorch framework, and facilitating the maintenance of large model training of heterogeneous clusters, so as to achieve the unification of different implementation logics and software call stacks of various manufacturers.

[0043] In another embodiment, obtaining a unified standardized adaptation module registered according to a private keyword provided by a training framework includes: At least the interface names, parameters, implementation methods and calling logic of the adapter modules provided by the heterogeneous acceleration cards are unified to obtain a unified standardized adapter module; according to the private keywords provided by the training framework, the unified standardized adapter module is registered in the training framework.

[0044] Specifically, define a unified interface standard for each heterogeneous accelerator card to ensure that the adapter modules of different hardware platforms can be developed in accordance with the same specifications, and unify the parameters, implementation methods and calling logic of the adapter modules provided by the heterogeneous accelerator cards. Implement specific adaptation logic in the adapter module, including but not limited to hardware initialization, task scheduling, performance optimization, etc. According to the defined unified interface, develop adapter modules, which will encapsulate the implementation details of each heterogeneous accelerator card. Use the private keywords provided by the training framework to register the developed adapter module to the training framework. Ensure that the adapter module can be recognized and called by the training framework, so that hardware from different manufacturers can be more easily integrated with the PyTorch framework, improve adaptation efficiency, and reduce the cost of manufacturers during adaptation and maintenance.

[0045] In some of the embodiments, in a unified standardized adapter module, the hardware layer software stack of each heterogeneous accelerator card is accessed through a factory mode, including: Define abstract classes corresponding to the functions and sub-functions of the heterogeneous accelerator cards, and encapsulate the common underlying implementation of the heterogeneous accelerator cards through the abstract classes; wherein the abstract classes contain multiple virtual functions, and the multiple virtual functions are used to encapsulate the functional interfaces of the accelerator cards; Create a factory class and manage the abstract class through the factory class. Each heterogeneous accelerator card implements a specific factory, which inherits from the abstract factory. Rewrite the virtual functions in the abstract class to access the hardware layer software stack of each heterogeneous accelerator card. Register the specific factory of each heterogeneous accelerator card in the unified standardized adaptation module through the registration mechanism.

[0046] Specifically, the standardized device interface is the first part of the unified standardized adaptation module of the adaptation layer, and is connected to the hardware layer software stack of each AI accelerator card through registration and factory mode. The hardware layer of the AI ​​accelerator card is abstracted, and corresponding abstract classes are defined for each function and sub-function to encapsulate the common underlying implementation of different AI accelerator cards. Each abstract class contains multiple virtual functions, which are used to encapsulate the functional interfaces specific to each accelerator card. A factory class is created to manage the abstract class. Each AI accelerator card creates a specific factory instance that inherits the abstract class. When implementing unique functions, the virtual functions in the abstract class are rewritten to access the hardware layer software stack of the AI ​​accelerator card. A registration mechanism is introduced to dynamically access the specific implementations of different types of AI accelerator cards, allowing developers to expand support for more types of AI accelerator cards by simply registering new hardware implementations without modifying existing code.

[0047] In another embodiment, the computing task distribution mechanism of the training framework is extended in the form of a plug-in, including: The operator adaptation interface of the training framework is accessed and expanded in the form of plug-ins, and the operator adaptation interfaces of various heterogeneous accelerator cards are unified to obtain a unified operator adaptation interface. The training framework is expanded in the form of plug-ins to support the access of self-developed operators and compilers to the training framework. The operator registration and distribution mechanism is extended to the target heterogeneous accelerator card, and the search recommendation algorithm is added to the operator calling module according to the limitations of operator performance, video memory occupancy and communication time, so that the training framework can autonomously optimize the target heterogeneous accelerator card.

[0048] Specifically, Figure 3 As shown in Figure 1, the extension module includes (1) unification of adaptation interfaces such as operators, communications, and graph fusion; (2) extension of operator adaptation and distribution mechanisms; and (3) adaptation of custom operators and compilers. Figure 4 As shown in the figure, the extension module is connected to the PyTorch framework through private keywords and C++ extensions. The extension module unifies the names and input and output of the heterogeneous AI accelerator card operator adapter interface, and reserves a custom operator adapter interface. The computing task can be distributed to any AI accelerator card in the node where the framework is located through the extended distribution mechanism, that is, the registration distribution mechanism of the extended operator is extended to the target heterogeneous accelerator card, and an autonomous optimization algorithm is added to the operator call module. The search recommendation algorithm is introduced to automatically optimize the applicable heterogeneous AI accelerator card operator according to the operator performance, video memory occupancy, communication time and other requirements, or a custom operator such as the triton operator that is universal for AI accelerators.

[0049] In some of the embodiments, conditional compilation is performed according to the operating environment where the unified standardized adaptation module is located to access the target heterogeneous accelerator card and be used for autonomous optimization of the training framework, including: Based on the hardware environment and compilation requirements of the unified standardized adapter module, the target heterogeneous accelerator card is polled and connected to the training framework.

[0050] Specifically, the compilation script includes detection of the hardware environment and compilation requirements. The detection results are the hardware labels or CPU compilation flags of different accelerator cards of the corresponding nodes. When compiling a unified framework and adaptation code, the hardware and software stack corresponding to the target AI accelerator card connected to the node is polled according to the detection results, or only the CPU version is compiled.

[0051] In another embodiment, obtaining a unified test module and verifying the adaptation status of the unified standardized adaptation module and each target heterogeneous accelerator card according to the unified test module includes: Obtain a unified test module, perform unit tests on various heterogeneous accelerator cards at the same time according to the unified test module, or perform large model training tests, obtain test results, and adjust and optimize the unified standardized adaptation module based on the test results.

[0052] Specifically, a unified test module is used to perform unit tests or large model training tests on each heterogeneous accelerator card. Unit testing can be performed on each independent component of the adapter module. During the test, data on the performance, stability, error rate, etc. of the adapter module and AI accelerator card are collected. These data will serve as the basis for evaluating the status of the adapter module and the AI ​​accelerator card. The collected test results are analyzed to identify the performance differences of the unified adapter module on different heterogeneous accelerator cards. According to the test results, necessary adjustments and optimizations are made to the unified standardized adapter module. This may include improving the automatic optimization operator method, optimizing resource allocation, and enhancing the error handling mechanism. After adjustment and optimization, the test is repeated to verify the effectiveness of the improvement measures. At the same time, the status of each AI accelerator card is compared to provide feedback to the manufacturer.

[0053] Based on the same concept, an artificial intelligence acceleration card heterogeneous cluster adaptation system is also provided in this embodiment, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated here. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the systems described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0054] Figure 7 : is a schematic diagram of the framework of the artificial intelligence acceleration card heterogeneous cluster adaptation system of this embodiment. Figure 7 As shown, the system 70 includes: An acquisition module 71 is used to acquire a unified standardized adaptation module registered according to a private keyword provided by a training framework; the training framework is a training framework for a heterogeneous cluster model; the unified standardized adaptation module is registered to the training framework through the private keyword provided by the training framework, and is used to connect the hardware layer software stack of each heterogeneous acceleration card to the training framework; The accelerator card access module 72 is used to access the hardware layer software stack of each heterogeneous accelerator card through the factory mode in the unified standardized adapter module; Extension module 73, used to extend the computing task distribution mechanism of the training framework in the form of a plug-in, so that the training framework can autonomously optimize the target heterogeneous accelerator card; A compiling module 74 is used to perform conditional compilation according to the operating environment where the unified standardized adaptation module is located, so as to access the target heterogeneous accelerator card and be used for autonomous optimization of the training framework; And the test module 75 is used to verify the adaptation status of the unified standardized adaptation module and each target heterogeneous acceleration card.

[0055] Figure 8 is a timing diagram of the artificial intelligence acceleration card heterogeneous cluster adaptation method of this embodiment, such as Figure 8As shown, the PyTorch framework layer requests the Python interface layer of the adaptation layer to call the device interface base class. The Python interface layer interface is bound to the C++ abstract layer interface through Pybind. The C++ abstract layer requests to obtain the factory instance through the registration function of the factory registrar. The factory registrar creates a specific factory instance by calling the specific implementation of the C++ implementation layer. The C++ implementation layer calls the hardware layer software stack interface of the heterogeneous cluster, and calls the AI ​​accelerator card hardware layer software stack through the hardware layer software stack interface to obtain the return result, and returns the result to the C++ implementation layer, C++ abstract layer, Python interface layer, and PyTorch framework layer in sequence, thereby completing the non-invasive modification of the PyTorch framework to access the AI ​​accelerator card hardware layer software stack.

[0056] In this embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0057] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0058] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program: S1, obtaining a unified standardized adaptation module registered according to a private keyword provided by a training framework; the training framework is a training framework for a heterogeneous cluster model; the unified standardized adaptation module is registered to the training framework through the private keyword provided by the training framework, and is used to connect the hardware layer software stack of each heterogeneous accelerator card to the training framework; S2, in a unified standardized adapter module, accesses the hardware layer software stack of each heterogeneous accelerator card through the factory mode.

[0059] S3, expands the computing task distribution mechanism of the training framework through plug-ins, so that the training framework can autonomously optimize the target heterogeneous accelerator card.

[0060] S4, conditionally compiles according to the operating environment where the unified standardized adapter module is located to access the target heterogeneous accelerator card and is used for autonomous optimization of the training framework.

[0061] S5: Verify the adaptation status of the unified standardized adaptation module and each target heterogeneous accelerator card according to the unified test module.

[0062] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0063] In addition, in combination with the artificial intelligence accelerator card heterogeneous cluster adaptation method provided in the above embodiment, a storage medium can also be provided in this embodiment to implement. The storage medium stores a computer program; when the computer program is executed by the processor, any one of the artificial intelligence accelerator card heterogeneous cluster adaptation methods in the above embodiment is implemented.

[0064] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0065] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0066] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0067] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0068] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0069] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A method for adapting a heterogeneous cluster of artificial intelligence accelerator cards, characterized in that: The method comprises: Obtain a unified standardized adaptation module registered according to a private keyword provided by a training framework; the training framework is a training framework for a heterogeneous cluster model; the unified standardized adaptation module is registered to the training framework through the private keyword provided by the training framework, and is used to connect the hardware layer software stack of each heterogeneous acceleration card to the training framework; In the unified standardized adaptation module, the hardware layer software stack of each heterogeneous acceleration card is accessed through the factory mode; Expanding the computing task distribution mechanism of the training framework in the form of a plug-in, so that the training framework can autonomously optimize the target heterogeneous accelerator card; Conditionally compile according to the operating environment where the unified standardized adaptation module is located, so as to access the target heterogeneous accelerator card and use it for autonomous optimization of the training framework; The adaptation status of the unified standardized adaptation module and each of the target heterogeneous acceleration cards is verified according to the unified test module.

2. The method for adapting a heterogeneous cluster of artificial intelligence accelerator cards according to claim 1, characterized in that: The step of obtaining a unified standardized adaptation module registered according to a private keyword provided by a training framework includes: Obtain a unified standardized adaptation module registered according to the private keyword 1, private keyword 2 and private keyword 3 provided by the training framework.

3. The method for adapting a heterogeneous cluster of artificial intelligence accelerator cards according to claim 1, characterized in that: The step of obtaining a unified standardized adaptation module registered according to a private keyword provided by a training framework includes: At least unify the interface name, parameters, implementation method and calling logic of the adapter modules provided by the heterogeneous accelerator cards to obtain the unified standardized adapter module; The unified standardized adaptation module is registered into the training framework according to the private keywords provided by the training framework.

4. The method for adapting a heterogeneous cluster of artificial intelligence accelerator cards according to claim 1, characterized in that: In the unified standardized adaptation module, the hardware layer software stack of each heterogeneous acceleration card is accessed through the factory mode, including: Define an abstract class corresponding to each function and sub-function of each heterogeneous accelerator card, and encapsulate the common underlying implementation of each heterogeneous accelerator card through the abstract class; wherein the abstract class includes a plurality of virtual functions, and the plurality of virtual functions are used to encapsulate the functional interface of each accelerator card; Creating a factory class, and managing the abstract class through the factory class, wherein each heterogeneous accelerator card implements a specific factory, and the specific factory inherits from the abstract factory; Rewrite the virtual functions in the abstract class to access the hardware layer software stack of each heterogeneous accelerator card; The specific factories of the heterogeneous acceleration cards are registered in the unified standardized adaptation module through a registration mechanism.

5. The method for adapting a heterogeneous cluster of artificial intelligence accelerator cards according to claim 1, characterized in that: The computing task distribution mechanism of the training framework is extended in the form of a plug-in, including: The operator adaptation interface of the training framework is accessed and expanded in the form of a plug-in, and the operator adaptation interfaces of the heterogeneous acceleration cards are unified to obtain a unified operator adaptation interface; Expand the training framework in the form of plug-ins to support the access of self-developed operators and compilers to the training framework; The registration and distribution mechanism of the operator is extended to the target heterogeneous accelerator card, and a search recommendation algorithm is added to the operator calling module according to the limitations of operator performance, video memory occupancy and communication time, so that the training framework can autonomously optimize the target heterogeneous accelerator card.

6. The method for adapting a heterogeneous cluster of artificial intelligence accelerator cards according to claim 1, characterized in that: The conditional compilation is performed according to the operating environment where the unified standardized adaptation module is located, so as to access the target heterogeneous acceleration card and be used for autonomous optimization of the training framework, including: Based on the hardware environment and compilation requirements of the unified standardized adaptation module, the target heterogeneous acceleration card is polled and connected to the training framework.

7. The method for adapting a heterogeneous cluster of artificial intelligence accelerator cards according to claim 1, characterized in that: The verifying the adaptation status of the unified standardized adaptation module and each of the target heterogeneous acceleration cards according to the unified test module includes: Obtain the unified test module, perform unit tests on each heterogeneous acceleration card at the same time according to the unified test module, or perform large model training tests to obtain test results, and adjust and optimize the unified standardized adaptation module according to the test results.

8. An artificial intelligence accelerator card heterogeneous cluster adaptation system, characterized in that: The system comprises: An acquisition module, used to acquire a unified standardized adaptation module registered according to a private keyword provided by a training framework; the training framework is a training framework for a heterogeneous cluster model; the unified standardized adaptation module is registered to the training framework through the private keyword provided by the training framework, and is used to connect the hardware layer software stack of each heterogeneous acceleration card to the training framework; An accelerator card access module, used to access the hardware layer software stack of each heterogeneous accelerator card in the unified standardized adaptation module through a factory mode; An extension module, used to extend the computing task distribution mechanism of the training framework in the form of a plug-in, so that the training framework can autonomously optimize the target heterogeneous accelerator card; A compilation module, used for conditional compilation according to the operating environment where the unified standardized adaptation module is located, so as to access the target heterogeneous acceleration card and be used for autonomous optimization of the training framework; and a testing module, used to verify the adaptation status of the unified standardized adaptation module and each of the target heterogeneous acceleration cards.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for adapting a heterogeneous cluster of artificial intelligence accelerator cards described in any one of claims 1 to 7 are implemented.

Images