Method and device for accessing hardware equipment to model reasoning acceleration framework and electronic equipment

Abstract

The invention discloses a method and device for accessing hardware equipment to a model reasoning acceleration framework and electronic equipment, and relates to the technical field of computers, and the method comprises the steps: setting a bidirectional decoupling middle layer between target hardware equipment and the model reasoning acceleration framework, a unified memory operation interface and a unified kernel function execution interface of different hardware devices are arranged in a middle layer, and kernel functions of the different hardware devices are registered to a model reasoning acceleration framework through the middle layer, so that the kernel functions of the model reasoning acceleration framework are separated from a system during hardware running. Different hardware devices can realize access to the model reasoning acceleration framework through the intermediate layer, and the codes, operators and even model structures of the model reasoning acceleration framework do not need to be adjusted to adapt to different hardware devices, so that the problem of poor universality of the model reasoning acceleration framework in related technologies is solved, and the universality of the model reasoning acceleration framework is improved. The technical effect of improving the universality of the model reasoning acceleration framework is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, and electronic device for accessing a hardware device to a model inference acceleration framework. Background Art

[0002] With the widespread application of large language models (LLMs), the very large language model (vLLM), as an open source large model inference acceleration framework, significantly improves the efficiency of large model inference by efficiently managing key-value cache measurement of the attention mechanism and custom operators.

[0003] In related technologies, when different hardware devices support the vLLM framework, due to the different hardware architectures and instruction sets of different hardware devices, it is necessary to modify the code of the vLLM framework, reconstruct the operators in the vLLM framework, and even adjust the model structure to adapt to different hardware devices, resulting in poor versatility of the vLLM framework. Summary of the Invention

[0004] The present application provides a method, apparatus, and electronic device for accessing a hardware device to a model reasoning acceleration framework, so as to at least solve the problem of poor versatility of the vLLM framework in related technologies.

[0005] This application provides a method for accessing a hardware device to a model inference acceleration framework, including:

[0006] Set up an intermediate layer between the target hardware device and the model inference acceleration framework;

[0007] A unified memory operation interface for different hardware devices is set up in the middle layer so that when the model inference acceleration framework needs to perform memory operations during the model inference process, it can call the unified memory operation interface to implement memory operations on different hardware devices.

[0008] A unified kernel function execution interface for different hardware devices is set in the middle layer, so that when the model inference acceleration framework needs to perform model inference acceleration during the model inference process, the unified kernel function execution interface is called to trigger different hardware devices to execute kernel functions, and obtain the results returned by the execution of kernel functions by different hardware devices. The middle layer registers the kernel functions of different hardware devices with the model inference acceleration framework;

[0009] Based on the middle layer, the target hardware device is connected to the model inference acceleration framework.

[0010] This application also provides an apparatus for hardware device access to a model inference acceleration framework, comprising:

[0011] The memory management module is used to provide a unified memory operation interface. The unified memory operation interface is used by the model inference acceleration framework to implement memory operations on different hardware devices when memory operations are required during the model inference process.

[0012] Unified kernel function execution interface, used for model inference acceleration framework. When model inference acceleration is needed during the model inference process, the unified kernel function execution interface is used to trigger different hardware devices to execute kernel functions and obtain the results returned by the kernel functions executed by different hardware devices.

[0013] The kernel function registration module is used to register the kernel functions of different hardware devices to the model inference acceleration framework.

[0014] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the method of accessing any of the above-mentioned hardware devices to a model inference acceleration framework when executing the computer program.

[0015] The present application also provides a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, the steps of the method of connecting any of the above-mentioned hardware devices to the model inference acceleration framework are implemented.

[0016] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method of connecting any of the above-mentioned hardware devices to the model inference acceleration framework.

[0017] Through this application, a bidirectionally decoupled middle layer is set between the target hardware device and the model reasoning acceleration framework, and a unified memory operation interface and a unified kernel function execution interface for different hardware devices are set in the middle layer. The kernel functions of different hardware devices are registered to the model reasoning acceleration framework through the middle layer to separate the kernel functions of the model reasoning acceleration framework from the system during hardware runtime. Different hardware devices can access the model reasoning acceleration framework through the middle layer without adjusting the code, operators or even model structure of the model reasoning acceleration framework to adapt to different hardware devices. This solves the problem of poor versatility of the model reasoning acceleration framework in related technologies and achieves the technical effect of improving the versatility of the model reasoning acceleration framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A schematic diagram of the structure of a system for accessing a model inference acceleration framework using hardware devices provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of a method for accessing a hardware device to a model inference acceleration framework provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the structure of a system for accessing a model inference acceleration framework with a hardware device according to an embodiment of the present application;

[0022] Figure 4 A flowchart of a method for accessing a model inference acceleration framework using a hardware device according to an embodiment of the present application;

[0023] Figure 5 A structural block diagram of an apparatus for accessing a model inference acceleration framework for a hardware device provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0026] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0027] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0028] With the development of artificial intelligence technology, LLM models are widely used in production and life scenarios. As a benchmark framework for open-source large-model inference acceleration, the vLLM framework improves the throughput of model inference by efficiently managing the key-value cache tensor of the attention mechanism. It deeply optimizes basic operations such as matrix operations and activation functions with custom operators, making up for the performance shortcomings of standard deep learning frameworks (such as PyTorch or TensorFlow) in large-model inference. The vLLM framework improves computing efficiency, accelerates model inference, and optimizes memory management, which makes the vLLM framework perform excellently in high-concurrency inference scenarios. It currently supports multiple large language models, greatly expanding the engineering implementation space of the model.

[0029] In the past, the core computing power of the vLLM framework was highly dependent on the Compute Unified Device Architecture (CUDA) of the NVIDIA graphics processing unit (GPU). This resulted in a single-tier hardware ecosystem for the vLLM framework. However, this design greatly accelerated the engineering implementation of large-model reasoning in the early stages.

[0030] However, with the continuous advancement of technology, chip types and manufacturing processes have gradually diversified, and more and more chip manufacturers have launched their own artificial intelligence (AI) chips, providing the market with more options. However, the hardware architecture and instruction sets of AI chips designed by different chip manufacturers vary, resulting in different solutions for AI chips designed by different chip manufacturers to support the vLLM framework. Taking attention mechanism calculations as an example, NVIDIA's GPU relies on warp-level thread synchronization to implement calculations, Ascend's Neural Network Processing Unit (NPU) uses specialized Cube instructions to implement calculations, and Cambricon's Machine Learning Unit (MLU) relies on task queue sharding to implement calculations. As a result, in order to adapt the vLLM framework to these different AI chips, a large number of core operators must be rewritten separately for each chip, which exponentially increases development costs. Moreover, due to the non-uniform interfaces of different chips and the lack of unified management, the vLLM framework has to maintain many different versions, resulting in chaotic version branches. This not only increases the maintenance cost, but also reduces the versatility, extensibility, and code readability of the vLLM framework, making it very difficult for new users to get started.

[0031] It can be seen that although AI chips from mainstream chip manufacturers currently support the vLLM framework, in order to adapt them to their own AI chips, they have all made significant modifications to the native architecture and code of the vLLM framework, even requiring the redesign of operator implementations, computational processes, and adjustments to model structures. Different chip manufacturers have different implementations of the same operator, or functional module (such as Attention calculation), and the resulting modified vLLM code files also differ significantly in style and logic. Furthermore, the different modifications to the vLLM framework code have led to fragmentation in the vLLM framework code generated by each chip manufacturer in terms of architectural design, programming paradigms, and interface standards. Furthermore, due to the deep binding to the chip's underlying instruction set, each chip manufacturer's vLLM framework adaptation can only serve its own hardware ecosystem. In other words, each chip manufacturer's modified vLLM framework can only adapt to its own AI chips, creating a technological island effect and poor compatibility and versatility of the vLLM framework.

[0032] Moreover, when the vLLM framework is upgraded, different chip manufacturers need to adjust their own codes according to the vLLM upgrade plan. This leads to further differences in the vLLM frameworks adapted by different chip manufacturers, and the adaptation cost also increases exponentially, seriously hindering the universal development of the vLLM framework among different chips.

[0033] To solve the above problems, an embodiment of the present application provides a method, apparatus and electronic device for accessing a hardware device to a model reasoning acceleration framework, the method comprising: setting an intermediate layer between the target hardware device and the model reasoning acceleration framework; setting a unified memory operation interface for different hardware devices in the intermediate layer, so that when the model reasoning acceleration framework needs to perform memory operations during the model reasoning process, it calls the unified memory operation interface to implement memory operations on different hardware devices; setting a unified kernel function execution interface for different hardware devices in the intermediate layer, so that when the model reasoning acceleration framework needs to perform model reasoning acceleration during the model reasoning process, it calls the unified kernel function execution interface to trigger different hardware devices to execute kernel functions, and obtain the results returned by the execution of kernel functions by different hardware devices, wherein the intermediate layer registers the kernel functions of different hardware devices to the model reasoning acceleration framework; based on the intermediate layer, the target hardware device is connected to the model reasoning acceleration framework. The method provided by the above solution establishes a bidirectional decoupled middle layer between the vLLM framework and the hardware device, separates the core operators of the vLLM framework from the hardware runtime system, defines cross-hardware abstract interface specifications, maintains the kernel function execution status and memory management strategy in the vLLM framework upward, calls the hardware runtime system of different AI chips downward, and connects different hardware devices through standardized functional function interfaces to ensure that the vLLM framework can adapt to different hardware devices. This solves the problem of poor versatility of the model reasoning acceleration framework in related technologies and achieves the technical effect of improving the versatility of the model reasoning acceleration framework.

[0034] In combination with the specific application environment architecture or specific hardware architecture on which the execution of the method of accessing the hardware device to the model reasoning acceleration framework depends, the specific application environment architecture or specific hardware architecture is described here.

[0035] The method, device and electronic device provided in the embodiment of the present application for accessing the model reasoning acceleration framework for hardware devices are applicable to accessing the model reasoning acceleration framework for different hardware devices. Figure 1 As shown, this is a structural diagram of the system for accessing the model reasoning acceleration framework of the hardware device based on this application. The system for accessing the model reasoning acceleration framework of the hardware device includes a target hardware device, an intermediate layer and a model reasoning acceleration framework, wherein the target hardware device accesses the model reasoning acceleration framework through the intermediate layer, and the intermediate layer realizes the separation of the model reasoning acceleration framework and the target hardware device, so that the model reasoning acceleration framework can adapt to different hardware devices. Specifically, by setting a unified memory operation interface for different hardware devices, a unified kernel function execution interface for different hardware devices, and registering the kernel functions of different hardware devices to the model reasoning acceleration framework in the intermediate layer, the separation of the model reasoning acceleration framework and the target hardware device is realized, ensuring that the vLLM framework can adapt to different hardware devices and improving the versatility of the vLLM framework.

[0036] The embodiment of the present application provides a method for hardware device access to a model inference acceleration framework, which is applied to a server. Figure 2 A flowchart of a method for accessing a model inference acceleration framework for a hardware device provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method of connecting the hardware device to the model inference acceleration framework includes:

[0037] Step S201: Setting an intermediate layer between the target hardware device and the model inference acceleration framework.

[0038] Among them, the target hardware device is AI chip, and the model inference acceleration framework is vLLM framework.

[0039] Step S202, setting a unified memory operation interface for different hardware devices in the middle layer, so that when the model inference acceleration framework needs to perform memory operations during the model inference process, it calls the unified memory operation interface to implement memory operations on different hardware devices.

[0040] in, Figure 3 This is a schematic diagram of the structure of the system for accessing the model inference acceleration framework provided by the embodiment of this application. Figure 3 As shown, different hardware devices such as device 0, device 1, device 2, device 3, etc. are connected to the vLLM framework through the middle layer, where the middle layer includes a memory management module, a unified kernel function execution interface, a kernel function registration module, and a device attribute query module.

[0041] The memory management module is used to provide a unified memory operation interface, which includes a unified memory asynchronous copy interface, a unified memory synchronous copy interface, a unified memory allocation interface, and a unified memory release interface.

[0042] The unified kernel function execution interface and kernel function registration module realize the access of kernel functions. The unified kernel function execution interface is generated based on the kernel function configuration file and the automatically generated code template, and the kernel function registration module registers the kernel function based on the kernel function configuration file and the automatically generated code template.

[0043] The device attribute query module is used to obtain device information of different hardware devices.

[0044] In the related art, memory operations on different hardware devices all have their own vendor-specific application programming interfaces (APIs). To mask the differences in memory operations across different hardware devices, this application provides unified management of memory operations across different hardware devices. Specifically, a unified memory operation interface for different hardware devices is set up in the memory management module. When the vLLM framework needs to perform memory operations during model inference, it only needs to call this unified memory operation interface to implement memory operations on different hardware devices.

[0045] The unified memory operation interface automatically converts the framework's memory operation requests into specific operations suitable for the hardware device based on the type of the hardware device.

[0046] In this way, the model inference acceleration framework can more efficiently use the memory of different hardware devices, without having to write complex memory operation code for each hardware device. This not only improves development efficiency, but also enhances the versatility and scalability of the framework, making it more adaptable to different hardware devices.

[0047] Step S203: Set a unified kernel function execution interface for different hardware devices in the middle layer, so that when the model inference acceleration framework needs to perform model inference acceleration during the model inference process, it calls the unified kernel function execution interface to trigger different hardware devices to execute kernel functions, and obtains the results returned by the execution of kernel functions by different hardware devices. The middle layer registers the kernel functions of different hardware devices to the model inference acceleration framework.

[0048] Kernel functions, also known as custom operators or core operators, are the fundamental building blocks for implementing the various computational tasks required for model inference, including but not limited to matrix multiplication, activation function application, and attention mechanism calculation. These computational operations are the core components of vLLM model inference. The better the kernel function design and optimization, the faster and more efficient the model inference process. Efficient kernel functions enable the vLLM framework to complete inference tasks faster than other frameworks under the same hardware conditions.

[0049] The middle layer registers the kernel functions of different hardware devices with the model inference acceleration framework. This means that the specific functions of each hardware device are recorded and can be called by the model inference acceleration framework when needed.

[0050] Since AI chips from different chip manufacturers have different hardware architectures, instruction sets, etc., they have multiple ways of writing the same kernel function. The embodiment of this application unifies the kernel function execution interface of the same kernel function, so that all hardware devices use the same API to start the kernel function, increasing the semantic transparency of the code. The unification of the kernel function execution interface facilitates the access of different operators, and the relevant code can be automatically generated by code automatic generation tools.

[0051] When the vLLM framework needs to accelerate model inference during the model inference process, it does not need to deeply understand the underlying details of each hardware device. Instead, it simply calls the kernel function through a unified kernel function execution interface. This interface selects the appropriate hardware device based on the current context or configuration and triggers the execution of the registered kernel function on that device, improving development efficiency. Operator logic does not need to be aware of underlying hardware differences. That is, when developers write operator logic (the code logic that implements specific computing functions), they do not need to consider the characteristics and differences of different hardware devices. This allows the operator logic to be universally applicable across different hardware devices, improving the portability and versatility of the code and making it easier for the vLLM framework to adapt to a variety of different hardware devices.

[0052] Step S204: Based on the middle layer, the target hardware device is connected to the model reasoning acceleration framework.

[0053] Among them, through the intermediate layer corresponding to the above steps S202 to S203, it is possible to directly connect the target hardware device to the model reasoning acceleration framework without modifying the model reasoning acceleration framework.

[0054] The method for hardware device access to the model reasoning acceleration framework provided in the embodiment of the present application breaks through the paradigm bottleneck of the tight coupling between the vLLM framework and the hardware device in the related art. By constructing a universal middle layer for heterogeneous hardware devices, the core operators of the vLLM framework are separated from the hardware runtime system. Different hardware devices can be uniformly scheduled through the middle layer. When a new hardware device needs to be connected, the hardware device can access the vLLM framework by simply connecting to the middle layer. Define cross-hardware abstract interface specifications, maintain the kernel function execution status and memory management strategy in the vLLM framework upward, call the hardware runtime system of different AI chips downward, hide the differences between different hardware devices through standardized function function interfaces, connect different hardware devices, and support multi-hardware device operations, reduce dependence on hardware devices of a single manufacturer, ensure that the vLLM framework can adapt to different hardware devices, improve the high flexibility and scalability of the vLLM framework, avoid different modifications to the vLLM framework by different manufacturers, and speed up development. It solves the problem of poor universality of the model reasoning acceleration framework in the related art and achieves the technical effect of improving the universality of the model reasoning acceleration framework.

[0055] By hiding the differences between different hardware devices through standardized interfaces, developers can focus on algorithm logic rather than the specific implementation details of the underlying hardware, which simplifies the developer's work. The code can be written once and run on different hardware devices without the need to write and optimize code separately for each hardware device.

[0056] The embodiment of the present application provides a method for hardware device access to a model inference acceleration framework, which is applied to a server. Figure 4 A flowchart of a method for accessing a model inference acceleration framework for a hardware device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method of connecting the hardware device to the model inference acceleration framework includes:

[0057] Step S401: Set up an intermediate layer between the target hardware device and the model inference acceleration framework. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0058] Step S402: Setting a unified memory operation interface for different hardware devices in the middle layer so that when the model inference acceleration framework needs to perform memory operations during the model inference process, it calls the unified memory operation interface to implement memory operations on different hardware devices.

[0059] Specifically, the above step S402 includes:

[0060] Step S4021, setting a unified memory allocation interface for different hardware devices in the middle layer, so that when the model inference acceleration framework needs to perform memory allocation operations during the model inference process, it calls the unified memory allocation interface to implement memory allocation operations on different hardware devices.

[0061] Step S4022, setting a unified memory copy interface for different hardware devices in the middle layer, so that when the model inference acceleration framework needs to perform a memory copy operation during the model inference process, it calls the unified memory copy interface to implement the memory copy operation on different hardware devices.

[0062] Step S4023, setting a unified memory release interface for different hardware devices in the middle layer, so that when the model reasoning acceleration framework needs to perform memory release operations during the model reasoning process, it calls the unified memory release interface to implement memory release operations on different hardware devices.

[0063] Among them, the unified memory operation interface includes a unified memory allocation interface, a unified memory copy interface and a unified memory release interface.

[0064] like Figure 3 As shown, the unified memory copy interface includes a unified memory asynchronous copy interface and a unified memory synchronous copy interface.

[0065] Different hardware devices only need to access the unified memory allocation interface, unified memory copy interface, and unified memory release interface of the memory management module, and the vLLM framework can implement memory operations on different hardware devices, greatly improving development efficiency.

[0066] Step S403: Set up a unified kernel function execution interface for different hardware devices in the middle layer, so that when the model inference acceleration framework needs to perform model inference acceleration during the model inference process, it calls the unified kernel function execution interface to trigger different hardware devices to execute kernel functions and obtain the results returned by the execution of kernel functions by different hardware devices. The middle layer registers the kernel functions of different hardware devices with the model inference acceleration framework. For details, see Figure 2 Step S203 of the illustrated embodiment will not be described in detail here.

[0067] Step S404: Based on the middle layer, connect the target hardware device to the model inference acceleration framework. Figure 2 Step S204 of the illustrated embodiment will not be described in detail here.

[0068] The method for connecting hardware devices to the model inference acceleration framework provided in this embodiment of the application shields the memory operation differences between different hardware devices through a unified memory operation interface. When calling the unified memory operation interface, the model inference acceleration framework does not need to consider the specific conditions of the underlying hardware and can run smoothly on a variety of hardware devices, making the model inference acceleration framework more adaptable to different hardware environments and improving compatibility.

[0069] In some optional implementations, the method for connecting the hardware device to the model inference acceleration framework further includes:

[0070] Step a1: Set a unified memory allocation strategy for different hardware devices in the middle layer. The unified memory allocation strategy is to pre-allocate memory blocks that exceed a preset memory threshold to the model reasoning acceleration task from the memory pool of the hardware device, and divide the memory blocks into multiple memory sub-blocks. When the model reasoning acceleration framework needs to use memory during the model reasoning process, at least one memory sub-block is allocated to the model reasoning acceleration framework for use. When the model reasoning acceleration framework no longer uses the memory sub-block allocated to it, the memory sub-block allocated to it is released.

[0071] Among them, in order to further optimize the model reasoning performance of the vLLM framework and avoid memory fragmentation caused by frequent allocation and release of memory, the memory management module in the middle layer of the embodiment of the present application unifies the memory allocation strategy between different hardware devices through the design of the memory pool, that is, pre-allocates large blocks of memory, and divides the pre-allocated large blocks of memory according to a fixed size to create a series of small blocks of the same size. When the vLLM framework needs memory during model reasoning, it can obtain at least one small block from the memory pool. When it is no longer needed, it is released for subsequent reuse, thereby improving the efficiency of memory allocation and release and reducing memory fragmentation. At the same time, the memory management between hardware devices can be logically unified, avoiding problems caused by different memory management systems of different hardware devices and simplifying the programming model.

[0072] It is understandable that the preset memory amount threshold is set by technical personnel and is not specifically limited here.

[0073] The method for connecting a hardware device to a model inference acceleration framework, provided in an embodiment of the present application, pre-allocates a memory block exceeding a preset memory threshold from the hardware device's memory pool and divides it into multiple memory sub-blocks. When the model inference acceleration framework needs to use memory, it can directly allocate at least one memory sub-block, eliminating the need for complex memory application and allocation operations each time. This significantly improves the speed and efficiency of memory allocation and helps accelerate the model inference process.

[0074] In some optional implementations, the method for connecting the hardware device to the model inference acceleration framework further includes:

[0075] Step b1: setting an automatic code generation template for a unified kernel function execution interface in the middle layer.

[0076] Step b2: When the model inference acceleration framework is upgraded and a new kernel function is added, a unified kernel function execution interface corresponding to the new kernel function is generated based on the automatically generated code template and the configuration file of the new kernel function.

[0077] The configuration file includes the name of the new kernel function, input and output parameters, etc.

[0078] The automatic code generation template automatically reads the information in the configuration file of the new kernel function, and can quickly generate a new unified kernel function execution interface to improve development efficiency.

[0079] The method for hardware device access to the model inference acceleration framework provided in the embodiment of the present application uniformly manages the unified kernel function execution interface. Multiple unified kernel function execution interfaces can share an automatically generated code template. Kernel function access can be automatically accessed through code automatic generation technology. If new kernel function requirements arise during the version upgrade of the vLLM framework, the middle layer can perform semantic analysis on the new kernel function and automatically generate adaptation code, that is, automatically generate the corresponding unified kernel function execution interface, thereby reducing upgrade and maintenance costs and greatly improving the access work efficiency of developers.

[0080] In some optional implementations, the method for connecting the hardware device to the model inference acceleration framework further includes:

[0081] In step c1, when registering the kernel functions of different hardware devices to the model inference acceleration framework in the middle layer, the hardware device names when registering the kernel functions of different hardware devices are set to a unified hardware device name, so that when the model inference acceleration framework needs to perform model inference acceleration during the model inference process, it calls the corresponding unified kernel function execution interface based on the unified hardware device name.

[0082] In the related art, there are certain differences in the registration of kernel functions of different hardware devices. For example, when NVIDIA devices register kernel functions, the hardware device name of the kernel function must be uniformly set to kCUDA. The kernel function registered by this registration method cannot be run on other hardware devices. In order to shield the differences between different hardware devices and enable the model inference acceleration framework to adapt to different hardware devices, the embodiment of the present application also unifies the kernel function registration, that is, when the kernel functions of different hardware devices are registered to the vLLM framework, the hardware device names of the kernel functions of different hardware devices when registered are set to a unified hardware device name, so that all kernel functions can be called to different devices through this unified hardware device name. In this way, regardless of the hardware device, a unified hardware device name can be used to register kernel functions, which solves the problem of inconsistent kernel function registration methods for different hardware devices and achieves a unified naming method.

[0083] The unified hardware device name can be a custom device, which is represented by kPrivateUse.

[0084] By unifying the hardware device names used for kernel function registration, the model inference acceleration framework no longer needs to consider the differences in kernel function registration methods across different hardware devices when calling kernel functions, reducing development and maintenance challenges. This also improves the framework's compatibility and versatility across different hardware devices, facilitating integration of various chips into the framework and enabling better utilization of computing resources across different hardware devices, promoting the efficient operation of model inference on a wider range of hardware devices.

[0085] In some optional implementations, the method for connecting the hardware device to the model inference acceleration framework further includes:

[0086] In step d1, a device attribute query module is set in the middle layer. The device attribute query module is used to obtain the device information of different hardware devices and send the device information of different hardware devices to the model reasoning acceleration framework so that the model reasoning acceleration framework can issue model reasoning acceleration tasks based on the device information of different hardware devices.

[0087] Among them, device information includes computing power, memory size, etc. Since the middle layer of the embodiment of the present application can connect to AI chips from different manufacturers, it is very important to classify the attributes of the devices. By sending the device information of different hardware devices to the scheduler in the model inference acceleration framework, the scheduler can issue model inference acceleration tasks based on the device information of different hardware devices.

[0088] The scheduler determines how to allocate limited resources to different model inference acceleration tasks to optimize performance and improve efficiency. Device information can help adjust the allocation of model inference acceleration tasks. For example, some model inference acceleration tasks may rely more on video memory, while others may require more central processing unit (CPU) computing power. The scheduler allocates the corresponding tasks based on this information.

[0089] In addition, the scheduler also needs to ensure load balancing between the various computing nodes to avoid situations where some nodes are overloaded while others are idle. Specifically, load balancing first considers the characteristics of each hardware device, such as the computing power of the hardware device, memory bandwidth (affects the data transmission rate, and mainly affects the model loading and inference speed in large model inference), video memory capacity (how much storage space is available), etc., and sets priorities for different types of tasks based on the characteristics of different hardware devices. At the same time, in order to cope with the dynamic changes in workload, some dynamic adjustment strategies can be adopted. By collecting monitoring indicators, such as the number of requests processed per second, average response time, device utilization, etc., the current workload of the hardware device is analyzed to determine whether load adjustment is needed. For example, if it is found that the utilization of a GPU is close to saturation and the utilization of another GPU is low, some tasks can be transferred from the high-load GPU to the low-load GPU. In addition, when reasoning with large models, the vLLM framework can optimize load distribution through model parallelism (deploying different parts of the model to different hardware devices for execution, which is suitable for situations where the model is very large and cannot be accommodated by a single hardware device), data parallelism (deploying different copies of the same model to multiple hardware devices and processing different data batches in parallel, which is suitable for tasks that can be processed in batches), and hybrid parallelism (combining the advantages of model parallelism and data parallelism). The middle layer of the embodiment of the present application can support the corresponding strategies, including data sending, data receiving, data reduction, data dispersion, etc., by meeting the corresponding data transmission requirements, and supports the corresponding reasoning strategies by implementing the corresponding interfaces.

[0090] Device information can be obtained through hardware interfaces provided by different manufacturers. This facilitates the vLLM framework's allocation of model inference acceleration tasks, allowing users to obtain inference-related data in a timely manner, evaluate metrics such as model inference speed and real-time load, and reduce the difficulty of using the vLLM framework.

[0091] The method for connecting hardware devices to the model inference acceleration framework provided in this embodiment of the application obtains device information through a device attribute query module. Based on this device information, the model inference acceleration framework can assign appropriate model inference acceleration to the most appropriate hardware device. For example, for computationally intensive tasks, the vLLM framework will prioritize hardware devices with strong computing power; for tasks with high memory requirements, it will be assigned to hardware devices with large memory, thereby improving overall inference efficiency.

[0092] In some optional implementations, the method for connecting the hardware device to the model inference acceleration framework further includes:

[0093] Step e1: If the target hardware device fails to be connected to the model reasoning acceleration framework based on the intermediate layer, the step of connecting the target hardware device to the model reasoning acceleration framework based on the intermediate layer is repeated until the target hardware device is successfully connected to the model reasoning acceleration framework.

[0094] In step e2, if the number of repeated executions of the step of connecting the target hardware device to the model reasoning acceleration framework based on the intermediate layer exceeds the preset number threshold, and the target hardware device still fails to successfully connect to the model reasoning acceleration framework, an alarm is issued.

[0095] The preset number threshold is set by technical personnel and is not specifically limited here.

[0096] Since the hardware device access process may be affected by various factors, such as network fluctuations, unstable device initialization status, etc., a single access failure does not mean that the hardware device cannot be accessed. The method of the hardware device access model inference acceleration framework provided in the embodiment of the present application can effectively overcome temporary problems and ensure the reliability of hardware device access by continuously repeating the access steps in the event of a single access failure.

[0097] The method for hardware device access model inference acceleration framework provided in the embodiment of the present application can promptly notify relevant personnel so that they can quickly pay attention to and start to troubleshoot the problem, thereby avoiding the problem existing for a long time and affecting the normal operation of the system and business development.

[0098] In some optional implementations, the method for connecting the hardware device to the model inference acceleration framework further includes:

[0099] Step f1, set up a hardware device status monitoring module in the middle layer, and the hardware device status monitoring module is used to obtain the status information of different hardware devices in real time. When the status information of any hardware device indicates that the hardware device is abnormal, the abnormal information of the hardware device is determined, and the abnormal information of the hardware device is sent to the model reasoning acceleration framework, so that the model reasoning acceleration framework adjusts the delivery strategy of the model reasoning acceleration task based on the abnormal information of the hardware device.

[0100] The status information may include temperature, power consumption, load, and other information of the hardware device.

[0101] If any hardware device encounters an abnormality, the model reasoning acceleration framework will no longer issue model reasoning acceleration tasks to the hardware device. When the hardware device returns to normal working state and reconnects to the model reasoning acceleration framework, the model reasoning acceleration framework can issue model reasoning acceleration tasks to the hardware device.

[0102] The method for connecting a hardware device to a model reasoning acceleration framework provided in an embodiment of the present application promptly determines the abnormal information and informs the model reasoning acceleration framework when an abnormality occurs in the hardware device. The model reasoning acceleration framework can adjust the task issuance strategy based on the abnormal information and no longer issue the model reasoning acceleration task to the abnormal device, thereby avoiding problems such as incorrect reasoning results and task execution interruption caused by the execution of tasks by abnormal devices, thereby ensuring the smooth progress of the model reasoning task.

[0103] The method for connecting hardware devices to a model inference acceleration framework provided in the embodiments of the present application enables different types of hardware devices to be connected to the vLLM framework in a more flexible manner. By decoupling the vLLM framework from the hardware devices and unifying memory management and kernel function access using an intermediate layer, the vLLM framework supports access to a variety of different types, models, and styles of hardware devices, improving hardware adaptation efficiency, enhancing the compatibility of the vLLM framework, and reducing the cost of connecting new hardware devices.

[0104] Furthermore, the method for integrating hardware devices into the model inference acceleration framework provided by the present embodiment decouples hardware devices while maintaining unified and highly readable code at the middle layer, accelerating integration through automatic code generation. Furthermore, automatic code generation can effectively adapt to new operators introduced by vLLM version upgrades, accelerating development progress.

[0105] It should be noted that the method of hardware device access to the model reasoning acceleration framework provided in the embodiment of the present application can not only be applied to hardware device access to the vLLM framework, but can also be applied to other deep learning frameworks, such as SGLang, pytorch, paddlepaddle, etc., and it can be achieved by adaptively modifying the above-mentioned method of hardware device access to the model reasoning acceleration framework.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0107] The embodiment of the present application also provides a device for hardware device access to the model reasoning acceleration framework, such as Figure 5 Shown, including:

[0108] The memory management module is used to provide a unified memory operation interface. The unified memory operation interface is used by the model inference acceleration framework to implement memory operations on different hardware devices when memory operations are required during the model inference process.

[0109] The unified memory operation interface includes a unified memory allocation interface, a unified memory copy interface, and a unified memory release interface. The unified memory copy interface includes a unified memory asynchronous copy interface and a unified memory synchronous copy interface.

[0110] The unified kernel function execution interface is used for the model inference acceleration framework. When model inference acceleration is needed during the model inference process, the unified kernel function execution interface is used to trigger different hardware devices to execute kernel functions and obtain the results returned by different hardware devices executing the kernel functions.

[0111] The kernel function registration module is used to register the kernel functions of different hardware devices to the model inference acceleration framework.

[0112] It can be understood that the apparatus for hardware device access to the model reasoning acceleration framework can be the middle layer in the above-mentioned method embodiment of the hardware device access to the model reasoning acceleration framework.

[0113] In some optional implementations, the apparatus for accessing a hardware device to a model inference acceleration framework further includes:

[0114] The device attribute query module is used to obtain the device information of different hardware devices and send the device information of different hardware devices to the model reasoning acceleration framework so that the model reasoning acceleration framework can issue model reasoning acceleration tasks based on the device information of different hardware devices.

[0115] In some optional implementations, the apparatus for accessing a hardware device to a model inference acceleration framework further includes:

[0116] The hardware device status monitoring module is used to obtain the status information of different hardware devices in real time. When the status information of any hardware device indicates that the hardware device is abnormal, the abnormal information of the hardware device is determined and sent to the model reasoning acceleration framework so that the model reasoning acceleration framework can adjust the delivery strategy of the model reasoning acceleration task based on the abnormal information of the hardware device.

[0117] In some optional embodiments, the memory management module is also used to set a unified memory allocation strategy for different hardware devices. The unified memory allocation strategy is to pre-allocate memory blocks exceeding a preset memory threshold to the model inference acceleration task from the memory pool of the hardware device, and divide the memory blocks into multiple memory sub-blocks. When the model inference acceleration framework needs to use memory during the model inference process, at least one memory sub-block is allocated to the model inference acceleration framework for use. When the model inference acceleration framework no longer uses the memory sub-block allocated to it, the memory sub-block allocated to it is released.

[0118] In some optional implementations, the apparatus for accessing a hardware device to a model inference acceleration framework further includes:

[0119] The automatic generation of code templates for the unified kernel function execution interface is used to generate the unified kernel function execution interface corresponding to the new kernel function based on the automatic generation of code templates and the configuration files of the new kernel function when the model inference acceleration framework is upgraded and a new kernel function is added.

[0120] In some optional implementations, the kernel function registration module is further configured to:

[0121] When registering the kernel functions of different hardware devices to the model inference acceleration framework in the middle layer, the hardware device names when registering the kernel functions of different hardware devices are set to a unified hardware device name, so that when the model inference acceleration framework needs to perform model inference acceleration during the model inference process, it calls the corresponding unified kernel function execution interface based on the unified hardware device name.

[0122] For the description of the features in the embodiment corresponding to the apparatus for hardware device access to the model reasoning acceleration framework, please refer to the relevant description of the intermediate layer in the embodiment corresponding to the method for hardware device access to the model reasoning acceleration framework, which will not be repeated here.

[0123] The embodiment of the present application also provides an electronic device, such as Figure 6As shown, it includes a processor 601 and a memory 602, in which a computer program is stored. The processor 601 is configured to run the computer program to execute the steps in any of the above-mentioned method embodiments of the hardware device accessing the model inference acceleration framework.

[0124] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned method embodiments of the hardware device accessing the model inference acceleration framework when running.

[0125] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0126] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in the method embodiment of any of the above-mentioned hardware devices accessing the model inference acceleration framework are implemented.

[0127] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in the method embodiment of any of the above-mentioned hardware devices accessing the model inference acceleration framework.

[0128] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0129] The above is a detailed introduction to the method, device and electronic device for accessing a hardware device to a model reasoning acceleration framework provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A method for accessing a hardware device to a model inference acceleration framework, characterized in that: include: Set up an intermediate layer between the target hardware device and the model inference acceleration framework; A unified memory operation interface for different hardware devices is set in the middle layer, so that when the model reasoning acceleration framework needs to perform memory operations during the model reasoning process, the unified memory operation interface is called to implement memory operations on different hardware devices; A unified kernel function execution interface for different hardware devices is set in the middle layer, so that when the model reasoning acceleration framework needs to perform model reasoning acceleration during the model reasoning process, the unified kernel function execution interface is called to trigger different hardware devices to execute kernel functions, and obtain the results returned by the execution of the kernel functions by different hardware devices, wherein the middle layer registers the kernel functions of different hardware devices with the model reasoning acceleration framework; Based on the intermediate layer, the target hardware device is connected to the model reasoning acceleration framework.

2. The method according to claim 1, characterized in that The unified memory operation interface for different hardware devices is provided in the middle layer, including: A unified memory allocation interface for different hardware devices is set in the middle layer, so that when the model reasoning acceleration framework needs to perform memory allocation operations during the model reasoning process, the unified memory allocation interface is called to implement memory allocation operations on different hardware devices; A unified memory copy interface for different hardware devices is set in the middle layer, so that when the model reasoning acceleration framework needs to perform a memory copy operation during the model reasoning process, the unified memory copy interface is called to implement the memory copy operation on different hardware devices; A unified memory release interface for different hardware devices is set in the middle layer, so that when the model reasoning acceleration framework needs to perform a memory release operation during the model reasoning process, the unified memory release interface is called to implement the memory release operation for different hardware devices; The unified memory operation interface includes a unified memory allocation interface, a unified memory copy interface and a unified memory release interface.

3. The method according to claim 2, characterized in that The method further comprises: A unified memory allocation strategy for different hardware devices is set in the middle layer. The unified memory allocation strategy is to pre-allocate memory blocks exceeding a preset memory threshold to the model reasoning acceleration task from the memory pool of the hardware device, and divide the memory blocks into multiple memory sub-blocks. When the model reasoning acceleration framework needs to use memory during the model reasoning process, at least one memory sub-block is allocated to the model reasoning acceleration framework for use. When the model reasoning acceleration framework no longer uses the memory sub-block allocated to it, the memory sub-block allocated to it is released.

4. The method according to claim 1, wherein The method further comprises: Setting an automatic code generation template of the unified kernel function execution interface in the middle layer; When the model reasoning acceleration framework is upgraded and a new kernel function is added, a unified kernel function execution interface corresponding to the new kernel function is generated based on the automatically generated code template and the configuration file of the new kernel function.

5. The method according to claim 1, wherein The method further comprises: When the intermediate layer registers the kernel functions of different hardware devices to the model reasoning acceleration framework, the hardware device names when the kernel functions of the different hardware devices are registered are set to a unified hardware device name, so that when the model reasoning acceleration framework needs to perform model reasoning acceleration during the model reasoning process, it calls the corresponding unified kernel function execution interface based on the unified hardware device name.

6. The method according to claim 1, characterized in that The method further comprises: A device attribute query module is set in the middle layer, which is used to obtain device information of different hardware devices and send the device information of different hardware devices to the model reasoning acceleration framework, so that the model reasoning acceleration framework can issue model reasoning acceleration tasks based on the device information of different hardware devices.

7. The method according to claim 1, characterized in that The method further comprises: If the target hardware device fails to be connected to the model reasoning acceleration framework based on the intermediate layer, the step of connecting the target hardware device to the model reasoning acceleration framework based on the intermediate layer is repeated until the target hardware device is successfully connected to the model reasoning acceleration framework; If the number of times the step of connecting the target hardware device to the model reasoning acceleration framework based on the intermediate layer is repeated exceeds a preset threshold, and the target hardware device still fails to successfully connect to the model reasoning acceleration framework, an alarm is issued.

8. A device for hardware device access to a model inference acceleration framework, characterized in that: include: A memory management module is used to provide a unified memory operation interface. The unified memory operation interface is used by the model inference acceleration framework to implement memory operations on different hardware devices when memory operations are required during the model inference process. A unified kernel function execution interface is used for triggering different hardware devices to execute kernel functions when the model inference acceleration framework needs to perform model inference acceleration during the model inference process, and obtaining the results returned by the execution of the kernel functions by different hardware devices through the unified kernel function execution interface; The kernel function registration module is used to register the kernel functions of different hardware devices to the model inference acceleration framework.

9. The device according to claim 8, characterized in that The device further comprises: The device attribute query module is used to obtain the device information of different hardware devices and send the device information of the different hardware devices to the model reasoning acceleration framework so that the model reasoning acceleration framework can issue model reasoning acceleration tasks based on the device information of different hardware devices.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for accessing a model inference acceleration framework to a hardware device as described in any one of claims 1 to 7 when executing the computer program.

Images