Method and system for accelerating model reasoning, electronic equipment and storage medium

Abstract

The invention provides a method and a system for accelerating model reasoning, electronic equipment and a storage medium. The method comprises the following steps: calling a GPU (Graphics Processing Unit) to execute reasoning of an ith calculation unit according to a weight parameter required by the ith calculation unit of a specified model; if it is detected that reasoning of the ith calculation unit is completed, weight parameters needed by the (i + n) th calculation unit of the specified model are transmitted to the GPU from the CPU memory; and adding 1 to i, returning to execute and call the GPU to execute the reasoning of the ith calculation unit according to the weight parameter required by the ith calculation unit of the specified model until the reasoning of all calculation units of the specified model is completed, and improving the model reasoning speed on the premise of not influencing the model reasoning precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a method and system for accelerating model inference, an electronic device and a storage medium. BACKGROUND

[0002] Artificial intelligence generated content (AIGC) models are widely used in content creation such as images and videos, and diffusion models (DiT) using transformers as network structures are common AIGC models.

[0003] However, DiT has increasingly demanding requirements for hardware resources (GPU memory, host memory, storage I / O and transmission bandwidth), and conventional consumer-grade graphics cards are difficult to meet the requirements of DiT, thereby affecting model inference accuracy and model inference speed. SUMMARY

[0004] Therefore, embodiments of the present application provide a method and system for accelerating model inference, an electronic device and a storage medium to improve model inference accuracy and model inference speed.

[0005] To achieve the above object, the embodiments of the present application provide the following technical solutions.

[0006] The first aspect of the embodiments of the present application discloses a method for accelerating model inference, comprising:

[0007] invoking a GPU to perform inference of an i-th calculation unit according to weight parameters required by the i-th calculation unit of a specified model, and dividing the weight parameters of the specified model according to any granularity of calculation units in advance;

[0008] if it is detected that the inference of the i-th calculation unit is completed, transferring the weight parameters required by an i+n-th calculation unit of the specified model from a CPU memory to the GPU, where n is greater than 1;

[0009] increasing i by 1 and returning to perform the step of invoking the GPU to perform inference of the i-th calculation unit according to the weight parameters required by the i-th calculation unit of the specified model until the inference of all calculation units of the specified model is completed.

[0010] Preferably, the calculation unit is a block.

[0011] If it is detected that the inference of the i-th calculation unit is completed, transferring the weight parameters required by an i+n-th calculation unit of the specified model from a CPU memory to the GPU, comprises:

[0012] In the case of enabling the static graph optimization mode, if it is detected that the inference of the i-th computing unit is completed, the weight parameters required by the i+n-th computing unit of the specified model are transmitted from the CPU memory to the first buffer corresponding to the fixed address in the GPU created in advance;

[0013] In the case of not enabling the static graph optimization mode, if it is detected that the inference of the i-th computing unit is completed, the second buffer in which the weight parameters required by the i-th computing unit are stored in the GPU is released;

[0014] The third buffer is recreated;

[0015] The weight parameters required by the i+n-th computing unit of the specified model are transmitted from the CPU memory to the third buffer.

[0016] Preferably, further comprising:

[0017] In the process of transmitting the weight parameters required by the i+n-th computing unit of the specified model from the CPU memory to the GPU, the weight parameters required by the i+m-th computing unit of the specified model in the hard disk are preloaded to the CPU memory, and m is greater than n.

[0018] Preferably, before calling the GPU to perform the inference of the i-th computing unit according to the weight parameters required by the i-th computing unit of the specified model, further comprising:

[0019] Loading the weight parameters of the specified model to the shared memory pool;

[0020] Calling the specified allocator to perform the data representation operation of the weight parameters of the specified model.

[0021] Preferably, the process of detecting that the inference of the i-th computing unit is completed comprises:

[0022] Judging whether the calculation completion identifier of the i-th computing unit is listened to;

[0023] When the calculation completion identifier is listened to, it is determined that the inference of the i-th computing unit is completed.

[0024] The second aspect of the embodiment of the application discloses a system for accelerating model inference, and the system comprises:

[0025] An inference module is configured to call the GPU to perform the inference of the i-th computing unit according to the weight parameters required by the i-th computing unit of the specified model, and the weight parameters of the specified model are divided according to any granularity of computing unit in advance;

[0026] transmitting a weight parameter required by an i+nth computing unit of the specified model from a CPU memory to the GPU if it is detected that inference of the ith computing unit is completed, wherein n is greater than 1;

[0027] The processing module is configured to increase i by 1 and return to execute the inference module until inference of all computing units of the specified model is completed.

[0028] Preferably, the computing unit is a block, and the transmitting module comprises:

[0029] The first transmitting submodule is configured to, if the static graph optimization mode is enabled, transmit the weight parameter required by the i+nth computing unit of the specified model from the CPU memory to a first buffer area corresponding to a fixed address in the GPU if it is detected that inference of the ith computing unit is completed.

[0030] The releasing submodule is configured to, if the static graph optimization mode is not enabled, release a second buffer area in the GPU storing the weight parameter required by the ith computing unit if it is detected that inference of the ith computing unit is completed.

[0031] The creating submodule is configured to re-create a third buffer area.

[0032] The second transmitting submodule is configured to transmit the weight parameter required by the i+nth computing unit of the specified model from the CPU memory to the third buffer area.

[0033] Preferably, the method further comprises:

[0034] The pre-reading module is configured to, in the process of transmitting the weight parameter required by the i+nth computing unit of the specified model from the CPU memory to the GPU, pre-load a weight parameter required by an i+mth computing unit of the specified model in a hard disk to the CPU memory, wherein m is greater than n.

[0035] A third aspect of the embodiment of the application discloses a computer device, comprising a processor and a memory, which are connected through a bus; wherein the processor is configured to call and execute a program stored in the memory; and the memory is configured to store the program, which is used to implement the method for accelerating model inference disclosed in the first aspect of the embodiment of the application.

[0036] A fourth aspect of the embodiment of the application discloses a storage medium, which stores computer executable instructions, and the computer executable instructions are used to execute the method for accelerating model inference disclosed in the first aspect of the embodiment of the application.

[0037] Based on the above, the embodiment of the application provides a method, system, electronic device and storage medium for accelerating model inference. The method comprises the following steps: calling a GPU to perform inference of an i-th calculation unit of a specified model according to weight parameters required by the i-th calculation unit; if it is detected that the inference of the i-th calculation unit is completed, transmitting the weight parameters required by an i+n-th calculation unit of the specified model from a CPU memory to the GPU; increasing i by 1 and returning to perform the inference of the i-th calculation unit of the specified model according to the weight parameters required by the i-th calculation unit, until the inference of all the calculation units of the specified model is completed. In the scheme, the weight parameters required by the i+n-th calculation unit are prepared in advance when the inference of the i-th calculation unit of the specified model is performed. After the inference of the i-th calculation unit is completed, the weight parameters required by the i+n-th calculation unit prepared in advance are transmitted from the CPU memory to the GPU, so that the model inference speed is improved without affecting the model inference accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0039] Figure 1 A flowchart of a method for accelerating model inference provided by the embodiment of the present application;

[0040] Figure 2 An example diagram of a multi-stage asynchronous cache pipeline provided by the embodiment of the present application;

[0041] Figure 3 A flowchart of transmitting weight parameters from a CPU memory to a GPU provided by the embodiment of the present application;

[0042] Figure 4 A flowchart of inference of a specified model provided by the embodiment of the present application;

[0043] Figure 5 A schematic diagram of a static computation graph switching mechanism provided by the embodiment of the present application;

[0044] Figure 6 An example diagram of a shared memory pool mechanism provided by the embodiment of the present application;

[0045] Figure 7 A structural block diagram of a system for accelerating model inference provided by the embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0048] Artificial Intelligence Generated Content (AIGC) models are widely used in the creation of content such as images and videos. Among them, the Diffusion Model (DiT), which uses a transformer as its network structure, is a common AIGC model. The capabilities of a model are positively correlated with the number of its parameters. With a large number of parameters, the model's weights are extremely large, making it difficult for conventional consumer-grade graphics cards to run DiT.

[0049] The current conventional strategy is to reduce the computational precision of the model to reduce GPU memory usage. This results in a loss of quality in the model's inference performance and relatively low GPU computing utilization.

[0050] To address this, embodiments of the present invention propose a method, system, electronic device, and storage medium for accelerating model inference. When executing the inference of the i-th computational unit of a specified model, the weight parameters required for the (i+n)-th computational unit are prepared in advance. Upon completion of the inference of the i-th computational unit, the pre-prepared weight parameters required for the (i+n)-th computational unit are transferred from CPU memory to the GPU, thereby improving the model inference speed without affecting the model inference accuracy.

[0051] This solution can specifically optimize the memory dependency problem of the Dit model, maximizing GPU computing utilization in common inference scenarios such as single card with low memory and multiple cards on a single machine without affecting the model's inference accuracy.

[0052] See Figure 1 The flowchart illustrates a method for accelerating model inference provided by an embodiment of the present invention, the method comprising:

[0053] Step S101: Call the GPU to perform inference for the i-th computational unit based on the weight parameters required for the i-th computational unit of the specified model.

[0054] It should be noted that the weight parameters of the specified model are pre-divided according to computational units of arbitrary granularity. That is, the weight parameters of the specified model are divided and stored according to "computational units of arbitrary granularity". Here, the specified model can be a DiT model, and the "computational unit of arbitrary granularity" can be a layer, block, or module.

[0055] In the specific implementation step S101, the GPU is invoked through the compute stream process to perform inference of the i-th compute unit according to the weight parameters required by the i-th compute unit of the specified model.

[0056] It should be noted that during the inference process of the specified model by the GPU, fine-grained synchronization primitives (CUDAEvents) are used to identify whether various events have been completed.

[0057] Specifically, when performing inference for the i-th computation unit through the computation stream, it is determined whether a computation completion flag (compute event) for the i-th computation unit has been detected. If a computation completion flag (compute event) is detected, it is determined that the inference for the i-th computation unit has been completed; otherwise, if no computation completion flag (compute event) is detected, it is determined that the inference for the i-th computation unit has not yet been completed.

[0058] Among them, the compute stream process can directly use the main thread.

[0059] Step S102: If the inference of the i-th computation unit is detected to be completed, the weight parameters required for the i+n-th computation unit of the specified model are transferred from the CPU memory to the GPU, where n is greater than 1.

[0060] In the specific implementation of step S102, if the inference of the i-th computation unit is detected to be completed, the weight parameters required for the i+n-th computation unit of the specified model are transferred from CPU memory (CPURAM) to GPU through the transfer stream process, where n is greater than 1.

[0061] In other words, when the inference of the i-th computation unit ends, the weight parameters required by the (i+1)-th computation unit should be transmitted on the transport stream, and the transport stream should then begin transmitting the subsequent weight parameters (the weight parameters required by the (i+n)-th computation unit).

[0062] Understandably, after the transfer stream process successfully transfers the weight parameters required for the i+nth computation unit from the CPU memory to the GPU, it will trigger a transfer event (indicating that the weight parameters have been transferred to the GPU).

[0063] In some specific embodiments, during the process of transferring the weight parameters required for the i+nth computational unit of the specified model from CPU memory to GPU via the transfer stream process, the weight parameters required for the i+mth computational unit of the specified model from the hard disk (such as SSD) to CPU memory via the prefetch stream process, where m is greater than n.

[0064] It is understandable that after the prefetch stream process successfully preloads the weight parameters required for the i+mth computational unit of the specified model from the hard disk into the CPU memory, it will trigger a prefetch event (indicating that the weight parameters have been transferred to the CPU memory).

[0065] Multi-stream parallel scheduling of weight parameters is performed through prefetch stream, transfer stream, and compute stream processes.

[0066] For example, the computing unit is a layer. Through the synchronization primitive (CUDA Events), when the GPU is computing the i-th layer, the transfer stream process is moving or has moved the weight parameters of the i+1-th layer to the GPU, and the prefetch stream process is reading or has read the weight parameters of the i+2-th layer to the CPU memory.

[0067] Through the multi-stream parallel scheduling method described above, the weight parameters required by each computing unit are continuously loaded onto the GPU through a multi-level asynchronous cache pipeline of "prefetch stream -> transport stream -> computation stream" for inference of the specified model. This can completely achieve I / O and computation overlap, so that the GPU's computational utilization is no longer limited by I / O.

[0068] Furthermore, this multi-level asynchronous cache pipeline of "prefetch stream -> transport stream -> compute stream" corresponds to "SSD -> CPU RAM -> GPU VRAM". Cold data resides on the SSD, hot data is prefetched to the CPU RAM, and computed data is asynchronously streamed to the GPU VRAM.

[0069] Here, computational data refers to the weight parameters of the computational unit currently inference; hot data refers to the weight parameters required by several subsequent computational units after the computational unit currently inference; and cold data refers to the weight parameters of the computational unit that has completed inference, as well as the weight parameters required by computational units that are far from the "computational unit currently inference" and have not yet performed inference.

[0070] For example Figure 2 As shown in the example diagram of the multi-level asynchronous caching pipeline, when the specified model starts inference, the prefetch stream continuously preloads the weight parameters of layers 1 to 4 into the transport stream; when the transport stream passes the weight parameters of layer 1 to the compute stream for inference, the transport stream has already prepared the weight parameters of layer 2. After the compute stream completes the inference of layer 1, the transport stream then passes the weight parameters of layer 2 to the compute stream; and so on, until the inference of all layers is completed.

[0071] Step S103: Increment i by 1, return to step S101, and continue until the inference of all computational units of the specified model is completed.

[0072] In the specific implementation of step S103, after transferring the weight parameters required for the (i+n)th computational unit of the specified model from the CPU memory to the GPU, i=i+1, and return to step S101 to continue the inference of subsequent computational units until the inference of all computational units of the specified model is completed.

[0073] In some specific embodiments, this solution also provides a data representation (tensor) allocation and shared memory mechanism, wherein the data representation (tensor) is used to describe a multidimensional array of a specific shape and numerical type.

[0074] Before executing step S101, an initialization phase is required. This initialization phase includes: loading the weight parameters of the specified model into the shared memory pool (such as the shm shared memory pool); calling the specified allocator (such as the tensor allocator) to perform data representation operations (tensor operations) on the weight parameters of the specified model. These data representation operations include, but are not limited to: determining whether to bind to the shm shared memory address, address page alignment, and registering pin_memory in CUDA.

[0075] Specifically, to maximize CPU-to-GPU data transfer performance, and addressing the additional space overhead caused by the power-of-two alignment method natively used in the framework for pin memory, this solution implements a splitting and merging mechanism and a custom allocator using a compact allocation strategy with non-power-of-two alignment for memory management, eliminating the additional memory overhead. For multi-process weight reuse scenarios, an efficient shared memory mechanism is implemented. The management process establishes a shared memory pool in the host memory, loads weight parameters to create shared objects, and supports subsequent pin memory in-place registration in the application. Child processes receive weight information and perform zero-copy mapping, achieving reuse of the same physical space.

[0076] Furthermore, a shared memory pool is designed for scenarios with multiple GPUs and multiple processes. It allocates a shared memory space (i.e., a shared memory pool) that can be read by different processes to store the same data. The shared object refers to the weight parameters, and the weight information of the weight parameters refers to their address, size, and data type in the shared memory space. After the shared memory space is allocated, other processes can read the weight parameters from the shared memory space based on the received weight information.

[0077] For example: Suppose that the specified model has a total of 3 weight parameters (denoted as weight parameter A, weight parameter B and weight parameter C), and these 3 weight parameters occupy a total of 1GB; therefore, the management process allocates more than 1GB of shared memory space; the address of weight parameter A in the shared memory space is 0x00, and the memory size occupied by weight parameter A is 200M; the weight information of weight parameters B and C will not be illustrated one by one.

[0078] In this embodiment of the invention, when performing inference for the i-th computational unit of a specified model, the weight parameters required for the (i+n)-th computational unit are prepared in advance. After the inference for the i-th computational unit is completed, the pre-prepared weight parameters required for the (i+n)-th computational unit are transferred from the CPU memory to the GPU, thereby improving the model inference speed without affecting the model inference accuracy.

[0079] It should be noted that research has found that CUDA Graph records the physical memory address pointers of all kernel operations during capture. The structure, parameter shape, data type, address, and device of the computation graph must remain consistent in each replay. If regular offload is used, the new memory address allocated for loading weight parameters will be different each time, which will cause the graph to become invalid. Using dynamic memory or memory allocation behavior without handling this and without combining asynchronous loading of parameters will also lead to incompatibility issues.

[0080] To support maximized CUDA graph performance optimization, this solution proposes a graph swap mechanism (static graph switching mechanism). For the DiT model, the computationally intensive part (i.e., the part with the largest computational cost) consists of multiple consecutive blocks with consistent structure (each block is a set containing multiple layers and weight parameters). The structure and computation process of each block are the same, only the values ​​of the weight parameters in each block are different. That is, these multiple blocks can be represented as the same computation process but using different weight parameters and data inputs. Therefore, a fixed small number of static resident buffers can be pre-allocated. After compilation optimization, the address of the buffer is kept fixed (i.e., fixed address or static address). During subsequent inference, the weight parameters required by each block are synchronized to the fixed buffer by sequential switching, and then the compiled static graph can be reused for computation. In extreme optimization scenarios, for blocks with different structures, sub-graphs can be captured separately and launched on demand at runtime.

[0081] Based on the above static graph switching mechanism, this invention provides an embodiment for... Figure 1 The step S102, which involves "transferring the weight parameters required for the (i+n)th computational unit of the specified model from CPU memory to GPU", can be found in [link to relevant documentation]. Figure 3 , Figure 3 This illustrates a flowchart of the transfer of weight parameters from CPU memory to GPU according to an embodiment of the present invention. Figure 3 In the example, the unit of computation is a block. Figure 3 Includes the following steps:

[0082] Step S301: Determine whether to enable static graph optimization mode. If static graph optimization mode is enabled, proceed to step S302; if static graph optimization mode is not enabled, proceed to step S303 and subsequent steps.

[0083] Step S302: When static graph optimization mode is enabled, if the inference of the i-th computation unit is detected to be completed, the weight parameters required for the i+n-th computation unit of the specified model are transferred from CPU memory to the first buffer corresponding to a fixed address that is pre-created in the GPU.

[0084] It should be noted that only blocks with the same structure can share a buffer (e.g., a graph buffer). For example, 40 blocks in a certain type of DiT model have the same structure, while another type of DiT model has two blocks with different structures.

[0085] In the specific implementation step S302, when the static graph optimization mode is enabled, the computational units of different structures (in this embodiment, the computational unit is a block) are first statically compiled (torch.compile fullgraph); if the inference of the i-th computational unit is detected to be completed, the weight parameters required for the i+n-th computational unit of the specified model are transferred from the CPU memory to the "first buffer corresponding to the fixed address pre-created in the GPU" through the transport stream.

[0086] Specifically, a fixed address (i.e., a static address) is pre-created, and the fixed address corresponds to the first buffer of the GPU. If the inference of the i-th computation unit is detected to be completed, the weight parameters required for the (i+n)-th computation unit are transmitted to the corresponding first buffer via a transport stream, according to the type of the (i+n)-th computation unit.

[0087] Step S303: Without enabling static graph optimization mode, if the inference of the i-th computation unit is detected to be completed, release the second buffer in the GPU that stores the weight parameters required for the i-th computation unit.

[0088] It should be noted that different addresses correspond to different buffers in the GPU, and the "weight parameters required for the i-th computing unit" are stored in the GPU's second buffer.

[0089] In the specific implementation of step S303, if the inference of the i-th computation unit is detected to be completed without enabling the static graph optimization mode, the address corresponding to the second buffer is released, thereby releasing the second buffer.

[0090] Step S304: Recreate the third buffer.

[0091] In the specific implementation step S304, the address is recreated, and the buffer corresponding to the recreated address is called the third buffer.

[0092] Step S305: Transfer the weight parameters required for the i+nth computational unit of the specified model from the CPU memory to the third buffer.

[0093] In the specific implementation of step S305, the "weight parameters required for the (i+n)th computing unit" are transmitted to the third buffer of the GPU via a transport stream.

[0094] The above embodiments of the present invention Figure 3 This is an instruction on how to transfer weight parameters from CPU memory to the GPU.

[0095] To better understand the inference process of a given model, taking a block as the computational unit, we will use... Figure 4The inference flowchart of the specified model is shown as an example for illustration. Figure 4 Includes the following steps:

[0096] Step S401: Perform the initialization phase.

[0097] In the specific implementation step S401, based on the specified model and preset cache configuration (e.g., whether caching is enabled, cache size, and granularity of weight parameter splitting), a cache manager (e.g., a pin memory allocator) is initialized. This cache manager collects and records the weight parameters of the specified model at granularity. If multiple GPUs and processes are involved, a shared memory pool is created, the weight parameters of the specified model are loaded into the shared memory pool, and the weight information (address, size, and data type) of the weight parameters is recorded. The cache manager calls the tensor allocator to perform operations on the weight parameter tensors.

[0098] It should be noted that the cache manager initializes the prefetch stream process, the transfer stream process, the compute stream process, and CUDA Events.

[0099] If only two levels of caching are configured, the transport stream and compute stream are initialized. The transport stream needs to be bound to the CUDA Tream, and transfer events and compute events are created and assigned at the granularity of the weight parameters.

[0100] Transfer events and compute events are used to identify state and synchronize between different streams. If the computation unit is a block, a transfer event and a compute event are created for each block.

[0101] If a three-level cache is configured, a prefetch stream process is added, and prefetch events are created and assigned.

[0102] Each of the above processes runs in the background upon creation. Based on the buffer limit of the GPU memory, it copies the weight parameters into the GPU memory. For example, if the limit is two blocks, the weight parameters required for the first two blocks are copied into the buffer first, and the transfer event of the two blocks is marked.

[0103] Step S402: Divide and store the weight parameters.

[0104] Step S403: Execute the cache scheduler.

[0105] Step S404: Asynchronously load each process.

[0106] In the specific implementation step S404, the prefetch stream process, transfer stream process, and compute stream process are loaded asynchronously.

[0107] Step S405: Determine whether to enable static graph optimization mode. If enabled, proceed to step S406; if not enabled, proceed to step S407.

[0108] Step S406: Transfer the weight parameters required for the next block to the buffer corresponding to the fixed address, and then execute step S408.

[0109] Step S407: Release the address involved in the currently completed inference block, and create a new address to transfer the weight parameters required for the next block to the buffer corresponding to the new address, and then proceed to step S408.

[0110] Step S408: Perform blcok inference.

[0111] Step S409: Release or mark the GPU cache and update the cache scheduler status.

[0112] It should be noted that steps S403 to S409 are the Loops during the inference runtime of the specified model.

[0113] Step S410: Determine if the reasoning has ended. If not, return to step S403; if ended, output the reasoning result.

[0114] Based on the content shown in steps S401 to S410 above, the inference calculation process of the specified model is as follows: starting from the first block, if the first block is involved in caching, then check whether the transfer event of the first block is completed; if the transfer event is completed, it means that the weight parameters have been copied to the GPU; if the transfer event is not completed, then wait.

[0115] After the computation stream completes inference on the first block, it marks a compute event. The transport stream started by the cache manager checks in sequence whether the compute events of the corresponding blocks in the loaded buffers have been completed. For blocks that have completed compute events (inference finished), the weight parameters required by the block that has finished inference are removed from the buffer, thereby freeing up space to load the weight parameters required by subsequent blocks.

[0116] If the buffer is limited to two blocks, in the initial stage, the transport stream loads the weight parameters required for the two blocks into the buffer. After the first block completes the compute event, the inference of the second block is performed. When the transport stream detects that the first block has completed inference (i.e., completed the compute event), it releases the weight parameters required for the first block from the buffer and begins to transmit the weight parameters required for the third block into the buffer. Similarly, when the transport stream detects that the second block has completed inference, it releases the weight parameters required for the second block from the buffer and begins to transmit the weight parameters required for the fourth block into the buffer.

[0117] Throughout the entire inference process described above, the computation stream only adds wait times for transfer events and compute event flags for each block, without requiring any changes to other operations within the computation stream. The transport stream runs continuously after starting in the background. Since the computation time is greater than the CPU-to-GPU transfer time, the transport stream always waits for the computation stream; there is no situation where the computation stream waits for the transport stream after completing inference. Therefore, this does not reduce overall GPU memory usage or affect computation time.

[0118] When static graph optimization mode is enabled, blocks with different structures are first statically compiled, and then the weight parameters required by the block are copied to the buffer corresponding to the fixed address according to the type of the block. Since the buffer is associated with the fixed address, the transport stream only needs to transmit data.

[0119] Without enabling static graph optimization mode, the buffer can correspond to different addresses. Therefore, after the inference of a block is completed, the addresses involved in the block that has been inferred must be released first, and then new addresses must be created. The weight parameters required by subsequent blocks are then transferred to the buffer corresponding to the new address.

[0120] For example: The buffer corresponding to the current address 1 stores the weight parameters required by the block currently being inferred; after the block currently being inferred completes its inference, the current address 1 is released, a new address 2 is created, and the weight parameters required by the subsequent block are transferred to the buffer corresponding to address 2.

[0121] After completing the inference for the specified model, release the cache, terminate all processes, and destroy the cache manager.

[0122] As can be seen from the above reasoning process, this solution introduces the division and management of weight parameters for the specified model during the initialization phase, and integrates a cache scheduler to perform asynchronous multi-stream operations during model inference. If a static graph optimization mode is used, a buffer swap mechanism is added. The context and forward overloads are provided through object encapsulation, eliminating the need to adjust the model flow code.

[0123] Combination Figure 2 As shown in the example diagram of the multi-level asynchronous cache pipeline, this scheme manages the lifecycle of weight parameters through a three-level pipeline and limits the resident memory usage by defining a limited number of ring buffers.

[0124] Specifically, for a given model, K buffers of fixed size can be configured according to blocks (the same applies to layers and modules) to store the weight parameters of K blocks. These K buffers are denoted as GPU Buffer[0,... K-1], and each buffer is used to store the corresponding weight parameters.

[0125] The prefetch stream process is responsible for data transfer from SSD to CPU RAM, the transfer stream process is responsible for data copying from CPU RAM to GPU, and the compute stream process is responsible for GPU computation. Each of these processes executes asynchronously, and dependency synchronization is achieved by binding to "CUDA events" to ensure that loading and releasing buffers do not cause conflicts.

[0126] During pipeline scheduling, assuming the specified model has a total of L layers, when computing the inference of the i-th layer, the computation stream is using the weight parameters in the GPU Buffer[i%K] to perform the inference of the i-th layer; the transport stream writes the weight parameters required for the i+1-th layer (or later layers, which need to be controlled according to the number of buffers K) from the CPU to the GPU Buffer[(i+1)%K]; the prefetch stream loads the weight parameters after the transport stream (such as the weight parameters required for the i+2-th layer) from the disk to the CPU.

[0127] The prefetch stream, transport stream, and computation stream are finely synchronized through events. Before the computation stream begins inference, it must wait for the transport stream to complete the transmission of the weight parameters of the i-th layer. Before the transport stream transmits the weight parameters of the (i+1)-th layer, it must wait for the computation of the i-th layer to complete (so that the buffer storing the weight parameters of the i-th layer can be released to load the weight parameters of the (i+1)-th layer), thereby preventing data races.

[0128] For a given model, after dividing the weight parameters according to granularity, the computation time for operations on the same buffer is much longer than the transmission time, and the transmission overhead can be completely covered, thus achieving full-speed computation performance.

[0129] It should be noted that during the pipeline scheduling process, to maximize the transfer performance from CPU to GPU, CUDA pin_memory is used for page-lock registration. However, the native PyTorch implementation uses a power-of-two strategy, resulting in excessive memory usage compared to the actual needs. This solution optimizes tensor memory allocation using a chunk split & pack mechanism and a custom CUDA extension. The split mechanism reduces the overall memory footprint by dividing a tensor of the original size into multiple chunks conforming to power-law specifications. For example, a 68MB tensor is split into "64MB + 4MB", and after pinning, the original 128MB of space required is reduced to the actual 68MB needed.

[0130] The `pack` function can integrate small-parameter tensors into a single object for management. For example, two tensors "5M + 3M" can be packed into an 8M object, and after pinning, the space occupied is reduced from "8+4=12M" to 8M. Overall, by splitting large sizes and integrating small sizes towards a power-law standard, it eliminates extra space usage while reducing object data, thus reducing the number of subsequent copy instructions `cudamemcpy` to reduce overhead. Custom extensions can bypass framework limitations by directly allocating a pinned memory of a specified size using `cudaHostAlloc`.

[0131] To address the issue of combining dynamic weight loading with static graph optimization, this scheme proposes a static buffer switching mechanism for compatibility mechanisms in computational performance optimization.

[0132] Specifically, fixed address anchoring is first performed, meaning that for a given model block, a fixed set of K buffers can be created (if the block structure is the same, two buffers can be reserved for asynchronous switching). These K buffers are not released or reallocated throughout the overall inference lifecycle. Static graph capture optimization tools (such as torchfulgraph compile) are applied to the allocated blocks. During the actual inference process, the transport stream transmits the weight parameters to these fixed buffers.

[0133] like Figure 5The schematic diagram of the static computation graph switching mechanism shows that two buffers (buffer A and buffer B) with fixed addresses are created. The weight parameters required for the 1st, 2+1st, and 2i+1st blocks are transferred to buffer A, and the weight hyperparameters required for the 2nd, 4th, and 2ith blocks are transferred to buffer B. After the inference of all blocks is completed, other post-processing layers of the specified model are performed and the final result is output.

[0134] The computation stream performs static graph replay on the compiled graph (obtained by statically compiling blocks). The weight parameters in the buffer are updated as blocks switch, but the physical addresses of each weight parameter remain unchanged. The static graph can correctly compute the current block's result without recompiling. Different blocks use the buffer in a round-robin fashion during transmission and computation to avoid conflicts between different streams while simultaneously overlapping computation and transmission.

[0135] To address the issue of memory reuse in multi-process inference, this solution proposes a shared memory pool mechanism for multiple processes. This shared memory pool mechanism consists of two parts: "initialization" and "child process usage".

[0136] Initialization section: such as Figure 6 As shown in the example diagram of the shared memory pool mechanism, to avoid process conflicts, a single process is used for memory pre-allocation management. The management process reads the configuration file, parses out the parameter dimensions, quantity, and type of each computational layer of the specified model, and calculates the total memory size S required for all parameters by performing upward page-alignment for each parameter in combination with the page_size (usually 4K). It then uses "linux shm api" or "mmap" with the MAP_SHARED flag to request a shared memory region of the corresponding size. When loading the weight parameters, the management process reads the model weight file from the hard drive, creates Tensor objects corresponding to each weight parameter in the requested shared memory region in sequence, binds them to the corresponding memory addresses, and writes the data content to the corresponding Tensor.

[0137] Child process usage: such as Figure 6The illustrated shared memory pool mechanism diagram shows two inference subprocesses, process A and process B. Each inference subprocess can independently perform computational operations to map the same weight parameters using the same shared memory region. Each inference subprocess receives Tensor description information (including the shape, type, and memory address of each weight parameter) from the management process. Based on this Tensor description information, the inference subprocess reconstructs all Tensor objects corresponding to the parameter set. Different inference subprocesses bind to the same physical memory region during the construction process; that is, Tensor objects correspond to the same physical memory space in different inference subprocesses, thus forming a shared memory pool. It should be noted that... Figure 6 Virtual memory in the system is a resource isolation provided by the operating system for each inference subprocess. The virtual memory address of the Tensor object shared by each inference subprocess is different, but it corresponds to the same physical address.

[0138] To maximize the "CPU->GPU transfer speed" through direct memory access via PCIe, each inference subprocess calls the CudaHostRegister API to lock (pinned) the parameter Tensor memory region. After mapping registration, since the weight parameters only need to be read during inference, the inference subprocesses do not interfere with each other and no inter-process communication is required. Combined with a caching mechanism, the "CPU->GPU transfer" and "GPU computation" are asynchronously overlapped, and performance is not affected by memory sharing. This part mainly focuses on maximizing performance in platform-type server deployment scenarios, which generally have sufficient memory space for a single set of weight parameters for a specified model, eliminating the need to consider multi-level disk caching. Furthermore, in extreme cases, a multi-level caching mechanism can be used independently for each inference subprocess, with different inference subprocesses sharing the same full SSD weights. The size of the memory buffer within the inference subprocess can be controlled to reduce the overall memory requirements.

[0139] As can be seen from the above embodiments, this solution has the following beneficial effects:

[0140] 1. Significantly reduces resource dependence without affecting inference accuracy or additional time consumption, while also exhibiting high compatibility in computational performance optimization.

[0141] 2. By dynamically allocating weight parameters to GPU memory, RAM, and SSD, and using asynchronous multi-stream transmission and overlapping communication and computation, it supports flexible parameter configuration for specific models and computing devices, which can significantly reduce GPU memory usage and greatly lower the threshold for model accessibility.

[0142] 3. Multi-level asynchronous caching is natively implemented on the general framework PyTorch. Aside from the non-static parameters introduced by dynamic data transfer, it does not introduce other impacts, thus exhibiting high compatibility with the framework's native performance optimization strategies and seamless compatibility with mainstream libraries such as Transformers / Diffusers. To support maximum performance optimization, this solution further addresses the impact of non-static parameters, such as incompatibility issues arising from computation graph optimization, through static Buffer Swap. It supports a fixed number of static allocations at the block scale. During inference, weight parameters in memory are dynamically and alternately loaded into the buffer. For static graph optimization, after static compilation and allocation of 2-4 blocks, weights are asynchronously and alternately loaded into different buffers during computation, with transfer and computation overlapping to achieve maximum optimization.

[0143] 4. For devices with limited memory, multi-level caching stores the main parameters at the SSD level, requiring only a small amount of cache space and significantly reducing memory constraints. Simultaneously, in maximizing CPU->GPU performance transfer, this solution supports both chunk split & pack (splitting or integrating the original data into a more suitable size) and custom allocation creation, achieving more efficient space utilization and reducing additional memory overhead. For single-machine multi-GPU inference deployment scenarios, this solution introduces a shared memory pool mechanism. By sharing the same weight parameters across different inference subprocesses, it supports pin_memory registration for shm memory, without affecting CPU->GPU transfer efficiency. Overall, while maintaining high performance, it significantly reduces memory usage in multi-GPU deployments (multi-instance or multi-concurrency).

[0144] In practical applications, this solution enables models that originally required large amounts of video memory to run on consumer-grade graphics cards with low video memory and limited RAM without affecting computing performance, reducing the computational bottleneck for ordinary creators in generating high-quality content. In multi-model hybrid deployment and scheduling scenarios, the low resident video memory usage supports flexible dynamic hot-switching of models, enabling a single graphics card to efficiently support the output of multiple models. Combined with a multi-process shared memory mechanism, it can further improve the effective utilization of resources for single-machine multi-card deployment with multiple models and high concurrency.

[0145] Corresponding to the method for accelerating model inference provided in the above embodiments of the present invention, see also... Figure 7 The present invention also provides a structural block diagram of a system for accelerating model inference, the system including: an inference module 701, a transmission module 702 and a processing module 703;

[0146] The inference module 701 is used to call the GPU to perform inference for the i-th computational unit according to the weight parameters required for the i-th computational unit of the specified model, and to pre-divide the weight parameters of the specified model according to computational units of arbitrary granularity.

[0147] The transmission module 702 is used to transfer the weight parameters required for the (i+n)th computation unit of the specified model from the CPU memory to the GPU if the inference of the i-th computation unit is detected to be completed, where n is greater than 1.

[0148] In some embodiments, the transmission module 702 detects the completion of the inference process of the i-th computing unit, including: determining whether the computation completion flag of the i-th computing unit is detected; when the computation completion flag is detected, determining that the inference of the i-th computing unit is completed.

[0149] The processing module 703 is used to increment i by 1 and return to the execution inference module 701 until the inference of all computational units of the specified model is completed.

[0150] In this embodiment of the invention, when performing inference for the i-th computational unit of a specified model, the weight parameters required for the (i+n)-th computational unit are prepared in advance. After the inference for the i-th computational unit is completed, the pre-prepared weight parameters required for the (i+n)-th computational unit are transferred from the CPU memory to the GPU, thereby improving the model inference speed without affecting the model inference accuracy.

[0151] Preferred, combined Figure 7 The content shown is calculated in blocks. The transmission module 702 includes a first transmission submodule, a release submodule, a creation submodule, and a second transmission submodule. The execution principle of each submodule is as follows:

[0152] The first transmission submodule is used to, when the static graph optimization mode is enabled, transfer the weight parameters required for the i+nth computation unit of the specified model from the CPU memory to a first buffer pre-created in the GPU corresponding to a fixed address if the inference of the i-th computation unit is detected to be completed.

[0153] The release submodule is used to release the second buffer in the GPU that stores the weight parameters required for the i-th computation unit if the inference of the i-th computation unit is detected to be completed, without enabling the static graph optimization mode.

[0154] Create a submodule to recreate the third buffer.

[0155] The second transmission submodule is used to transfer the weight parameters required for the i+nth calculation unit of the specified model from the CPU memory to the third buffer.

[0156] Preferred, combined Figure 7The system also includes, as shown, the following:

[0157] The prefetch module is used to preload the weight parameters required for the i+nth computational unit of the specified model from the hard disk to the CPU memory during the process of transferring the weight parameters required for the i+mth computational unit of the specified model from the CPU memory to the GPU, where m is greater than n.

[0158] Preferred, combined Figure 7 The system also includes, as shown, the following:

[0159] The shared module is used to load the weight parameters of a specified model into a shared memory pool; and to call a specified allocator to perform data representation operations on the weight parameters of a specified model.

[0160] Preferably, the present invention also provides a computer device, including: a processor and a memory, the processor and the memory being connected via a bus; wherein, the processor is used to call and execute a program stored in the memory; the memory is used to store the program, the program being used to implement the method for accelerating model inference provided in the above method embodiments.

[0161] Preferably, embodiments of the present invention also provide a storage medium storing computer-executable instructions for executing the accelerated model inference method provided in the above-described method embodiments.

[0162] In summary, the embodiments of the present invention provide a method, system, electronic device, and storage medium for accelerating model inference. When performing inference for the i-th computational unit of a specified model, the weight parameters required for the (i+n)-th computational unit are prepared in advance. After the inference for the i-th computational unit is completed, the pre-prepared weight parameters required for the (i+n)-th computational unit are transferred from the CPU memory to the GPU, thereby improving the model inference speed without affecting the model inference accuracy.

[0163] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0164] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0165] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for accelerating model inference, characterized in that, The method includes: The GPU is invoked to perform inference for the i-th computational unit based on the weight parameters required for the i-th computational unit of the specified model. The weight parameters of the specified model are pre-divided according to computational units of arbitrary granularity. If the inference of the i-th computation unit is detected to be completed, the weight parameters required for the (i+n)-th computation unit of the specified model are transferred from the CPU memory to the GPU, where n is greater than 1; Increment i by 1, return to the execution call, and the GPU will perform inference for the i-th computational unit according to the weight parameters required for the i-th computational unit of the specified model, until the inference for all computational units of the specified model is completed.

2. The method according to claim 1, characterized in that, The unit of computation is a block; If inference for the i-th computational unit is detected to be complete, the weight parameters required for the (i+n)-th computational unit of the specified model are transferred from CPU memory to the GPU, including: When static graph optimization mode is enabled, if the inference of the i-th computation unit is detected to be completed, the weight parameters required for the i+n-th computation unit of the specified model are transferred from CPU memory to a first buffer pre-created in the GPU corresponding to a fixed address. Without enabling static graph optimization mode, if the inference of the i-th computation unit is detected to be completed, the second buffer in the GPU that stores the weight parameters required for the i-th computation unit is released. Recreate the third buffer; The weight parameters required for the (i+n)th computational unit of the specified model are transferred from CPU memory to the third buffer.

3. The method according to claim 1, characterized in that, Also includes: During the process of transferring the weight parameters required for the (i+n)th computational unit of the specified model from the CPU memory to the GPU, the weight parameters required for the (i+m)th computational unit of the specified model from the hard disk are preloaded into the CPU memory, where m is greater than n.

4. The method according to claim 1, characterized in that, Before invoking the GPU to perform inference for the i-th computational unit based on the weight parameters required for the i-th computational unit of the specified model, the following steps are also included: Load the weight parameters of the specified model into the shared memory pool; Invoke the specified allocator to perform data representation of the weight parameters of the specified model.

5. The method according to claim 1, characterized in that, The process of detecting the completion of the inference of the i-th computational unit includes: Determine whether the completion flag of the i-th computation unit has been detected; When the computation completion flag is detected, it is determined that the inference of the i-th computation unit has been completed.

6. A system for accelerating model inference, characterized in that, The system includes: The inference module is used to call the GPU to perform inference for the i-th computational unit based on the weight parameters required by the i-th computational unit of the specified model, and to pre-divide the weight parameters of the specified model according to computational units of arbitrary granularity. The transmission module is used to transfer the weight parameters required for the (i+n)th computation unit of the specified model from the CPU memory to the GPU if the inference of the i-th computation unit is detected to be completed, where n is greater than 1; The processing module is used to increment i by 1 and return to execute the inference module until the inference of all computational units of the specified model is completed.

7. The system according to claim 6, characterized in that, The computational unit is a block; The transmission module includes: The first transmission submodule is used to, when the static graph optimization mode is enabled, transfer the weight parameters required for the i+nth computation unit of the specified model from the CPU memory to a first buffer pre-created in the GPU corresponding to a fixed address if the inference of the i-th computation unit is detected to be completed. The release submodule is used to release the second buffer in the GPU that stores the weight parameters required for the i-th computation unit if the inference of the i-th computation unit is detected to be completed without enabling the static graph optimization mode. Create a submodule to recreate the third buffer; The second transmission submodule is used to transfer the weight parameters required by the i+nth calculation unit of the specified model from the CPU memory to the third buffer.

8. The system according to claim 6, characterized in that, Also includes: The pre-fetch module is used to preload the weight parameters required for the (i+n)th computation unit of the specified model from the hard disk to the CPU memory during the process of transferring the weight parameters required for the (i+m)th computation unit of the specified model from the CPU memory to the GPU, where m is greater than n.

9. A computer device, characterized in that, include: A processor and a memory are connected via a bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program for implementing the method for accelerating model inference as described in any one of claims 1-5.

10. A storage medium, characterized in that, The storage medium stores computer-executable instructions for performing the method for accelerating model inference as described in any one of claims 1-5.

Images