Data processing method and system

Abstract

Embodiments of the present specification provide a data processing method and system, the method comprising: a reasoning framework, in response to a cold start instruction sent by a cloud platform, cold starting on a target graphics processor, sending an opening hijacking command to a hijacking module, and during the process of loading first model weights of a first reasoning model, initiating a video memory application to the target graphics processor; the hijacking module, in response to the opening hijacking command, hijacking the video memory application, and redirecting the video memory application to a first virtual address in a fixed virtual address space; the reasoning framework, in a case where it is determined that the first model weights are completed loading, sending an ending hijacking command to the hijacking module, and based on the first model weights in the first virtual address, constructing a first reasoning execution graph to execute a first reasoning task using the first reasoning execution graph. By hijacking the CUDA video memory allocation and redirecting it to a pre-reserved fixed address, the address of the model weights is ensured to be persistent and stable, so that the CUDA Graph continues to be effective in multiple loadings and hot switching.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to a data processing method and system. BACKGROUND

[0002] In the current MaaS (Model as a Service) platform, service providers usually provide multiple models for users to choose from. In order to improve service flexibility and user experience, it has become a common requirement for users to frequently switch models without interrupting service. This requires the inference service system to be able to implement online hot switching of the running model inference service in the same container.

[0003] At the same time, modern inference frameworks widely use CUDA Graph technology to optimize model inference performance. However, the construction process of CUDA Graph is time-consuming, and once constructed, the internal recorded memory addresses are strongly bound to the physical memory locations of the model weights and cannot be dynamically updated. If the address of the model weight in the memory changes, the constructed CUDA Graph will be invalid and must be rebuilt, resulting in significant performance overhead. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a data processing method, and one or more embodiments of the present specification also relate to a data processing system, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a data processing method is provided, applied to a data processing system, the data processing system comprising an inference framework and a hijacking module, the method comprising:

[0006] The inference framework, in response to a cold start instruction sent by a cloud platform, cold starts on a target graphics processor, sends an opening hijacking command to the hijacking module, and initiates a memory application to the target graphics processor during the process of loading the first model weight of the first inference model;

[0007] The hijacking module, in response to the opening hijacking command, hijacks the memory application and redirects the memory application to a first virtual address of a fixed virtual address space, wherein the fixed virtual address space is a pre-reserved continuous virtual address range in the virtual address space of the target graphics processor, and the first virtual address is a continuous sub-region divided from the fixed virtual address space, and the first virtual address is mapped to the memory of the target graphics processor and used to store the first model weight;

[0008] The inference framework sends an end hijacking command to the hijacking module in a case where it is determined that the loading of the first model weight is completed, and constructs a first inference execution graph based on the first model weight in the first virtual address, to execute a first inference task using the first inference execution graph.

[0009] According to a second aspect of the embodiments of the present specification, a data processing system is provided, comprising an inference framework and a hijacking module, wherein,

[0010] The inference framework is configured to respond to a cold start instruction sent by a cloud platform, to cold start on a target graphics processor, to send an open hijacking command to the hijacking module, and to initiate a video memory application to the target graphics processor in a process of loading first model weight of a first inference model.

[0011] The hijacking module is configured to respond to the open hijacking command, to hijack the video memory application, and to redirect the video memory application to a first virtual address of a fixed virtual address space, wherein the fixed virtual address space is a pre-reserved continuous virtual address range in a virtual address space of the target graphics processor, the first virtual address is a continuous sub-region divided from the fixed virtual address space, and the first virtual address is mapped to a video memory of the target graphics processor and used to store the first model weight.

[0012] The inference framework is configured to send an end hijacking command to the hijacking module in a case where it is determined that the loading of the first model weight is completed, and to construct a first inference execution graph based on the first model weight in the first virtual address, to execute a first inference task using the first inference execution graph.

[0013] According to a third aspect of the embodiments of the present specification, a computing device is provided, comprising:

[0014] a memory and a processor;

[0015] The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-mentioned data processing method.

[0016] According to a fourth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores computer programs / instructions, which, when executed by a processor, implement the steps of the data processing method.

[0017] According to a fifth aspect of the embodiments of the present specification, a computer program product is provided, comprising computer programs / instructions, which, when executed by a processor, implement the steps of the above-mentioned data processing method.

[0018] An embodiment of the specification implements a data processing method applied to a data processing system, the data processing system comprising an inference framework and a hijacking module, the method comprising: the inference framework, in response to a cold start instruction sent by a cloud platform, cold starting on a target graphics processor, sending an opening hijacking command to the hijacking module, and initiating a video memory application to the target graphics processor in the process of loading first model weights of a first inference model; the hijacking module, in response to the opening hijacking command, hijacking the video memory application, and redirecting the video memory application to a first virtual address of a fixed virtual address space, wherein the fixed virtual address space is a pre-reserved continuous virtual address range in a virtual address space of the target graphics processor, the first virtual address is a continuous sub-region divided from the fixed virtual address space, and the first virtual address is mapped to the video memory of the target graphics processor for storing the first model weights; and the inference framework, in the case where it is determined that the first model weights are completed, sending an ending hijacking command to the hijacking module, and constructing a first inference execution graph based on the first model weights in the first virtual address, to execute a first inference task using the first inference execution graph.

[0019] The data processing method uses the hijacking module to redirect the video memory allocation of the model weights to the first virtual address of the pre-reserved fixed virtual address space by hijacking the CUDA video memory management API, ensures that different model weights can be accurately placed in the persistent fixed address in the video memory of the target GPU when loaded, and solves the problem of CUDA Graph invalidation caused by changes in the video memory address of the model weights. In combination with efficient video memory layout management, a plurality of model weights of models can be reasonably allocated to non-conflicting storage areas within the pre-reserved fixed virtual address space, realizing the address stability of the model weights. This enables the inference execution graph (CUDA Graph) constructed in advance for each model to be continuously valid throughout the entire life cycle of the model, without the need for re-construction even in the process of multiple loading and hot switching, realizes efficient and lossless hot switching between different models, and significantly improves the response speed and resource utilization of the MaaS service. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of a data processing method provided by an embodiment of the specification;

[0021] Figure 2 is a two-stage workflow diagram of a data processing method provided by an embodiment of the specification;

[0022] Figure 3 is a structural schematic diagram of a data processing system provided by an embodiment of the specification;

[0023] Figure 4 is a structural block diagram of a computing device provided by one embodiment of the present specification. DETAILED DESCRIPTION

[0024] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples, and it is understood that the scope of the present specification is not limited to the details below.

[0025] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0026] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, without departing from the scope of one or more embodiments of the present specification, first can be termed second, and similarly, second can be termed first. The term "if' as used herein can be interpreted as meaning "when" or "in response to determining" depending on the context.

[0027] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0028] In one or more embodiments of the present specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, billions, tens of billions, hundreds of billions, or even tens of billions of model parameters. The large model can also be referred to as a foundation model. Through large-scale unlabeled corpus pre-training, a pre-trained model with hundreds of millions of parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models (LLM) and multi-modal pre-training models.

[0029] In practical applications, a large model can be widely used in natural language processing (NLP) and computer vision fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), image generation, and natural language processing tasks such as text-based sentiment classification, text summarization generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0030] First, the technical terms related to one or more embodiments of the present specification are explained.

[0031] CPU: Central Processing Unit, central processor or central processing unit, is the core component of a computer, responsible for executing instructions and processing general computing tasks, coordinating the work of various hardware in the system.

[0032] GPU: Graphics Processing Unit, graphics processor or graphics processing unit, is an electronic computing device designed for handling complex graphics operations and parallel processing tasks, widely used in graphics rendering, scientific computing and machine learning fields.

[0033] Video memory: Video memory is a GPU-specific memory device used to store data and textures used by the GPU, improving the speed and efficiency of graphics processing.

[0034] CUDA: Compute Unified Device Architecture, is a general-purpose parallel computing architecture that enables GPUs to solve complex computational problems, and is the de facto standard in the field of AI (Artificial Intelligence).

[0035] API: Application Programming Interface, application programming interface.

[0036] CUDA API: is a set of function, type and constant interfaces provided by NVIDIA to developers, which is used to manage GPU devices, allocate and operate memory, start GPU kernel, control execution flow, synchronization, error handling, etc.

[0037] CRIU gpu: Checkpoint Restore in Userspace for GP, is a process checkpoint and recovery tool for GPU.

[0038] SGLang: Structured Generation Language for LLMs, is a programming language and runtime system designed for structured output of large language models (LLM).

[0039] KVCache: Key-Value Cache, is a key optimization technology for large language models in generative inference process, which caches the calculated Key and Value vectors to avoid repeated calculation at each step, significantly reducing computational redundancy, improving generation speed and reducing latency.

[0040] cudaMalloc: CUDA memory allocation function or GPU memory allocation function, which allocates a block of memory space of a specified size in the GPU's memory and returns a device pointer pointing to the memory space.

[0041] cudaMemcpyAsync: is a function in the CUDA runtime API (Application Programming Interface) that asynchronously copies data between the host (Host) and the device (Device), or within the device. Unlike the synchronous version of cudaMemcpy, cudaMemcpyAsync does not block the host thread, but instead places the copy operation into the specified CUDA stream and returns immediately, allowing the host to continue executing subsequent code.

[0042] LD_PRELOAD: Allows users to force a specified shared library (.so file) to be loaded before the program starts, taking precedence over the system default library. This way, you can intercept, replace, or enhance the calls to standard library functions (such as malloc, open, read, etc.) in the program.

[0043] In the prior art, the sleep wake-up mechanism of vLLM allows vLLM to dump the current model's weights and KVcache's memory into CPU memory, making the current vLLM service enter a "sleep" state; when the service is subsequently woken up, the memory of these two parts is loaded into the GPU again. This process is similar to model hot switching, but this solution can only operate on a single model and cannot be used in multi-model switching project scenarios.

[0044] In the container-based scheduling solution, a separate container is created for each model to be switched within the same POD (container group), and the container is directly scheduled during the model switching process. When adding, deleting, or modifying the model set, the POD container needs to be operated accordingly, resulting in higher subsequent operation and maintenance costs. Moreover, since this solution is completely independent of the inference framework, it has weak awareness of the upper elements (such as the use of memory), resulting in repeated construction and deletion of many context resources during the model switching process, which reduces the overall operation efficiency.

[0045] To solve the above technical problems, in the embodiments of the present specification, a data processing method is provided, and one or more embodiments of the present specification simultaneously relate to a data processing system, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0046] The data processing method provided by the embodiments of the present specification implements an efficient model hot switching method based on CUDA API hijacking. First, based on the hijacking of the CUDA memory API, the memory allocation operation in the model loading and switching process is hijacked by implanting a bottom-layer hook function. When the inference framework calls the memory allocation function such as cudaMalloc, the hijacking module will intercept the call and redirect it to the pre-reserved fixed virtual address space, thereby realizing the controllability and consistency of address allocation. Secondly, the method realizes the consistency management of the memory address of the model weight, and designs a set of memory layout management mechanism for the model weight. The mechanism pre-divides multiple independent address intervals in the GPU memory, allocates a dedicated storage area for each model through an address mapping table, ensures that the weight addresses within a single model and between different models do not conflict, and keeps the memory address persistent during multiple loading and switching processes, thereby avoiding the forced reconstruction of the CUDA Graph due to address changes. In addition, the method introduces an explicit memory management interface in the inference framework, and provides a model weight memory pre-allocation interface by extending the memory management module of the inference framework. Before the model is loaded, the controlled memory allocation process is triggered through the interface to ensure that all weight allocation requests can be hijacked correctly, avoiding the failure of hijacking of the memory allocation API in the inference framework.

[0047] Among them, implanting a bottom-layer hook function refers to a specific implementation method of intercepting CUDA API calls at the bottom layer of the system, such as through dynamic link library injection (DLL injection), function pointer modification (function hooking), etc. to enable the hijacking module to be inserted into the normal CUDA call process.

[0048] The memory management interface is a set of APIs or a module added to the original inference framework (such as vLLM, TensorRT) code. Its core function is to send commands to the hijacking module for active management and notification, rather than directly allocating memory. Before the model is loaded, this interface knows that "we are about to allocate weights for model A", and knows the weight size of model A and where it should be placed in the pre-reserved fixed virtual address space (such as address segment 0x1000-0x2000). It will actively "greet" the hijacking module before the cudaMalloc in the inference framework bottom layer is called, that is, issue an instruction such as: "Attention! The next memory allocation is for model A, please hijack it to address 0x1000".

[0049] Referring to Figure 1 , Figure 1A flowchart of a data processing method according to one embodiment of the present specification is shown, the data processing method is applied to a data processing system including an inference framework and a hijacking module, and the method specifically includes the following steps.

[0050] Step 102: The inference framework, in response to the cold start instruction sent by the cloud platform, cold starts on the target graphics processor, sends an opening hijacking command to the hijacking module, and initiates a video memory application to the target graphics processor in the process of loading the first model weight of the first inference model.

[0051] The data processing method provided by one or more embodiments of the present specification can be applied to any scenario of efficient model hot switching in large model inference process, including but not limited to large-scale model as a service cloud platform: on the MaaS platform, service providers need to provide a variety of models for a large number of users, through this method, the MaaS platform can host all these models in one service instance / container. When the user switches the model, realize millisecond level, no perceptual hot switching. Real-time recommendation system and advertising delivery engine: usually need to run multiple models in parallel or quickly in series (such as recall model, rough sorting model, fine sorting model, CTR estimation model, etc.), and the model needs to be frequently updated online. This method allows all related models to be resident in the fixed address of the GPU video memory. The switching overhead between models is reduced, ensuring the end-to-end low latency of the recommendation system. Autonomous driving and robot edge real-time systems: on the edge side, computing resources (GPU video memory) are particularly valuable, but the system needs to rely on multiple neural network models (such as target detection, lane line identification, path planning, etc.) to make real-time decisions; this method ensures that multiple key models can be on standby at any time with the highest performance (using CUDA Graph) on limited edge GPU resources, and does not introduce any jitter or delay when switching, greatly enhancing the real-time performance and reliability of the system. Of course, it can also be applied to other applicable scenarios, which are not exhausted here.

[0052] The inference framework can be understood as a software system for loading and running an artificial intelligence model, such as vLLM and the like. In the embodiments of the present specification, the inference framework is an inference framework that is extended on the basis of an original memory management module and integrates a display memory management interface. The inference framework is a deeply customized and extended inference framework, which is modified by injecting an explicit memory management interface. That is, on the basis of the original inference function of the inference framework, a memory management subsystem responsible for collaborative work with the underlying hijacking module is integrated, that is, the newly added explicit memory management interface, so that the inference framework contains two core parts: the original inference engine: responsible for model loading, computation graph construction, inference execution and other core functions, this part of the capability is retained; the newly added memory management interface: this is the code injected or integrated into the inference framework, which collaborates with the underlying hijacking module.

[0053] The cloud platform can be understood as an external resource management and scheduling system, such as a Kubernetes (container orchestration system, referred to as K8s) based cluster management system or an internal function calculation platform. The cloud platform is responsible for deciding when and where to start an inference task.

[0054] The cold start instruction can be understood as an instruction issued by the cloud platform for instructing the inference framework to load and initialize an inference model from the beginning. In contrast to hot start (quickly recover from existing state), cold start means that the inference model is not currently activated in the GPU and needs to go through a complete loading process (such as reading model weights from disk or memory, allocating video memory, building a computation graph, etc.).

[0055] The target graphics processor can be understood as a target GPU specified by the cloud platform, which specifies that the inference model will be cold started on which physical or logical GPU device.

[0056] The hijacking module can be understood as a separate module that is usually located at the bottom (such as dynamic link library injection). Its core function is interception and operation. The hijacking module listens to and intercepts CUDA runtime API calls such as cudaMalloc, cudaFree, etc.; when it receives an instruction from the memory management interface of the inference framework, it waits. Once it detects a video memory application call, it will be redirected to a specified fixed address according to the instruction instead of being randomly allocated by the CUDA driver. The hijacking module and the memory management interface of the inference framework work together to ensure that the model weights can be accurately allocated to the fixed video memory address, thereby laying a solid foundation for efficient model hot switching.

[0057] The opening hijacking command can be understood as a command issued by an explicit memory management interface within the inference framework to the hijacking module, which informs the hijacking module that the upcoming memory application is for model weight allocation, and prepares to hijack and redirect to the reserved fixed virtual address space.

[0058] The first inference model can be understood as a target large model that needs to be loaded, and the first model weight can be understood as parameter data corresponding to the first inference model, and the first model weight is static and read-only and will not change in the life cycle of a single inference task.

[0059] The memory application can be understood as a request for a continuous space in the memory to store the first model weight by calling the CUDA runtime API to request the target GPU driver to allocate a continuous space in the memory when the inference framework loads the first model weight.

[0060] Specifically, after receiving the cold start instruction from the cloud platform, the inference framework first performs environment initialization on the target GPU. The explicit memory management interface integrated in the inference framework actively sends an opening hijacking command to the hijacking module, which can include a pre-allocated fixed virtual address space. Then, when the inference framework starts to load the first model weight and calls cudaMalloc and other CUDA APIs to apply for memory, the hijacking module in standby state immediately intercepts this call (memory application) through the underlying hook function. After the interception, the hijacking module does not perform the real memory allocation, but directly redirects it to the first virtual address of the fixed virtual address space specified in the command, and returns the first virtual address to the inference framework. The inference framework loads the model weight data to the fixed first virtual address without awareness.

[0061] Step 104: The hijacking module, in response to the opening hijacking command, hijacks the memory application and redirects the memory application to the first virtual address of the fixed virtual address space.

[0062] The fixed virtual address space is a continuous virtual address range pre-reserved in the virtual address space of the target GPU, and the first virtual address is a continuous sub-region divided from the fixed virtual address space, and the first virtual address is mapped to the memory of the target GPU and used to store the first model weight.

[0063] Specifically, the redirection can be understood as that after intercepting the memory application, the hijacking module does not perform the real memory allocation, but directly returns a first virtual address in a pre-reserved fixed virtual address space to the inference framework, so that the inference framework writes the first model weight to the first virtual address.

[0064] The first virtual address can be understood as the specific address returned by the hijacking module to the inference framework when performing redirection. It is the starting position of a continuous address segment specially divided for the currently loaded model (the first inference model) from the fixed virtual address space.

[0065] The fixed virtual address space can be understood as a continuous and address-fixed virtual address range reserved in advance in the global virtual address space of the target GPU memory during the initialization stage, which is a resource pool for the fixed addresses of all model weights.

[0066] In actual application, during the initialization stage or the first start, a continuous address range is reserved in advance in the global virtual address space of the target GPU as the fixed virtual address space by calling the CUDA virtual memory management API. Subsequently, a fixed mapping relationship is established between this reserved fixed virtual address space and a specific physical memory block carved out from the target GPU memory, thereby forming a persistent memory region dedicated to model weight storage. This memory region will not be occupied by regular memory allocation operations during runtime, and its virtual starting address and corresponding physical memory location remain unchanged throughout the entire life cycle, providing basic support for subsequent fixed address loading of model weights. When the first model weight needs to be loaded, the memory layout manager will carve out a small continuous and address-fixed virtual address segment as the first virtual address from this global fixed virtual address space according to the size of the model weight and the preset memory alignment requirement, and allocate it to the first model.

[0067] Specifically, first, the hijacking module will reserve a large and continuous fixed virtual address space in the target GPU's memory as a global address pool when initializing through the CUDA virtual memory management API. When the hijacking module receives the "start hijacking command" from the inference framework, it immediately enters a standby state. This command can include a preset fixed virtual address space for the current model (the first inference model). The hijacking module points its internal memory allocation pointer to the fixed virtual address space, preparing to start address allocation. Then, when the inference framework starts loading the model weights and calling cudaMalloc to apply for memory, the hijacking module's hook function will immediately intercept this call. Instead of submitting a memory application to the GPU driver, the hijacking module executes its redirection logic: according to the current request size, it carves out a continuous address segment from the previously set fixed virtual address space that meets the size and alignment requirements, i.e., the first virtual address, and moves the internal pointer forward by the corresponding distance to prepare for the next allocation. Then the hijacking module returns this calculated first virtual address located in the fixed address space to the inference framework. The inference framework is completely unaware of this process and copies the first model weight of the first inference model to the "designated" first virtual address hijacked by the hijacking module. When all the first model weights are loaded, the inference framework sends the "end hijacking command", and the hijacking module stops intercepting, thus completing a complete hijacking task that ensures that the model weights are loaded to a fixed memory address.

[0068] Step 106: The inference framework, upon determining that the first model weight loading is complete, sends an end hijacking command to the hijacking module, and constructs a first inference execution graph based on the first model weight in the first virtual address to execute a first inference task using the first inference execution graph.

[0069] The end hijacking command can be understood as a control command sent by the inference framework to the hijacking module, indicating that the first model weight loading is complete, and the hijacking module can stop intercepting subsequent memory applications based on this end hijacking command.

[0070] The first inference execution graph can be understood as a CUDA Graph constructed based on the first model weight fixed at the first virtual address, which is a pre-compiled, static, and efficient inference computation graph.

[0071] The first inference task can be understood as using the first inference execution graph to perform model inference computation on the actual input data of the first inference model.

[0072] Specifically, after completing the loading process of the first model weight of the first inference model, the inference framework confirms through its internal state tracking mechanism that all weight tensors (i.e., each tensor data block constituting the first model weight) have been accurately written to the preset first virtual address. Then, the inference framework sends an "end hijacking command" to the hijacking module through its explicit memory management interface. The end hijacking command instructs the hijacking module to immediately release the interception state of the CUDA memory allocation API, so that it restores the normal memory allocation logic.

[0073] Secondly, the inference framework enters the performance optimization phase and uses the CUDA Graph capture mechanism to start a dedicated graph capture stream in which the forward inference process of the first inference model is completely executed once. Since the memory addresses of all weights of the first inference model have been fixed at this time, all kernel launches and memory operations recorded in this capture process internally reference fixed and persistent memory addresses. After the capture is completed, the inference framework instantiates this stream into a static and repeatable first inference execution graph. The construction of the first inference execution graph is a one-time overhead. Once constructed, the inference framework only needs to simply start and repeatedly execute this graph for multiple inference requests in the entire life cycle of the first inference model, without the need for dynamic graph construction for each inference. This significantly reduces the GPU kernel launch overhead and improves the submission efficiency of computing instructions.

[0074] Since GPU memory is a relatively valuable and limited resource, when multiple inference models that far exceed the capacity of a single GPU memory need to be hosted, all inference models cannot be resident in memory. Therefore, a cold switching mechanism is needed to allow the inference model that is not currently in use to be unloaded from the GPU memory, thereby freeing up space for other inference models. This allows limited GPU hardware to support a larger number of model services. The specific implementation is as follows:

[0075] After constructing the first inference execution graph based on the first model weight in the first virtual address to execute the first inference task using the first inference execution graph, the method further includes:

[0076] In response to the model cold switching command sent by the cloud platform, the inference framework sends a release memory command to the hijacking module and releases the coupled data related to the first inference model.

[0077] In response to the release memory command, the hijacking module releases the mapping relationship between the first virtual address and the memory of the target GPU, releases the first model weight in the first virtual address, and retains the first virtual address.

[0078] The model cold switching command can be understood as a command issued by the cloud platform for switching from a currently running first inference model to another inference model.

[0079] The release video memory command can be understood as a release command issued by the inference framework to the hijacking module for triggering the release of video memory resources corresponding to the first virtual address in the fixed virtual address space occupied by the first model weight.

[0080] The coupling data related to the first inference model can be understood as runtime data related to the first inference model, including but not limited to KV Cache (key value cache, used to accelerate the autoregressive generation process) and temporary intermediate activation values.

[0081] Specifically, when the inference framework receives the model cold switching command issued by the cloud platform, it first starts the resource cleaning process. The memory management interface in the inference framework synchronously sends a release video memory command to the hijacking module, which can contain the model identifier corresponding to the first model weight to be released. Secondly, the inference framework is responsible for cleaning the runtime data tightly coupled with the first model, such as KV Cache.

[0082] After receiving the release video memory command, the hijacking module locates the starting address and size of the precise paragraph of the first model weight in the first virtual address space by querying the internally maintained address mapping table. Subsequently, the hijacking module calls the CUDA virtual memory management API (such as cuMemUnmap) to remove the mapping relationship between the segment of virtual addresses and physical video memory pages. That is, this operation only releases the physical video memory resources, while the original occupied virtual address range (i.e., the first virtual address in the fixed virtual address space) is still reserved and marked as idle, ensuring that the first virtual address can be reused as is in the subsequent model, thereby maintaining the persistence of the address.

[0083] The data processing method provided by one or more embodiments of the present specification ensures that the useless video memory is immediately released during model switching by the hijacking module for targeted release of the fixed address space, and the inference framework for cleaning of KV Cache and other coupling data, freeing up sufficient space for subsequent model loading, significantly improving the utilization rate of GPU video memory resources. The release operation only removes the physical mapping while preserving the virtual address segment, ensuring that the fixed virtual address allocated for each model remains unchanged during the life cycle of multiple loading and unloading, thereby allowing the CUDA Graph previously constructed for the model to be rebuilt without changing the address, solving the high first inference delay problem caused by Graph rebuilding in the cold switching scenario.

[0084] After releasing the first model weight in the first virtual address, the physical display memory corresponding to the first virtual address can be used by other inference models. The specific implementation is as described below.

[0085] After releasing the first model weight in the first virtual address, the physical display memory corresponding to the first virtual address can be used by other inference models. The specific implementation is as described below.

[0086] The inference framework sends the start hijacking command to the hijacking module, and initiates a display memory application to the target GPU during the process of loading the second model weight of the second inference model.

[0087] The hijacking module, in response to the start hijacking command, hijacks the display memory application and redirects the display memory application to a second virtual address in the fixed virtual address space, wherein the second virtual address is mapped to the display memory of the target GPU and used to store the second model weight.

[0088] The inference framework, upon determining that the second model weight loading is complete, sends the end hijacking command to the hijacking module, and based on the second model weight in the second virtual address, constructs a second inference execution graph to execute a second inference task using the second inference execution graph.

[0089] The second inference model can be understood as another inference model that needs to be loaded, which is different from the first inference model in structure or function.

[0090] The second model weight can be understood as the parameter data of the second inference model, which needs to be loaded into the target GPU display memory.

[0091] The second virtual address can be understood as another continuous address segment in the fixed virtual address space for the second model weight, which does not overlap with the first virtual address.

[0092] The second inference execution graph can be understood as a CUDA graph constructed based on the second model weight in the second virtual address, used to execute the second inference task.

[0093] The second inference task can be understood as a model inference calculation using the second inference execution graph on actual input data of the second inference model.

[0094] Specifically, when the cloud platform needs to switch to the second inference model, the inference framework responds to the model cold switching command. The inference framework first sends an open hijacking command to the hijacking module, which can include a fixed virtual address space. During the process of starting to load the second model weight, when the inference framework calls the CUDA memory allocation API, the hijacking module in standby state immediately intercepts the call. The hijacking module divides another continuous address segment from the pre-reserved fixed virtual address space carried in the open hijacking command as a second virtual address according to the address allocation strategy maintained internally, and redirects the memory application to the second virtual address. The inference framework loads the second model weight of the second inference model into the physical memory where the second virtual address exists in a mapping relationship without awareness.

[0095] After the inference framework determines that the second model weight is completed, the inference framework sends an end hijacking command to the hijacking module to stop intercepting the memory application. Subsequently, the inference framework starts the CUDA Graph capture process based on the second model weight stored in the second virtual address, and constructs a second inference execution graph for the second inference model. The second inference execution graph records the complete calculation process on the fixed weight address, and can be used for efficient execution of a large number of second inference tasks after construction.

[0096] The data processing method provided by one or more embodiments of the present specification first allocates independent fixed address segments for different inference models to ensure that the CUDA Graphs of each inference model do not interfere with each other and are persistent and effective; second, the unified address space management mechanism enables efficient turnover of limited GPU memory resources among multiple inference models. Through this method, when supporting complex multi-inference model project scenarios, the inference performance of each inference model can be guaranteed to be optimal, and the CUDA Graph reconstruction overhead caused by address changes can be avoided during inference model switching, thereby providing scalable and high-performance service capabilities for MaaS platforms.

[0097] In order to realize fast model switching and seamless service switching, in the case of loading the first model weight and the second model weight to the memory of the target GPU, the first model weight and the second model weight can also be backed up to the target central processor asynchronously. The specific implementation is as follows:

[0098] After the end hijacking command is sent to the hijacking module, the method further includes:

[0099] The inference framework backs up the first model weight in the first virtual address to a target central processor; and

[0100] The inference framework backs up the second model weight in the second virtual address to the target central processor.

[0101] The target central processing unit can be understood as a target CPU, and is configured to persistently store backup copies of the first model weight and the second model weight.

[0102] Specifically, after sending the end hijacking command to the hijacking module and completing the loading and CUDA Graph construction of the current model, the inference framework asynchronously starts the backup process of the model weight. The inference framework can use the memory copy interface of CUDA to copy the first model weight stored in the first virtual address from the target GPU memory corresponding to the first virtual address to the memory of the target central processing unit in an asynchronous transmission manner. Similarly, after the second model weight is loaded, the inference framework also backs up the second model weight in the target GPU memory corresponding to the second virtual address to the CPU memory.

[0103] The data processing method provided by one or more embodiments of the present specification builds a hierarchical model storage system by backing up the model weight in the fixed virtual address to the CPU. First, a reliable disaster recovery mechanism is provided, which can quickly recover from the CPU backup when unexpected situations cause data damage or loss in the video memory, thereby ensuring service continuity. Second, through this mechanism, only the model weight of the active model can be retained in the GPU video memory, while the model weight of the non-active but possibly called model is temporarily stored in the CPU, which significantly expands the model capacity that can be managed by a single card. In combination with the fixed virtual address space design, the model weight recovered from the CPU can be accurately loaded back to its original memory location, ensuring that the existing CUDA Graph is completely effective without the need for reconstruction, thereby realizing fast model rotation and seamless service switching.

[0104] In addition, when the inference framework receives a model hot switching command sent by the cloud platform, the second model weight in the second virtual address is also released to free up the physical video memory of the target GPU corresponding to the second virtual address. The specific implementation is as follows:

[0105] After constructing the second inference execution graph based on the second model weight in the second virtual address and using the second inference execution graph to execute the second inference task, the method further includes:

[0106] The inference framework sends the release memory command to the hijacking module and releases the coupled data related to the second inference model in response to the model hot switching command sent by the cloud platform.

[0107] The hijacking module releases the mapping relationship between the second virtual address and the video memory of the target GPU and releases the second model weight in the second virtual address in response to the release memory command, and retains the second virtual address.

[0108] The model hot switching command can be understood as an instruction issued by the cloud platform, indicating switching from a currently running inference model to another model without interrupting service.

[0109] The second inference model related coupling data can be understood as dynamic data related to the second inference model, including but not limited to KV Cache (key value cache), intermediate activation value and other non-weight data.

[0110] The disassociation relationship can be understood as the hijacking module disconnecting the association between the second virtual address and the physical memory page of the target GPU through the CUDA virtual memory management API, thereby releasing the physical memory resource.

[0111] Specifically, when the inference framework receives the model hot switching command sent by the cloud platform, it immediately starts the resource release process. The inference framework first sends a release memory command for the second model weight to the hijacking module, which can contain the model identification information of the second inference model. Then, the inference framework cleans up the second inference model related coupling data in parallel, such as cleaning up KV Cache and other dynamic runtime data through the standard CUDA memory release interface, to ensure that these temporary resources are recycled in time.

[0112] The hijacking module, after receiving the release memory command, accurately locates the position range of the second virtual address in the fixed virtual address space by querying the internally maintained address mapping table, etc. The hijacking module can release the mapping relationship between the second virtual address and the physical memory page by calling the CUDA virtual memory management API, thereby releasing the physical memory resource actually occupied by the second model weight of the second inference model. The second virtual address itself remains in the fixed virtual address space, and only the binding with the physical memory is released, maintaining address consistency for subsequent possible reloading.

[0113] The data processing method provided by one or more embodiments of the present specification ensures that the useless memory is immediately released during model switching by the hijacking module for targeted release of the fixed address space, in combination with the inference framework for cleaning up KV Cache and other coupling data, freeing up sufficient space for subsequent model loading, and significantly improving the utilization rate of GPU memory resources. The release operation only releases the physical mapping while retaining the virtual address segment, ensuring that the fixed virtual address allocated for each model remains unchanged during the life cycle of multiple loading and unloading, thereby making it unnecessary to rebuild the CUDA Graph previously constructed for the model due to address changes, solving the high first inference delay problem caused by Graph rebuilding in the hot switching scenario.

[0114] After releasing the second model weight in the second virtual address, model hot switching can be achieved in the hot start phase, and the specific implementation manner is as follows:

[0115] After releasing the second model weight in the second virtual address, the method further includes:

[0116] The inference framework sends the hijacking command to the hijacking module, and during the process of loading the first model weight of the first inference model, reads the first model weight from the target central processing unit, and creates a tensor on the target graphics processor to represent the first model weight through the target interface, thereby triggering a request for video memory on the target graphics processor.

[0117] The hijacking module, in response to the hijacking command, hijacks the video memory request and redirects the video memory request to the first virtual address of the fixed virtual address space;

[0118] When the inference framework determines that the first model weights have been loaded, it sends the end hijacking command to the hijacking module and executes the first inference task using the first inference execution graph.

[0119] The target interface can be understood as the Torch C++ API, which refers to the C++ interface of the PyTorch framework. Specifically, it is the torch::from_blob() function, which is used to create tensors on the GPU without copying the data.

[0120] Creating a tensor on the target graphics processor to represent the weights of the first model can be understood as creating a tensor on the target graphics processor to represent the weights of the first model using the torch::from_blob() function. That is, using the from_blob() method provided by PyTorch to create a tensor object from an existing memory block. This process will trigger underlying video memory allocation.

[0121] Triggering a memory allocation request for the target graphics processor can be understood as triggering a CUDA API call. When torch::from_blob() is called to create a tensor, PyTorch will automatically call CUDA APIs such as cudaMalloc to allocate memory space for the target GPU.

[0122] Hijacking the aforementioned memory allocation can be understood as the hijacking module intercepting CUDA API calls, that is, the hijacking module intercepting memory allocation calls such as cudaMalloc issued by the underlying PyTorch framework.

[0123] Specifically, in response to the model hot switching command sent by the cloud platform, the inference framework first sends an opening hijacking command to the hijacking module after releasing the target GPU memory resource. Then, the inference framework reads the backup data of the first model weight of the first inference model from the target CPU memory, and prepares to create a corresponding tensor object on the target GPU through the torch::from_blob() interface of the PyTorch C++ API. This creation process triggers the PyTorch underlying to call the CUDA API to apply for memory.

[0124] At this time, the hijacking module that has been activated immediately intercepts the CUDA API call and redirects it to the preset first virtual address in the fixed virtual address space. The inference framework associates the first model weight in the CPU with the redirected fixed memory address through torch::from_blob() without awareness, and completes the recovery and loading of the first model weight. After confirming that all the first model weights of the first inference model have been loaded, the inference framework sends an end hijacking command to the hijacking module to stop intercepting the CUDA API. Since the first model weight is accurately restored to the original fixed address: the first virtual address, the inference framework can directly reuse the pre-constructed first inference execution graph and immediately start processing the first inference task.

[0125] The data processing method provided by one or more embodiments of the present specification realizes non-invasive memory address control of the inference framework by integrating the torch::from_blob() interface of the PyTorch framework and the CUDA API hijacking mechanism. First, the PyTorch API is used to trigger memory allocation; second, the underlying CUDA API call is intercepted to accurately control the memory location of the weight tensor, realize address fixation, solve the problem of invalid CUDA Graph due to address change, and provide a high-performance and highly compatible model hot switching solution for the PyTorch-based MaaS platform.

[0126] The data processing method provided by one or more embodiments of the present specification first realizes controllability and consistency of address allocation by redirecting the memory application operation in the model loading and switching process to the pre-reserved fixed virtual address space based on hijacking of the CUDA memory API, ensuring the stability of the location of the model weight in the memory. Secondly, by realizing the weight memory address consistency management, cooperating with the designed model weight memory layout management, the weight address conflict between different models is ensured, and the physical address remains unchanged in multiple switching. Through the mechanism of pre-reserved fixed virtual address space + long-occupied physical memory page bound to this part of address space + only unmapping without releasing the physical page during switching, the same model can always be allocated to the same virtual address in the multiple load-unload cycles, and the physical memory page mapped behind the virtual address is always the original one. The CUDA Graph records the virtual address, and the mapping relationship between the virtual address and the physical address does not change throughout the life cycle of the Graph, so the Graph can continue to run efficiently without rebuilding, thereby avoiding the forced rebuilding of the CUDA Graph due to address changes, ensuring the continuous optimization of the inference performance. In addition, the explicit memory management interface introduced in the inference framework can actively trigger the controlled memory allocation process, avoiding the hijacking failure caused by the uncertainty of the memory application API call in the inference framework, and improving the reliability and stability. Specifically, the method can realize efficient hot switching between multiple models on the basis of ensuring the persistence and effectiveness of the CUDA Graph, significantly improving the GPU resource utilization and inference service performance.

[0127] Specifically, after the cold start instruction is sent to the inference framework by the cloud platform according to the analysis result after the cloud platform analyzes the first inference task sent by the user, the specific implementation manner is as follows:

[0128] The data processing system further comprises the cloud platform.

[0129] Before the cold start instruction sent by the cloud platform, it further comprises:

[0130] The cloud platform analyzes the first inference task sent by the user, determines the first inference model for executing the first inference task and the inference framework, and allocates the target graphics processor for the first inference model.

[0131] In the case where it is determined that the inference framework is not started on the target graphics processor, a cold start instruction is sent to the inference framework.

[0132] The first inference task can be understood as a specific inference task instruction initiated by a user and requiring a large model for processing. The first inference task can include task content and related parameters. For example, the first inference task can be a user sending "generate a picture of a man running on the lake shore".

[0133] The inference framework is not started on the target GPU, which can be understood as that there is neither a process instance of the inference framework running on the target GPU nor the first inference model required by the first inference task loaded on the target GPU, and the target GPU is in an initial state of being completely unused.

[0134] In specific implementation, when a user submits a first inference task to a cloud platform, the cloud platform first analyzes the specific requirements of the first inference task, intelligently identifies a first inference model suitable for processing the first inference task and an inference framework corresponding to the first inference model. Specifically, the cloud platform can maintain a model-framework registry, which records the inference framework corresponding to each deployed model and the resource requirements thereof. When analyzing the first inference task, the cloud platform extracts the model identifier or task type label in the first inference task as a query key and performs matching lookup in the framework registry, so as to determine the specific model and the inference framework bound thereto that should be called. Subsequently, the cloud platform allocates an appropriate target GPU for the first inference task from a pool of available GPU resources. Specifically, the cloud platform pre-maintains a resource pool including real-time states (such as total memory, remaining memory, computing power type, and current load) of GPUs. When receiving the first inference task, the scheduler of the cloud platform analyzes the resource configuration requirements (such as required memory size and computing accuracy requirement) of the first inference model corresponding to the first inference task, and then selects a GPU that simultaneously satisfies the resource specifications and has a lighter current load as the target GPU from the resource pool based on a preset strategy (such as optimal matching and load balancing). After the target GPU is allocated, the cloud platform checks whether the inference framework is loaded on the target GPU for the first time (that is, whether there is an inference service instance that can directly process the first inference task and is ready). If so, the cloud platform immediately sends a cold start instruction to the inference framework.

[0135] The data processing method provided by one or more embodiments of the present specification can intelligently schedule the cloud platform. When receiving a first inference task of a user, the cloud platform allocates appropriate GPU resources for a first inference model corresponding to the first inference task. If it is determined that the inference framework has not been started on the target GPU, the cloud platform triggers cold start of the inference framework, realizes precise pre-allocation of GPU resources and rapid initialization of the inference framework, ensures that the first inference task of the user can be responded to immediately, avoids idle GPU resources, and significantly improves GPU utilization and response efficiency of the first inference task.

[0136] Specifically, the parsing of the first inference task sent by the user, the determination of the first inference model for executing the first inference task and the inference framework, and the allocation of the target GPU for the first inference model, comprise:

[0137] The cloud platform parses the model identification parameter carried in the first inference task sent by the user, determines the first inference model for executing the first inference task and the inference framework corresponding to the first inference model;

[0138] According to the model parameters of the first inference model, the target GPU is allocated for the first inference model.

[0139] The model identification parameter can be understood as identification information carried in the first inference task for determining the first inference model, for example, the model identification parameter can be the model name, version number or identifier of the first inference model.

[0140] The model parameters of the first inference model can be understood as resource demand characteristics of the first inference model, such as scale, memory demand and computational complexity.

[0141] In specific implementation, after the cloud platform receives the first inference task sent by the user, the model identification parameter carried in the first inference task is first extracted, and the first inference model to be called and the inference framework corresponding to the first inference model are determined by querying the model registration center and the like. Secondly, the cloud platform analyzes the parameter scale and resource demand characteristics of the first inference model, including model parameters such as memory occupation and computational complexity. Then, based on the resource demand information obtained by analyzing the model parameters, the cloud platform intelligently selects a more matched GPU from the available GPU resource pool as the target GPU, to ensure that the hardware performance accurately corresponds to the demand of the first inference model.

[0142] The data processing method provided by one or more embodiments of the present specification intelligently parses the model identification parameter carried in the first inference task sent by the user, and automatically matches appropriate GPU computing resources for the first inference task. The user only needs to specify the required first inference model, and the cloud platform can automatically complete the GPU resource allocation and service preparation, significantly simplifying the use process of AI services, and the user can quickly obtain a good AI service experience without understanding the underlying technical details, while ensuring efficient use of GPU computing resources.

[0143] In the process of cold starting the inference framework on the target GPU, the first model weight of the first inference model is loaded to the memory of the target GPU, and is hijacked and redirected to the first virtual address in the fixed virtual address space corresponding to the first inference model by the hijacking module. When the inference framework determines that the first model weight is loaded, the hijacking module sends an end hijacking command, and constructs a first inference execution graph based on the first model weight in the first virtual address to execute the first inference task using the first inference execution graph. The specific implementation is as follows:

[0144] The constructing a first inference execution graph based on the first model weight in the first virtual address to execute the first inference task using the first inference execution graph includes:

[0145] The inference framework constructs a first inference execution graph based on the first model weight in the first virtual address, and executes the first inference task using the first inference execution graph by using the first inference model.

[0146] In specific implementation, after the loading of the first model weight is completed, the inference framework starts the construction process of the first inference execution graph. The inference framework can first create a dedicated CUDA stream as a graph capture stream, and then call the capture start API of the CUDA Graph to start recording the calculation operation. In this capture state, the inference framework performs the complete forward inference process of the first inference model: the example input data is passed through the first model weight loaded to the first virtual address, and all calculation layers defined by the model are executed in turn, including matrix multiplication, convolution operation, activation function and all CUDA kernels. Since all weight data is located in the fixed first virtual address, the memory address referenced in all calculation kernels remains consistent during the entire capture process. After the capture is completed, the inference framework calls the graph instantiation API to compile and optimize the recorded operation sequence into a static and directly executable first inference execution graph.

[0147] In subsequent processing of the first inference task, the inference framework no longer performs a dynamic forward calculation process, but directly starts the constructed first inference execution graph. For each input data, the inference framework only needs to copy the data to the input node memory address of the graph, and then execute the start command of the first inference execution graph. The entire calculation process will be efficiently executed on the GPU, avoiding the kernel start overhead and the uncertainty of dynamic memory addressing.

[0148] The data processing method provided by one or more embodiments of the present specification executes the first inference task through the inference execution graph constructed based on the fixed address weight, and achieves significant optimization of task execution efficiency. The inference execution graph constructed based on the fixed address weight can be directly executed without the need for dynamic memory addressing and kernel start overhead, and the execution efficiency of the inference task is improved. Figure 1The constructed model can be repeatedly called in task execution, avoiding the overhead of rebuilding the calculation graph every time the inference task is executed, thereby greatly reducing the task processing delay and improving the real-time response speed.

[0149] In addition, the model cold switching command received by the inference framework is issued by the cloud platform in the case that the first inference model does not process the first inference task sent by the user within a preset time period, and the specific implementation manner is described as follows.

[0150] The method further comprises:

[0151] The cloud platform sends a model cold switching command to the inference framework in the case that the first inference model does not receive a new first inference task sent by the user within a preset time period.

[0152] The preset time period can be understood as a pre-set continuous time threshold, which is used to determine whether the target GPU computing resource is in an underutilized state. For example, the preset time period can be set to 5 minutes or 10 minutes, etc.

[0153] The model cold switching command can be understood as a command issued by the cloud platform to the inference framework, requiring the first inference model to be unloaded and the related resources to be released, in order to prepare for switching to other inference models or entering a standby state.

[0154] Specifically, the cloud platform continuously monitors the task receiving state of the first inference model, records the time stamp of the last time the first inference task is received through an internal timer, etc. When the time difference between the current time and the last time the first inference task is received exceeds the preset time period, the cloud platform determines that the first inference model is in an idle state, and then generates a model cold switching command for the first inference model and sends it to the inference framework through a network interface. After receiving the model cold switching command, the inference framework starts the cold switching process: first, sends a release video memory command to the hijacking module to release the mapping relationship between the first model weight and the fixed virtual address; then, cleans up the coupled data (such as KV Cache) related to the first inference model; finally, marks the model state as unloaded, and confirms the switching completion to the cloud platform.

[0155] The data processing method provided in one or more embodiments of this specification achieves efficient recycling and dynamic allocation of GPU memory resources by utilizing an automatic cold switching mechanism based on idle detection. First, through intelligent judgment of preset time windows, idle memory resources occupied by inference models can be released in a timely manner, avoiding resource waste and significantly improving GPU utilization and overall service throughput. Second, this automated resource management reduces the need for operational intervention, enabling the platform to more flexibly cope with fluctuating workloads and support efficient rotation of more models. Furthermore, the precise triggering of cold switching commands ensures that inference models can enter a sleep state when there are no tasks, and combined with fixed addresses, maintains an address consistency basis for subsequent inference model loading, thereby maintaining rapid response capabilities while saving resources.

[0156] Furthermore, the hot-swap command is triggered by the cloud platform when it receives a new first inference task from the user, and is sent to the inference framework. The specific implementation is as follows:

[0157] The method further includes:

[0158] The cloud platform receives a new first inference task sent by the user and sends a model hot-switching command to the inference framework.

[0159] The model hot-switching command can be understood as a command issued by the cloud platform to the inference framework, requiring it to switch to the first inference model to handle the new first inference task while maintaining service continuity.

[0160] Specifically, when the cloud platform receives a new first inference task from a user, it first checks the current state of the first inference model. If it detects that the first inference model is not currently in an active state, the cloud platform can send a model hot-switching command to the corresponding inference framework instance through a pre-established communication channel. This model hot-switching command can include metadata such as the model identifier to ensure that the inference framework can accurately identify the first inference model that needs to be switched.

[0161] The data processing method provided in one or more embodiments of this specification achieves a balance between service response speed and resource efficiency through an intelligent model hot-switching mechanism triggered by user requests. Firstly, it ensures that model switching has a clear target orientation, avoiding resource idleness and significantly improving service responsiveness and user experience. Secondly, the model hot-switching process maintains service continuity, and users do not need to be aware of the underlying model scheduling process, achieving a seamless service experience.

[0162] See Figure 2 , Figure 2 This is a two-stage workflow diagram of a data processing method provided in one embodiment of this specification, which specifically includes the following steps.

[0163] Phase 1: Cold start.

[0164] Step 202: The inference framework sends an open hijack command to the hijack module in response to the cold start instruction sent by the cloud platform, which can contain the model identifier of Model-A and the corresponding fixed virtual address space.

[0165] Step 204: The inference framework starts loading the model weights of Model-A from the disk, and in this process, initiates a standard process to request memory allocation.

[0166] Step 206: The hijack module monitors memory allocation calls in real time, intercepts the memory application issued by the inference framework, and redirects it to the specified location of the pre-reserved fixed virtual address space.

[0167] Step 208: After the inference framework completes the loading of all model weights of Model-A, it sends an end hijack command to the hijack module, notifying the hijack module to stop the interception operation of memory application.

[0168] Step 210: When the inference framework needs to perform model cold switching, it sends a release memory command for Model-A to the hijack module.

[0169] Step 212: The hijack module responds to the release memory command, removes the mapping relationship between the virtual address corresponding to the Model-A weight and the physical memory page, and releases the physical memory resource.

[0170] Step 214: The inference framework cleans up the runtime coupling data related to Model-A, including releasing the memory space occupied by the KV cache.

[0171] Step 216: The inference framework sends an open hijack command to the hijack module, which can contain the model identifier of Model-B and the corresponding fixed virtual address space.

[0172] Step 218: The inference framework starts loading the model weights of Model-B from the disk, and in this process, initiates a standard process to request memory allocation.

[0173] Step 220: The hijack module monitors memory allocation calls in real time, intercepts the memory application issued by the inference framework, and redirects it to the specified location of the pre-reserved fixed virtual address space.

[0174] Step 222: After the inference framework completes the loading of all model weights of Model-B, it sends an end hijack command to the hijack module, notifying the hijack module to stop the interception operation of memory application.

[0175] Phase 2: Hot switching.

[0176] Step 224: The inference framework sends a release GPU memory command for Model-B to the hijacking module when it needs to perform model hot switching.

[0177] Step 226: The hijacking module responds to the release GPU memory command by removing the mapping relationship between the virtual address corresponding to the Model-B weight and the physical GPU memory page, and releasing the physical GPU memory resource.

[0178] Step 228: The inference framework cleans up the runtime coupling data related to Model-B, including releasing the GPU memory space occupied by the KV cache.

[0179] Step 230: The inference framework sends an enable hijacking command to the hijacking module, which can include the model identifier of Model-A and the corresponding fixed virtual address space.

[0180] Step 232: The inference framework explicitly triggers a GPU memory allocation request by calling the torch::from_blob() method of the Torch C++ API and passing in the Model-A weight data pointer loaded from the CPU. This operation drives the PyTorch underlying framework to automatically call the CUDA GPU memory allocation API, thereby actively activating and executing the subsequent hijacking process.

[0181] Step 234: The hijacking module monitors GPU memory allocation calls in real time, intercepts GPU memory applications issued by the inference framework, and redirects them to the specified location in the pre-reserved fixed virtual address space.

[0182] Step 236: After completing the loading of all model weights of Model-A, the inference framework sends an end hijacking command to the hijacking module, notifying the hijacking module to stop the interception operation on GPU memory applications.

[0183] The data processing method provided by one or more embodiments of the present specification pre-allocates a large-capacity, fixed-mapping virtual GPU memory address space and constructs GPU layout management inside it, effectively solving the limitation of the vLLM's built-in sleep-wake mechanism, which only supports single-model operation, and meeting the frequent multi-model on-demand loading and switching requirements in the MaaS platform. When switching models, only the target model weight needs to be loaded from the CPU memory to the GPU, and it is ensured that it falls within the pre-set address interval, thereby realizing fast model hot switching without affecting the effectiveness of the CUDA Graph.

[0184] Furthermore, compared to container-based scheduling solutions, this method can directly perform hot model switching within the current container of the current POD, avoiding container-level add, delete, and modify operations, thus significantly reducing subsequent operation and maintenance costs. By allowing the inference framework to deeply participate in the model hot switching process, it overcomes the weakness of the underlying system's ability to perceive upper-level elements in existing solutions, significantly improving the reusability of context resources, efficiently fulfilling project requirements with minimal operational costs, and ensuring consistently superior inference performance while enhancing service flexibility.

[0185] Corresponding to the above method embodiments, this specification also provides data processing system embodiments. Figure 3 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification. Figure 3 As shown, the system includes:

[0186] The inference framework 302 is used to respond to the cold start command sent by the cloud platform, perform a cold start on the target graphics processor, send an enable hijacking command to the hijacking module 304, and initiate a video memory request to the target graphics processor during the process of loading the first model weight of the first inference model.

[0187] The hijacking module 304 is configured to, in response to the hijacking command, hijack a video memory request and redirect the video memory request to a first virtual address in a fixed virtual address space, wherein the fixed virtual address space is a pre-reserved continuous virtual address range in the virtual address space of the target graphics processor, the first virtual address is a continuous sub-region divided from the fixed virtual address space, and the first virtual address is mapped to the video memory of the target graphics processor for storing the first model weights;

[0188] The inference framework 302 is used to send a termination hijacking command to the hijacking module 304 when it is determined that the first model weight has been loaded, and to construct a first inference execution graph based on the first model weight in the first virtual address, so as to use the first inference execution graph to execute the first inference task.

[0189] Optionally, the inference framework 302 is further configured to, in response to the model cold switching command sent by the cloud platform, send a release memory command to the hijacking module 304 and release the coupling data related to the first inference model;

[0190] The hijacking module 304 is further configured to, in response to the release memory command, release the mapping relationship between the first virtual address and the video memory of the target graphics processor, release the first model weight in the first virtual address, and retain the first virtual address.

[0191] Optionally, the inference framework 302 is further configured to send the start hijacking command to the hijacking module 304, and initiate a video memory application to the target GPU in a process of loading second model weights of a second inference model.

[0192] The hijacking module 304 is further configured to, in response to the start hijacking command, hijack the video memory application, and redirect the video memory application to a second virtual address in the fixed virtual address space, wherein the second virtual address is mapped to a video memory of the target GPU, and used to store the second model weights.

[0193] The inference framework 302 is further configured to, in a case where it is determined that the loading of the second model weights is completed, send the end hijacking command to the hijacking module 304, and construct a second inference execution graph based on the second model weights in the second virtual address, so as to execute a second inference task by using the second inference execution graph.

[0194] Optionally, the inference framework 302 is further configured to backup the first model weights in the first virtual address to a target central processing unit (CPU); and

[0195] The inference framework 302 is further configured to backup the second model weights in the second virtual address to the target CPU.

[0196] Optionally, the inference framework 302 is further configured to, in response to a model hot switching command sent by a cloud platform, send the release video memory command to the hijacking module 304, and release coupled data related to the second inference model.

[0197] The hijacking module 304 is further configured to, in response to the release video memory command, release a mapping relationship between the second virtual address and the video memory of the target GPU, release the second model weights in the second virtual address, and retain the second virtual address.

[0198] Optionally, the inference framework 302 is further configured to send the start hijacking command to the hijacking module 304, and in a process of loading first model weights of the first inference model, read the first model weights from the target CPU, and create a tensor located on the target GPU and used to represent the first model weights through a target interface, to trigger a video memory application to the target GPU.

[0199] The hijacking module 304 is further configured to, in response to the start hijacking command, hijack the video memory application, and redirect the video memory application to a first virtual address in the fixed virtual address space.

[0200] The inference framework 302 is further configured to send the end hijacking command to the hijacking module 304 and execute the first inference task using the first inference execution graph in a case where it is determined that the loading of the first model weight is completed.

[0201] Optionally, the data processing system further comprises the cloud platform.

[0202] The cloud platform is configured to parse the first inference task sent by the user, determine the first inference model for executing the first inference task and the inference framework 302, and allocate the target GPU to the first inference model.

[0203] In a case where it is determined that the inference framework 302 is not started on the target GPU, a cold start instruction is sent to the inference framework 302.

[0204] Optionally, the inference framework 302 is further configured to construct a first inference execution graph based on the first model weight in the first virtual address, and execute the first inference task using the first inference model and the first inference execution graph.

[0205] Optionally, the cloud platform is further configured to parse a model identifier parameter carried in the first inference task sent by the user, determine the first inference model for executing the first inference task and the inference framework 302 corresponding to the first inference model.

[0206] The target GPU is allocated to the first inference model according to the model parameter of the first inference model.

[0207] Optionally, the cloud platform is configured to send a model cold switching command to the inference framework 302 in a case where it is detected that the first inference model does not receive a new first inference task sent by the user within a preset time period.

[0208] Optionally, the cloud platform is configured to receive a new first inference task sent by the user and send a model hot switching command to the inference framework 302.

[0209] The above is a schematic scheme of the data processing system of the embodiment. It should be noted that the technical scheme of the data processing system and the technical scheme of the data processing method described above belong to the same concept, and the details of the technical scheme of the data processing system that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0210] Referring to Figure 4 , Figure 4 is a structural block diagram of a computing device 400 provided by an embodiment of the present specification.

[0211] The components of computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0212] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of these and other networks such as the Internet. The access device 440 can include one or more of any type of network interface (for example, a network interface card (NIC)), wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC).

[0213] In one embodiment of the present specification, the above-mentioned components of the computing device 400 and other components not shown in the Figure 4 may be connected to each other, for example, via a bus. It should be understood that Figure 4 The computing device structure diagram shown is for the purpose of example only and is not a limitation on the scope of the present specification. Other components can be added or replaced as needed by those skilled in the art.

[0214] The computing device 400 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 400 can also be a mobile or stationary server.

[0215] The processor 420 is configured to execute the following computer-executable instructions, which implement the steps of the data processing method described above.

[0216] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the data processing method described above belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0217] An embodiment of the present specification further provides a computer-readable storage medium storing computer programs / instructions, which implement the steps of the data processing method described above when executed by a processor.

[0218] The above is a schematic scheme of the computer-readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the data processing method described above belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0219] An embodiment of the present specification further provides a computer program product comprising computer programs / instructions, which implement the steps of the data processing method described above when executed by a processor.

[0220] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the data processing method described above belong to the same concept, and the details of the technical scheme of the computer program product that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0221] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order described in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous or possible.

[0222] The computer program / instructions can include a computer program code, which can be in a form of source code, object code, executable file, or some intermediate form etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of patent practice, for example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0223] It should be noted that, for the foregoing method embodiments, in order to facilitate description, they are all described as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present specification are not limited by the order of the described actions, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of the present specification.

[0224] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0225] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and limit the invention to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and use the present specification. The present specification is limited by the claims and their entire scope and equivalents.

Claims

1. A data processing method applied to a data processing system, the data processing system including an inference framework and a hijacking module, the method comprising: The inference framework, in response to the cold start command sent by the cloud platform, performs a cold start on the target graphics processor, sends an enable hijacking command to the hijacking module, and initiates a video memory request to the target graphics processor during the process of loading the first model weight of the first inference model. The hijacking module, in response to the hijacking command, hijacks a video memory request and redirects the video memory request to a first virtual address in a fixed virtual address space. The fixed virtual address space is a pre-reserved continuous virtual address range in the virtual address space of the target graphics processor. The first virtual address is a continuous sub-region divided from the fixed virtual address space, and the first virtual address is mapped to the video memory of the target graphics processor to store the first model weights. When the inference framework determines that the first model weight has been loaded, it sends a command to end the hijacking to the hijacking module and constructs a first inference execution graph based on the first model weight in the first virtual address, so as to use the first inference execution graph to execute the first inference task.

2. The data processing method according to claim 1, after constructing a first inference execution graph based on the first model weight in the first virtual address, and using the first inference execution graph to execute the first inference task, further includes: In response to the model cold switching command sent by the cloud platform, the inference framework sends a command to release video memory to the hijacking module and releases the coupling data related to the first inference model. In response to the command to release video memory, the hijacking module releases the mapping relationship between the first virtual address and the video memory of the target graphics processor, releases the first model weight in the first virtual address, and retains the first virtual address.

3. The data processing method according to claim 2, after releasing the first model weight in the first virtual address, further comprising: The inference framework sends the hijacking command to the hijacking module and, during the process of loading the second model weights of the second inference model, initiates a video memory request to the target graphics processor. The hijacking module, in response to the hijacking command, hijacks the video memory request and redirects the video memory request to a second virtual address in the fixed virtual address space, wherein the second virtual address is mapped to the video memory of the target graphics processor for storing the second model weights; When the inference framework determines that the second model weights have been loaded, it sends the end hijacking command to the hijacking module and constructs a second inference execution graph based on the second model weights in the second virtual address, so as to use the second inference execution graph to execute the second inference task.

4. The data processing method according to claim 3, after sending the end-hijacking command to the hijacking module, further includes: The inference framework backs up the first model weight in the first virtual address to a target central processing unit; And The inference framework backs up the second model weight in the second virtual address to the target central processing unit.

5. The data processing method of claim 4, after the constructing a second inference execution graph based on the second model weight in the second virtual address to execute a second inference task using the second inference execution graph, further comprising: The inference framework, in response to a model hot switching command sent by the cloud platform, sends the release video memory command to the hijacking module, and releases the coupled data related to the second inference model; The hijacking module, in response to the release video memory command, releases the mapping relationship between the second virtual address and the video memory of the target graphics processing unit, releases the second model weight in the second virtual address, and retains the second virtual address.

6. The data processing method of claim 5, after the releasing the second model weight in the second virtual address, further comprising: The inference framework sends the start hijacking command to the hijacking module, reads the first model weight from the target central processing unit during the process of loading the first model weight of the first inference model, and creates a tensor on the target graphics processing unit for representing the first model weight through a target interface to trigger a video memory application for the target graphics processing unit; The hijacking module, in response to the start hijacking command, hijacks the video memory application and redirects the video memory application to the first virtual address of the fixed virtual address space; The inference framework, in the case of determining that the first model weight is completed, sends the end hijacking command to the hijacking module, and executes the first inference task using the first inference execution graph.

7. The data processing method of any one of claims 1-6, the data processing system further comprising the cloud platform; Before the responding to the cold start instruction sent by the cloud platform, further comprising: The cloud platform parses a first inference task sent by a user, determines the first inference model and the inference framework for executing the first inference task, and allocates the target graphics processing unit for the first inference model; In the case of determining that the inference framework is not started on the target graphics processing unit, sends a cold start instruction to the inference framework.

8. The data processing method of any one of claims 1-6, the constructing a first inference execution graph based on the first model weight in the first virtual address to execute a first inference task using the first inference execution graph, comprising: The inference framework constructs a first inference execution graph based on the first model weight in the first virtual address, and executes the first inference task using the first inference execution graph by using the first inference model.

9. The data processing method according to claim 7, wherein parsing the first inference task sent by the user, determining the first inference model and the inference framework for executing the first inference task, and allocating the target graphics processor to the first inference model, comprises: The cloud platform parses the model identifier parameters carried in the first inference task sent by the user to determine the first inference model for executing the first inference task and the inference framework corresponding to the first inference model. Based on the model parameters of the first inference model, the target graphics processor is assigned to the first inference model.

10. The data processing method according to claim 7, further comprising: When the cloud platform detects that the first inference model has not received a new first inference task from the user within a preset time period, it sends a model cold switch command to the inference framework.

11. The data processing method according to claim 7, further comprising: The cloud platform receives a new first inference task sent by the user and sends a model hot-switching command to the inference framework.

12. A data processing system, comprising an inference framework and a hijacking module, wherein, The inference framework is used to respond to the cold start command sent by the cloud platform, perform a cold start on the target graphics processor, send an enable hijacking command to the hijacking module, and initiate a video memory request to the target graphics processor during the loading of the first model weight of the first inference model. The hijacking module is configured to, in response to the hijacking command, hijack a video memory request and redirect the video memory request to a first virtual address in a fixed virtual address space, wherein the fixed virtual address space is a pre-reserved continuous virtual address range in the virtual address space of the target graphics processor, the first virtual address is a continuous sub-region divided from the fixed virtual address space, and the first virtual address is mapped to the video memory of the target graphics processor for storing the first model weights; The inference framework is used to send a command to end the hijacking to the hijacking module when it is determined that the first model weight has been loaded, and to construct a first inference execution graph based on the first model weight in the first virtual address, so as to use the first inference execution graph to execute the first inference task.

13. A computing device, comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the inference framework and the hijacking module in any one of claims 1-11.

14. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the inference framework and hijacking module in the method of any one of claims 1-11.

15. A computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the inference framework and the hijacking module in the method of any one of claims 1-11.

Images