Memory management method and device for inference system

Abstract

One or more embodiments of the invention provide a memory management method and device for an inference system. The method is applied to an inference engine in the inference system. Computing resources of the inference engine comprise a GPU (Graphic Processing Unit) carried on computing equipment for deploying the inference engine; the reasoning engine maintains a scheduling queue for scheduling a reasoning request set; the method comprises the following steps: determining a memory management time window according to data processing duration associated with a reasoning request set which is being executed in a scheduling queue; calculating a GPU memory demand quantity corresponding to the reasoning request set in the memory management time window, and distributing a GPU memory for the reasoning request set according to the GPU memory demand quantity; and when the memory management time window ends, determining the next memory management time window corresponding to the memory management time window again according to the data processing duration associated with the inference request set which is being executed in the scheduling queue.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a memory management method and device for an inference system. Background Art

[0002] An inference system is a computer program that uses logical rules and known facts to draw new conclusions or decisions. An inference system is an important part of the field of artificial intelligence, mainly used to simulate the human decision-making process. It derives conclusions based on a set of defined knowledge bases and inference engines. An inference system can execute the inference request it obtains and output the corresponding inference results.

[0003] A typical reasoning system usually consists of the following parts: Knowledge Base, Inference Engine, User Interface, and Explanation Facility. The knowledge base includes all the facts and rules known to the storage system. These facts can be about the state of the world, object properties, etc., while the rules are logical expressions that describe how to draw new conclusions from known facts. The inference engine is the core component of the reasoning system. It is responsible for performing logical operations in the reasoning process, that is, drawing new conclusions or decisions from a given knowledge base; the inference engine uses a series of rules and known facts to derive new knowledge to help the system solve problems or make decisions. The user interface allows users to interact with the system, input queries, or observe the results of the reasoning process. The explanation mechanism is used to explain how the system draws specific conclusions, which is very important for transparency and trust.

[0004] Inference engines usually use computing resources, storage resources, network resources, etc. (e.g., GPU, GPU memory, storage, network interface, etc.) to perform inference tasks. The efficient use of these resources directly affects the performance of the inference engine. Therefore, it is expected to manage the resources used by the inference engine better and more flexibly. Summary of the invention

[0005] One or more embodiments of the present application provide the following technical solutions: The present application provides a memory management method for an inference system, which is applied to an inference engine in the inference system; the computing resources of the inference engine include a GPU mounted on a computing device for deploying the inference engine; the inference engine maintains a scheduling queue for scheduling an inference request set; the method includes: Determining a memory management time window according to a data processing duration associated with a set of inference requests being executed in the scheduling queue; Calculating a GPU memory requirement corresponding to the inference request set within the memory management time window, and allocating GPU memory to the inference request set according to the GPU memory requirement; When the memory management time window ends, a subsequent memory management time window corresponding to the memory management time window is determined again based on the data processing duration associated with the inference request set being executed in the scheduling queue.

[0006] The present application also provides a memory management device for an inference system, which is applied to an inference engine in the inference system; the computing resources of the inference engine include a GPU mounted on a computing device for deploying the inference engine; the inference engine maintains a scheduling queue for scheduling an inference request set; the device includes: A time window determination module determines a memory management time window according to a data processing duration associated with a set of inference requests being executed in the scheduling queue; A GPU memory allocation module calculates the GPU memory requirement corresponding to the reasoning request set within the memory management time window, and allocates GPU memory to the reasoning request set according to the GPU memory requirement; The time window determination module is also used to determine, when the memory management time window ends, a subsequent memory management time window corresponding to the memory management time window based on the data processing duration associated with the set of inference requests being executed in the scheduling queue.

[0007] The present application also provides an electronic device, including: processor; a memory for storing processor-executable instructions; The processor implements the steps of any of the above methods by running the executable instructions.

[0008] The present application also provides a computer-readable storage medium having computer instructions stored thereon, which implement the steps of any of the above methods when executed by a processor.

[0009] In the above technical solution, the inference engine in the inference system can use the GPU mounted on the computing device where it is located as a computing resource to execute inference requests and manage GPU memory. Specifically, the inference engine can maintain a scheduling queue for scheduling inference request sets, and determine the memory management time window according to the data processing duration associated with the inference request set being executed in the scheduling queue, and subsequently calculate the GPU memory demand corresponding to the inference request set within the memory management time window, and allocate GPU memory to the inference request set according to the GPU memory demand, and after the end of the memory management time window, the next memory management time window corresponding to the memory management time window can be determined again according to the data processing duration associated with the inference request set being executed in the scheduling queue, so as to perform GPU memory management again within the next memory management time window.

[0010] By adopting the above method, there is no need to reserve a large amount of GPU memory for the inference engine when it is started. Instead, during the process of the inference engine executing inference requests in batches, the memory management time window can be continuously set, and the GPU memory demand corresponding to the inference request set being executed within the memory management time window can be predicted, so as to allocate GPU memory to the inference request set according to the GPU memory demand. This not only ensures that the inference engine has sufficient GPU memory available during the process of batch executing inference requests, but also avoids the waste of GPU memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The following is a description of the accompanying drawings required for describing the exemplary embodiments, wherein: Figure 1 It is a schematic diagram of an inference system shown in an exemplary embodiment of the present application.

[0012] Figure 2 It is a schematic diagram of another reasoning system shown in an exemplary embodiment of the present application.

[0013] Figure 3 It is a flowchart of a memory management method of an inference system shown in an exemplary embodiment of the present application.

[0014] Figure 4 It is a structural schematic diagram of a device shown in an exemplary embodiment of the present application.

[0015] Figure 5 It is a block diagram of a memory management device of an inference system shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0016] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. On the contrary, they are only examples consistent with some aspects of one or more embodiments of the present application.

[0017] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the steps included in the method may be more or less than those described in this application. In addition, a single step described in this application may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this application may also be combined into a single step for description in other embodiments.

[0018] In this application, the inference engine in the inference system can be deployed on a computing device, and the computing resources, storage resources, network resources, etc. (e.g., GPU, GPU memory, storage, network interface, etc.) of the computing device are used to perform calculations in the inference process and execute inference tasks. Alternatively, the inference engine in the inference system can be deployed on a computing cluster to meet the needs of processing large-scale data, high-concurrency requests, or high-performance computing, thereby achieving higher computing power, better fault tolerance, and more flexible resource management.

[0019] In general, the inference engine may use a CPU as a computing resource, where the CPU may be a CPU mounted on a computing device for deploying the inference engine.

[0020] With the development of artificial intelligence technology, reasoning systems based on large models (such as Large Language Model) are becoming more and more widely used.

[0021] Large models refer to machine learning models with a large number of parameters, such as various variants of the Transformer architecture, including but not limited to natural language processing models such as GPT and BERT. These models achieve powerful representation learning capabilities through large amounts of training data and complex architectures.

[0022] In a large model-based reasoning system, the large model can be viewed as a large knowledge base that contains information learned from a large amount of data. The large model learns a large number of patterns and features during training, which represent complex relationships in the data. Therefore, to some extent, the information stored inside the large model can be viewed as a form of knowledge.

[0023] The inference engine designed for large models is responsible for calling the large model to perform specific inference work, that is, it can manage the loading of large models, execute inference operations of large models, and manage the interaction with hardware (for example: CPU, GPU or other accelerators). The inference engine can also include optimization algorithms to improve the speed and efficiency of inference.

[0024] In practical applications, the large model can be deployed separately from the reasoning system, or it can be integrated into the reasoning system and the reasoning engine in the system can be used to call the large model to efficiently perform reasoning tasks.

[0025] Since large models are artificial intelligence applications that require a lot of computing resources, the inference engine usually uses GPU as a computing resource when calling large models to perform inference tasks.

[0026] GPUs are designed for parallel processing and can process multiple data points at the same time. This is particularly useful for deep learning models, as they often need to perform the same operation on large amounts of data. Modern AI models, especially neural network models, require a large number of floating-point operations. GPUs generally perform better than CPUs on floating-point operations, especially when processing large-scale matrix operations, which are common operations in deep learning. GPUs generally have higher memory bandwidth than CPUs, which means they are able to read and write data from memory faster. This is very important for applications that need to frequently access large amounts of data. Many GPU vendors (such as NVIDIA) have optimized their hardware and software stacks for machine learning tasks. For example, they provide hardware specifically designed to accelerate tensor operations, as well as programming models like CUDA to take full advantage of these hardware features. GPUs can increase inference speed. For applications deployed at large scale, using GPUs can reduce latency and increase system responsiveness.

[0027] During the operation of the inference engine, whether the computing resources, storage resources, network resources and other resources of the computing device can be efficiently utilized usually directly affects the performance of the inference engine.

[0028] For example, during the reasoning process of a large model, KV Cache (Key-Value Cache) is usually used to store the generated tokens.

[0029] It should be noted that the reasoning process of a large model refers to the process of using a trained large machine learning model to generate corresponding output results (i.e., reasoning results) based on given input data (i.e., data contained in the reasoning request). In the field of natural language processing, especially in large language models, the reasoning process usually involves generating a continuous text sequence based on a given input prompt, and generating a token for each iteration. The token here usually refers to a series of small units into which text is segmented in natural language processing, such as words or characters, depending on the word segmentation method of the model.

[0030] Specifically, a piece of text can be provided to the model as input, which is called a Prompt; the model receives the Prompt and converts it into a form that can be processed internally, usually by converting each Token in the Prompt into a corresponding digital representation, such as an embedding vector. An embedding vector is a numerical vector in a high-dimensional space that is used to represent the semantic information of each Token; based on the current Prompt, the model predicts the next most likely Token through complex calculations (such as multi-layer neural network operations); the model adds the predicted Token to the existing Prompt to form a new Prompt; the above steps can be repeated multiple times, and each repetition is called an iteration. In each iteration, the model predicts the next Token based on the latest Prompt until the set stop condition is reached, such as a certain number of Tokens are generated or a specific end mark is encountered.

[0031] KV Cache is a data storage method that is usually used to speed up data access. It is a form of data storage as key-value pairs, where the key is a unique identifier used to find data, and the value is the data associated with it or the information pointing to the location of the data. When an application requests specific data, the system can quickly retrieve the corresponding value through the key, thereby improving the system's response speed.

[0032] In large models, especially those used to perform natural language processing tasks, KV Cache is used to store previously calculated results for rapid reuse during decoding, thereby accelerating the inference process.

[0033] In actual applications, using KV Cache to store generated tokens consumes a lot of GPU memory. For an inference engine based on a large model, when the inference engine is started, a certain amount of GPU memory needs to be reserved for the inference engine based on past experience and actual conditions, for subsequent model inference in the inference engine. The allocation ratio is usually 80% to 90% of the total GPU memory. In addition, due to the uncertainty of the intermediate data generated during the inference process of the large model, a certain amount of GPU memory needs to be reserved for these intermediate data.

[0034] However, the traffic of the inference engine is not fixed, that is, the number of inference tasks that need to be performed by the inference engine may change over time. For example, more inference requests are sent to the inference engine in the current time period, while fewer inference requests are sent to the inference engine in the next time period. In this case, the GPU memory reserved for the inference engine will be idle when the traffic is small and cannot be used by other services; in addition, the remaining 10% to 20% of the GPU memory that is not reserved for the inference engine cannot be used by the inference engine. Therefore, the GPU memory of the computing device used to deploy the inference engine cannot be used efficiently, which affects the performance of the inference engine.

[0035] In the technical solution provided by one or more embodiments of the present application, the inference engine in the inference system can use the GPU mounted on the computing device where it is located as a computing resource to execute inference requests and manage GPU memory. Specifically, the inference engine can maintain a scheduling queue for scheduling inference request sets, and determine the memory management time window based on the data processing duration associated with the inference request set being executed in the scheduling queue, and subsequently calculate the GPU memory demand corresponding to the inference request set within the memory management time window, and allocate GPU memory to the inference request set based on the GPU memory demand, and after the memory management time window ends, the next memory management time window corresponding to the memory management time window can be determined based on the data processing duration associated with the inference request set being executed in the scheduling queue, so as to perform GPU memory management again within the next memory management time window.

[0036] By adopting the above method, there is no need to reserve a large amount of GPU memory for the inference engine when it is started. Instead, during the process of the inference engine executing inference requests in batches, the memory management time window can be continuously set, and the GPU memory demand corresponding to the inference request set being executed within the memory management time window can be predicted, so as to allocate GPU memory to the inference request set according to the GPU memory demand. This not only ensures that the inference engine has sufficient GPU memory available during the process of batch executing inference requests, but also avoids the waste of GPU memory.

[0037] Please refer to Figure 1 and Figure 2 , Figure 1 and Figure 2 They are respectively schematic diagrams of an inference system shown by an exemplary embodiment of the present application.

[0038] like Figure 1 As shown, the above reasoning system may include an API server (API Server) and a reasoning engine.

[0039] An API server is a server specifically designed to process client requests and return responses. It is usually an indispensable component of an application. It is responsible for processing client requests, executing business logic, generating responses, and ensuring the security, scalability, and performance of the system.

[0040] Specifically, the API server usually serves as the unified entrance of the reasoning system, handles all requests from clients, and can distribute requests to different backend services to achieve decoupling between services. The API server can be responsible for authentication and authorization to ensure that only legitimate requests can access system resources, and implement fine-grained permission control to protect sensitive data and operations. The API server can also work with the load balancer to evenly distribute requests to multiple backend instances to improve the availability and performance of the reasoning system.

[0041] The above-mentioned inference engine can be deployed on an independent computing device or on a computing cluster composed of at least one computing node. The computing device can be a physical or virtual local computing device or a physical or virtual cloud computing device.

[0042] A computing cluster is a system that is connected by a network and works together. These computing devices work together to complete computing tasks, providing stronger computing power and higher availability than a single computing device. Computing nodes are one of the core components of a computing cluster, usually referring to computing devices used to perform computing tasks, and the GPUs installed on them can be used as computing resources.

[0043] Correspondingly, the above-mentioned API server can be deployed on the computing device where the above-mentioned inference engine is located; or, the above-mentioned API server can be deployed on the head node in the above-mentioned computing cluster, where the head node usually refers to the computing node responsible for the management and scheduling of the computing cluster, and can also be deployed on other computing devices outside the computing cluster.

[0044] like Figure 2As shown, for each computing device in an independent computing device or a computing cluster used to deploy the above-mentioned inference engine, at least one computing instance can be deployed on the computing device. Among them, the computing instance refers to a dedicated resource allocated on demand through virtualization technology in a local data center or cloud computing platform. The resources of these computing instances may include computing resources, storage resources, network resources, etc. (for example: GPU, GPU memory, storage, network interface, etc.), so that each computing instance can use these resources to perform specific computing tasks. Computing instances can be created, started, stopped or deleted according to actual needs, and their resources can also be adjusted according to demand.

[0045] The computing resource of a computing instance may be a GPU of the computing device on which it is located. Specifically, a computing device may be equipped with multiple GPUs, and a GPU may be allocated to a computing instance as a computing resource of the computing instance. Alternatively, a computing device may be equipped with only one GPU, and the computing resource allocated to a computing instance may be a part of the computing unit, memory, etc. of the GPU.

[0046] In the inference engine based on the big model, each computing instance can call the big model to perform specific inference work, that is, the big model can be loaded into each computing instance, and each computing instance performs the inference operation of the big model. Specifically, the big model can be read from a local or cloud storage medium (e.g., local disk, cloud storage, etc.) to the GPU memory of each computing instance to perform inference based on the big model on the GPU resources of each computing instance.

[0047] In actual applications, the Model Runtime framework can be installed in the computing instance, and after the large model is read into the computing instance, the model service is configured according to the Model Runtime framework and the large model, so that the computing instance can be used as an independently run model service instance. Among them, the model runtime refers to the environment and framework used to perform model reasoning after the model is deployed. This environment usually includes a series of steps such as model loading, input data processing, model execution, and output result processing, covering the entire life cycle of the model from loading to execution and then unloading.

[0048] For the above-mentioned inference engine including at least one computing instance, a load-aware scheduling method may be adopted to implement scheduling of inference requests on the inference engine.

[0049] Load-aware scheduling is a strategy for allocating tasks in a computing system based on the current load of each node or resource. This scheduling method aims to optimize resource usage efficiency, reduce waiting time and processing delays, and improve the overall throughput of the system.

[0050] Load-aware scheduling is usually used in distributed systems, cloud computing platforms, data centers and other scenarios. Its purpose is to dynamically balance the workload of each node and avoid the situation where some nodes are overloaded while other nodes are idle. By monitoring the load of each node in real time, the load-aware scheduler can make more reasonable task allocation decisions.

[0051] For example, in the above-mentioned inference system, the above-mentioned API server and the above-mentioned inference engine can communicate with each other. In this case, the API server can obtain the inference request from the request queue, and select a computing instance with a suitable GPU load according to the GPU load of each computing instance, and schedule the obtained inference request to the computing instance to use the GPU resources of the computing instance to execute the inference request.

[0052] By deploying computing instances on computing devices, on the one hand, the resources of the computing devices can be fully utilized to accelerate the execution of computing tasks, thereby improving the computing power of the inference engine; on the other hand, different computing instances are independent of each other, which can ensure the stability and security of computing tasks, and can achieve resource isolation and management, which is conducive to the effective use of resources and avoid resource requisition; on the third hand, for computing tasks that need to be processed in parallel, parallel computing can be used to speed up the completion time of computing tasks; on the fourth hand, elastic scaling can be achieved. When a large number of computing tasks need to be processed, computing instances can be added, and when the number of computing tasks that need to be processed decreases, some computing instances can be terminated to release some resources and realize flexible allocation of resources.

[0053] For the above-mentioned API server, the API server may include two components, namely, a request router component responsible for routing inference requests to forward the inference requests to the above-mentioned inference engine, and a monitor component responsible for monitoring the operation-related information of all computing instances in the inference engine (for example: GPU memory utilization, GPU memory bandwidth utilization, the number of inference requests waiting in the scheduling queue and the number of running inference requests, etc.).

[0054] For a computing device used to deploy the above-mentioned inference engine, in order to facilitate the maintenance, management and scheduling of the computing instances deployed on the computing device, an agent may also be deployed on the computing device.

[0055] The API server can communicate with the agent deployed on the computing device. In this case, the API server can obtain the inference request from the request queue, and select the computing instance with the appropriate GPU load according to the GPU load of each computing instance, and send the obtained inference request to the agent deployed on the computing device where the computing instance is located, and the agent further schedules the inference request to the computing instance to use the GPU resources of the computing instance to execute the inference request.

[0056] In practical applications, in order to ensure the reliability and correctness of the inference request scheduling, the above agent can maintain a scheduling queue.

[0057] A scheduling queue is a special type of task queue, which is mainly used to manage and schedule tasks with specific execution time or priority (in this application, it is the inference task specified by the inference request). The main functions of the scheduling queue include task scheduling, task management, and resource optimization. Among them, task scheduling means that the scheduling queue can ensure that the task is executed at a specified time point or time range, and support the scheduling of periodic tasks; and the priority of the task can be set to ensure that the high-priority tasks in the scheduling queue are executed first. Task management refers to the management of all tasks in one queue for easy monitoring and control, and the execution status of the task can be tracked, including unexecuted, executing, completed, and failed states. Resource optimization means that by reasonably arranging the execution time of tasks, the system can be prevented from being overloaded at a certain moment, and system resources can be ensured to be effectively utilized to avoid resource waste.

[0058] Through the above scheduling queue and the above computing instance, the above agent has queue and batch processing capabilities. That is, each inference request in an inference request set (also called a batch of inference requests) can be scheduled to each computing instance through the scheduling queue, so that each inference request is executed by each computing instance, which enables different inference requests in the inference request set to be executed in parallel.

[0059] In addition, the above agent can also contain two components, namely, the memory predictor and the dynamic memory management component. The memory predictor can continuously execute the memory consumption prediction algorithm and dynamically manage the GPU memory in combination with the dynamic memory management component.

[0060] For each computing instance, the computing instance may contain two components: a KV Cache component used to cache intermediate data and results of reasoning to avoid repeated calculations, and an Executor component responsible for calling the large model to execute reasoning tasks corresponding to the reasoning request.

[0061] Please Figure 1 and Figure 2 Based on the reference Figure 3 , Figure 3 It is a flowchart of a memory management method of an inference system shown in an exemplary embodiment of the present application.

[0062] In this embodiment, the memory management method of the above reasoning system can be applied to Figure 1 The inference engine shown.

[0063] like Figure 3 As shown, the memory management method of the above reasoning system may include the following steps: Step 302: Determine a memory management time window according to a data processing duration associated with a set of inference requests being executed in the scheduling queue.

[0064] In this embodiment, after each inference request in an inference request set is scheduled to each computing instance for execution through the above-mentioned scheduling queue, the execution status of each inference request in the inference request set maintained by the scheduling queue can be updated from an unexecuted state to an executing state. For the inference request set being executed in the scheduling queue, the memory management time window can be determined based on the data processing duration associated with the inference request set. Subsequently, the memory management time window can be used as the time unit of the GPU memory management task, that is, GPU memory management is performed once within the memory management time window.

[0065] In some embodiments, in order to improve the accuracy and adaptability of the determined memory management time window, when determining the memory management time window based on the data processing duration associated with the inference request set being executed in the above-mentioned scheduling queue, the corresponding data processing duration can be first determined by the execution stage of each inference request in the inference request set, and then the memory management time window can be determined based on the data processing duration.

[0066] It should be noted that the reasoning process may include multiple stages in some cases. For example, for a reasoning system based on a large model, the reasoning process of the large model (especially the generative model in natural language processing) may include the Prefill stage and the Decode stage.

[0067] In the reasoning process of a large model, the Prefill stage and the Decode stage refer to different steps in generating text.

[0068] The prefill stage usually occurs at the initial stage of generating text, and its main purpose is to provide an initial context for the model so that the model can better understand and generate subsequent text. In the prefill stage, the model usually generates an initial text fragment that contains the starting part of the generated sequence. This stage may involve using some predefined strategies or algorithms to select the most appropriate starting vocabulary or phrases. For example, when dealing with a conditional generation task, the model may first generate a part of the text as a basis based on the input conditions (such as summary generation, question and answer, dialogue, etc.).

[0069] The Decode stage is the main stage of generating text. The model gradually generates new words or characters in this stage until a complete output sequence is generated. In the Decode stage, the model generates new content word by word or word by word based on the existing text. In each iteration, the model predicts the next most likely word and adds it to the current text sequence. This process continues until the preset end condition is reached (for example, reaching the maximum text length, generating a text end marker, etc.). Various strategies may be used in the Decode stage to optimize the quality of generation, such as Top-k sampling, Top-p (also known as Nucleus Sampling) sampling, and other techniques, which can help the model avoid generating content that is too mundane or meaningless.

[0070] In practical applications, the two stages of Prefill and Decode are closely linked. The Prefill stage provides the initial text basis, while the Decode stage is responsible for gradually expanding on this basis to generate a complete text output. These two stages together determine the quality and coherence of the final generated text.

[0071] Specifically, it may be first determined whether there is an inference request in the Prefill stage in the set of inference requests being executed in the scheduling queue.

[0072] If there are inference requests in the Prefill stage, the number of tokens contained in each inference request in the above inference request set can be compared to determine the largest number, which is the maximum number of tokens corresponding to the inference requests in the inference request set, so that the token processing time corresponding to the maximum number of tokens can be determined as the length of the above memory management time window.

[0073] If there is no inference request in the Prefill stage, it means that all inference requests in the above inference request set are in the Decode stage. In this case, the token generation duration corresponding to each inference request in the inference request set can be compared, where the token generation duration is the duration for generating a token based on the Prompt contained in the inference request, so as to determine the longest token generation duration corresponding to the inference requests in the inference request set, so that the longest token generation duration can be determined as the length of the above memory management time window.

[0074] Step 304: Calculate the GPU memory requirement corresponding to the inference request set within the memory management time window, and allocate GPU memory to the inference request set according to the GPU memory requirement.

[0075] In this embodiment, when the above-mentioned memory time window is determined, the GPU memory requirement corresponding to the above-mentioned inference request set within the memory management time window can be calculated, so that the GPU memory can be allocated to the inference request set according to the GPU memory requirement. In this way, since the GPU memory can be allocated to the inference engine according to the actual GPU memory requirement during the process of executing the inference request in the inference request set, it can not only ensure that the inference engine has enough GPU memory to use during the process of executing the inference request, but also avoid the waste of GPU memory.

[0076] In practical applications, the memory predictor may execute a memory consumption prediction algorithm, that is, determine a memory management time window according to the data processing duration associated with the inference request set being executed in the scheduling queue, and calculate the GPU memory demand corresponding to the inference request set within the memory management time window, while the dynamic memory management component may execute specific GPU memory allocation, that is, allocate GPU memory to the inference request set according to the GPU memory demand.

[0077] Step 306: When the memory management time window ends, a next memory management time window corresponding to the memory management time window is determined again based on the data processing duration associated with the inference request set being executed in the scheduling queue.

[0078] In this embodiment, after the GPU memory management execution in the above memory management time window ends, the end of the memory management time window can be waited. When the memory management time window ends, the next memory management time window corresponding to the memory management time window can be determined again based on the data processing duration associated with the inference request set being executed in the above scheduling queue.

[0079] Specifically, assuming that the above-mentioned memory management time window is the Tth memory management time window determined according to the data processing duration associated with the inference request set being executed in the above-mentioned scheduling queue, the GPU memory requirement corresponding to the inference request set in the Tth memory management time window can be calculated, and GPU memory can be allocated to the inference request set based on the GPU memory requirement; at the end of the Tth memory management time window, the T+1th memory management time window can be re-determined based on the data processing duration associated with the inference request set being executed in the scheduling queue, so that the GPU memory requirement corresponding to the inference request set in the T+1th memory management time window can be calculated, and GPU memory can be allocated to the inference request set based on the GPU memory requirement; and so on.

[0080] In some embodiments, when calculating the GPU memory demand corresponding to the reasoning request set within the above-mentioned memory management time window, the static GPU memory usage corresponding to the reasoning request set within the memory management time window can be calculated first, and then the maximum GPU memory usage in the previous memory management time window corresponding to the memory management time window and the sum of the static GPU memory usage corresponding to the reasoning request set in each target memory management time window are determined, and the difference between the maximum GPU memory usage and the sum of the static GPU memory usage is determined as the dynamic GPU memory usage corresponding to the reasoning request set within the memory management time window, wherein the target memory management window is a memory management window located before the memory management time window during the execution of the reasoning request set, and finally the sum of the static GPU memory usage and the dynamic GPU memory usage is determined as the GPU memory demand corresponding to the reasoning request set within the memory management time window.

[0081] Specifically, assuming that the above memory management time window is the Tth memory management time window determined according to the data processing duration associated with the inference request set being executed in the above scheduling queue, the previous memory management time window corresponding to the Tth memory management time window is the T-1th memory management time window, and the target memory management window located before the Tth memory management time window during the execution of the inference request set is the 1st to T-1th memory management time windows. In this case, StaticResource[T] represents the static GPU memory usage within the Tth memory management time window, DynamicResource[T] represents the dynamic GPU memory usage within the Tth memory management time window, MaxResource[T] represents the maximum GPU memory usage within the Tth memory management time window, and NeedResource[T] represents the GPU memory demand within the Tth memory management time window, then the following formula is used: DynamicResource[T] = MaxResource[T-1]– (StaticResource[0] +StaticResource[1]+ … + StaticResource[T-1]); NeedResource[T] = StaticResource[T]+DynamicResource[T].

[0082] That is, for the current memory management time window mentioned above, the dynamic GPU memory usage within the memory management time window can be predicted based on the maximum GPU memory usage in the previous memory management time window corresponding to the memory management time window, and the sum of the static GPU memory usage in all memory management time windows before the memory management time window.

[0083] It should be noted that the above-mentioned static GPU memory usage refers to the GPU memory that is foreseeable to be used during the inference process, such as: the GPU memory usage required for the tokens contained in the Prompt, the GPU memory usage required for new tokens generated based on the Prompt, etc.; the above-mentioned dynamic GPU memory usage refers to the GPU memory that is unforeseeable to be used during the inference process, such as: the GPU memory usage required for the uncertain intermediate data generated during the inference process of a large model.

[0084] In practical applications, in order to improve the compatibility of the technical solution in this application, when the above-mentioned inference engine is started, a sufficient amount of virtual GPU memory can be reserved for the inference engine, and the above-mentioned memory management method is used in the subsequent inference process to allocate physical GPU memory by calling the API provided by CUDA (Compute Unified Device Architecture).

[0085] In some embodiments, in order to improve the accuracy and adaptability of the determined static GPU memory usage, when calculating the static GPU memory usage corresponding to the above-mentioned inference request set within the above-mentioned memory management time window, the static GPU memory usage within the memory management time window can be determined according to the execution stage of each inference request in the inference request set.

[0086] Specifically, it may be first determined whether there is an inference request in the Prefill stage in the set of inference requests being executed in the scheduling queue.

[0087] If there is an inference request in the Prefill stage, the GPU memory usage required for the token corresponding to each inference request in the above inference request set (which may be referred to as the first GPU memory usage) and the GPU memory usage required to generate a token based on each inference request in the inference request set (which may be referred to as the second GPU memory usage) may be determined, and the sum of the first GPU memory usage and the second GPU memory usage may be determined as the static GPU memory usage corresponding to the inference request set within the above memory management time window. The GPU memory usage required for the token corresponding to an inference request may be the GPU memory usage required for the token contained in the Prompt in the inference request; the GPU memory usage required to generate a token based on an inference request may be the GPU memory usage required to generate a token based on the Prompt contained in the inference request.

[0088] If there is no inference request in the Prefill stage, it means that all inference requests in the above inference request set are in the Decode stage. In this case, the GPU memory usage required to generate a token based on each inference request in the inference request set can be determined, and the GPU memory usage can be determined as the static GPU memory usage corresponding to the inference request set within the memory management time window.

[0089] In some embodiments, in order to ensure the rationality and availability of GPU memory allocation, when allocating GPU memory for the above-mentioned inference request set according to the above-mentioned GPU memory demand, the current GPU memory usage corresponding to the inference request set can be obtained first. Then, according to the current GPU memory usage and the GPU memory demand, it can be determined whether the GPU memory allocation condition and the GPU memory release condition are met.

[0090] It should be noted that the above GPU memory allocation condition and the above GPU memory release condition are two mutually exclusive conditions. That is, if the GPU memory allocation condition is met, it means that the GPU memory release condition is not met; if the GPU release allocation condition is met, it means that the GPU memory allocation condition is not met.

[0091] If the above GPU memory allocation conditions are met, GPU memory can be allocated to the above inference request set according to the above GPU memory demand.

[0092] If the above-mentioned GPU memory release conditions are met, the GPU memory used to execute the above-mentioned inference request set can be released, and the above-mentioned current CPU memory usage can be updated, so as to allocate GPU memory to the inference request set according to the GPU memory demand when it is determined that the above-mentioned GPU memory allocation conditions are met based on the updated current GPU memory usage and the above-mentioned GPU memory demand.

[0093] It should be noted that the released GPU memory usage can be determined according to the executed GPU memory release task, and the released GPU memory usage can be subtracted from the above current GPU memory usage to obtain the updated current GPU memory usage. However, in actual applications, in order to obtain a more accurate current GPU memory usage, the current GPU memory usage can be obtained in real time or periodically through tools such as DCGM (Data Center GPU Manager). DCGM is a tool for managing and monitoring GPU resources in a data center. It can help effectively manage large-scale GPU clusters and provide in-depth insights into GPU performance, health status, and utilization.

[0094] In some embodiments, the above-mentioned memory allocation condition may be that the sum of the current GPU memory usage and the GPU memory demand is less than a preset threshold (which may be referred to as the first threshold); the above-mentioned GPU memory release condition may be that the current GPU memory usage is greater than a preset threshold (which may be referred to as the second threshold). The specific values ​​of the first threshold and the second threshold may be the same or different, and this application does not impose any special restrictions on this.

[0095] Specifically, let NeedResource[T] represent the GPU memory demand within the T-th memory management time window, CurrentResource[T] represent the current GPU memory usage obtained when performing GPU memory management within the T-th memory management time window, Threshold1 represent the first threshold mentioned above, and Threshold2 represent the second threshold mentioned above. Then, it can be determined that the above memory allocation condition is satisfied when CurrentResource[T] + NeedResource[T] < Threshold1, and it can be determined that the above memory release condition is satisfied when CurrentResource[T] > Threshold2.

[0096] In practical applications, the period when CurrentResource[T] + NeedResource[T] > Threshold1 and CurrentResource[T] < Threshold2 can be used as the buffer time for releasing GPU memory. If during this period, CurrentResource[T] changes by itself, such that the updated CurrentResource[T] + NeedResource[T] < Threshold1, then since the above memory allocation condition is satisfied, GPU memory can be allocated to the above inference request set according to the above GPU memory demand. If during this period, CurrentResource[T] changes by itself, such that CurrentResource[T] > Threshold2, then since the above memory release condition is satisfied, the GPU memory used to execute the above inference request set can be released. In this way, the number of times of executing the GPU memory release task can be reduced to a certain extent.

[0097] In some embodiments, various ways can be adopted to implement the release of GPU memory. For example, when releasing the GPU memory used to execute the above inference request set, the specific GPU memory release method can be selected according to the number of inference requests in the inference request set.

[0098] Specifically, it can first be determined whether the number of inference requests in the above inference request set is greater than a preset threshold (which can be referred to as the third threshold).

[0099] If the number of inference requests in the above inference request set is greater than the above third threshold, then the tokens corresponding to the respective inference requests in the inference request set can be offloaded from the GPU memory to the CPU memory to release the GPU memory occupied by these tokens.

[0100] If the number of inference requests in the inference request set is not greater than the third threshold, the inference request may be updated based on the tokens corresponding to the respective inference requests in the inference request set to release the GPU memory occupied by the tokens. Wherein, updating the inference request based on the token corresponding to an inference request means updating the Prompt using the token generated based on the Prompt included in the inference request to form a new Prompt.

[0101] In the above technical solution, the inference engine in the inference system can use the GPU mounted on the computing device where it is located as a computing resource to execute inference requests and manage GPU memory. Specifically, the inference engine can maintain a scheduling queue for scheduling inference request sets, and determine the memory management time window according to the data processing duration associated with the inference request set being executed in the scheduling queue, and subsequently calculate the GPU memory demand corresponding to the inference request set within the memory management time window, and allocate GPU memory to the inference request set according to the GPU memory demand, and after the end of the memory management time window, the next memory management time window corresponding to the memory management time window can be determined again according to the data processing duration associated with the inference request set being executed in the scheduling queue, so as to perform GPU memory management again within the next memory management time window.

[0102] By adopting the above method, there is no need to reserve a large amount of GPU memory for the inference engine when it is started. Instead, during the process of the inference engine executing inference requests in batches, the memory management time window can be continuously set, and the GPU memory demand corresponding to the inference request set being executed within the memory management time window can be predicted, so as to allocate GPU memory to the inference request set according to the GPU memory demand. This not only ensures that the inference engine has sufficient GPU memory available during the process of batch executing inference requests, but also avoids the waste of GPU memory.

[0103] Corresponding to the aforementioned method embodiments, the present application also provides device embodiments.

[0104] Please refer to Figure 4 , Figure 4It is a structural diagram of a device shown in an exemplary embodiment of the present application. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408 and a non-volatile memory 410, and of course may also include other required hardware. One or more embodiments of the present application can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present application do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0105] Please refer to Figure 5 , Figure 5 It is a block diagram of a memory management device of an inference system shown in an exemplary embodiment of the present application.

[0106] The memory management device of the above reasoning system can be applied to Figure 4 The device shown in the figure is used to implement the technical solution of the present application. The device can deploy an inference engine in an inference system, and the memory management device of the inference system can be specifically applied to the inference engine; the computing resources of the inference engine include a GPU mounted on a computing device for deploying the inference engine; the inference engine maintains a scheduling queue for scheduling an inference request set; the device includes: A time window determination module 502 determines a memory management time window according to a data processing duration associated with a set of inference requests being executed in the scheduling queue; A GPU memory allocation module 504 calculates the GPU memory requirement corresponding to the reasoning request set within the memory management time window, and allocates GPU memory to the reasoning request set according to the GPU memory requirement; The time window determination module 502 is also used to determine, when the memory management time window ends, a subsequent memory management time window corresponding to the memory management time window based on the data processing duration associated with the set of inference requests being executed in the scheduling queue.

[0107] In some embodiments, determining the memory management time window according to the data processing duration associated with the set of inference requests being executed in the scheduling queue includes: In the set of inference requests being executed in the scheduling queue, determining whether there is an inference request in the Prefill stage; If there is an inference request in the Prefill phase, determine a maximum number of tokens corresponding to the inference request in the inference request set, and determine a memory management time window according to a token processing duration corresponding to the maximum number of tokens; If there is no inference request in the Prefill phase, a longest token generation duration corresponding to the inference requests in the inference request set is determined, and a memory management time window is determined according to the longest token generation duration.

[0108] In some embodiments, calculating the GPU memory requirement corresponding to the inference request set within the memory management time window includes: Calculating a static GPU memory usage corresponding to the inference request set within the memory management time window; Determine a maximum GPU memory usage in a previous memory management time window corresponding to the memory management time window, and a sum of static GPU memory usages corresponding to the inference request set in each target memory management time window, and determine a difference between the maximum GPU memory usage and the sum of the static GPU memory usages as a dynamic GPU memory usage corresponding to the inference request set in the memory management time window; wherein the target memory management window is a memory management window located before the memory management time window during the execution of the inference request set; The sum of the static GPU memory usage and the dynamic GPU memory usage is determined as the GPU memory demand corresponding to the inference request set within the memory management time window.

[0109] In some embodiments, calculating the static GPU memory usage corresponding to the inference request set within the memory management time window includes: In the set of inference requests being executed in the scheduling queue, determining whether there is an inference request in the Prefill stage; If there are inference requests in the Prefill stage, determine a first GPU memory usage required for a token corresponding to each inference request in the inference request set, and a second GPU memory usage required to generate a token based on each inference request in the inference request set, and determine the sum of the first GPU memory usage and the second GPU memory usage as a static GPU memory usage corresponding to the inference request set within the memory management time window; If there is no inference request in the Prefill stage, determine the GPU memory usage required to generate a token based on each inference request in the inference request set, and determine the GPU memory usage as the static GPU memory usage corresponding to the inference request set within the memory management time window.

[0110] In some embodiments, allocating GPU memory to the inference request set according to the GPU memory requirement includes: Obtain the current GPU memory usage corresponding to the inference request set; Determine whether a GPU memory allocation condition and a GPU memory release condition are met according to the current GPU memory usage and the GPU memory demand; If the GPU memory allocation condition is met, allocating GPU memory to the inference request set according to the GPU memory requirement; If the GPU memory release condition is met, the GPU memory used to execute the inference request set is released, and the current CPU memory usage is updated, so that when it is determined that the GPU memory allocation condition is met based on the updated current GPU memory usage and the GPU memory demand, GPU memory is allocated to the inference request set according to the GPU memory demand.

[0111] In some embodiments, the GPU memory allocation condition includes: The sum of the current GPU memory usage and the GPU memory requirement is less than a preset first threshold; The GPU memory release conditions include: The current GPU memory usage is greater than a preset second threshold.

[0112] In some embodiments, releasing GPU memory used to execute the set of inference requests includes: Determining whether the number of inference requests in the inference request set is greater than a preset third threshold; If the number is greater than the third threshold, unloading the token corresponding to each inference request in the inference request set to the CPU memory to release the GPU memory occupied by the token; If the number is not greater than the third threshold, based on the tokens corresponding to the respective inference requests in the inference request set, the inference requests are updated to release the GPU memory occupied by the tokens.

[0113] For the device embodiment, it basically corresponds to the method embodiment, so the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the technical solution of the present application.

[0114] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0115] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0116] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0117] Computer readable media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0118] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0119] The above specific embodiments of the present application are described. Other embodiments are within the scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0120] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings. The term "and / or" refers to and includes any or all possible combinations of one or more associated listed items.

[0121] The description of the terms "one embodiment", "some embodiments", "example", "specific example" or "an implementation method" etc. used in one or more embodiments of the present application means that the specific features or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present application. The schematic description of these terms is not necessarily for the same embodiment. Moreover, the specific features or characteristics described can be combined in a suitable manner in one or more embodiments of the present application. In addition, different embodiments and specific features or characteristics in different embodiments can be combined without contradicting each other.

[0122] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0123] The above description is merely a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

[0124] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A memory management method for an inference system, applied to an inference engine in the inference system; the computing resources of the inference engine include a GPU mounted on a computing device for deploying the inference engine; The inference engine maintains a scheduling queue for scheduling a set of inference requests; the method comprises: Determining a memory management time window according to a data processing duration associated with a set of inference requests being executed in the scheduling queue; Calculating a GPU memory requirement corresponding to the inference request set within the memory management time window, and allocating GPU memory to the inference request set according to the GPU memory requirement; When the memory management time window ends, a subsequent memory management time window corresponding to the memory management time window is determined again based on the data processing duration associated with the inference request set being executed in the scheduling queue.

2. The method according to claim 1, wherein determining the memory management time window according to the data processing duration associated with the set of inference requests being executed in the scheduling queue comprises: In the set of inference requests being executed in the scheduling queue, determining whether there is an inference request in the Prefill stage; If there is an inference request in the Prefill phase, determine a maximum number of tokens corresponding to the inference request in the inference request set, and determine a memory management time window according to a token processing duration corresponding to the maximum number of tokens; If there is no inference request in the Prefill phase, a longest token generation duration corresponding to the inference requests in the inference request set is determined, and a memory management time window is determined according to the longest token generation duration.

3. The method according to claim 1, wherein the step of calculating the GPU memory requirement corresponding to the inference request set within the memory management time window comprises: Calculating a static GPU memory usage corresponding to the inference request set within the memory management time window; Determine a maximum GPU memory usage in a previous memory management time window corresponding to the memory management time window, and a sum of static GPU memory usages corresponding to the inference request set in each target memory management time window, and determine a difference between the maximum GPU memory usage and the sum of the static GPU memory usages as a dynamic GPU memory usage corresponding to the inference request set in the memory management time window; wherein the target memory management window is a memory management window located before the memory management time window during the execution of the inference request set; The sum of the static GPU memory usage and the dynamic GPU memory usage is determined as the GPU memory demand corresponding to the inference request set within the memory management time window.

4. The method according to claim 3, wherein the step of calculating the static GPU memory usage corresponding to the inference request set within the memory management time window comprises: In the set of inference requests being executed in the scheduling queue, determining whether there is an inference request in the Prefill stage; If there are inference requests in the Prefill stage, determine a first GPU memory usage required for a token corresponding to each inference request in the inference request set, and a second GPU memory usage required to generate a token based on each inference request in the inference request set, and determine the sum of the first GPU memory usage and the second GPU memory usage as a static GPU memory usage corresponding to the inference request set within the memory management time window; If there is no inference request in the Prefill stage, determine the GPU memory usage required to generate a token based on each inference request in the inference request set, and determine the GPU memory usage as the static GPU memory usage corresponding to the inference request set within the memory management time window.

5. The method according to claim 1, allocating GPU memory to the inference request set according to the GPU memory requirement, comprising: Obtain the current GPU memory usage corresponding to the inference request set; Determine whether a GPU memory allocation condition and a GPU memory release condition are met according to the current GPU memory usage and the GPU memory demand; If the GPU memory allocation condition is met, allocating GPU memory to the inference request set according to the GPU memory requirement; If the GPU memory release condition is met, the GPU memory used to execute the inference request set is released, and the current CPU memory usage is updated, so that when it is determined that the GPU memory allocation condition is met based on the updated current GPU memory usage and the GPU memory demand, GPU memory is allocated to the inference request set according to the GPU memory demand.

6. The method according to claim 5, wherein the GPU memory allocation condition comprises: The sum of the current GPU memory usage and the GPU memory requirement is less than a preset first threshold; The GPU memory release conditions include: The current GPU memory usage is greater than a preset second threshold.

7. The method according to claim 5, wherein releasing GPU memory used to execute the set of inference requests comprises: Determining whether the number of inference requests in the inference request set is greater than a preset third threshold; If the number is greater than the third threshold, unloading the token corresponding to each inference request in the inference request set to the CPU memory to release the GPU memory occupied by the token; If the number is not greater than the third threshold, based on the tokens corresponding to the respective inference requests in the inference request set, the inference requests are updated to release the GPU memory occupied by the tokens.

8. A memory management device for an inference system, applied to an inference engine in the inference system; the computing resources of the inference engine include a GPU mounted on a computing device for deploying the inference engine; The inference engine maintains a scheduling queue for scheduling an inference request set; the device comprises: A time window determination module determines a memory management time window according to a data processing duration associated with a set of inference requests being executed in the scheduling queue; A GPU memory allocation module calculates the GPU memory demand corresponding to the reasoning request set within the memory management time window, and allocates GPU memory to the reasoning request set according to the GPU memory demand; The time window determination module is also used to determine, when the memory management time window ends, a subsequent memory management time window corresponding to the memory management time window based on the data processing duration associated with the set of inference requests being executed in the scheduling queue.

9. An electronic device, comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 by running the executable instructions.

10. A computer-readable storage medium having computer instructions stored thereon, wherein the instructions are executed by a processor to implement the method according to any one of claims 1 to 7.

Images