GPU video memory and computing unit multi-model sharing method
By splitting the model into general and differentiated layers and dynamically scheduling them on GPU computing units, the problems of low GPU resource utilization and scheduling lag are solved, video memory reuse and cross-node calls are achieved, and the flexibility and throughput of model services are improved.
Patent Information
- Application Number
- CN202510717138.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-30
- Publication Date
- 2025-09-12
AI Technical Summary
The resource utilization rate in existing GPU computing power deployment is low, making it difficult to meet the flexible demands of different AI models for computing power and video memory resources. In addition, resource scheduling has a high lag, resulting in resource waste and inflexible model service response.
The model is split into a general layer and a differentiated layer, which are deployed on different GPU computing units respectively. Dynamic scheduling is achieved through load balancing strategies to achieve multi-model sharing of GPU resources and cross-physical node calls.
It improves GPU resource utilization, reduces video memory waste, enhances the response flexibility and real-time performance of model services, supports the decoupling of CPU and GPU resources, and enhances the throughput of algorithm requests.
Smart Images
Figure CN120634838A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for sharing multiple models of GPU display memory and computing units. Background Art
[0002] Existing GPU computing power deployment is generally based on bare metal physical machine deployment, allocating fixed video memory based on model size, and calling computing power based on user requests for model inference. After the model is deployed, it is difficult to schedule and expand computing power. The model's exclusive use of the GPU results in low GPU resource utilization in most cases. Alternatively, by virtualizing the GPU, the physical GPU is divided into multiple virtual GPUs at a fixed ratio. Each virtual GPU has equal video memory and shared computing power, but it is difficult to meet the flexible computing power and video memory resource requirements of different AI models. In addition, GPU resources are limited to containers or virtual machines on a single physical machine. In essence, resources are still exclusive and cannot be dynamically adjusted.
[0003] Alternatively, GPUs could be further pooled to provide heterogeneous computing control and data plane capabilities, enabling data center-level O&M capabilities such as resource discovery, resource scheduling, resource recovery, heterogeneous pool management, and log monitoring within a global resource pool environment. However, even with virtualization or containerization, the recovery of idle resources and the provisioning of high-scheduled algorithm model services still lag and lack flexibility. Summary of the Invention
[0004] The present invention aims to realize dynamic calling of GPU resources to meet the different demands of different models for GPU resources at different times, reduce the waste of GPU and video memory resources, and improve the real-time and flexibility of the algorithm model service. It provides a GPU video memory and computing unit multi-model sharing method.
[0005] To achieve this object, the present invention adopts the following technical solutions:
[0006] A method for sharing multiple models of GPU display memory and computing units is provided, comprising the following steps:
[0007] S1, splitting each network layer for data processing of different models into a general layer and a distinguishing layer, and deploying each general layer and / or each distinguishing layer of each model on a corresponding GPU computing unit;
[0008] S2, identifying the network layer split structure of the model to be run, and calling the first GPU computing unit through the load balancing strategy to perform data processing on the identified general layer, and calling the second GPU computing unit to perform data processing on the identified difference layer.
[0009] Preferably, the general layer includes an input layer and a feature extraction layer, the input layer is used to preprocess the data input to the model; the feature extraction layer is used to extract features from the data output after passing through the input layer.
[0010] Preferably, the distinction layer includes a decision layer and an output layer, wherein the decision layer is used to make target decisions based on the data features extracted by the feature extraction layer in the general layer; and the output layer is used to generate the output of the decision layer as a processing result of the model and output it.
[0011] Preferably, the load balancing strategy includes the steps of:
[0012] A1, screening out the GPU computing units of each network layer where the model to be run is deployed, where the number of the screened out GPU computing units is greater than or equal to 1;
[0013] A2, determining whether the number of the filtered GPU computing units is 1,
[0014] If yes, the data is processed by the algorithm using the model deployed in the selected GPU computing unit;
[0015] If not, go to step A3;
[0016] A3, determining whether the current load of each of the screened GPU computing units exceeds a first load threshold,
[0017] If yes, go to step A4;
[0018] If not, calling the GPU computing unit with the lowest load among the GPU computing units screened in step A1 to perform the model processing task;
[0019] A4, screening out each first GPU computing unit that has the general layer in the model to be run but has not the distinguishing layer deployed, and screening out each second GPU computing unit that has the distinguishing layer in the model to be run but has not the general layer deployed;
[0020] A5: Transmit the data to be processed to each of the first GPU computing units for general layer data processing, and forward the output of each of the first GPU computing units to each of the second GPU computing units for distinguished layer data processing.
[0021] Preferably, in step A5, the first GPU computing unit that performs general layer data processing on the data to be processed is: the first GPU computing unit with the lowest load among the first GPU computing units screened in step A4;
[0022] In step A5, the second GPU computing unit that performs difference layer data processing on the output of each first GPU computing unit is: the second GPU computing unit with the lowest load among the second GPU computing units screened in step A4.
[0023] Preferably, the load balancing strategy further comprises the steps of:
[0024] A6: Determine whether there is a GPU computing unit whose current load has dropped below the first load threshold among the GPU computing units screened in step A1.
[0025] If yes, then selecting the model deployed in the GPU computing unit with the lowest load among the GPU computing units whose load is reduced to the first load threshold to perform subsequent data processing tasks;
[0026] If not, continue to step A5.
[0027] The present invention has the following beneficial effects:
[0028] 1. The size of the video memory determines the size and number of models that the GPU can load and deploy. After conventional model deployment occupies most of the video memory, the remaining video memory is insufficient to deploy other complete models. This method splits the model and deploys it based on the model structure. The models use the same feature extraction layer and reuse the model feature structure to avoid video memory monopoly and achieve video memory reuse. This allows the deployment of more models and fully utilizes the fragmented video memory space.
[0029] 2. Load scheduling is performed based on the load table and routing table, linking together the fragmented distributed deployment models. The scheduling algorithm selects the optimal call path. When a computing service's call peak arrives, only the decision layer of the algorithm needs to be loaded into the graphics memory, eliminating the need to deploy the entire algorithm container or virtual machine service.
[0030] 3. Supports cross-physical node GPU calls, thereby supporting the decoupling of CPU and GPU resources. The scope of the resource pool is expanded from a single physical node to the entire data center, and can be further combined with new chips such as DPUs.
[0031] 4. Each GPU has the same input layer and feature extraction layer. The deployed algorithm model uses a common input layer and feature extraction layer to preprocess and extract features of the data, and then forward it to the decision layer and output layer GPU with the target algorithm.
[0032] 5. Deploying a single complete model or simply splitting the model cannot reuse the model due to differences in the input and feature extraction layers. This solution, however, uses the same input and feature extraction layers, allowing GPUs 2 and 3 to perform operations on the input and feature extraction layers, while GPU 1 performs the decision and output layer calculations, increasing the throughput of algorithm requests. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.
[0034] Figure 1 It is a schematic diagram of the connection relationship between the input layer, feature extraction layer, decision layer and output layer of the model;
[0035] Figure 2 This is an example diagram of the gateway calling each GPU computing unit;
[0036] Figure 3 This is a schematic diagram of the gateway performing load balancing on each GPU computing unit;
[0037] Figure 4 This is a diagram of the implementation steps of a GPU display memory and computing unit multi-model sharing method provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0038] The technical solution of the present invention will be further described below with reference to the accompanying drawings and through specific implementation methods.
[0039] Among them, the drawings are only used for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting this patent; in order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.
[0040] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "inside", "outside" and the like indicate an orientation or position relationship based on the orientation or position relationship shown in the drawings, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0041] In the description of the present invention, unless otherwise expressly specified or limited, when the term "connection" or the like appears to indicate a connection relationship between components, such term should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be internal communication between two components or an interaction between two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood in specific circumstances.
[0042] The size of the GPU memory determines the size and number of models that can be loaded and deployed on the GPU. After conventional model deployment occupies most of the GPU memory, the remaining memory is insufficient to deploy other complete models. When conventional models are not working, the occupied GPU memory resources waste memory for running models. In addition, the call of computing power services has a tidal ebb and flow periodicity. The fixed model deployment mode requires time to start up when the call peak of a certain algorithm model occurs, even if a virtual machine or containerization is used, which is not flexible.
[0043] This embodiment provides a GPU memory and computing unit multi-model sharing method, including the following technical steps:
[0044] 1. Model Splitting
[0045] Split the model into Figure 1 The input layer, feature extraction layer, decision layer, and output layer are shown.
[0046] The input layer is used to perform data preprocessing, data standardization and other operations on the data input to the model.
[0047] The feature extraction layer is connected to the input layer and is used to extract features from the data processed by the input layer. The same model reuses the same feature extraction layer for different detection tasks.
[0048] Decision layer: responsible for making decisions based on the features extracted by the feature extraction layer, such as classifying and identifying the types of objects in the input image.
[0049] The output layer is used to post-process the data from the decision layer according to different tasks, meet the format encapsulation requirements, and output the model processing results.
[0050] 2. Model Training
[0051] 1. Use the same data preprocessing method to normalize the input data;
[0052] 2. Use labeled data of various targets to conduct large sample training and improve the training parameters of the feature extraction layer and decision layer;
[0053] 3. Depending on the task, freeze the parameters of the feature extraction layer and fine-tune the parameters of the decision layer using the target dataset of the specific task;
[0054] 4. According to different tasks, construct the corresponding output layer that meets the task requirements.
[0055] 3. Algorithm Deployment
[0056] 1. Split the algorithm model for deployment. A complete single-body algorithm service model can be deployed on a GPU computing unit, or only any one or more of the input layer, feature extraction layer, decision layer, and output layer can be deployed. For example, the network layer of the first algorithm model includes the input layer, feature extraction layer, first decision layer, and first output layer; the network layer of the second algorithm model includes the input layer, feature extraction layer, second decision layer, and second output layer; the network layer of the third algorithm model includes the input layer, feature extraction layer, third decision layer, and third output layer. It can be seen that in this embodiment, each algorithm model can reuse the same input layer and the same feature extraction layer, but the decision layer can be the same or different, and the output layer can also be the same or different.
[0057] 2. The paths, routing tables, and load tables of each network layer (including input layer, feature extraction layer, decision layer, and output layer) of the gateway maintenance model.
[0058] 4. Algorithm Call
[0059] 1. The gateway receives the algorithm request and selects the calling path with low load according to the load table. For an example of the gateway calling each GPU computing unit, see Figure 2 For an example of load balancing, see Figure 3 .
[0060] 2. An algorithm call can be completed on a single GPU computing unit or split across multiple GPU computing units. The idle GPU computing unit processes the data of the input layer and feature extraction layer, and then forwards the processed data to the GPU computing unit with the decision layer and output layer of the algorithm model.
[0061] 3. For example, the request of the first algorithm model can be executed through the input layer, feature extraction layer, first decision layer, and first output layer deployed in the first GPU computing unit. It can also be first processed by the input layer and feature extraction layer of the second GPU computing unit, and then forwarded to the first decision layer and first output layer of the first GPU computing unit. It can also be first processed by the input layer and feature extraction layer of the second GPU computing unit, and then forwarded to the first decision layer and first output layer of the third GPU computing unit for processing. It should be noted here that the first decision layer and the first output layer in the first GPU computing unit, the second GPU computing unit, and the third GPU computing unit are the decision layer and the output layer in the network layer structure of the first algorithm model, which can execute the data decision analysis and output process of the first algorithm model.
[0062] 4. Algorithm scheduling is prioritized on the same GPU computing unit to reduce cross-GPU communication consumption. When an algorithm model is called frequently, the input layer and feature extraction layer of the algorithm model are processed on a GPU computing unit without the algorithm's decision layer and output layer. The output of the feature extraction layer is then forwarded to a GPU computing unit with the algorithm's decision layer and output layer for further data analysis and processing. This allows the GPU computing unit with the algorithm's decision layer and output layer to focus on decision-making and output, while other GPU computing units perform data input and feature extraction, thereby increasing the throughput of algorithm requests.
[0063] 5. The gateway monitors the load of each GPU computing unit and preferentially selects the GPU path with low load. The specific method is: the operation scheduling of the algorithm model is given priority to being performed on the same GPU computing unit. If the same algorithm model is fully deployed on two or more GPU computing units (the input layer, feature extraction layer, decision layer and output layer are deployed at the same time), load balancing is performed and the one with low load is called first. However, when the load of the fully deployed GPU computing units exceeds the first load threshold, the GPU computing units that have not deployed the decision layer and output layer of the algorithm model but have deployed the input layer and feature extraction layer of the algorithm model and whose load is lower than the second load threshold are allowed to participate in the calculation of the input layer and feature extraction layer of the algorithm model, and the calculation results are forwarded to the GPU computing units that have deployed the decision layer and output layer of the algorithm model to continue the calculation. Until the load of the GPU computing unit with the algorithm model fully deployed is lower than the first load threshold, the GPU computing unit with the lowest load is selected from the GPU computing units with the algorithm model fully deployed below the first load threshold for model calculation.
[0064] In summary, this embodiment provides a GPU memory and computing unit multi-model sharing method, such as Figure 1 As shown, the steps include:
[0065] S1, split each network layer for data processing of different models into common layers and distinct layers, and deploy each common layer and / or each distinct layer of each model on the corresponding GPU computing unit;
[0066] S2, identifying the network layer split structure of the model to be run, and calling the first GPU computing unit to perform data processing on the identified common layer through the load balancing strategy, and calling the second GPU computing unit to perform data processing on the identified difference layer.
[0067] The general layer includes an input layer and a feature extraction layer. The input layer is used to preprocess the data input to the model; the feature extraction layer is used to extract features from the data output after the input layer.
[0068] The distinguishing layer includes a decision layer and an output layer. The decision layer is used to make target decisions based on the data features extracted by the feature extraction layer in the general layer; the output layer is used to generate the output of the decision layer as the processing result of the model and output it.
[0069] The load balancing strategy includes the following steps:
[0070] A1: Filter out the GPU computing units for each network layer where the model to be run is deployed. The number of filtered GPU computing units must be greater than or equal to 1.
[0071] A2, determines whether the number of filtered GPU computing units is 1,
[0072] If yes, the model deployed in the selected GPU computing unit is used to process the data algorithmically;
[0073] If not, go to step A3;
[0074] A3, determining whether the current load of each filtered GPU computing unit exceeds a first load threshold,
[0075] If yes, go to step A4;
[0076] If not, then call the GPU computing unit with the lowest load among the GPU computing units screened in step A1 to perform the model processing task;
[0077] A4, screening out each first GPU computing unit that has deployed the common layer of the model to be run but has not deployed the distinguishing layer, and screening out each second GPU computing unit that has deployed the distinguishing layer of the model to be run but has not deployed the common layer;
[0078] A5: The data to be processed is transmitted to each first GPU computing unit for general layer data processing, and the output of each first GPU computing unit is forwarded to each second GPU computing unit for distinguishing layer data processing.
[0079] Preferably, in step A5, the first GPU computing unit that performs general layer data processing on the data to be processed is: the first GPU computing unit with the lowest load among the first GPU computing units screened in step A4;
[0080] In step A5, the second GPU computing unit that performs distinguishing layer data processing on the output of each first GPU computing unit is: the second GPU computing unit with the lowest load among the second GPU computing units screened in step A4.
[0081] Preferably, the load balancing strategy further includes the steps of:
[0082] A6: Determine whether there is a GPU computing unit whose current load has dropped below a first load threshold among the GPU computing units screened in step A1.
[0083] If yes, then select the model deployed in the GPU computing unit with the lowest load among the GPU computing units whose load is reduced to the first load threshold to perform subsequent data processing tasks;
[0084] If not, continue to step A5.
[0085] In summary, the present invention has the following beneficial effects:
[0086] 1. The size of the video memory determines the size and number of models that the GPU can load and deploy. After conventional model deployment occupies most of the video memory, the remaining video memory is insufficient to deploy other complete models. This method splits the model and deploys it based on the model structure. The models use the same feature extraction layer and reuse the model feature structure to avoid video memory monopoly and achieve video memory reuse. This allows the deployment of more models and fully utilizes the fragmented video memory space.
[0087] 2. Load scheduling is performed based on the load table and routing table, linking together the fragmented distributed deployment models. The scheduling algorithm selects the optimal call path. When a computing service's call peak arrives, only the decision layer of the algorithm needs to be loaded into the graphics memory, eliminating the need to deploy the entire algorithm container or virtual machine service.
[0088] 3. Supports cross-physical node GPU calls, thereby supporting the decoupling of CPU and GPU resources. The scope of the resource pool is expanded from a single physical node to the entire data center, and can be further combined with new chips such as DPUs.
[0089] 4. Each GPU has the same input layer and feature extraction layer. The deployed algorithm model uses a common input layer and feature extraction layer to preprocess and extract features of the data, and then forward it to the decision layer and output layer GPU with the target algorithm.
[0090] 5. Deploying a single complete model or simply splitting the model cannot reuse the model due to differences in the input and feature extraction layers. This solution, however, uses the same input and feature extraction layers, allowing GPUs 2 and 3 to perform operations on the input and feature extraction layers, while GPU 1 performs the decision and output layer calculations, increasing the throughput of algorithm requests.
[0091] It should be noted that the above-described specific embodiments are merely preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that various modifications, equivalent substitutions, and variations may be made to the present invention. However, as long as these modifications do not depart from the spirit of the present invention, they are intended to be within the scope of protection of the present invention. Furthermore, certain terms used in the specification and claims of this application are not intended to be limiting and are provided solely for ease of description.
Claims
1. A GPU memory and computing unit multi-model sharing method, characterized in that: Including steps: S1, splitting each network layer for data processing of different models into a general layer and a distinguishing layer, and deploying each general layer and / or each distinguishing layer of each model on a corresponding GPU computing unit; S2, identifying the network layer split structure of the model to be run, and calling the first GPU computing unit through the load balancing strategy to perform data processing on the identified general layer, and calling the second GPU computing unit to perform data processing on the identified difference layer.
2. A GPU memory and computing unit multi-model sharing method according to claim 1, characterized in that: The general layer includes an input layer and a feature extraction layer. The input layer is used to preprocess the data input to the model; the feature extraction layer is used to extract features from the data output after passing through the input layer.
3. A GPU memory and computing unit multi-model sharing method according to claim 1 or 2, characterized in that: The distinguishing layer includes a decision layer and an output layer. The decision layer is used to make target decisions based on the data features extracted by the feature extraction layer in the general layer; the output layer is used to generate the output of the decision layer as a processing result of the model and output it.
4. A GPU memory and computing unit multi-model sharing method according to claim 1, characterized in that: The load balancing strategy includes the following steps: A1, screening out the GPU computing units of each network layer where the model to be run is deployed, where the number of the screened out GPU computing units is greater than or equal to 1; A2, determining whether the number of the filtered GPU computing units is 1, If yes, the data is processed by the algorithm using the model deployed in the selected GPU computing unit; If not, go to step A3; A3, determining whether the current load of each of the screened GPU computing units exceeds a first load threshold, If yes, go to step A4; If not, calling the GPU computing unit with the lowest load among the GPU computing units screened in step A1 to perform the model processing task; A4, screening out each first GPU computing unit that has the general layer in the model to be run but has not the distinguishing layer deployed, and screening out each second GPU computing unit that has the distinguishing layer in the model to be run but has not the general layer deployed; A5: Transmit the data to be processed to each of the first GPU computing units for general layer data processing, and forward the output of each of the first GPU computing units to each of the second GPU computing units for distinguished layer data processing.
5. A GPU memory and computing unit multi-model sharing method according to claim 4, characterized in that: In step A5, the first GPU computing unit that performs general layer data processing on the data to be processed is: the first GPU computing unit with the lowest load among the first GPU computing units screened in step A4; In step A5, the second GPU computing unit that performs difference layer data processing on the output of each first GPU computing unit is: the second GPU computing unit with the lowest load among the second GPU computing units screened in step A4.
6. A GPU memory and computing unit multi-model sharing method according to claim 4, characterized in that: The load balancing strategy further comprises the steps of: A6: Determine whether there is a GPU computing unit whose current load has dropped below the first load threshold among the GPU computing units screened in step A1. If yes, then selecting the model deployed in the GPU computing unit with the lowest load among the GPU computing units whose load is reduced to the first load threshold to perform subsequent data processing tasks; If not, continue to step A5.