GPU extended storage system supporting multiple users
By optimizing the storage of user key-value cache data in main video memory and migrating it to extended video memory when the interaction state meets the conditions, the problem of limited main video memory capacity in multi-user scenarios is solved, and the concurrency and computational efficiency of large model inference services are improved.
Patent Information
- Application Number
- CN202410507806.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-25
- Publication Date
- 2025-11-04
AI Technical Summary
In multi-user scenarios, existing extended video memory solutions suffer from limited main video memory capacity, resulting in intermediate data for multiple users being placed in the less efficient extended video memory, which affects the concurrency of large model inference services.
By storing the target user's key-value cache data in the main video memory and migrating it to the extended video memory when the interaction state meets the conditions, the storage location is optimized to improve the utilization of the main video memory, ensuring that each user's input data is processed in the main video memory, and supporting inference services for more users.
It increases the number of concurrent connections that large model inference services can support, improves computational efficiency and storage resource utilization, and reduces hardware costs.
Smart Images
Figure CN120892169A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data storage, in particular to a GPU extended storage system supporting multiple users. BACKGROUND
[0002] With the rapid development of artificial intelligence technology, large models with a large number of model parameters have shown excellent performance. When performing inference tasks, large models can use the KV cache method for inference acceleration. However, the KV cache requires a large amount of storage space.
[0003] At present, one method of the prior art is to store the KV cache in the CPU memory and transfer it to the GPU in batches as needed through PCIE for calculation. However, due to the bandwidth and latency limitations of the PCIE channel, it cannot effectively support the computing supply requirements of the GPU, that is, it affects the computing efficiency of the GPU. Another method of the prior art is to use a multi-GPU interconnection method to obtain a larger video memory capacity. However, the increase in the number of GPU chips greatly increases the hardware cost, and the additional network hardware requirements caused by interconnection will further increase the training and inference costs of large models. In order to overcome the above problems, an effective solution is to expand the video memory scheme, that is, to add an extended video memory to the main video memory, so that the KV cache data obtained by intermediate calculation during the execution of the large model inference task can be stored in the main video memory and the extended video memory. However, considering the use cost, the extended video memory scheme usually uses a small amount of storage with high efficiency and low capacity as the main video memory and a storage with low efficiency and high capacity as the extended video memory.
[0004] However, when the extended video memory scheme is applied to a multi-user scenario, due to the limited capacity of the main video memory, the intermediate data of most users will be placed in the extended video memory with poor storage efficiency, resulting in poor ability to provide inference services for multiple users, and thus the number of concurrent users that the large model inference service can support is low. Therefore, how to optimize storage and improve the number of concurrent users that the large model inference service can support has become a problem to be solved. SUMMARY
[0005] To solve the above technical problems, the technical scheme adopted by the present application is as follows:
[0006] A GPU extended storage system supporting multiple users, the system comprising: a main GPU chip, an extended video memory G2, a database, a processor, and a memory storing a computer program, wherein the main GPU chip corresponds to a main video memory G1, the database comprises weight data X of a large model, input data sequence Y=[y1, y2, …, y n , …, y N ] of the large model, and the processor is configured to perform inference on the large model based on the weight data X and the input data sequence Y. nFor the nth input data, n is an integer in the range of [1, N], N is the number of input data in Y, the capacity D(G1) of the main memory G1 is greater than the amount D(X) of the weight data X, the weight data X is stored in the main memory G1, and when the computer program is executed by the processor, the following steps are implemented:
[0007] S101, determining the target data for executing the inference task of the large model from the input data sequence Y, and determining the target user corresponding to the target data.
[0008] S102, using the weight data X and the target data, calculating the KV cache data corresponding to the target user, storing the KV cache data of the target user in the main memory G1, and obtaining the time point of completing the calculation as the first time point.
[0009] S103, determining the waiting time period according to the first time point and the preset waiting time length.
[0010] S104, detecting the interaction state of the target user at a fixed frequency during the waiting time period.
[0011] S105, if the interaction state of the target user is the second interaction state during the waiting time period, moving the KV cache data of the target user from the main memory G1 to the extended memory G2, deleting the target data from the input data sequence Y, and updating the input data sequence Y.
[0012] S106, returning to steps S101 to S105 until the input data sequence Y is empty.
[0013] Compared with the prior art, the GPU extension storage system supporting multiple users provided by the application has obvious beneficial effects, and can achieve considerable technical progress and practicality, and has wide industrial utilization value, and at least has the following beneficial effects:
[0014] The application provides a GPU extension storage system supporting multiple users, which comprises a main GPU chip, an extended memory G2, a database, a processor and a memory storing a computer program, wherein the main GPU chip corresponds to a main memory G1, the database comprises weight data X of a large model, an input data sequence Y=[y1, y2, …, y n , …, y N ] and y nFor the nth input data, n is an integer in the range of [1, N], N is the number of input data in Y, the capacity D(G1) of the main memory G1 is greater than the amount D(X) of the weight data X, the weight data X is stored in the main memory G1, when the computer program is executed by the processor, the following steps are implemented: determining the target data for executing the inference task of the large model from the input data sequence Y, determining the target user corresponding to the target data as the target user, using the weight data X and the target data, calculating the KV cache data corresponding to the target user, storing the KV cache data of the target user in the main memory G1, and obtaining the time point of completing the calculation as the first time point, determining the waiting time period according to the first time point and the preset waiting time, detecting the interaction state of the target user at a fixed frequency during the waiting time period, if the interaction state of the target user is the second interaction state during the waiting time period, then the KV cache data of the target user is moved from the main memory G1 to the extended memory G2, and the target data is deleted from the input data sequence Y, the input data sequence Y is updated, and the target user is selected and the inference service is performed until the input data sequence Y is empty.
[0015] It can be known that when the large model executes the inference task, the KV cache data corresponding to the target user is first stored in the main memory with higher performance, so as to ensure that the calculation efficiency when the KV cache data is applied is higher, after the inference task of the target data corresponding to the target user is completed, the interaction state of the target user is detected, when the interaction state meets the first preset condition, the KV cache data corresponding to the target user is moved to the extended memory for storage, and the target user is re-determined, so that when the input data of each user is processed, the corresponding KV cache data is in the main memory, and a limited main memory capacity can be used to provide inference service for more users, thereby improving the concurrent quantity that can be supported by the large model inference service. BRIEF DESCRIPTION OF DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.
[0017] Figure 1 A flowchart of steps implemented by a computer program when executed by a processor in a GPU extended storage system supporting multiple users provided by the first embodiment of the present application.
[0018] Figure 2 A flowchart of steps implemented by a computer program when executed by a processor in a GPU extended storage system based on user interaction detection provided by the second embodiment of the present application. DETAILED DESCRIPTION
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] Embodiment 1 of the present invention provides a GPU extended storage system supporting multiple users. The system includes: a main GPU chip, extended video memory G2, a database, a processor, and a memory storing computer programs. The main GPU chip corresponds to the main video memory G1. The database includes weight data X of a large model and an input data sequence Y = [y1, y2, ..., y...]. n , ..., y N ], y n Let n be the nth input data, where n is an integer in the range [1, N], and N is the number of input data in Y. The capacity D(G1) of the main video memory G1 is greater than the amount D(X) of the weight data X. The weight data X is stored in the main video memory G1. See [link to documentation]. Figure 1 This is a flowchart illustrating the steps implemented when a computer program is executed by a processor in a multi-user GPU extended storage system according to Embodiment 1 of the present invention. When the computer program is executed by the processor, the following steps are implemented:
[0021] S101, determine the target data for performing the inference task of the large model from the input data sequence Y, and determine the user corresponding to the target data as the target user;
[0022] S102, using the weight data X and the target data, calculate the KV cache data corresponding to the target user, store the KV cache data of the target user in the main video memory G1, and obtain the time point when the calculation is completed as the first time point;
[0023] S103, determine the waiting time period based on the first time point and the preset waiting time;
[0024] S104, during the waiting period, the interaction status of the target user is detected at a fixed frequency;
[0025] S105, if the target user's interaction state is the second interaction state during the waiting period, then the target user's KV cache data is moved from the main video memory G1 to the extended video memory G2, and the target data is deleted from the input data sequence Y, and the input data sequence Y is updated;
[0026] S106, return to execute steps S101 to S105 until the input data sequence Y is an empty sequence.
[0027] The extended memory solution uses a main memory with higher storage performance and an extended memory with lower storage performance for implementation. The main memory can use HBM storage particles, and the extended memory can use storage particles with lower cost or lower bandwidth than HBM storage particles.
[0028] In this embodiment, the capacity D(G1) of the default main memory G1 is greater than the amount D(X) of the weight data X, that is, the main memory can store not only the large model weight data but also the intermediate data when the large model provides inference services for users.
[0029] It should be noted that before determining the target user, the implementer can compare the remaining capacity in the current main memory G1 with the expected amount of user intermediate data. If the remaining capacity in the current main memory G1 is greater than the expected amount of user intermediate data, then step S101 is started. The expected amount of user intermediate data can be determined according to the amount of intermediate data corresponding to the historical inference services provided by the large model, for example, by taking the average of the amount of intermediate data corresponding to the historical inference services provided by the large model as the expected amount of user intermediate data. The implementer can select historical inference services within a preset time period for calculation according to actual conditions, or use statistical values such as median and mode to determine the expected amount of user intermediate data. The above adjustments are within the protection scope of the present application.
[0030] In this embodiment, the KV cache data of the target user can refer to the KV cache data corresponding to the target data and the inference result, that is, the KV cache data of the target user is the KV cache data after the inference service is completed based on the target data.
[0031] In this embodiment, the time point of completion of calculation is taken as the first time point, which can represent the termination time of the inference service. The waiting time length can be used to wait for the user to input information again. Let the first time point be t1 and the waiting time length be T, then the waiting time period is [t1, t1+T].
[0032] The embodiment sets the fixed frequency to 1 second for 5 detections, for example, when the waiting time is 2 seconds, a total of 10 detections of the interactive state of the target user are performed, if the detection results of the ten times are all the second interactive state, the KV cache data of the target user is moved from the main memory to the extended memory, since the cost of the extended memory is smaller, the capacity of the extended memory is larger by default, moving the KV cache data of the target user to the extended memory can provide the historical context information for the inference service with the KV cache data stored in the extended memory, ensuring the accuracy of the inference service, in addition, when the interactive state of the target user is stable for a certain length of time, it can be considered that the inference service does not need to be provided for the target user in real time, the position occupied by the KV cache data in the main memory can be vacated to support the inference service calculation of the next user, thereby improving the number of concurrent inference tasks that can be supported by the large model.
[0033] In a specific embodiment, the target data for performing the inference task of the large model is determined from the input data sequence Y, including:
[0034] The first corresponding input data in the input data sequence Y is selected as the target data.
[0035] In the embodiment, the input data is represented in a sequence manner, so that the input data has an order, which can be directly used as the order of providing the inference service by the large model, that is, the first corresponding input data in the input data sequence Y is selected as the target data, and after the target data is processed, the input data sequence Y is updated by deleting the target data.
[0036] In a specific embodiment, when the computer program is executed by the processor, the following steps are also implemented:
[0037] When the database receives new input data, the new input data is inserted after the end of the input data sequence Y, and the input data sequence Y is updated.
[0038] In a specific embodiment, when the computer program is executed by the processor, the following steps are also implemented:
[0039] Step S201, determining that the user corresponding to the KV cache data stored in the extended memory G2 is a candidate user;
[0040] Step S202, detecting the interactive state of each candidate user at the fixed frequency;
[0041] Step S203, when the interaction state of any candidate user is the first interaction state, the KV cache data of the candidate user is moved from the extended graphics memory G2 to the main graphics memory G1.
[0042] In this way, the large model can continue to provide inference services for another input data of the candidate user, and thus when it is detected that the interaction state of the candidate user is the first interaction state, i.e., it is detected that the candidate user starts to input information, the KV cache data of the candidate user is moved from the extended graphics memory G2 to the main graphics memory G1, so as to respond to the candidate user in time.
[0043] It should be noted that, if the target user is a candidate user, when step S102 is performed, the existing KV cache data, weight data X and target data are used to calculate updated KV cache data corresponding to the target user, the KV cache data of the target user is stored in the main graphics memory G1, and the time point when the calculation is completed is obtained as the first time point.
[0044] In one specific embodiment, the first interaction state is an input state.
[0045] In this way, the input state can be determined by detecting the interactive box, and if the user selects the interactive box and inputs data, it can be considered that the user starts to input information.
[0046] In one specific embodiment, the second interaction state is an uninput state.
[0047] In one specific embodiment, the determination of the waiting time period according to the first time point and the preset waiting time length comprises:
[0048] Obtaining a user priority p corresponding to the target user;
[0049] Determining a waiting time length T corresponding to the target user according to the user priority p;
[0050] Determining the waiting time period according to the first time point and the waiting time length corresponding to the target user.
[0051] In this way, the user priority can be determined according to the payment information, use frequency and the like of the user, for example, the more the user pays, the higher the user priority, or the higher the use frequency of the user, the higher the user priority, so as to be suitable for the commercial large model to provide different experiences for different users when providing inference services, so as to support low-latency inference of high-priority users at a limited cost through reasonable data migration.
[0052] In one specific implementation, the waiting time T corresponding to the target user satisfies the condition: T = q / [exp(-k1p)+1], where q is the preset maximum waiting time and k1 is the first adjustment coefficient.
[0053] In this embodiment, q can be set to 2 and k1 can be set to 1. The implementer can adjust the values of q and k1 according to the actual situation. It should be noted that since the calculation result of T can be a decimal, the implementer can perform rounding operation according to the actual situation.
[0054] In this embodiment, when the large model performs inference tasks, the key-value cache data corresponding to the target user is first stored in the high-performance main video memory to ensure high computational efficiency when applying the key-value cache data. After the inference task corresponding to the target user's target data is completed, the interaction state of the target user is detected. When the interaction state meets the preset conditions, the key-value cache data corresponding to the target user is moved to the extended video memory for storage, and then the target user is re-determined. This ensures that when the input data of each user is processed, the corresponding key-value cache data is in the main video memory, and inference services can be provided to more users with limited main video memory capacity, thereby increasing the number of concurrent requests that the large model inference service can support.
[0055] Based on the GPU extended storage system supporting multiple users provided in Embodiment 1 of this invention, Embodiment 2 of this invention provides a GPU extended storage system based on user interaction detection. The system includes: a main GPU chip, extended video memory G2, a database, a processor, and a memory storing computer programs. The main GPU chip corresponds to the main video memory G1. The database includes weight data X of a large model and an input data sequence Y = [y1, y2, ..., y...]. n , ..., y N ], y n Let n be the nth input data, where n is an integer in the range [1, N], and N is the number of input data in Y. The capacity D(G1) of the main video memory G1 is greater than the amount D(X) of the weight data X. The weight data X is stored in the main video memory G1. See [link to documentation]. Figure 2 This is a flowchart illustrating the steps implemented when a computer program is executed by a processor in a GPU extended storage system based on user interaction detection, according to Embodiment 2 of the present invention. When the computer program is executed by the processor, the following steps are implemented:
[0056] S301, determine the target data for performing the inference task of the large model from the input data sequence Y, and determine the user corresponding to the target data as the target user;
[0057] S302, using the weight data X and the target data, calculating the KV cache data corresponding to the target user, and obtaining the time point of completing the calculation as a first time point;
[0058] S303, determining a second time point according to the first time point and a preset waiting duration, and forming a waiting time period from the first time point and the second time point;
[0059] S304, detecting the interaction state of the target user at a fixed frequency during the waiting time period;
[0060] S305, if the interaction state of the target user from any time point in the waiting time period to the second time point is the third interaction state, the KV cache data of the target user is stored in the extended video memory G2, otherwise, the KV cache data of the target user is stored in the main video memory G1, the target data is deleted from the input data sequence Y, and the input data sequence Y is updated;
[0061] S306, return to execute steps S301 to S305 until the input data sequence Y is empty.
[0062] Wherein, the first time point and the preset waiting duration are added to determine the addition result as the second time point.
[0063] If the interaction state of the target user from any time point in the waiting time period to the second time point is the third interaction state, it means that the target user is likely to input data immediately, and the KV cache data of the target user is directly stored in the extended video memory G2.
[0064] In an embodiment, the interaction state of the target user can be monitored at a fixed frequency when the target data is inferred and calculated, and the KV cache data corresponding to the target user can be divided into stored KV cache data and new KV cache data. When the new KV cache data is obtained, it is determined whether the state of the target user is the third interaction state. If the state of the target user is not the third interaction state, the new KV cache data is stored in the main video memory as the stored KV cache data. If the state of the target user is the third interaction state, the new KV cache data is stored in the extended video memory, and the stored KV cache data of the target user in the main video memory is moved to the extended video memory. This embodiment can timely change the storage location of the KV cache data of the target user during inference and calculation, and further improve the storage efficiency of the extended video memory scheme.
[0065] In a specific embodiment, the target data for executing the inference task of the large model is determined from the input data sequence Y, comprising:
[0066] selecting input data corresponding to a first position in the input data sequence Y as the target data.
[0067] In a specific implementation, when the computer program is executed by the processor, the following steps are further implemented:
[0068] When the database receives new input data, the new input data is inserted after the last position of the input data sequence Y, and the input data sequence Y is updated.
[0069] In a specific implementation, when the computer program is executed by the processor, the following steps are further implemented:
[0070] determining that a user corresponding to the KV cache data stored in the extended graphics memory G2 is a candidate user;
[0071] detecting the interaction state of each candidate user at the fixed frequency;
[0072] When the interaction state of any candidate user is a first interaction state, the KV cache data of the candidate user is moved from the extended graphics memory G2 to the main graphics memory G1.
[0073] The first interaction state is an input state.
[0074] In a specific implementation, the third interaction state is an interaction page non-stay state.
[0075] The interaction page non-stay state can include that the interaction page is switched, the interaction page is closed, or the interaction page is slid such that the interaction box for input cannot be selected.
[0076] In a specific implementation, step S305 further includes the following steps:
[0077] S401, if the interaction state of the target user is the third interaction state from any time point in the waiting time period to the second time point, the user priority p of the target user is obtained;
[0078] S402, the storable cache quantity w of the main graphics memory G1 is obtained;
[0079] S403, using the mapping relationship between the storable cache quantity and the preset reference priority, the storable cache quantity w is mapped to the reference priority r;
[0080] S404, when p < r, storing the KV cache data of the target user in the extended graphics memory G2 directly, otherwise, storing the KV cache data of the target user in the main graphics memory G1, deleting the target data from the input data sequence Y, and updating the input data sequence Y.
[0081] The user optimization level can be determined according to user payment information, use frequency, etc. For example, the more the user pays, the higher the user priority, or the higher the user use frequency, the higher the user priority, so as to adapt to the commercial large model to provide different experiences for different users when providing inference services according to user priority.
[0082] In a specific embodiment, the method further includes:
[0083] obtaining the remaining graphics memory capacity D(s) of the main graphics memory G1.
[0084] According to the remaining graphics memory capacity and the preset reference cache data amount D(h), the storable cache amount w is calculated as w=D(s) / D(h).
[0085] The remaining graphics memory capacity D(s)=D(G1)-D(X)-D(other), and D(other) can be the data amount corresponding to other data that the main graphics memory needs to store. The other data that the main graphics memory needs to store can include input data, intermediate results of calculation, etc.
[0086] In an embodiment, since the inference service is usually performed in batches, the service amount f of a single batch also needs to be considered when calculating the storable cache amount, so w=D(s) / [f*D(h)].
[0087] In a specific embodiment, the mapping relationship between the storable cache amount and the reference priority is r=max(p)*exp(-k2w), and k2 is a second adjustment coefficient.
[0088] max(p) is the maximum priority, and k2 is set to 0.5 in this embodiment. The implementer can adjust the value of k2 according to the actual situation.
[0089] In this embodiment, when the large model performs an inference task, after obtaining the KV cache data of the target user, the interaction detection is performed on the target user, and when the interaction state of the target user meets the second preset condition, the KV cache data corresponding to the target user is directly stored in the extended display memory with low performance, so as to avoid unnecessary overhead caused by subsequent KV cache data transfer, and make the storage space of the main display memory better supply the inference service with strong real-time performance, improve the utilization efficiency of the main display memory, and then improve the efficiency of the large model in providing the inference service.
[0090] Although some specific embodiments of the present application have been described in detail by way of example with reference to the drawings, it is to be understood that the examples are for illustrative purposes only and are not to be construed as limiting the scope of the present application. Those skilled in the art should understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.
Claims
1. A GPU extended storage system supporting multiple users, characterized in that, The system includes: a main GPU chip, extended video memory G2, a database, a processor, and a memory storing computer programs. The main GPU chip corresponds to the main video memory G1. The database includes weight data X of a large model and an input data sequence Y = [y1, y2, ..., y...]. n , ..., y N ], y n Let n be the nth input data, where n is an integer in the range [1, N], and N is the number of input data in Y. The capacity D(G1) of the main video memory G1 is greater than the amount D(X) of the weight data X. The weight data X is stored in the main video memory G1. When the computer program is executed by the processor, the following steps are implemented: S101, determine the target data for performing the inference task of the large model from the input data sequence Y, and determine the user corresponding to the target data as the target user; S102, using the weight data X and the target data, calculate the KV cache data corresponding to the target user, store the KV cache data of the target user in the main video memory G1, and obtain the time point when the calculation is completed as the first time point; S103, determine the waiting time period based on the first time point and the preset waiting time; S104, during the waiting period, the interaction status of the target user is detected at a fixed frequency; S105, if the target user's interaction state is the second interaction state during the waiting period, then the target user's KV cache data is moved from the main video memory G1 to the extended video memory G2, and the target data is deleted from the input data sequence Y, and the input data sequence Y is updated; S106, return to steps S101 to S105 until the input data sequence Y is an empty sequence.
2. The GPU extended storage system supporting multiple users according to claim 1, characterized in that, The step of determining the target data for performing the inference task of the large model from the input data sequence Y includes: The first input data corresponding to the first element in the input data sequence Y is selected as the target data.
3. The GPU extended storage system supporting multiple users according to claim 2, characterized in that, When the computer program is executed by the processor, the following steps are also performed: When the database receives new input data, it inserts the new input data after the end of the input data sequence Y and updates the input data sequence Y.
4. The GPU extended storage system supporting multiple users according to claim 3, characterized in that, When the computer program is executed by the processor, the following steps are also performed: Step S201: Determine the user corresponding to the KV cache data stored in the extended video memory G2 as the candidate user; Step S202: Detect the interaction status of each candidate user at the fixed frequency; Step S203: When the interaction state of any candidate user is the first interaction state, move the KV cache data of the candidate user from the extended video memory G2 to the main video memory G1.
5. The GPU extended storage system supporting multiple users according to claim 4, characterized in that, The first interactive state is the input state.
6. The GPU extended storage system supporting multiple users according to claim 1, characterized in that, The second interaction state is the no-input state.
7. The GPU extended storage system supporting multiple users according to claim 1, characterized in that, The step of determining the waiting time period based on the first time point and the preset waiting time includes: Obtain the user priority p corresponding to the target user; Based on the user priority p, determine the waiting time T corresponding to the target user; The waiting time period is determined based on the first time point and the waiting time corresponding to the target user.
8. The GPU extended storage system supporting multiple users according to claim 7, characterized in that, The waiting time T for the target user satisfies the condition: T = q / [exp(-k1p)+1], where q is the preset maximum waiting time and k1 is the adjustment coefficient.