Large model weight data storage system based on GPU (Graphics Processing Unit) extension
By storing large model weight data in extended video memory and loading it cyclically into main video memory, the problem of large model weight data exceeding the capacity of main video memory is solved, achieving efficient storage and task execution.
Patent Information
- Application Number
- CN202410507803.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-25
- Publication Date
- 2025-11-04
AI Technical Summary
Existing technologies suffer from low storage efficiency when dealing with large model weight data volumes exceeding the main GPU memory capacity, making it impossible to efficiently execute training and inference tasks.
Large model weight data is stored in extended video memory, and the weight data is loaded into high-performance main video memory in a loop according to the order of use of the decoder, thus dividing the storage space to improve storage efficiency.
While increasing the amount of large model weight data supported by GPUs, the computing power is not reduced, the efficiency of GPU memory storage is improved, and the high efficiency of task execution is ensured.
Smart Images
Figure CN120892166A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data storage, in particular to a large model weight data storage system based on GPU extension. BACKGROUND
[0002] With the rapid development of artificial intelligence technology, large models with a large number of model parameters have shown excellent performance, such as GPT, LLaMA, PaLM, etc. Large models all include hundreds of billions of model parameters, and a large number of model parameters also correspondingly bring challenges to the storage efficiency and computing efficiency of GPU chips.
[0003] At present, the prior art proposes an extension memory scheme, that is, adding an extension memory to the main memory to improve the memory storage capacity. Considering the use cost, the extension memory scheme usually uses a small amount of memory with high storage efficiency and low capacity as the main memory and memory with low storage efficiency and high capacity as the extension memory.
[0004] However, when facing the case that the data amount of the large model weight data is greater than the capacity of the main memory, the large model weight data needs to be stored in the main memory and the extension memory. When performing the training task and the inference task of the large model, the weight data needs to be read from the extension memory, which leads to low efficiency of the large model when performing the task, that is, the storage efficiency of the memory is low when using the extension memory scheme. Therefore, how to improve the storage efficiency of the memory has become a problem to be solved. SUMMARY
[0005] To solve the above technical problems, the technical scheme adopted by the present application is as follows:
[0006] A large model weight data storage system based on GPU extension, the system comprises a main memory G1, an extension memory G2, a database, a processor and a memory storing a computer program, wherein the database comprises weight data X of a large model, X={x1, x2, …, x m , …, x M}, x m is the weight sub-data corresponding to the mth decoder in the large model, m is an integer in the range of [1, M], M is the total number of decoders in the large model, the data amount of the weight data X is D(X), the capacity of the main memory G1 is D(G1), the number of weight sub-data that the main memory G1 can store is N, the capacity of the extension memory G2 is D(G2), D(G2)≥D(X)≥D(G1), and the weight data X is stored in the extension memory G2. When the computer program is executed by the processor, the following steps are implemented:
[0007] S101, x1, x2, …, x NLoad x
[0008] S102, initialize target identifier i = 1.
[0009] S103, use x i Execute the task of the large model, and obtain intermediate data y i after execution. i Store in the main memory G1.
[0010] S104, x i is released from the main memory G1, and x N+i is loaded from the extended memory G2 to the main memory G1.
[0011] S105, update the target identifier i = i + 1, return to execute steps S103 to S104, until N + i = M + 1, use each weight sub-data stored in the main memory G1 to execute the task of the large model in turn, and obtain the task execution result of the large model.
[0012] The present application has obvious beneficial effects compared with the prior art. By the above technical scheme, the large model weight data storage system based on GPU extension provided by the present application can achieve considerable technical progress and practicality, and has wide industrial utilization value, and at least has the following beneficial effects:
[0013] The present application provides a large model weight data storage system based on GPU extension, the system comprising: a main memory G1, an extended memory G2, a database, a processor and a memory storing a computer program, wherein the database comprises weight data X = {x1, x2, …, x m , …, x M} of a large model, x m is a weight sub-data corresponding to the mth decoder in the large model, m is an integer in the range of [1, M], M is the total number of decoders in the large model, the data amount of the weight data X is D(X), the capacity of the main memory G1 is D(G1), the number of weight sub-data that the main memory G1 can store is N, the capacity of the extended memory G2 is D(G2), D(G2) ≥ D(X) ≥ D(G1), and the weight data X is stored in the extended memory G2, when the computer program is executed by the processor, the following steps are realized: x1, x2, …, x N is loaded from the extended memory G2 to the main memory G1, the target identifier i is initialized to 1, x i is used to execute the task of the large model, and intermediate data y i is obtained after execution. iStored in the main video memory G1, x i Release from the main video memory G1, and x N+i Load the extended video memory G2 into the main video memory G1, update the target identifier i = i + 1, return to execute steps S103 to S104 until N + i = M + 1, use the weight sub-data stored in the main video memory G1 to execute the task of the large model in sequence, and obtain the task execution result of the large model.
[0014] It is known that initially, all large model weight data is stored in extended video memory. The storage space of the main video memory is divided into a part for storing weight data and a part for storing intermediate data. The large model weight data is split into units based on the weight data of a single decoder. According to the order of use of the decoder, the weight data of the decoder is loaded into the main video memory in a loop. This ensures that the large model weight data can always be read from the high-performance main video memory when applied. Thus, while increasing the amount of large model weight data that the GPU can support using the GPU extension solution, the computing power is not reduced and the video memory storage efficiency is improved. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a flowchart illustrating the execution of a computer program by a processor in a GPU-based large model weight data storage system, as provided in an embodiment of the present invention. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] This embodiment provides a large model weight data storage system based on GPU extension. The system includes: main video memory G1, extended video memory G2, a database, a processor, and a memory storing computer programs. The database includes the weight data X = {x1, x2, ..., x...} of a large model. m , ..., x M}, x mFor the weight sub-data corresponding to the mth decoder in the large model, m is an integer in the range of [1, M], M is the total number of decoders in the large model, the data amount of the weight data X is D(X), the capacity of the main memory G1 is D(G1), the number of weight sub-data that the main memory G1 can store is N, the capacity of the extended memory G2 is D(G2), D(G2)≥D(X)≥D(G1), the weight data X is stored in the extended memory G2, see Figure 1 The flowchart of the computer program based on the GPU extension of the large model weight data storage system provided by the embodiment of the application is executed by the processor, and the following steps are implemented when the computer program is executed by the processor:
[0019] S101, x1, x2, …, x N Loaded from the extended memory G2 to the main memory G1;
[0020] S102, initialize the target identifier i=1;
[0021] S103, use x i Execute the task of the large model, and obtain intermediate data y i Load the intermediate data y i Stored in the main memory G1;
[0022] S104, x i Release from the main memory G1, and load x N+i From the extended memory G2 to the main memory G1;
[0023] S105, update the target identifier i=i+1, return to execute step S103 to step S104, until N+i=M+1, use each weight sub-data stored in the main memory G1 to execute the task of the large model in turn, and obtain the task execution result of the large model.
[0024] In this embodiment, the extended memory scheme is applied, and the extended memory scheme usually uses a main memory with higher storage performance and an extended memory with lower storage performance.
[0025] Due to the success of the GPT model, the mainstream large model, or large language model (LLM), usually adopts a decoder-only architecture. In this embodiment, the large model is also described as a priori decoder-only architecture, so the weight data X of the large model includes the weight sub-data of M decoders.
[0026] In an implementation, the implementer can also apply the present embodiment to a large model using an encoder-decoder architecture. Since the encoder and the decoder still have a fixed order of use during task execution, the method provided in the present embodiment can still be used for storing the weight data of the large model.
[0027] It should be noted that the present embodiment is applied to a scenario where the data amount D(X) of the weight data X is greater than or equal to the capacity D(G1) of the main memory G1. In such a scenario, it is inevitable to store the weight data in the extended memory because all the weight data X cannot be directly stored in the main memory G1. Since the capacity of the extended memory increases the cost, the capacity D(G2) of the extended memory is greater than or equal to the data amount D(X) of the weight data X by default.
[0028] In an implementation, the application scenario can also be that the data amount D(X) of the weight data X is greater than or equal to the available capacity of the main memory G1. The available capacity can be obtained by subtracting the preset intermediate data amount from the capacity D(G1) of the main memory G1, so that the main memory can reserve storage locations for intermediate data that needs to be continuously used, thereby improving the efficiency of model task execution.
[0029] Since the large model of the decoder has a fixed order of use during task execution, in the present embodiment, the position of the weight data in the weight data X represents its use order. In the present embodiment, the data amount of the weight sub-data of different decoders is the same by default, and the number of weight sub-data that the main memory G1 can store is N. When the large model task is started, the first N weight sub-data x1, x2, …, xN of the decoders in the weight data X can be loaded into the main memory G1. N Load from the extended memory G2 to the main memory G1.
[0030] The intermediate data can include intermediate activation data generated in the forward calculation process of the large model training task, gradient data obtained by the backward propagation calculation of the large model, optimizer state data, and KV cache data in the large model inference task, etc. The intermediate activation data can include mask (Mask) data used for dropout operation on model parameters.
[0031] It should be noted that, in general, the main memory also needs to store the input data and the intermediate calculation result of the large model, and the memory capacity occupied by the two is small, and the intermediate calculation result will be released as soon as possible after use.
[0032] Specifically, the release and calculation of the weight sub-data of the decoder are performed synchronously, the release of the calculated weight sub-data does not affect the calculation process of the uncalculated weight sub-data, and when N+i=M+1, it indicates that all the weight sub-data of the decoder has been loaded into the main memory in the execution process of the large model task, and the task of the large model is executed in sequence using the weight sub-data stored in the main memory G1, so that the execution result of the large model is obtained.
[0033] In a specific embodiment, the database further comprises an intermediate data amount D(Y) when the large model executes the task, D(G1)>D(Y), and when the computer program is executed by the processor, the following steps are further implemented:
[0034] According to the intermediate data amount D(Y), the capacity D(G1) of the main memory G1, the data amount D(X) of the weight data X, and the total number M of decoders in the large model, the number N of weight sub-data that can be stored in the main memory G1 is determined.
[0035] In this embodiment, it is assumed that D(G1)>D(Y), that is, the data amount of the intermediate data is less than the capacity of the main memory, so as to reserve storage space for the intermediate data that needs to be continuously used, thereby improving the efficiency of the model task execution. Therefore, according to the intermediate data amount D(Y), the capacity D(G1) of the main memory G1, the data amount D(X) of the weight data X, and the total number M of decoders in the large model, the number N of weight sub-data that can be stored in the main memory G1 is determined.
[0036] In a specific embodiment, the number N of weight sub-data that can be stored in the main memory G1 satisfies the condition: N=f{[D(G1)-D(Y)] / [D(X) / M]}, wherein f{} is a floor function.
[0037] In this embodiment, it is assumed that the data amount of the weight sub-data of different decoders is the same, so the data amount of a single weight sub-data can be represented as D(X) / M, the available capacity of the main memory is D(G1)-D(Y), and then [D(G1)-D(Y)] / [D(X) / M] can represent the number of weight sub-data that can be stored in the main memory. However, [D(G1)-D(Y)] / [D(X) / M] may be a non-integer, and in this embodiment, the weight data is split according to the number of decoders. Therefore, [D(G1)-D(Y)] / [D(X) / M] is processed by the floor function to update the representation of the number of weight sub-data, that is, N=f{[D(G1)-D(Y)] / [D(X) / M]}.
[0038] In a specific embodiment, the main memory G1 uses HBM storage particles.
[0039] The graphics memory using the HBM memory particles is produced by a vertical stack semiconductor process, can realize low power consumption and an ultra-wideband communication channel, has higher storage efficiency, but has higher processing cost and use cost, and therefore is used as the main graphics memory.
[0040] In a specific embodiment, the extended graphics memory G2 uses memory particles with lower cost than the HBM memory particles.
[0041] The storage efficiency and cost of the extended graphics memory are lower than those of the main graphics memory.
[0042] In a specific embodiment, the intermediate data includes KV cache data.
[0043] In this embodiment, when the decoder-only architecture large model performs an inference task, the KV cache (K-VCache) can be used to accelerate the inference process, and at this time, the KV cache data is also stored in the main graphics memory, that is, the method provided in this embodiment can support the large model to use the KV cache scheme when performing an inference task from the graphics memory level, and avoid the problem that the KV cache cannot be applied to large model inference acceleration due to insufficient graphics memory capacity.
[0044] In a specific embodiment, the task of the large model is a training task, and accordingly, the intermediate data further includes intermediate gradient data and intermediate optimizer state data.
[0045] In a specific embodiment, the x i performing the task of the large model, and obtaining intermediate data y i after the execution.
[0046] According to the intermediate data stored in the main graphics memory G1, the x i performing the task of the large model, and obtaining intermediate data y i after the execution.
[0047] The intermediate data can be used for subsequent calculation of the task execution of the large model, for example, in a training task, the intermediate activation value data can be used for calculation of a subsequent reverse gradient, etc.
[0048] In the embodiment, all large model weight data is initially stored in the extended graphics memory, the storage space of the main graphics memory is divided into a part for storing weight data and a part for storing intermediate data, the large model weight data is split in units of weight data of a single decoder, the weight data of the decoder is loaded into the main graphics memory in a loop according to the use order of the decoder, so that the large model weight data can always be read from the high-performance main graphics memory when applied, thereby increasing the amount of large model weight data that can be supported by the GPU while not reducing the computing power and improving the storage efficiency of the graphics memory when using the GPU extension scheme.
[0049] Although some specific embodiments of the present application have been described in detail by way of example with reference to the drawings, it is to be understood that the examples are for illustrative purposes only and are not to be construed as limiting the scope of the present application. It will be appreciated by persons skilled in the art that various modifications can be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.
Claims
1. A large model weight data storage system based on GPU extension, characterized in that, The system includes: main video memory G1, extended video memory G2, a database, a processor, and a memory storing computer programs, wherein the database includes weight data X = {x1, x2, ..., x...} of a large model. m , ..., x M }, x m Let X be the weight sub-data corresponding to the m-th decoder in the large model, where m is an integer in the range [1, M], and M is the total number of decoders in the large model. The data size of the weight data X is D(X). The capacity of the main video memory G1 is D(G1), and the number of weight sub-data that the main video memory G1 can store is N. The capacity of the extended video memory G2 is D(G2), where D(G2) ≥ D(X) ≥ D(G1). The weight data X is stored in the extended video memory G2. When the computer program is executed by the processor, the following steps are implemented: S101, set x1, x2, ..., x N The extended video memory G2 is loaded into the main video memory G1; S102, Initialize target identifier i = 1; S103, using x i The task of the large model is executed, and intermediate data y is obtained after the execution is completed. i , the intermediate data y i Stored in the main video memory G1; S104, x i Release from the main video memory G1, and x N+i Load from the extended video memory G2 to the main video memory G1; S105, update the target identifier i = i + 1, return to steps S103 to S104, until N + i = M + 1, use the weight sub-data stored in the main memory G1 to execute the task of the large model in sequence, and obtain the task execution result of the large model.
2. The GPU-extended large model weight data storage system according to claim 1, characterized in that, The database also includes intermediate data D(Y) during the execution of the large model task, where D(G1) > D(Y). When the computer program is executed by the processor, the following steps are also implemented: Based on the intermediate data volume D(Y), the capacity D(G1) of the main video memory G1, the data volume D(X) of the weight data X, and the total number of decoders M in the large model, determine the number N of weight sub-data that the main video memory G1 can store.
3. The GPU-extended large model weight data storage system according to claim 2, characterized in that, The number of weighted sub-data N that can be stored in the main video memory G1 satisfies the condition: N = f{[D(G1)-D(Y)] / [D(X) / M]}, where f{} is the floor function.
4. The large model weight data storage system based on GPU extension according to claim 1, characterized in that, The main video memory G1 uses HBM storage chips.
5. The GPU-extended large model weight data storage system according to claim 4, characterized in that, The extended video memory G2 uses memory chips that are less expensive than the HBM memory chips.
6. The GPU-extended large model weight data storage system according to claim 1, characterized in that, The task of the large model is inference, and accordingly, the intermediate data includes key-value cache data.
7. The large model weight data storage system based on GPU extension according to claim 1, characterized in that, The task of the large model is training; accordingly, the intermediate data also includes intermediate gradient data and intermediate optimizer state data.
8. The large model weight data storage system based on GPU extension according to claim 1, characterized in that, The use of x i The task of the large model is executed, and intermediate data y is obtained after the execution is completed. i ,include: Based on the intermediate data already stored in the main video memory G1, use x i The task of the large model is executed, and intermediate data y is obtained after the execution is completed. i .