Model execution method and device, storage medium and equipment
By dividing the weight data of a large neural network model into several parts and loading them one by one, the storage space and computational accuracy problems of loading large models on small memory devices are solved, enabling efficient inference of large models on small memory devices.
Patent Information
- Application Number
- CN202410958586.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-26
AI Technical Summary
When loading and inferring large neural network models on devices with limited memory, existing technologies require a large amount of memory, resulting in high computational costs and compromised computational accuracy.
The model's parameter layer weight data is divided into several parts and loaded onto the device one by one. Inference operations are performed according to the connection order of the parameter layers, and the used storage space is released to load the next part of the data.
It reduces the storage space occupied by the device, ensures the computational accuracy and inference efficiency of the model, and is suitable for devices with small memory.
Smart Images

Figure CN119025259A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine learning technology, and in particular to a model execution method, device, storage medium and equipment. Background Art
[0002] With the development of deep learning technology, large neural network models (hereinafter referred to as models) have become increasingly important in various applications. However, these models usually require a lot of memory to store weights and perform calculations, which is a challenge on small memory devices.
[0003] At present, the industry has proposed quantization, pruning and other methods to reduce memory usage. Taking quantization as an example, it reduces memory usage by sacrificing a small part of accuracy. Although quantization reduces the storage usage of the model, it will also introduce more calculations and cause errors in calculation accuracy. Operations such as pruning will also have a certain impact on the accuracy of the model. Summary of the invention
[0004] The problem to be solved by the present invention is: loading and reasoning a large model on a computing device with a small memory while ensuring that the calculation accuracy is intact.
[0005] To solve the above problem, an embodiment of the present invention provides a model loading method, the method comprising:
[0006] According to the model to be loaded, create a corresponding empty model on the first device, and determine the actual occupied space of the weight data of each parameter layer in the empty model;
[0007] According to the available storage space on the first device, the weight data of each parameter layer in the empty model is divided into several parts, and the storage space occupied by each weight data is smaller than the available storage space of the first device
[0008] According to the connection order of each parameter layer in the empty model, the weight data of each parameter layer in the empty model is loaded from the disk to the corresponding position one by one, and the inference operation is performed.
[0009] In a possible embodiment, dividing the weight data of each parameter layer in the empty model into several parts according to the available storage space on the first device includes:
[0010] When the available storage space on the first device is smaller than the actually occupied space, the weight data of each parameter layer in the empty model is divided into several parts according to the available storage space on the first device.
[0011] In a possible embodiment, each piece of weight data occupies a storage space that is half of the available storage space on the first device.
[0012] In a possible embodiment, the weight data of all parameter layers contained in at least one transformation module of the model to be loaded is used as one copy of weight data; wherein, the available storage space on the first device is larger than the space occupied by the weight data of all parameter layers contained in at least two transformation modules of the model to be loaded.
[0013] In a possible embodiment, loading the weight data of each parameter layer in the empty model from the disk to the corresponding position one by one according to the connection order of each parameter layer in the empty model includes:
[0014] After each piece of weight data is loaded, the storage space occupied by the loaded weight data is released to be used for loading the next piece of weight data.
[0015] In a possible embodiment, according to the connection order of each parameter layer in the empty model, loading the weight data of each parameter layer in the empty model from the disk to the corresponding position one by one, and performing the inference operation, includes:
[0016] Loading two portions of weight data for the first time, and assigning the first portion of weight data to the empty model and performing a local reasoning operation according to the connection order of each parameter layer in the empty model;
[0017] While performing local reasoning operations using the loaded weight data, the storage space occupied by the weight data used to perform the local reasoning operations is released, and the next piece of weight data is loaded.
[0018] In a possible embodiment, creating a corresponding empty model on the first device and determining the actual occupied space of each parameter layer weight data in the empty model includes:
[0019] Performing an initialization operation on the empty model on the first device to obtain metadata information of each parameter layer in the empty model, wherein the metadata information includes parameter dimensions and data types of each parameter layer in the empty model;
[0020] According to the metadata information of each parameter layer in the empty model, the actual occupied space of the weight data of each parameter layer in the empty model is determined.
[0021] The embodiment of the present invention further provides a model loading device, the device comprising:
[0022] An empty model creation unit, configured to create a corresponding empty model on the first device according to the model to be loaded;
[0023] An occupied space calculation unit, used to determine the actual occupied space of the weight data of each parameter layer in the empty model;
[0024] A weight division unit, configured to divide the weight data of each parameter layer in the empty model into a plurality of portions according to the available storage space on the first device, wherein the storage space occupied by each portion of the weight data is smaller than the available storage space on the first device;
[0025] The loading unit is used to load the weight data of each parameter layer in the empty model from the disk to the corresponding position one by one according to the connection order of each parameter layer in the empty model, and perform the inference operation.
[0026] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the steps of any of the above methods.
[0027] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of any of the above methods when running the computer program.
[0028] Compared with the prior art, the technical solution of the embodiment of the present invention has the following advantages:
[0029] By applying the solution of the present invention, the weight data of each parameter layer in the empty model is first divided into several parts, and then when loading the weight data, the weight data can be loaded from the disk one by one according to the connection order of each parameter layer in the empty model, rather than loading all of the weight data to the first device at one time, thereby reducing the occupancy of the storage space of the first device and realizing the loading and inference of large models on a computing device with small memory. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 is a flow chart of a model execution method in an embodiment of the present invention;
[0031] Figure 2 is a flow chart of another model execution method in an embodiment of the present invention;
[0032] Figure 3 It is a structural schematic diagram of a model execution device in an embodiment of the present invention. DETAILED DESCRIPTION
[0033] The traditional large language model consists of N transformer blocks, each of which consists of multiple sub-modules such as root mean square normalization (RMSNorm), attention mechanism (Attention), feedforward network (FeedForward), etc. The attention mechanism sub-module consists of parameter layers such as linear layer (linear), matrix multiplication layer (matmul), Softmax activation function layer, rotation position encoding layer (RoPE), etc. The feedforward network sub-module consists of parameter layers such as linear layer (linear), matrix multiplication (matmul), Silu activation function layer, etc.
[0034] The input data is loaded through the input embedding layer (InputEmbedding), and the input data is processed by N conversion blocks in sequence, and then processed by the root mean square normalization layer, linear layer and Softmax activation function layer outside the N conversion blocks to obtain the output of the large language model.
[0035] Normally, when loading a large language model, you need to first create a corresponding empty model in memory, then initialize the empty model, obtain the metadata information of the parameter layer in the empty model through the initialization operation, then load the trained model parameter weight data from the disk, and finally move the model parameter weight data loaded from the disk to the corresponding location. After loading is complete, you can use the loaded large language model to perform inference operations.
[0036] In practical applications, when loading trained model parameter weight data from disk, the model parameter weight data that needs to be used is usually loaded into the memory at one time (that is, all). Taking the common large language model LLaMA2-70b as an example, the model parameter weight data stored in FP16 format will occupy about 140GB of storage space on the disk. If all are loaded into the computer's memory, a commonly used graphics processing unit (GPU) requires the GPU to have more than 140GB of video memory, which does not include the video memory occupied by the cache part, which undoubtedly greatly increases the cost of the graphics processor.
[0037] To address this problem, the present invention provides a model execution method. When the model is loaded using this method, the weight data of each parameter layer in the empty model is first divided into several parts, and then loaded one by one, so that the weight data can be taken and used immediately, thereby reducing the occupancy of the storage space of the first device. Moreover, in the process of loading the weight data, it is loaded according to the connection order of each parameter layer in the empty model, thereby avoiding affecting the calculation accuracy of the empty model.
[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0039] Reference Figure 1 , an embodiment of the present invention provides a model execution method, the method may include the following steps:
[0040] Step 11: Create a corresponding empty model on the first device according to the model to be loaded, and determine the actual occupied space of the weight data of each parameter layer in the empty model.
[0041] In a specific implementation, the empty model refers to the model structure of the model to be loaded. The model structure includes the parameter layers of the model to be loaded and the connection relationship between the parameter layers. When creating an empty model, the weight data of each parameter layer can be occupied by a placeholder without loading the actual weight data. The actual weight data can be loaded in the subsequent loading process.
[0042] In a specific implementation, if the model to be loaded is finally run on the central processing unit (CPU) of the first device, an empty model can be created on the memory of the first device. If the model to be loaded is finally run on the GPU of the first device, an empty model can be created on the video memory of the first device.
[0043] In one embodiment, creating a corresponding empty model on the first device and determining the actual occupied space of each parameter layer weight data in the empty model may include:
[0044] Performing an initialization operation on the empty model on the first device to obtain metadata information of each parameter layer in the empty model, wherein the metadata information includes parameter dimensions and data types of each parameter layer in the empty model;
[0045] According to the metadata information of each parameter layer in the empty model, the actual occupied space of the weight data of each parameter layer in the empty model is determined.
[0046] Specifically, on the first device, an initialization operation is performed on the empty model, that is, an initialization operation is performed on each parameter layer of the empty model, such as random initialization, Gaussian distribution initialization, all-zero initialization, etc. of the input embedding layer and the linear layer. After the initialization operation, metadata information of each parameter layer in the empty model can be loaded, which can specifically include the parameter dimension and data type of each parameter layer in the empty model. Among them, the data type of each parameter layer can be float32, int32, etc.
[0047] Based on the metadata information of each parameter layer in the empty model, the actual occupied space of the weight data of each parameter layer in the empty model can be calculated.
[0048] Step 12: divide the weight data of each parameter layer in the empty model into several parts according to the available storage space on the first device, and the storage space occupied by each part of the weight data is smaller than the available storage space of the first device.
[0049] In a specific implementation, after calculating the actual occupied space of the weight data of each parameter layer in the empty model, the actual occupied space can be compared with the available storage space on the first device. When the available storage space on the first device is smaller than the actual occupied space, the weight data of each parameter layer in the empty model can be divided into several parts according to the available storage space on the first device.
[0050] When creating an empty model on the memory of the first device, the actual occupied space is compared with the available storage space on the first device, that is, the actual occupied space is compared with the available storage space on the memory of the first device. When creating an empty model on the video memory of the first device, the actual occupied space is compared with the available storage space on the first device, that is, the actual occupied space is compared with the available storage space on the video memory of the first device.
[0051] In a specific implementation, the weight data of each parameter layer in the empty model can be divided into at least two parts, or three parts or more, without limitation here. The size of the space occupied by each weight data can be set according to the size of the available storage space of the first device. Among them, the space occupied by each weight data should be smaller than the available storage space of the first device.
[0052] In a specific implementation, the weight data of each parameter layer in the empty model can be divided in a variety of ways, which are not limited here.
[0053] In one embodiment, each portion of weight data may be set to occupy half of the available storage space on the first device. That is, weight data occupying half of the available storage space on the first device is taken as one portion. For example, when the available storage space on the first device is 4G, each 2G of weight data in the weight data of each parameter layer in the empty model may be taken as one portion.
[0054] In another embodiment, the weight data of all parameter layers contained in at least one transformation module of the model to be loaded can be used as one piece of weight data; wherein, the available storage space on the first device is larger than the space occupied by the weight data of all parameter layers contained in at least two transformation modules of the model to be loaded.
[0055] For example, refer to Figure 1, the large language model includes 80 transformation modules, then the weight data of each parameter layer in the empty model can be divided into 80 parts, and the weight data of all parameter layers contained in each transformation module is used as one weight data. At this time, the available storage space on the first device is greater than the weight data of all parameter layers contained in 2 transformation modules, so that one weight data can be loaded while performing calculation operations using another weight data, thereby reducing the time loss caused by data loading.
[0056] Step 13, according to the connection order of each parameter layer in the empty model, load the weight data of each parameter layer in the empty model from the disk to the corresponding position one by one, and perform the inference operation.
[0057] In the specific implementation, the connection order of each parameter layer in the empty model refers to the order of execution in the actual operation of the model to be loaded. When loading weight data, the parameter layer that executes the calculation first, the weight data is loaded first, and the parameter layer that executes the calculation later, the weight data is loaded later. Therefore, the weight data is divided into several parts, and each weight data is arranged in sequence according to the execution order of the corresponding parameter layer.
[0058] In a specific implementation, the weight data in the disk is loaded to the corresponding position of the empty model. When the weight data of each parameter layer in the empty model is loaded from the disk one by one, the storage space occupied by the loaded weight data can be released after each weight data is loaded to be used for loading the next weight data.
[0059] In one embodiment, in order to further improve the loading efficiency, two portions of weight data may be loaded for the first time, and according to the connection order of each parameter layer in the empty model, the first portion of weight data may be assigned to the empty model and a local reasoning operation may be performed. Afterwards, while the loaded weight data is used to perform the local reasoning operation, the storage space occupied by the weight data used to perform the local reasoning operation is released, and the next portion of weight data is loaded.
[0060] Specifically, taking the weight data of all parameter layers contained in each transformation module as a piece of weight data as an example, the model to be loaded may include the first transformation module, the second transformation module, the third transformation module, ..., the 80th transformation module in the execution order. At this time, the loading process can be as follows:
[0061] First, load the input data, load the weight data of the first and second transformation modules, and use the first transformation module to perform the corresponding local reasoning operation.
[0062] After the calculation part corresponding to the first transformation module is calculated, the weight data of the first transformation module is released, and the weight data of the third transformation module is loaded. At the same time, the corresponding local reasoning operation is performed using the second transformation module.
[0063] After the calculation part corresponding to the second transformation module is calculated, the weight data of the second transformation module is released, and the weight data of the fourth transformation module is loaded. At the same time, the corresponding local reasoning operation is performed using the third transformation module.
[0064] …
[0065] This continues until the weight data of the 80th transformation module is loaded and all inference operations are completed.
[0066] In the above loading process, the storage space occupied each time is only the size of the weight data of two transformation modules. In addition, except for loading the weight data of the first and second transformation modules, the time for loading the weight data of the subsequent transformation modules is synchronized with the process of performing local reasoning operations using the weight data of the previous transformation module, thereby reducing the time loss caused by data loading.
[0067] Figure 2 FIG. 1 is a flow chart of a model execution method in another embodiment of the present invention. Figure 2 , the execution process in the embodiment of the present invention may be as follows:
[0068] Step 21, according to the model to be loaded, create a corresponding empty model on the first device, and determine the actual occupied space of the weight data of each parameter layer in the empty model.
[0069] Regarding step 21, specific implementation may refer to the above description of step 11, which will not be repeated here.
[0070] Step 22: Divide the weight data of each parameter layer in the empty model into several parts according to the available storage space on the first device.
[0071] Specifically, taking the example of dividing the weight data of each parameter layer in the empty model into n parts, the weight data of each part are w1, w2, ..., wn in sequence.
[0072] Taking the weight data of each parameter layer in the empty model as an example, the actual space occupied by the weight data is 160G, and the video memory of the first device is 8G, the weight data of each parameter layer in the empty model can be divided into 54 parts, each occupying 3GB of space. Among them, 6G of the storage space of the 8G available storage space of the first device is used to load the weight data, and the remaining 2G can be used to save the intermediate results during the inference operation.
[0073] Step 23, load input data x1.
[0074] In a specific implementation, the input data x1 can be loaded from the input embedding layer of the model to be loaded.
[0075] Step 24, loading the first weight data w1 and the second weight data w2 from the disk.
[0076] Step 25, load the first weight data w1 to the corresponding position of the empty model, and use the first weight data w1 to perform local reasoning operations on the input data x1 to obtain intermediate data x2 and save it.
[0077] Step 26, release the storage space occupied by the first weight data w1, and load the third weight data w3, and at the same time load the second weight data w2 to the corresponding position of the empty model, and use the second weight data w2 to perform local reasoning operations on the intermediate data x2 to obtain the intermediate data x3, and save it.
[0078] Step 27, release the storage space occupied by the second weight data w2, and load the fourth weight data w4, and at the same time load the third weight data w3 to the corresponding position of the empty model, and use the third weight data w3 to perform local reasoning operations on the intermediate data x3, and save it.
[0079] …
[0080] Step 28, load the nth weight data wn to the corresponding position of the empty model, and perform local reasoning operations on the intermediate data xn to obtain the output data y and save it.
[0081] By adopting the scheme of the present invention, the weight data is read one by one, and the process of loading and reasoning is cyclically performed until all the weights of the entire model are loaded and reasoning is completed. Assuming that the actual space occupied by all weight data is 120G, only 3G of weight data is loaded each time. Compared with loading 120G of weight data at one time, the occupancy of storage space can be greatly reduced, allowing small memory devices to load large models in a piece-by-piece manner, solving the problem that large models cannot be run on small memory devices, not only can limited memory resources be effectively utilized, but also the continuity and accuracy of model reasoning are guaranteed. Because the model is a sequential structure, all weight data are loaded at one time. In most cases, part of the weight data is in a state of being taken but not used. By adopting the scheme of the present invention, the model weight data can be taken as needed, thereby greatly reducing the occupancy of storage space.
[0082] In order to enable those skilled in the art to better understand and implement the present invention, the device, test system, electronic device and computer-readable storage medium corresponding to the above method are described in detail below.
[0083] Reference Figure 3The embodiment of the present invention further provides a model execution device 40, which may include: an empty model creation unit 41, an occupied space calculation unit 42, a weight division unit 43 and a loading unit 44. Wherein:
[0084] The empty model creation unit 41 is used to create a corresponding empty model on the first device according to the model to be loaded;
[0085] The occupied space calculation unit 42 is used to determine the actual occupied space of the weight data of each parameter layer in the empty model;
[0086] The weight division unit 43 is used to divide the weight data of each parameter layer in the empty model into a plurality of parts according to the available storage space on the first device, and the storage space occupied by each weight data part is smaller than the available storage space of the first device;
[0087] The loading unit 44 is used to load the weight data of each parameter layer in the empty model from the disk to the corresponding position one by one according to the connection order of each parameter layer in the empty model, and perform the inference operation.
[0088] Regarding the empty model creation unit 41, the occupied space calculation unit 42, the weight division unit 43 and the loading unit 44, they can be implemented specifically by referring to the description of the corresponding steps in the above-mentioned model execution method, and will not be repeated here.
[0089] In a specific implementation, the model execution device 40 may be integrated into the first device, or may be independently provided with the first device, which is not limited here.
[0090] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the steps of any of the above methods.
[0091] In a specific implementation, the computer-readable storage medium may include: ROM, RAM, magnetic disk or optical disk, etc.
[0092] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of any of the above methods when running the computer program.
[0093] Regarding the various modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, or hardware modules / units, or they can be partially software modules / units and partially hardware modules / units. For example, for various devices and products applied to or integrated in a chip, the various modules / units included therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units included therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units can be implemented in the form of hardware such as circuits. The element can be implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in the terminal, the various modules / units contained therein can be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits.
[0094] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.
Claims
1. A model execution method, characterized in that: include: According to the model to be loaded, create a corresponding empty model on the first device, and determine the actual occupied space of the weight data of each parameter layer in the empty model; According to the available storage space on the first device, the weight data of each parameter layer in the empty model is divided into a plurality of parts, and the storage space occupied by each part of the weight data is smaller than the available storage space of the first device; According to the connection order of each parameter layer in the empty model, the weight data of each parameter layer in the empty model is loaded from the disk to the corresponding position one by one, and the inference operation is performed.
2. The model execution method according to claim 1, characterized in that: The step of dividing the weight data of each parameter layer in the empty model into several parts according to the available storage space on the first device includes: When the available storage space on the first device is smaller than the actually occupied space, the weight data of each parameter layer in the empty model is divided into several parts according to the available storage space on the first device.
3. The model execution method according to claim 1 or 2, characterized in that: Each piece of weight data occupies half of the available storage space on the first device.
4. The model execution method according to claim 1 or 2, characterized in that: The weight data of all parameter layers contained in at least one transformation module of the model to be loaded is used as one copy of weight data; wherein the available storage space on the first device is larger than the space occupied by the weight data of all parameter layers contained in at least two transformation modules of the model to be loaded.
5. The model execution method according to claim 1 or 2, characterized in that: The step of loading the weight data of each parameter layer in the empty model from the disk to the corresponding position one by one according to the connection order of each parameter layer in the empty model comprises: After each piece of weight data is loaded, the storage space occupied by the loaded weight data is released to be used for loading the next piece of weight data.
6. The model execution method according to claim 5, characterized in that: According to the connection order of each parameter layer in the empty model, the weight data of each parameter layer in the empty model is loaded from the disk to the corresponding position one by one, and the reasoning operation is performed, including: Loading two portions of weight data for the first time, and assigning the first portion of weight data to the empty model and performing a local reasoning operation according to the connection order of each parameter layer in the empty model; While performing local reasoning operations using the loaded weight data, the storage space occupied by the weight data used to perform the local reasoning operations is released, and the next piece of weight data is loaded.
7. The model execution method according to claim 1 or 2, characterized in that: The step of creating a corresponding empty model on the first device and determining the actual occupied space of the weight data of each parameter layer in the empty model includes: Performing an initialization operation on the empty model on the first device to obtain metadata information of each parameter layer in the empty model, wherein the metadata information includes parameter dimensions and data types of each parameter layer in the empty model; According to the metadata information of each parameter layer in the empty model, the actual occupied space of the weight data of each parameter layer in the empty model is determined.
8. A model execution device, characterized in that: include: An empty model creation unit, configured to create a corresponding empty model on the first device according to the model to be loaded; An occupied space calculation unit, used to determine the actual occupied space of the weight data of each parameter layer in the empty model; A weight division unit, configured to divide the weight data of each parameter layer in the empty model into a plurality of portions according to the available storage space on the first device, wherein the storage space occupied by each portion of the weight data is smaller than the available storage space on the first device; The loading unit is used to load the weight data of each parameter layer in the empty model from the disk to the corresponding position one by one according to the connection order of each parameter layer in the empty model, and perform the inference operation.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the steps of the method according to any one of claims 1 to 7.
10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the method according to any one of claims 1 to 7 are performed.
Citation Information
Cited By
Terminal large model parameter collaborative loading method, scheduling method and system
CN120144490A
Model processing method, device and system and electronic equipment
CN121349368A
