Model weighting compression

Abstract

Provided in the embodiments of the present disclosure are model weighting compression methods, a device and a storage medium. A method comprises: acquiring a model weighting in a first data format of a neural network model; converting the model weighting in the first data format into a model weighting in a second data format, the data length of the second data format being smaller than the data length of the first data format; performing compression processing on the model weighting in the second data format; and storing the compressed model weighting in a video memory. The present disclosure converts the model weighting in the first data format of the neural network model into the model weighting in the second data format, so as to shorten the data length of the first data format; and the present application then performs compression processing on the model weighting in the second data format which has the shortened data length, and stores the compressed model weighting in the video memory.

Description

[0001] Model weight compression technology field

[0002]

[0001] The present disclosure relates to the field of artificial intelligence processing technology, and more particularly to model weight compression.

[0003]

[0002] Large Language Models (LLMs) have a large number of parameters and a complex structure, requiring powerful computing power and storage space for training and reasoning. Generally, high-performance computing devices such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs) are used, and distributed computing technology is used to accelerate the model training and reasoning process.

[0004] In the related art, for example, a large language model requires at least 350GB of video memory at FP16 precision and at least 700GB of video memory at FP32 precision. However, a common GPU currently only has 80GB of video memory per card. During the inference phase, at least five GPUs are required to complete the inference process for 350GB of parameters. Therefore, large language models currently require a large amount of GPU memory resources during inference.

[0005]

[0004] The embodiments of the present disclosure provide a model weight compression method, device and storage medium for reducing the amount of GPU memory occupied by a neural network model during the inference process.

[0006]

[0005] In a first aspect, an embodiment of the present disclosure provides a model weight compression method, the method comprising: obtaining a model weight of a neural network model in a first data format; converting the model weight in the first data format into a model weight in a second data format, wherein the data length of the second data format is smaller than the data length of the first data format; compressing the model weight in the second data format; and storing the compressed model weight in a video memory.

[0007]

[0006] In a second aspect, an embodiment of the present disclosure provides a model weight compression device, which includes: an acquisition module for acquiring model weights in a first data format of a neural network model; a conversion module for converting the model weights in the first data format into model weights in a second data format, wherein the data length of the second data format is smaller than the data length of the first data format; a compression module for compressing the model weights in the second data format; and a storage module for storing the compressed model weights in a video memory.

[0008]

[0007] In a third aspect, an embodiment of the present disclosure provides a model weight compression method, the method comprising: receiving a request triggered by a client device by calling a neural network model service provided by the cloud, the request including a model inference task; obtaining a model weight in a first data format of a neural network model; converting the model weight in the first data format into a model weight in a second data format, wherein the data length of the second data format is smaller than the data length of the first data format; compressing the model weight in the second data format; storing the compressed model weight in a video memory; executing the inference task according to the compressed model weight stored in the video memory; and feeding back the execution result of the inference task to the client device.

[0009]

[0008] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the model weight compression method described in the first aspect or the third aspect.

[0010]

[0009] In a fifth aspect, an embodiment of the present disclosure provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the model weight compression method described in the first aspect or the third aspect.

[0011]

[0010] In the model weight compression scheme provided by the embodiment of the present disclosure, by converting the model weights of the neural network model in a first data format into the model weights of the second data format, the data length of the first data format is shortened, the amount of data required for subsequent model weight compression is reduced, and the efficiency of subsequent compression processing is improved. Then, by compressing the model weights in the second data format after the shortened data length, and storing the compressed model weights in the video memory, the amount of data stored in the video memory is reduced, thereby reducing the GPU video memory resource requirements of the neural network model during the inference process, and saving video memory resources.

[0012]

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0013] FIG1 is a schematic diagram of a hardware execution environment for a model weight compression method provided in an embodiment of the present disclosure;

[0014] FIG2 is a schematic diagram of a cloud computing environment for a model weight compression method provided in an embodiment of the present disclosure;

[0015] FIG3 is a flow chart of a model weight compression method provided in an embodiment of the present disclosure;

[0016] FIG. 4 is an example diagram of model weights in a first data format according to an embodiment of the present disclosure;

[0017] FIG. 5 is an example diagram of model weights in a second data format according to an embodiment of the present disclosure;

[0018]

[0017] FIG6 is a flow chart of a method for processing an inference task provided in an embodiment of the present disclosure;

[0019]

[0018] FIG7 is an application example diagram of a model weight compression method provided in an embodiment of the present disclosure;

[0019] FIG8 is a structural schematic diagram of a model weight compression device provided in an embodiment of the present disclosure;

[0020]

[0020] FIG9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure.

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present disclosure without creative effort are within the scope of protection of the present disclosure.

[0022]

[0022] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0023]

[0023] Some embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. The following embodiments and features thereof may be combined with one another unless they conflict with each other. Furthermore, the sequence of steps in the following method embodiments is provided for illustrative purposes only and is not intended to be a strict limitation.

[0024]

[0024] First, the terms or concepts involved in the embodiments of the present disclosure are explained.

[0025]

[0025] rANS: is a data encoding and decoding method based on entropy, and is also a lossless compression algorithm that can achieve the purpose of compressing data by encoding and decoding the input data.

[0026]

[0026] FP16: Half-precision floating-point, a floating-point number represented by half precision based on the IEEE 754-2008 standard. It requires 16 bits to store in a computer. In simple terms, it is a floating-point number represented by 16 bits of binary.

[0027]

[0027] FP32: The full name is Single-precision floating-point, which is a full-precision floating-point type representation based on the IEEE 754-2008 standard. It requires 32 bits of storage in a computer. In simple terms, it is a floating-point number represented by 32 bits of binary.

[0028]

[0028] In order to reduce the memory space occupied by neural network models with large data volumes, such as LLM, the present disclosure provides a model weight compression method. This method can compress model weights to reduce the GPU memory space occupied by its weight parameters during the model inference phase, thereby saving memory resources.

[0029] The following is an introduction to the model weight compression scheme provided by the embodiments of the present disclosure.

[0029]

[0030] Figure 1 is a schematic diagram of a hardware execution environment of a model weight compression method provided by an embodiment of the present disclosure. As shown in Figure 1, the hardware execution environment of the model weight compression method can be composed of a client device 101 and / or a server device 102, and the client device 101 is communicatively connected to the server device 102.

[0030]

[0031] The server device 102 may be a cloud server of a cloud service provider.

[0031]

[0032] The client device 101 may be a laptop, tablet computer, PC, etc. Optionally, the execution process of the above-mentioned model weight compression method may be as follows: a user sends a model inference task to the server device 102 via the client device 101. The server device 102 (also referred to as the cloud) obtains the model weights of the neural network model in a first data format, converts the model weights in the first data format into model weights in a second data format (the data length of the second data format is shorter than the data length of the first data format), compresses the model weights in the second data format, stores the compressed model weights in video memory, executes the inference task based on the compressed model weights stored in the video memory, and then feeds back the execution result of the inference task to the client device 101.

[0032]

[0033] In practical applications, the client device 102 may be a cloud server maintained by a cloud service provider, also known as a computing node. In a cloud computing environment as shown in FIG2 , several distributed computing nodes (cloud servers) (illustrated as 201-1, 201-2, ... in FIG2 ) may be included, each of which has processing resources such as computing and storage. In a cloud computing environment, multiple computing nodes may be organized to provide a particular service. Of course, a single computing node may also provide one or more services, such as Service A, Service B, Service C, and Service D illustrated in FIG2 . Services in a cloud computing environment may be provided by providing a service interface, which client devices invoke to access the corresponding service. Service interfaces include software development kits (SDKs) and application programming interfaces (APIs), among others.

[0033]

[0034] The above services are deployed based on various virtualization technologies supported by cloud computing environments, such as virtual machines and containers. Taking container-based virtualization as an example, the containers corresponding to a service can be assembled into a container group (pod). For example, service B, shown in Figure 2, can be configured with one or more pods, each of which can include a proxy and one or more containers. The one or more containers in the pod are used to process requests related to one or more corresponding functions of the service, while the proxy in the pod is used to control network functions related to the service, such as routing and load balancing.

[0034]

[0035] During operation, executing a request from a client device may require invoking one or more services in the cloud computing environment. Executing one or more functions of one service may also require invoking one or more functions of another service. As shown in Figure 2, after receiving a request from a client device, service A can invoke service B, which in turn can request service D to execute one or more functions.

[0035]

[0036] The following describes in detail the execution process of the model weight compression method provided by the embodiment of the present disclosure in conjunction with the accompanying drawings. The model weight compression method can be executed by the above-mentioned client device or by the computing node in the above-mentioned cloud computing environment.

[0036]

[0037] FIG3 is a flow chart of a model weight compression method provided by an embodiment of the present disclosure. As shown in FIG3 , the method includes the following steps.

[0037]

[0038] 301. Obtain model weights of a neural network model in a first data format.

[0038]

[0039] 302. Convert the model weights in the first data format into model weights in the second data format, where the data length of the second data format is smaller than the data length of the first data format.

[0039]

[0040] 303. Compress the model weights in the second data format.

[0040]

[0041] 304. Store the compressed model weights in video memory.

[0041]

[0042] It should be noted that the methods provided in the embodiments of the present disclosure are applicable to any neural network model and are not limited to large language models. However, if the data volume of the neural network model's weight parameters is small and does not impose a significant burden on graphics memory, the methods provided in the embodiments of the present disclosure may not be used for model weight compression. Specifically, the methods provided in the embodiments of the present disclosure are primarily intended for neural network models whose weight parameter data volume exceeds a set threshold. The set threshold can be determined based on actual conditions (e.g., based on the memory capacity available on a graphics card) and is not limited here.

[0042]

[0043] In the embodiment of the present disclosure, the model weight in the first data format includes a sign bit, an exponent bit, and a first fraction bit, and the model weight in the second data format includes a sign bit, an exponent bit, and a second fraction bit, wherein the second fraction bit is truncated from the first fraction bit. Therefore, it can be seen that the data length corresponding to the second data format is smaller than the data length corresponding to the first data format.

[0043]

[0044] For ease of understanding, the solution of the present disclosure is described below by taking the first data format being FP16 as an example.

[0044]

[0045] First, let’s briefly introduce the data structure of FP16.

[0045]

[0046] The FP16 data structure consists of a sign bit, five exponent bits, and the first fraction bit. The first fraction bit consists of 10 fraction bits, as shown in Figure 4. In Figure 4, the data corresponding to the sign bit is "0", the data corresponding to the exponent bit is "0 1 1 00", and the data corresponding to the fraction bit is "0 1 00000000".

[0046]

[0047] The model weight in the second data format after conversion includes a sign bit, an exponent bit, and a second fraction bit, where the second fraction bit is truncated from the first fraction bit. For example, the first 7 bits of the FP16 first fraction bit can be truncated to form the second fraction bit. For details, please refer to Figure 5, which is an example diagram of the model weight in the second data format. In Figure 5, the model weight in the second data format consists of 1 sign bit, 5 exponent bits, and 7 second fraction bits, where the data corresponding to the sign bit is "0", the data corresponding to the exponent bit is "0 1 1 00", and the data corresponding to the second fraction bit is "0 1 0 00 00".

[0047]

[0048] It should be noted that in actual applications, intercepting the first 7 bits of the first fraction digit of FP16, that is, deleting the last three bits of the subsequent fraction digits to form the second fraction digit, has no significant impact on the loss of overall accuracy of the neural network model and can be ignored.

[0048]

[0049] In practical applications, the model weights corresponding to a first data format (e.g., FP16) of a neural network model are obtained and converted to model weights in a second data format. Specifically, the last three digits of the first fraction digit in the first data format are deleted. In this case, the data length of the second data format (13 bits) is smaller than the data length of the first data format (16 bits). Subsequently, the model weights in the second data format are compressed and stored in video memory. The compression of the model weights in the second data format can be performed by using a lossless compression method to compress all or part of the model weights in the second data format, thereby further reducing the storage space occupied by the model weights.

[0049]

[0050] Optionally, converting the model weights in the first data format to the model weights in the second data format includes: determining the model weights in the first data format of a first type of network layer and the model weights in the first data format of a second type of network layer of the neural network model, wherein the number of the first type of network layers in the neural network model is greater than the number of the second type of network layers; and converting the model weights in the first data format of the first type of network layer to the model weights in the second data format. Based on this, storing the compressed model weights in the video memory includes: storing the compressed model weights corresponding to the first type of network layer in the video memory, and storing the model weights in the first data format of the second type of network layer in the video memory.

[0050]

[0051] It is understood that, in practice, a network layer contains a large number of model weights, typically represented as matrices or tensors. Therefore, the model weights in the embodiments of this disclosure can be understood as elements contained in a weight matrix / tensor. To facilitate understanding, the following examples illustrate the contents of this embodiment.

[0051]

[0052] Taking the LLM as an example, the LLM input layer has several types of embedding network layers (also called embedding layers), such as the token embedding layer (also called the token embedding layer) and the position embedding layer (also called the position embedding layer). Above these embedding network layers, that is, in the middle part of the neural network, multiple transformer block layers are cascaded. Each transformer block layer is composed of multiple network layers (such as the attention network layer and the feedforward network layer). The reason why the model weight parameters are large is that the transformer block layer has a large number of parameters, while the embedding layer has a small number of parameters. Therefore, to improve compression efficiency, the embodiments of the present disclosure can only compress the transformer block layers with large parameters, while the model weights of the embedding layer are stored in the original data format.

[0052]

[0053] Therefore, the first type of network layer described above can be understood as a transformer block layer, and the second type of network layer can be understood as an embedding layer. In practical applications, the model weights in the transformer block layer (in the first data format) can be converted to model weights in the second data format, compressed, and stored in video memory. During this process, the model weights in the first data format of the embedding layer (i.e., the second type of network layer) do not require data format conversion or compression and can be directly stored in video memory.

[0053]

[0054] The above description only uses LLM as an example to illustrate the first and second types of network layers. In practice, the network layer whose data volume of the weight parameter is greater than the set threshold can be regarded as the second type of network layer, and the other network layers can be regarded as the first type of network layers.

[0054]

[0055] By dividing the network layers of the neural network into types and performing differentiated processing on the model weights of the first data format of the divided first and second types of network layers, that is, only performing data format conversion and compression processing on the model weights corresponding to the first type of network layers with a large number of network layers, while not performing data format conversion and compression processing on the model weights corresponding to the second type of network layers with a small number of network layers (but directly storing them), the amount of data to be compressed is reduced and the model compression efficiency is improved.

[0055]

[0056] In summary, the model weight compression method provided by the embodiments of the present disclosure converts the model weights of a neural network model in a first data format into model weights in a second data format. This shortens the data length of the first data format, reduces the amount of data required for subsequent model weight compression, and improves the efficiency of subsequent compression processing. Furthermore, by compressing the model weights in the second data format after the shortened data length, and storing the compressed model weights in video memory, the amount of data stored in the video memory is reduced, thereby reducing the GPU memory resource requirements during the neural network model inference process and conserving video memory resources.

[0056]

[0057] Due to the data characteristics of neural network model weights, their exponent bits generally contain a significant amount of redundancy. Therefore, in the disclosed embodiment, compression processing is performed on the model weights in the second data format, including lossless compression of the exponent bits in the model weights in the second data format. The data characteristics of the model weights are as follows: in the FP16 data format, the exponent portion is 5 bits long, corresponding to a large numerical range, but the exponent, or integer portion, of the model weights is distributed within a small range within this numerical range. This means that a large number of values ​​within this numerical range are not used in the model weights.

[0057]

[0058] In a specific implementation, for example, rANS encoding can be used to perform lossless compression on the exponent bits in the model weights in the second data format, but this is not limiting. For example, Huffman encoding, tANS encoding, and other encoding methods may also be used, not listed here, as long as they can achieve lossless compression of the exponent bits in the model weights in the second data format. Using rANS encoding, for example, exponent bits represented in binary format (e.g., occupying 5 bits) can be compressed into a data format with a smaller bit width, such as a decimal format.

[0058]

[0059] Furthermore, after compressing the model weights in the second data format, storing the compressed model weights in the video memory includes: determining multiple compressed model weights corresponding to the same network layer in the neural network model; determining multiple first storage spaces in the video memory for storing the compressed model weights of multiple network layers; and storing the compressed model weights of the multiple network layers in the multiple first storage spaces accordingly.

[0059]

[0060] In embodiments of the present disclosure, for multiple network layers in a neural network model requiring model weight compression, a first storage space corresponding to each network layer can be allocated in video memory on a per-layer basis, thereby obtaining multiple first storage spaces corresponding to the multiple network layers. For each network layer, there are multiple model weights contained in a corresponding matrix / tensor. Each model weight undergoes the format conversion and compression described above to obtain a compressed model weight. Multiple compressed model weights for the same network layer are stored in the first storage space allocated for that network layer, and the correspondence between the first storage space and the network layer is recorded to facilitate subsequent reading of the model weights based on this correspondence.

[0060]

[0061] In addition, as described above, in an optional embodiment, the compression processing of the model weight after format conversion may be performed only on the exponential part thereof. Therefore, a model weight may include a compressed exponent bit and a sign bit and a second fraction bit after format conversion.

[0061]

[0062] Based on this, the compressed model weights of multiple network layers are correspondingly stored in multiple first storage spaces, including: storing multiple compressed exponent bits in the multiple compressed model weights corresponding to the target network layer in the first sub-storage space in the target first storage space, and storing multiple sign bits and multiple second fraction bits in the multiple compressed model weights corresponding to the target network layer in the second sub-storage space in the target first storage space. The target network layer is any one of the multiple network layers.

[0062]

[0063] Specifically, the disclosed embodiment divides the first storage space corresponding to each network layer into a first sub-storage space and a second sub-storage space. The compressed exponent bits in the compressed model weights corresponding to the corresponding network layer are stored in the first sub-storage space, while the sign bits and second fraction bits in the compressed model weights corresponding to the network layer are stored in the second sub-storage space, thereby storing compressed data and uncompressed data separately. The compressed exponent bits, sign bits, and second fraction bits correspond to individual model weights: one model weight corresponds to one sign bit and one second fraction bit consisting of multiple bits, and one model weight corresponds to one exponent bit consisting of multiple bits.

[0064] It should be understood that in actual applications, the first storage space corresponding to a network layer can be a continuous storage space. The first storage space and the second storage space can be divided as follows: assuming that the network layer contains N model weights, based on the bit width occupied by the compressed exponent bit in a model weight, the first storage space is first divided into N continuous first sub-storage spaces for sequentially storing the compressed exponent bits of the N model weights. The remaining portion of the first storage space is then used as the second sub-storage space for sequentially storing the concatenation of the sign bit and the second fraction bit corresponding to each of the N model weights. Of course, the division method of the first sub-storage space and the second sub-storage space is not limited to this.

[0063]

[0065] Through the model weight compression process described in the above embodiment, the model weights of each network layer of the neural network model can be compressed and the compressed model weights of each network layer can be stored separately. By compressing the model weights, the memory space occupied by the model weights can be reduced.

[0064]

[0066] The following is a detailed description of the process of the neural network model performing specific reasoning tasks.

[0065]

[0067] FIG6 is a flowchart of a method for processing an inference task provided by an embodiment of the present disclosure. As shown in FIG6 , the method includes the following steps.

[0066]

[0068] 601. In response to an input of an inference task of a neural network model, determine a second storage space shared by multiple network layers in a video memory.

[0067]

[0069] 602. Obtain compressed model weights of multiple network layers from multiple first storage spaces layer by layer.

[0068]

[0070] 603. Decompress the multiple compressed model weights of the target network layer currently obtained.

[0069]

[0071] 604. Store the decompressed multiple model weights of the target network layer into the second storage space to obtain multiple model weights of the target network layer in a second data format.

[0070]

[0072] 605. Process the inference task according to the multiple model weights in the second data format of the target network layer.

[0071]

[0073] In this embodiment, it should be noted that the aforementioned model weight compression process can be performed immediately after the neural network model is trained. Then, when the neural network model is used to perform a certain inference task, the model weights of each network layer can be read from the video memory on demand to perform relevant calculations. Alternatively, the aforementioned model weight compression process can be performed only when the neural network model is used to perform a certain inference task, so as to store the model weights in the video memory for use by the GPU when performing the inference task.

[0072]

[0074] It is understood that after neural network model training is completed, the model weights are written to disk. During the model weight compression process, the CPU can load the full model weights into memory, perform the model weight compression process, and store the compressed model weights in video memory for use by the GPU during inference tasks. Therefore, the steps shown in Figure 6 below can be executed by the GPU.

[0073]

[0075] To facilitate understanding of the reasoning task, an example is given below.

[0074]

[0076] If the neural network model is a question-answering model, then the reasoning task is the question posed by the user. If the neural network model is a classification model, then the reasoning task is the input object to be classified, such as a document or image.

[0075]

[0077] In a specific implementation, in response to the input of an inference task, a second storage space shared by multiple network layers in a neural network model is determined in the video memory. The size of the second storage space is the same as the data size of the uncompressed model weights of one network layer. For example, if a network layer contains N model weights and the total data size before compression is 30 GB (which will be less than this value after the above compression processing), the storage capacity of the second storage space can match the 30 GB size.

[0076]

[0078] After determining the second storage space, the neural network model is inferred layer by layer: compressed model weights of multiple network layers can be retrieved layer by layer from the multiple first storage spaces, the currently retrieved compressed model weights of the target network layer can be decompressed, and the decompressed model weights of the target network layer can be stored in the second storage space to obtain multiple model weights of the target network layer in the second data format. "Layer by layer" means that during the inference process, the model weights of each network layer are sequentially read one at a time. For ease of description, it is assumed that the model weights of all network layers of the neural network model are compressed. Initially, the compressed model weights of network layer 1 are read, decompressed, and stored in the second storage space. The decompressed model weights of network layer 1 are read from the second storage space for relevant calculations. Subsequently, the model weights of network layer 1 are deleted from the second storage space. The compressed model weights of network layer 2 are read, decompressed, and stored in the second storage space. The decompressed model weights of network layer 2 are read from the second storage space for relevant calculations, and so on. It can be understood that, for the target network layer, what is obtained after decompression are multiple model weights in the second data format of the target network layer.

[0077]

[0079] Furthermore, the reasoning task is processed according to the multiple model weights in the second data format of the target network layer, including: converting the multiple model weights in the second data format of the target network layer into multiple model weights in the first data format of the target network layer, and processing the reasoning task according to the multiple model weights in the first data format of the target network layer.

[0078]

[0080] Continuing with the above example, assume the second data format is 13-bit data, while a typical GPU can only process standard FP16 16-bit data. In this case, the multiple model weights in the second data format need to be converted back to the first data format. Specifically, a "zero padding" operation can be performed after the 13-bit data. Specifically, the three-digit "000" data is added to the end of the second fraction digit of the 13-bit data. This ensures that subsequent inference tasks can proceed normally. It should be noted that performing the "zero padding" operation after the 13-bit data does not significantly affect the overall accuracy of the neural network model and can be ignored.

[0081] FIG7 is a diagram illustrating an application example of a model weight compression method provided by an embodiment of the present disclosure. As shown in FIG7 , assuming that the neural network model is a large language model, the method specifically includes:

[0079]

[0082] Determine whether the data volume of the trained large language model's model weights stored on the hard disk can be directly stored in the video memory. If not, that is, the video memory resources are insufficient to store the model weights, then read the entire model weights of the large language model into the memory.

[0080]

[0083] The model weights in the first data format of the large language model are converted into model weights in the second data format in memory, and the model weights in the second data format are compressed. The compressed model weights are stored in the video memory in units of network layers.

[0081]

[0084] Next, based on the input of the inference task, a request is made for shared graphics memory space for multiple network layers to obtain shared storage space for the multiple network layers in the graphics memory. Next, the compressed model weights of the multiple network layers are obtained layer by layer. The currently obtained compressed model weights of the target network layer are decompressed and stored in the shared storage space to obtain the model weights of the target network layer in the second data format. The inference task is then processed based on the model weights of the target network layer in the second data format.

[0082]

[0085] The model weight compression method provided by the embodiments of the present disclosure converts the model weights of a neural network model in a first data format into model weights in a second data format. This shortens the data length of the first data format, reduces the amount of data required for subsequent model weight compression, and improves the efficiency of subsequent compression processing. The model weights in the second data format after the shortened data length are then compressed and stored in video memory, thereby reducing the amount of data stored in the video memory. This, in turn, reduces the GPU memory resource requirements during the neural network model inference process, conserving video memory resources. By determining a second storage space shared by multiple network layers and sequentially obtaining the compressed model weights of multiple network layers from multiple first storage spaces, the method reduces video memory usage while ensuring the smooth execution of subsequent inference tasks.

[0083]

[0086] The following describes in detail the model weight compression device of one or more embodiments of the present disclosure. Those skilled in the art will understand that these devices can be configured using commercially available hardware components through the steps taught in this solution.

[0084]

[0087] FIG8 is a schematic structural diagram of a model weight compression device provided by an embodiment of the present disclosure. As shown in FIG8 , the device includes: an acquisition module 11, a conversion module 12, a compression module 13, and a storage module 14.

[0085]

[0088] The acquisition module 11 is used to obtain the model weight of the neural network model in a first data format.

[0086]

[0089] The conversion module 12 is configured to convert the model weights in the first data format into the model weights in the second data format, where the data length of the second data format is smaller than the data length of the first data format.

[0087]

[0090] The compression module 13 is configured to compress the model weights in the second data format.

[0091] The storage module 14 is used to store the compressed model weights in the video memory.

[0088]

[0092] Optionally, the model weight in the first data format includes a sign bit, an exponent bit, and a first fraction bit, and the model weight in the second data format includes the sign bit, the exponent bit, and a second fraction bit, wherein the second fraction bit is truncated from the first fraction bit.

[0089]

[0093] Optionally, the compression module 13 is specifically configured to perform lossless compression on the exponent bits in the model weights in the second data format.

[0090]

[0094] Among them, optionally, the storage module 14 is specifically used to: determine multiple compressed model weights corresponding to the same network layer in the neural network model; determine multiple first storage spaces in the video memory for storing the compressed model weights of multiple network layers; and store the compressed model weights of the multiple network layers correspondingly in the multiple first storage spaces.

[0091]

[0095] Optionally, the storage module 14 is specifically used to: store multiple compressed exponent bits in multiple compressed model weights corresponding to the target network layer into a first sub-storage space in the target first storage space, and store multiple sign bits and multiple second fraction bits in multiple compressed model weights corresponding to the target network layer into a second sub-storage space in the target first storage space, where the target network layer is any one of the multiple network layers.

[0092]

[0096] Among them, optionally, the device also includes: a processing module, which is used to determine a second storage space shared by the multiple network layers in the video memory in response to the input of the inference task of the neural network model; obtain the compressed model weights of the multiple network layers from the multiple first storage spaces layer by layer; decompress the multiple compressed model weights of the target network layer currently obtained; store the decompressed multiple model weights of the target network layer in the second storage space to obtain the multiple model weights of the target network layer in the second data format; and process the inference task according to the multiple model weights of the target network layer in the second data format.

[0093]

[0097] Optionally, the processing module is specifically used to: convert multiple model weights in the second data format of the target network layer into multiple model weights in the first data format of the target network layer; and process the inference task according to the multiple model weights in the first data format of the target network layer.

[0094]

[0098] Optionally, the conversion module 12 is specifically used to: determine the model weights of the first type of network layer and the model weights of the second type of network layer in the first data format of the neural network model, wherein the number of the first type of network layers in the neural network model is greater than the number of the second type of network layers; convert the model weights of the first type of network layer in the first data format into model weights of the second data format; storing the compressed model weights in the video memory includes: storing the compressed model weights corresponding to the first type of network layer in the video memory, and storing the model weights of the second type of network layer in the first data format in the video memory.

[0099] The device shown in FIG8 can execute the steps provided in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments and will not be repeated here.

[0095]

[0100] In one possible design, the structure of the model weight compression device shown in FIG8 can be implemented as an electronic device. As shown in FIG9 , the electronic device may include a processor 21, a memory 22, and a communication interface 23. Memory 22 stores executable code. When executed by processor 21, the executable code enables processor 21 to at least implement the model weight compression method provided in the aforementioned embodiments.

[0096]

[0101] In addition, an embodiment of the present disclosure provides a non-transitory machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the model weight compression method provided in the aforementioned embodiment.

[0097]

[0102] The apparatus embodiment described above is merely illustrative, and the network elements described as separate components may or may not be physically separate. Some or all of the modules may be selected based on actual needs to achieve the objectives of the solution in this embodiment. Persons of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0098]

[0103] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using a general-purpose hardware platform, or alternatively, through a combination of hardware and software. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant art, can be embodied in the form of a computer product. The present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099]

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

Claims 1. A model weight compression method, comprising: Obtaining model weights of a neural network model in a first data format; Converting the model weights in the first data format into model weights in a second data format, wherein the data length of the second data format is smaller than the data length of the first data format; compressing the model weights in the second data format; Store the compressed model weights in video memory.

2. The method according to claim 1, wherein: The model weight in the first data format includes a sign bit, an exponent bit, and a first fraction bit, and the model weight in the second data format includes the sign bit, the exponent bit, and a second fraction bit, wherein the second fraction bit is truncated from the first fraction bit.

3. The method according to claim 2, wherein: The compressing the model weights in the second data format includes: performing lossless compression on exponent bits in the model weights in the second data format.

4. The method according to claim 3, wherein: The storing of the compressed model weights in the video memory includes: determining a plurality of compressed model weights corresponding to the same network layer in the neural network model; determining a plurality of first storage spaces in the video memory for storing the compressed model weights of the plurality of network layers; and storing the compressed model weights of the plurality of network layers in the plurality of first storage spaces accordingly.

5. The method according to claim 4, wherein: The storing the compressed model weights of the multiple network layers correspondingly to the multiple first storage spaces includes: storing multiple compressed exponent bits in the multiple compressed model weights corresponding to the target network layer in the first sub-storage space in the target first storage space, and storing multiple sign bits and multiple second fraction bits in the multiple compressed model weights corresponding to the target network layer in the second sub-storage space in the target first storage space, wherein the target network layer is any one of the multiple network layers.

6. The method according to claim 4, further comprising: In response to an input of an inference task of the neural network model, determining a second storage space shared by the multiple network layers in the video memory; Acquire the compressed model weights of the plurality of network layers from the plurality of first storage spaces layer by layer; decompress the plurality of compressed model weights of the target network layer currently acquired; Storing the decompressed multiple model weights of the target network layer in the second storage space to obtain the multiple model weights of the target network layer in the second data format; Processing the inference task according to the multiple model weights of the second data format of the target network layer reason.

7. The method according to claim 6, wherein: The processing of the inference task according to the multiple model weights in the second data format of the target network layer includes: converting the multiple model weights in the second data format of the target network layer into multiple model weights in the first data format of the target network layer; processing the inference task according to the multiple model weights in the first data format of the target network layer.

8. The method according to any one of claims 1 to 7, wherein: The converting of the model weights in the first data format into the model weights in the second data format includes: determining the model weights in the first data format of the first type of network layer and the model weights in the first data format of the second type of network layer of the neural network model, wherein the number of the first type of network layers in the neural network model is greater than the number of the second type of network layers; converting the model weights in the first data format of the first type of network layer into the model weights in the second data format; storing the compressed model weights in the video memory includes: storing the compressed model weights corresponding to the first type of network layer in the video memory, and storing the model weights in the first data format of the second type of network layer in the video memory.

9. A model weight compression method, comprising: Receiving a request triggered by a client device by calling a neural network model service provided by the cloud, wherein the request includes a model inference task; Obtaining a model weight of a neural network model in a first data format; converting the model weight in the first data format into a model weight in a second data format, wherein a data length of the second data format is less than a data length of the first data format; compressing the model weights in the second data format; Store the compressed model weights in video memory; Executing the inference task according to the compressed model weights stored in the video memory; Feedback the execution result of the inference task to the client device.

10. An electronic device, comprising: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the model weight compression method according to any one of claims 1 to 8 or claim 9.

11. A non-transitory machine-readable storage medium, wherein: The non-transitory machine-readable storage medium stores 16 stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes the model weight compression method according to any one of claims 1 to 8 or claim 9.

Images