Multimodal large language model quantization method, device, equipment, storage medium and product

Abstract

The present application relates to the field of model deployment technology, and discloses a multimodal large language model quantization method, device, equipment, storage medium and product, including: dividing the large model to be quantized into multimodal processing blocks and language model blocks; constructing a processing submodule sequence according to the quantization error corresponding to each processing submodule in the multimodal processing block; quantizing each processing submodule in the multimodal processing block based on the processing submodule sequence; determining the language submodule sequence according to the quantized multimodal processing block; and quantizing each language submodule in the language model block according to the language submodule sequence. Since targeted module division can be performed according to the computational characteristics of the multimodal language large model, and quantitative adjustment is performed based on the module division order, it is ensured that the quantization conforms to the actual computational characteristics of the model, and the effect of quantitative deployment of the multimodal language large model is improved.

Description

Technical Field

[0001] The present application relates to the field of model deployment technology, and in particular to a multimodal large language model quantization method, apparatus, device, storage medium and product. Background Art

[0002] At present, when large models are quantitatively deployed, they generally need to be split and deployed due to their large size. In order to simplify the large model as much as possible, the model also needs to be quantized and the parameters reduced before corresponding deployment. This can reduce the computational cost, data bandwidth and storage space of the neural network model, allowing the neural network model to be applied faster and better on hardware with limited memory and computing power.

[0003] The current large language models all use multi-round autoregressive architectures, and dedicated accelerators are usually used for deployment and inference. These dedicated accelerators are often unable to handle conventional feature extraction tasks for modalities such as images or videos, which also makes the quantitative deployment of large multimodal language models more difficult. Summary of the invention

[0004] The main purpose of this application is to provide an invention name, which aims to solve the technical problems that large multimodal language models in related technologies are difficult to reasonably quantify and deploy, and the quantified deployment effect is poor.

[0005] To achieve the above objectives, the present application proposes a multimodal large language model quantization method, the method comprising:

[0006] Dividing the large model to be quantified into a multimodal processing block and a language model block, wherein the large model to be quantified is a language large model supporting multimodal input;

[0007] Constructing a processing submodule sequence according to the quantization errors corresponding to each processing submodule in the multimodal processing block, wherein the processing submodule includes at least one network layer in the multimodal processing block;

[0008] quantizing each processing submodule in the multimodal processing block based on the processing submodule sequence;

[0009] Determining a language submodule sequence according to the quantized multimodal processing blocks;

[0010] Each language submodule in the language model block is quantized according to the language submodule sequence, and the language submodule includes at least one layer of network in the language model block.

[0011] In a possible implementation of the present application, quantizing each processing submodule in the multimodal processing block based on the processing submodule sequence includes:

[0012] Traversing the processing submodule sequence according to the sequence order of the processing submodule sequence to obtain a current processing submodule;

[0013] quantizing the current processing submodule in the multimodal processing block;

[0014] After the current processing submodule is quantized, the parameters of the unquantized processing submodules in the multimodal processing block are adjusted.

[0015] In a possible implementation of the present application, before traversing the processing submodule sequence according to the sequence order of the processing submodule sequence to obtain the current processing submodule, the method further includes:

[0016] Obtaining the pre-quantization output of the multimodal processing block for a preset sample;

[0017] After the current processing submodule is quantized, adjusting the parameters of the unquantized processing submodules in the multimodal processing block includes:

[0018] After the current processing submodule is quantized, obtaining the quantized output of the multimodal processing block for a preset sample;

[0019] The parameters of the unquantized processing submodules in the multimodal processing block are adjusted based on the pre-quantization output and the post-quantization output.

[0020] In a possible implementation of the present application, adjusting the parameters of the unquantized processing submodule in the multimodal processing block based on the pre-quantization output and the post-quantization output includes:

[0021] Constructing a pre- and post-output error according to the pre-quantization output and the post-quantization output;

[0022] With the goal of minimizing the output before-after error, the parameters of the unquantized processing submodules in the multimodal processing block are adjusted.

[0023] In a possible implementation of the present application, quantizing the current processing submodule in the multimodal processing block includes:

[0024] Acquire characteristic distribution data of the current processing submodule;

[0025] The current processing submodule in the multimodal processing block is quantized according to the minimum value and / or the maximum value in the feature distribution data.

[0026] In a possible implementation of the present application, determining a language submodule sequence according to the quantized multimodal processing blocks includes:

[0027] Processing the preset samples by the quantized multimodal processing blocks to generate sample output feature values;

[0028] Determine the quantization error corresponding to each language submodule in the language model block based on the sample output feature value, wherein the quantization error corresponding to the language submodule is the output difference of the language model block for the sample output feature value before and after the language submodule in the language model block is quantized;

[0029] Based on the quantization errors corresponding to the language submodules, the language submodules in the language model block are sorted from large to small to generate a language submodule sequence.

[0030] In a possible implementation of the present application, quantizing each language submodule in the language model block according to the language submodule sequence includes:

[0031] Traversing the language submodule sequence according to the sequence order of the language submodule sequence to obtain a current language submodule;

[0032] quantizing the current language submodule in the language model block;

[0033] After the current language submodule is quantized, the parameters of the unquantized language submodules in the language model block are adjusted.

[0034] In a possible implementation of the present application, before traversing the language submodule sequence according to the sequence order of the language submodule sequence to obtain the current language submodule, the method further includes:

[0035] Obtaining the pre-quantization output of the language model block for the sample output feature value;

[0036] After the current language submodule is quantized, adjusting the parameters of the unquantized language submodule in the language model block includes:

[0037] After the current language submodule is quantized, obtaining the quantized output of the language model block for the sample output feature value;

[0038] Based on the pre-quantization output and the post-quantization output, parameters of the unquantized language submodule in the language model block are adjusted.

[0039] In a possible implementation of the present application, the step of constructing a processing submodule sequence according to the quantization error corresponding to each processing submodule in the multimodal processing block includes:

[0040] Obtaining a quantization error corresponding to each processing submodule in the multimodal processing block, where the quantization error corresponding to the processing submodule is the output difference of the multimodal processing block for the same sample before and after the processing submodule in the multimodal processing block is quantized;

[0041] Based on the quantization error, the processing submodules in the multimodal processing block are sorted from large to small to generate a processing submodule sequence.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a multimodal large language model quantization device, the device comprising:

[0043] A division module, used for dividing the large model to be quantized into a multimodal processing block and a language model block, wherein the large model to be quantized is a language large model supporting multimodal input;

[0044] A construction module, used to construct a processing submodule sequence according to the quantization error corresponding to each processing submodule in the multimodal processing block, wherein the processing submodule includes at least one network layer in the multimodal processing block;

[0045] A first quantization module, configured to quantize each processing submodule in the multimodal processing block based on the processing submodule sequence;

[0046] A determination module, configured to determine a language submodule sequence according to the quantized multimodal processing blocks;

[0047] The second quantization module is used to quantize each language submodule in the language model block according to the language submodule sequence, and the language submodule includes at least one layer of network in the language model block.

[0048] In addition, to achieve the above-mentioned objectives, the present application also provides a multimodal large language model quantization device, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the multimodal large language model quantization method as described above.

[0049] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the multimodal large language model quantization method as described above are implemented.

[0050] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the multimodal large language model quantization method as described above.

[0051] One or more technical solutions proposed in this application have at least the following technical effects:

[0052] Since targeted module division can be carried out according to the computing characteristics of the large multimodal language model, and quantitative adjustment can be performed based on the module division order, it can ensure that the quantization conforms to the actual computing characteristics of the model, thereby improving the effect of quantitative deployment of the large multimodal language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0055] Figure 1 A flowchart diagram of the first embodiment of the multimodal large language model quantization method of the present application;

[0056] Figure 2 This is a schematic diagram of the structure of a multimodal language large model according to an embodiment of the present application;

[0057] Figure 3 A schematic diagram of model block division according to an embodiment of the present application;

[0058] Figure 4 A flowchart diagram of the second embodiment of the multimodal large language model quantization method of the present application is provided;

[0059] Figure 5 A schematic diagram of related feature extraction of multimodal processing block quantization according to an embodiment of the present application;

[0060] Figure 6 A flowchart diagram of the third embodiment of the multimodal large language model quantization method of the present application;

[0061] Figure 7 This is a schematic diagram of the module structure of a multimodal large language model quantization device according to an embodiment of the present application;

[0062] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the multimodal large language model quantization method in the embodiment of the present application.

[0063] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0064] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0065] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0066] Based on this, the embodiment of the present application provides a multimodal large language model quantization method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the multimodal large language model quantization method of the present application.

[0067] In this embodiment, the multimodal large language model quantization method includes steps S10 to S50:

[0068] Step S10: Divide the large model to be quantized into multimodal processing blocks and language model blocks.

[0069] It should be noted that the executor of this embodiment may be the multimodal large language model quantization device, and the multimodal large language model quantization device may be an electronic device such as a personal computer or a server, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the multimodal large language model quantization method of this application is described by taking the multimodal large language model quantization device as an example.

[0070] It should be noted that neural network is a deep artificial neural network in the field of machine learning and deep learning, which can process large images. Convolutional neural networks usually include convolutional layers, pooling layers, fully connected layers, etc. Neural networks have made great progress in image classification, object detection, semantic segmentation and other fields.

[0071] The language model is a large artificial intelligence model based on deep learning technology, focusing on processing and understanding natural language. Through learning and training on massive text data, it can understand the semantics, grammar and pragmatics of natural language, thus possessing multiple capabilities such as language generation, comprehension, question answering, and translation. Language models usually have a large number of parameters and a complex neural network structure to achieve a high degree of simulation and precise processing of language.

[0072] The large model to be quantified in this embodiment can be a language large model that supports multimodal input (such as supporting the input of modules such as text, pictures, and videos at the same time), which can also be called a multimodal language large model. It is a model that further integrates multiple information expression methods on the basis of the language large model. In addition to language text, it also combines data from other modalities such as images, audio, and video. By integrating information from multiple modalities, the multimodal language large model can understand and process complex real-world scenarios and tasks more comprehensively and deeply, and can achieve cross-modal interaction and understanding, such as generating relevant text descriptions based on images, or generating corresponding images based on text, etc., providing richer and more diverse application experiences and functions.

[0073] In actual use, targeted module division can be carried out according to the computational characteristics of the large model to be quantized, and it can be divided into multimodal processing blocks and language model blocks.

[0074] For ease of understanding, now combined Figure 2 and 3 This is for illustration only, but not for limitation. Figure 2 Schematic diagram of the structure of the multimodal language model of this embodiment. Figure 3 This is a schematic diagram of the model block division of this embodiment.

[0075] like Figure 2 As shown in the figure, the large model to be quantized, that is, the multimodal language large model, usually includes three parts, namely the vision encoder, the projection and the language model.

[0076] Among them, the vision encoder encodes the input image features to obtain visual features, and then transforms the visual features into visual vectors (embedding) through projection, and then fuses them with the input language vectors (embedding), and then inputs the fused embedding into the language model for decoding, and finally obtains the language response output. In this type of architecture, for a single input, the vision encoder and projector only need to be calculated once, that is, only visual features need to be extracted once, while the language model needs to be called and calculated multiple times in a loop according to the pre-set output length. It is an autoregressive network. Due to the differences in the computing characteristics of the vision encoder and the language model, the traditional quantization deployment scheme for the vision encoder or the language model alone is no longer applicable, and a new quantization deployment scheme needs to be designed.

[0077] Based on this, Figure 3 As shown, it is possible to quantify the large model (such as Figure 3 The computational characteristics of the Model shown in the figure divide the large model to be quantized into a multimodal processing block (Model A) and a language model block (Model B).

[0078] Step S20: constructing a processing submodule sequence according to the quantization error corresponding to each processing submodule in the multimodal processing block.

[0079] It should be noted that in order to facilitate quantitative processing, the multimodal processing block can be further divided into multiple processing sub-modules (Blocks). The processing sub-module may include at least one network layer (Layer) in the multimodal processing block. The number of network layers of different processing sub-module networks may be the same or different.

[0080] Among them, the specific division method of the processing sub-modules can be set in advance by the manager of the multimodal large language model quantization device.

[0081] In actual use, in order to ensure the quantization effect, a processing sub-module sequence can be constructed according to the quantization error corresponding to each processing sub-module in the multimodal processing block, wherein the processing sub-module sequence is used to characterize the order of quantizing each processing sub-module when quantizing the multimodal processing block.

[0082] In a possible implementation of this embodiment, in order to ensure that the processing submodule with greater quantization influence is quantized first, step S20 in this embodiment may include:

[0083] Obtaining quantization errors corresponding to each processing submodule in the multimodal processing block;

[0084] Based on the quantization error, the processing submodules in the multimodal processing block are sorted from large to small to generate a processing submodule sequence.

[0085] It should be noted that the quantization error corresponding to the processing submodule is the difference in the output of the multimodal processing block for the same sample before and after the processing submodule in the multimodal processing block is quantized. The larger the quantization error, the greater the impact on the output of the multimodal processing block when the processing submodule is quantized. At this time, in order to ensure that the part with a large impact can be dealt with (such as parameter adjustment, etc.), the processing submodule with a greater quantization impact can be quantized first. Therefore, the processing submodules in the multimodal processing block can be sorted in order from large to small according to the corresponding quantization error to generate a processing submodule sequence.

[0086] For example: divide the multimodal processing block Model A into multiple processing sub-modules Block (Block 1 to Blockn), prepare a number of image samples, input these image samples to Model A, and save the output feature value Feat of the last Block in Model A (the last Block according to the network connection order);

[0087] Quantize each processing submodule in the multimodal processing block. When quantizing a block, keep the floating point of other blocks unchanged. Then input the image sample to Model A, extract the output Feat' of Model A, and then calculate the difference between Feat' and the previously extracted Feat (such as mean square error), which is recorded as Error;

[0088] The Error corresponding to each Block is recorded as Error_1, Error_2, ..., Error_n, and then the Blocks are sorted from large to small based on the corresponding Error to obtain the processing sub-module sequence. The indexes of each Error in the processing sub-module sequence are idx_1, idx_2, ..., idx_n.

[0089] In actual use, if the influence of each processing sub-module on the output of the entire multimodal processing block is not considered, the processing sub-modules in the multimodal processing block are directly sorted, such as directly sorting the processing sub-modules in ascending or descending order according to the network connection order in the multimodal processing block to generate a processing sub-module sequence.

[0090] Step S30: quantizing each processing submodule in the multimodal processing block based on the processing submodule sequence.

[0091] It should be noted that after determining the processing sub-module sequence, each processing sub-module in the multimodal processing block may be quantified according to the order of each processing sub-module recorded in the processing sub-module sequence.

[0092] Step S40: determining a language submodule sequence according to the quantized multimodal processing blocks.

[0093] It should be noted that, in order to facilitate quantization processing, the language model block can be further divided into multiple language sub-modules (Blocks), wherein the language sub-module can include at least one network layer (Layer) in the language model block.

[0094] In actual use, the quantized multimodal processing block can be used to generate features to be processed by the language model block (such as language vectors, text vectors, or vectors that are a fusion of language vectors and text vectors). Then, the quantization loss of each language sub-module in the language model block is determined by the feature, and a language sub-module sequence is constructed based on the quantization loss.

[0095] The language submodule sequence is used to represent the order in which the language submodules are quantized when the language model blocks are quantized.

[0096] In a possible implementation of this embodiment, in order to reasonably construct a language submodule sequence, step S40 in this embodiment may include:

[0097] Processing the preset samples by the quantized multimodal processing blocks to generate sample output feature values;

[0098] Determine the quantization error corresponding to each language submodule in the language model block based on the sample output feature value;

[0099] Based on the quantization errors corresponding to the language submodules, the language submodules in the language model block are sorted from large to small to generate a language submodule sequence.

[0100] It should be noted that the quantization error corresponding to the language submodule may be the output difference of the feature value output by the language model block for the sample before and after the language submodule in the language model block is quantized.

[0101] For example: divide the language model block Model B into several language sub-modules Block, after the multimodal processing block Model A is quantized, several image samples are input to the quantized Model A, and the output feature value image feat of Model A is saved, and then the image feat and the corresponding text language are input to the language model block Model B, and the output feature value language feat of the last Block (the last Block determined according to the network connection order) is saved during the first decoding calculation (that is, autoregressive decoding is not performed);

[0102] Each language submodule in the language model block is quantized separately. When quantizing a certain language submodule, the floating point of the other language submodules remains unchanged. Then the image feat and the corresponding text language are input to Model B, the output language feat' of Model B is extracted, and then the difference between language feat' and the previously extracted language feat (such as mean square error) is calculated. In this way, the quantization error corresponding to each language submodule in the language model block can be obtained.

[0103] Similarly, it is necessary to ensure that the language sub-modules that have a greater impact on the output of the language model block are quantized earlier. Therefore, the language sub-modules in the language model block can be sorted from large to small based on the quantization errors corresponding to each language sub-module to generate a language sub-module sequence.

[0104] In actual use, if the influence of each language sub-module on the output of the entire language model block is not considered, the language sub-modules in the language model block are directly sorted, such as directly sorting the language sub-modules in ascending or descending order according to the network connection order in the language model block to generate a language sub-module sequence.

[0105] Among them, in actual application, if the quantized multimodal processing block is used to process the preset sample to generate the sample output feature value, the quantization of the language model block needs to wait for the multimodal processing block to be quantized before it can start. If it is necessary to reduce the overall time consumption of quantizing the large model to be quantized, the unquantized multimodal processing block can also be used to process the preset sample to generate the sample output feature value, and then construct the language sub-module sequence according to the sample output feature value. In this way, it can be ensured that the quantization of the language model block does not need to wait for the multimodal processing block to be quantized. The quantization of the two can be carried out simultaneously, thereby reducing the overall time consumption of quantizing the large model to be quantized.

[0106] Step S50: quantizing each language submodule in the language model block according to the language submodule sequence.

[0107] It should be noted that after determining the language submodule sequence, each language submodule in the language model block may be quantized separately according to the order of each language submodule recorded in the language submodule sequence.

[0108] This embodiment provides a quantization method for a large multimodal language model. Since targeted module division can be performed according to the computing characteristics of the large multimodal language model, and quantization adjustment can be performed based on the module division order, it ensures that the quantization conforms to the actual computing characteristics of the model, thereby improving the effect of quantization deployment of the large multimodal language model.

[0109] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 4 , step S30 includes steps S301 to S303:

[0110] Step S301: traverse the processing submodule sequence according to the sequence order of the processing submodule sequence to obtain the current processing submodule.

[0111] It should be noted that the processing submodule sequence may be traversed according to the sequence order of the processing submodule sequence, and the traversed processing submodule may be used as the current processing submodule.

[0112] The sequence order of the processing submodule sequence is the sorting order of each processing submodule in the processing submodule sequence.

[0113] Step S302: quantizing the current processing submodule in the multimodal processing block.

[0114] It should be noted that, based on the traversal, each processing submodule in the multimodal processing block can be quantified one by one, and therefore, the current processing submodule in the multimodal processing block can be quantified.

[0115] In actual use, the algorithm used to quantize the current processing submodule can be selected according to the hardware support of the device used when deploying the large model to be quantized. In actual quantization, in addition to the maximum / minimum value quantization method, the method of minimizing the quantization error of the current processing submodule can also be used to quantize the current processing submodule.

[0116] Among them, in order to ensure that it can adapt to various different hardware and actual quantization requirements, a variety of quantization algorithms can be pre-set in the multimodal large language model quantization device, such as linear symmetric quantization method, linear asymmetric quantization method, Power-of-Two quantization method and other algorithms.

[0117] Step S303: After the current processing submodule is quantized, the parameters of the unquantized processing submodules in the multimodal processing block are adjusted.

[0118] It should be noted that the quantization of the current processing sub-module will actually affect the overall output of the multimodal processing block. When the current processing sub-module is quantized, the parameters of the unquantized processing sub-modules in the multimodal processing block can be adjusted. Through this parameter adjustment, the impact of the quantization of the current processing sub-module on the overall output can be reduced, thereby minimizing the impact of quantization.

[0119] In actual use, after adjusting the parameters of the unquantized processing submodules in the multimodal processing block, it is also possible to detect whether there are untraversed elements in the processing submodule sequence. If so, it is possible to return to step S301 to continue traversing;

[0120] If not, it can be determined that the multimodal processing block quantization is completed, and subsequent steps can be continued, such as executing step S40.

[0121] In a possible implementation of this embodiment, in order to ensure reasonable adjustment parameters, before step S301 in this embodiment, the following may also be included:

[0122] Obtaining the pre-quantization output of the multimodal processing block for a preset sample;

[0123] The step S303 may include:

[0124] After the current processing submodule is quantized, obtaining the quantized output of the multimodal processing block for a preset sample;

[0125] The parameters of the unquantized processing submodules in the multimodal processing block are adjusted based on the pre-quantization output and the post-quantization output.

[0126] It should be noted that, in order to reasonably adjust the parameters of the unquantized processing submodule, before the traversal begins, the output of the multimodal processing block for the preset sample can be recorded and used as the output before quantization;

[0127] At the same time, after the current processing submodule is quantized, the output of the multimodal processing block for the preset sample can be recorded as the quantized output.

[0128] Afterwards, based on the difference between the output before quantization and the output after quantization, the parameters of the unquantized processing submodules in the multimodal processing block are adjusted.

[0129] In a specific implementation, in order to facilitate judgment during adjustment, the step of adjusting the parameters of the unquantized processing submodule in the multimodal processing block based on the pre-quantization output and the post-quantization output described in this embodiment may include:

[0130] Constructing a pre- and post-output error according to the pre-quantization output and the post-quantization output;

[0131] With the goal of minimizing the output before-after error, the parameters of the unquantized processing submodules in the multimodal processing block are adjusted.

[0132] It should be noted that in order to ensure that there is a reasonable target during adjustment, the difference between the output before quantization and the output after quantization can be calculated, and the difference can be used as the error before and after the output. Then, with minimizing the error before and after the output, the parameters in the unquantized processing sub-modules in the multimodal processing block are fine-tuned to minimize the impact of the quantization of the processing sub-module on the output of the multimodal processing block.

[0133] The difference between the output before quantization and the output after quantization can be calculated by the mean square error. Of course, other algorithms with similar functions can also be used for calculation, such as root mean square error, mean absolute error, mean square logarithmic error and other algorithms.

[0134] For ease of understanding, the following example is given:

[0135] Assume that the multimodal processing block Model A is divided into n processing sub-modules Block, prepare a number of image samples, input these image samples to Model A, and save the output Feat of the last Block in Model A (the last Block according to the network connection order);

[0136] The index order in the processing submodule sequence can be represented as idx_1, idx_2, ..., idx_n. Then, when quantizing at this time, the order of each processing submodule can be represented as Block_idx_1, Block_idx_2, ..., Block_idx_n;

[0137] The first processing submodule to be traversed is Block_idx_1. At this time, after Block_idx_1 is quantized, the processing submodules that have not been quantized are Block_idx_2, ..., Block_idx_n. At this time, the parameters (such as weight parameters) of each layer in Block_idx_2, ..., Block_idx_n can be fine-tuned. The output of the multimodal processing block for the above image sample after parameter adjustment can be represented as Feat'. At this time, the mean square error between Feat' and Feat can be minimized. At this time, the quantization of Block_idx_1 is completed, and then the traversal continues. By analogy, the quantization of all processing submodules in the multimodal processing block can be completed.

[0138] The above examples are only for facilitating understanding of the scheme and do not limit the present scheme.

[0139] In a specific implementation, in order to ensure that quantization can be performed correctly, step S302 in this embodiment may include:

[0140] Acquire characteristic distribution data of the current processing submodule;

[0141] The current processing submodule in the multimodal processing block is quantized according to the minimum value and / or the maximum value in the feature distribution data.

[0142] It should be noted that the feature distribution data of the processing submodule can be determined together with the output of the multimodal processing block before quantization. At this time, the feature distribution data of the current processing submodule can be directly extracted from the previously determined content.

[0143] For ease of understanding, now combined Figure 5 This is for illustration only, but not for limitation. Figure 5 This is a schematic diagram of relevant feature extraction for block quantization of multimodal processing in this embodiment.

[0144] like Figure 5 As shown, the multimodal processing block Model A is divided into n processing sub-modules Block, and several image samples image are prepared. These image samples are input into Model A to extract the relevant features required for quantization. At this time, the feature distribution data C_i of the output of each layer of Model A can be recorded (C_i can at least include the maximum value C_il and the minimum value C_is, and other data can be included if necessary, i is the layer number of the network, i∈1, 2, ...., n), and the output Feat of the last Block in Model A (the last Block according to the network connection order) is saved.

[0145] In actual use, after determining the characteristic distribution data, the current processing submodule in the multimodal processing block can be quantized according to the minimum value and / or maximum value in the characteristic distribution data according to the requirements of the adopted quantization algorithm.

[0146] In actual use, if the quantization algorithm is a linear symmetric quantization method, the quantization can be characterized as:

[0147]

[0148] In the formula, if the quantization algorithm is an asymmetric quantization method, the quantization can be characterized as:

[0149]

[0150] If the quantization algorithm is the Power-of-Two quantization method, the quantization can be characterized as:

[0151]

[0152] In the above formula, X represents the original floating point data, C lrepresents the maximum value of the characteristic distribution data of the network layer included in the current processing submodule before the statistics, Cs represents the minimum value of the characteristic distribution data of the network layer included in the current processing submodule before the statistics, B w Indicates the bit width of quantization (fixed point), X Q Indicates the quantized (fixed-point) value.

[0153] This embodiment provides a multimodal large language model quantization method. Since it traverses the processing sub-module sequence, each processing sub-module in the multimodal processing block is quantized separately. During the quantization process, the parameters of the unquantized processing sub-module can also be adjusted to minimize the impact of quantization on the model output, thereby improving the reasoning effect of the model that is finally deployed.

[0154] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. Figure 6 , step S50 includes steps S501 to S503:

[0155] Step S501: traverse the language submodule sequence according to the sequence order of the language submodule sequence to obtain the current language submodule.

[0156] It should be noted that the language submodule sequence can be traversed according to the sequence order of the language submodule sequence, and the traversed language submodule is used as the current language submodule.

[0157] The sequence order of the language submodule sequence is the sorting order of each language submodule in the language submodule sequence.

[0158] Step S502: quantizing the current language submodule in the language model block.

[0159] It should be noted that, based on traversal, each language sub-module in the language model block can be quantized one by one, so the current language sub-module in the language model block can be quantized.

[0160] In actual use, the algorithm used to quantize the current language submodule can be selected according to the hardware support of the device used when deploying the large model to be quantized. In actual quantization, in addition to the maximum / minimum value quantization method, the method of minimizing the quantization error of the current language submodule can also be used to quantize the current language submodule.

[0161] Among them, in order to ensure that it can adapt to various different hardware and actual quantization requirements, a variety of quantization algorithms can be pre-set in the multimodal large language model quantization device, such as linear symmetric quantization method, linear asymmetric quantization method, Power-of-Two quantization method and other algorithms.

[0162] Specifically, the processing flow for quantizing the current language sub-module in the language model block is similar to the specific processing flow for quantizing the current processing sub-module in the multimodal processing block, such as selecting a quantization algorithm based on hardware support, obtaining feature distribution data in conjunction with the quantization algorithm, etc. The specific implementation can refer to the above description and will not be repeated here.

[0163] Step S503: After the current language submodule is quantized, the parameters of the unquantized language submodules in the language model block are adjusted.

[0164] It should be noted that the quantization of the current language sub-module will actually affect the overall output of the language model block. When the current language sub-module is quantized, the parameters of the unquantized language sub-module in the language model block can be adjusted. Through this parameter adjustment, the impact of the quantization of the current language sub-module on the overall output can be reduced, and the impact of quantization can be minimized.

[0165] In actual use, after adjusting the parameters of the unquantized language submodules in the language model block, it is also possible to detect whether there are untraversed elements in the language submodule sequence. If so, it is possible to return to step S301 to continue traversing;

[0166] If it does not exist, it can be determined that the language model block quantization has been completed, and it can be determined that the quantization of the large model to be quantized has been completed.

[0167] In a possible implementation of this embodiment, in order to ensure reasonable adjustment parameters, before step S501 in this embodiment, the following may also be included:

[0168] Obtaining the pre-quantization output of the language model block for the sample output feature value;

[0169] The step S503 may include:

[0170] After the current language submodule is quantized, obtaining the quantized output of the language model block for the sample output feature value;

[0171] Based on the pre-quantization output and the post-quantization output, parameters of the unquantized language submodule in the language model block are adjusted.

[0172] It should be noted that, in order to reasonably adjust the parameters of the unquantized language submodule, before the traversal begins, the output of the language model block for the sample output feature value can be recorded and used as the output before quantization;

[0173] At the same time, after the current language submodule is quantized, the output of the language model block for the sample output feature value can be recorded as the quantized output.

[0174] Afterwards, based on the difference between the output before quantization and the output after quantization, the parameters of the unquantized language submodule in the language model block are adjusted.

[0175] The sample output feature value may be an output feature after the quantized multimodal processing block processes the preset sample.

[0176] In actual use, in some cases, the input includes multimedia data and text data. At this time, when obtaining the pre-quantization output of the language model block for the sample output feature value, the sample output feature value and the text data can also be input into the language model block together, and then the obtained output of the language model block is used as the pre-quantization output.

[0177] Among them, when adjusting the parameters of the unquantized language sub-module in the language model block, the pre- and post-output error can also be constructed based on the output before quantization and the output after quantization, and then the parameters of the unquantized language sub-module in the language model block are adjusted with the goal of minimizing the pre- and post-output error. The specific implementation is similar to the relevant implementation method of adjusting the parameters of the unquantized processing sub-module in the multimodal processing block. Please refer to the above description for details, which will not be repeated here.

[0178] This embodiment provides a multimodal large language model quantization method. Since it traverses the language sub-module sequence, each language sub-module in the language model block is quantized separately. During the quantization process, the parameters of the unquantized language sub-module can also be adjusted to minimize the impact of quantization on the model output, thereby improving the reasoning effect of the model that is finally deployed.

[0179] This application also provides a multimodal large language model quantization device, please refer to Figure 7 , the multimodal large language model quantization device comprises:

[0180] A division module 10 is used to divide the large model to be quantized into a multimodal processing block and a language model block, wherein the large model to be quantized is a language large model supporting multimodal input;

[0181] A construction module 20, configured to construct a processing submodule sequence according to the quantization error corresponding to each processing submodule in the multimodal processing block, wherein the processing submodule includes at least one network layer in the multimodal processing block;

[0182] A first quantization module 30, configured to quantize each processing submodule in the multimodal processing block based on the processing submodule sequence;

[0183] A determination module 40, configured to determine a language submodule sequence according to the quantized multimodal processing blocks;

[0184] The second quantization module 50 is used to quantize each language submodule in the language model block according to the language submodule sequence, and the language submodule includes at least one layer of network in the language model block.

[0185] The multimodal large language model quantization device provided by the present application adopts the multimodal large language model quantization method in the above-mentioned embodiment, which can solve the technical problem that the multimodal language large model in the related art is difficult to reasonably quantize and deploy, and the quantization deployment effect is poor. Compared with the prior art, the beneficial effects of the multimodal large language model quantization device provided by the present application are the same as the beneficial effects of the multimodal large language model quantization method provided by the above-mentioned embodiment, and the other technical features in the multimodal large language model quantization device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0186] The present application provides a multimodal large language model quantization device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal large language model quantization method in the above-mentioned embodiment 1.

[0187] Reference below Figure 8 , which shows a schematic diagram of the structure of a multimodal large language model quantization device suitable for implementing the embodiment of the present application. The multimodal large language model quantization device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions: tablet computers), PMPs (Portable Media Players: portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8The multimodal large language model quantization device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0188] like Figure 8 As shown, the multimodal large language model quantization device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the multimodal large language model quantization device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the multimodal large language model quantization device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a multimodal large language model quantization device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0189] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0190] The multimodal large language model quantization device provided by the present application adopts the multimodal large language model quantization method in the above embodiment, which can solve the technical problems of the related art that the multimodal language large model is difficult to reasonably quantize and deploy, and the quantization deployment effect is poor. Compared with the prior art, the beneficial effects of the multimodal large language model quantization device provided by the present application are the same as the beneficial effects of the multimodal large language model quantization method provided by the above embodiment, and the other technical features in the multimodal large language model quantization device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0191] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0192] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0193] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, the computer-readable program instructions being used to execute the multimodal large language model quantization method in the above-mentioned embodiment.

[0194] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0195] The computer-readable storage medium may be included in the multimodal large language model quantization device; or may exist independently without being assembled into the multimodal large language model quantization device.

[0196] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a multimodal large language model quantization device, the multimodal large language model quantization device: divides the large model to be quantized into a multimodal processing block and a language model block, wherein the large model to be quantized is a large language model supporting multimodal input; constructs a processing submodule sequence according to the quantization error corresponding to each processing submodule in the multimodal processing block, wherein the processing submodule includes at least one network layer in the multimodal processing block; quantizes each processing submodule in the multimodal processing block based on the processing submodule sequence; determines a language submodule sequence according to the quantized multimodal processing block; and quantizes each language submodule in the language model block according to the language submodule sequence, wherein the language submodule includes at least one network layer in the language model block.

[0197] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0198] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0199] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0200] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned multimodal large language model quantization method, and can solve the technical problems of the related art that the multimodal language large model is difficult to reasonably quantize and deploy, and the quantization deployment effect is poor. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the multimodal large language model quantization method provided in the above-mentioned embodiment, and will not be repeated here.

[0201] The present application also provides a computer program product, including a computer program, which implements the steps of the multimodal large language model quantization method as described above when executed by a processor.

[0202] The computer program product provided by this application can solve the technical problem that it is difficult to reasonably quantify and deploy a large multimodal language model in the related art, and the quantization deployment effect is poor. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the multimodal large language model quantization method provided in the above embodiment, and will not be repeated here.

[0203] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A multimodal large language model quantization method, characterized in that: The method comprises: Dividing the large model to be quantified into a multimodal processing block and a language model block, wherein the large model to be quantified is a language large model supporting multimodal input, wherein the language large model generates a text description or an image according to the input, wherein the multimodal input includes a language text, or the multimodal input includes a language text and at least one of an image, an audio, and a video; Constructing a processing submodule sequence according to the quantization errors corresponding to each processing submodule in the multimodal processing block, wherein the processing submodule includes at least one network layer in the multimodal processing block; quantizing each processing submodule in the multimodal processing block based on the processing submodule sequence; Determining a language submodule sequence according to the quantized multimodal processing blocks; quantizing each language submodule in the language model block according to the language submodule sequence, wherein the language submodule includes at least one layer of network in the language model block; The step of determining a language submodule sequence according to the quantized multimodal processing blocks includes: Processing the preset samples by the quantized multimodal processing blocks to generate sample output feature values; Determine the quantization error corresponding to each language submodule in the language model block based on the sample output feature value, wherein the quantization error corresponding to the language submodule is the output difference of the language model block for the sample output feature value before and after the language submodule in the language model block is quantized; Based on the quantization errors corresponding to the language submodules, the language submodules in the language model block are sorted from large to small to generate a language submodule sequence.

2. The multimodal large language model quantization method according to claim 1, characterized in that: The quantizing each processing submodule in the multimodal processing block based on the processing submodule sequence includes: Traversing the processing submodule sequence according to the sequence order of the processing submodule sequence to obtain a current processing submodule; quantizing the current processing submodule in the multimodal processing block; After the current processing submodule is quantized, the parameters of the unquantized processing submodules in the multimodal processing block are adjusted.

3. The multimodal large language model quantization method according to claim 2, characterized in that: Before traversing the processing submodule sequence according to the sequence order of the processing submodule sequence to obtain the current processing submodule, the method further includes: Obtaining the pre-quantization output of the multimodal processing block for a preset sample; After the current processing submodule is quantized, adjusting the parameters of the unquantized processing submodules in the multimodal processing block includes: After the current processing submodule is quantized, obtaining the quantized output of the multimodal processing block for a preset sample; The parameters of the unquantized processing submodules in the multimodal processing block are adjusted based on the pre-quantization output and the post-quantization output.

4. The multimodal large language model quantization method according to claim 3, characterized in that: The adjusting the parameters of the unquantized processing submodule in the multimodal processing block based on the pre-quantization output and the post-quantization output includes: Constructing a pre- and post-output error according to the pre-quantization output and the post-quantization output; With the goal of minimizing the output before-after error, the parameters of the unquantized processing submodules in the multimodal processing block are adjusted.

5. The multimodal large language model quantization method according to claim 2, characterized in that: The quantizing the current processing submodule in the multimodal processing block includes: Acquire characteristic distribution data of the current processing submodule; The current processing submodule in the multimodal processing block is quantized according to the minimum value and / or the maximum value in the feature distribution data.

6. The multimodal large language model quantization method according to claim 1, characterized in that: The quantizing of each language submodule in the language model block according to the language submodule sequence includes: Traversing the language submodule sequence according to the sequence order of the language submodule sequence to obtain a current language submodule; quantizing the current language submodule in the language model block; After the current language submodule is quantized, the parameters of the unquantized language submodules in the language model block are adjusted.

7. The multimodal large language model quantization method according to claim 6, characterized in that: Before traversing the language submodule sequence according to the sequence order of the language submodule sequence to obtain the current language submodule, the method further includes: Obtaining the pre-quantization output of the language model block for the sample output feature value; After the current language submodule is quantized, adjusting the parameters of the unquantized language submodule in the language model block includes: After the current language submodule is quantized, obtaining the quantized output of the language model block for the sample output feature value; Based on the pre-quantization output and the post-quantization output, parameters of the unquantized language submodule in the language model block are adjusted.

8. The multimodal large language model quantization method according to any one of claims 1 to 6, characterized in that: The step of constructing a processing submodule sequence according to the quantization errors corresponding to each processing submodule in the multimodal processing block includes: Obtaining a quantization error corresponding to each processing submodule in the multimodal processing block, where the quantization error corresponding to the processing submodule is the output difference of the multimodal processing block for the same sample before and after the processing submodule in the multimodal processing block is quantized; Based on the quantization error, the processing submodules in the multimodal processing block are sorted from large to small to generate a processing submodule sequence.

9. A multimodal large language model quantization device, characterized in that: The device comprises: A division module, used for dividing the large model to be quantized into multimodal processing blocks and language model blocks, wherein the large model to be quantized is a language large model supporting multimodal input, wherein the language large model generates a text description or an image according to the input, wherein the multimodal input includes language text, or the multimodal input includes language text and at least one of an image, audio, and video; A construction module, used to construct a processing submodule sequence according to the quantization error corresponding to each processing submodule in the multimodal processing block, wherein the processing submodule includes at least one network layer in the multimodal processing block; A first quantization module, configured to quantize each processing submodule in the multimodal processing block based on the processing submodule sequence; A determination module, configured to determine a language submodule sequence according to the quantized multimodal processing blocks; A second quantization module, configured to quantize each language submodule in the language model block according to the language submodule sequence, wherein the language submodule includes at least one layer of network in the language model block; The determination module is further used to process a preset sample through the quantized multimodal processing block to generate a sample output feature value; determine the quantization error corresponding to each language submodule in the language model block based on the sample output feature value, the quantization error corresponding to the language submodule is the output difference of the language model block for the sample output feature value before and after the language submodule in the language model block is quantized; and sort the language submodules in the language model block from large to small based on the quantization errors corresponding to the language submodules to generate a language submodule sequence.

10. A multimodal large language model quantization device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multimodal large language model quantization method according to any one of claims 1 to 8.

11. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multimodal large language model quantization method according to any one of claims 1 to 8 are implemented.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the multimodal large language model quantization method according to any one of claims 1 to 8 are implemented.

Images