Multimodal large model compression method and device based on sparse codebook quantization
By optimizing the saliency distribution of visual encoder weights and dynamically allocating codewords through sparse codebook quantization, the computation and storage problems of large multimodal visual-language models deployed on resource-constrained devices are solved, achieving a balance between efficient model compression and inference performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2025-12-31
- Publication Date
- 2026-05-29
AI Technical Summary
When large vision-language multimodal models are deployed on resource-constrained edge devices, computational complexity and storage consumption are high. Traditional quantization methods have failed to effectively reduce the computational burden and have affected model performance.
The sparse codebook quantization method is adopted to optimize the weight saliency distribution of the visual encoder, dynamically allocate the number of codewords, and combine high-level candidate search and low-level subset refinement strategies to achieve the best balance between model compression and inference performance.
Without sacrificing model performance, it significantly reduces computational complexity and storage overhead, improves quantization compression efficiency, and is suitable for deployment on resource-constrained devices.
Smart Images
Figure CN122114016A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for compressing multimodal large models based on sparse codebook quantization. Background Technology
[0002] In recent years, large-scale visual-language multimodal models have made significant progress as a core component of multimodal artificial intelligence systems. These models can process both image and text information simultaneously, demonstrating superior reasoning and understanding capabilities in a variety of complex cross-modal tasks, such as visual question answering, image-text retrieval, image description generation, embodied command execution, and robot navigation. The success of large-scale visual-language multimodal models is mainly attributed to their massive parameter scale and training on massive amounts of multimodal data. By fusing information from different modalities, they possess powerful semantic understanding and context modeling capabilities. Furthermore, the design of many advanced architectures has also driven performance improvements; for example, the Transformer-based architecture enables models to maintain efficient information interaction and attention mechanisms under long sequences of input.
[0003] While large-scale visual-language multimodal models have demonstrated excellent accuracy and generalization capabilities across multiple tasks, their extremely high computational complexity and memory consumption have become key bottlenecks restricting their practical deployment. This is particularly pronounced on resource-constrained edge devices or mobile terminals, such as smartphones, embedded robots, or drone platforms, where high latency and high power consumption are especially significant. Furthermore, large-scale visual-language multimodal models typically employ autoregressive methods to generate responses word-by-word, requiring multiple forward propagation operations, which further exacerbates the overall computational burden and impacts the model's real-time performance and efficiency. Therefore, in practical applications, there is an urgent need to compress large-scale visual-language multimodal models to reduce their computational and storage overhead while preserving as much of their original performance as possible.
[0004] To achieve lightweight models, researchers have proposed various model compression techniques, including pruning, quantization, low-rank decomposition, and efficient architecture design. Among these, quantization has attracted significant attention due to its remarkable effectiveness in improving inference speed and reducing model size. This method significantly improves computational efficiency by replacing floating-point numbers with fixed-point numbers and using integer operations instead of multiplication and addition, making it one of the most promising compression methods currently available. Because large-scale vision-language multimodal models have strong data dependencies and extremely high retraining costs, post-training quantization has become a mainstream strategy. This method does not require access to the original training data; it only needs to freeze the network parameters using a small calibration set and then find the optimal quantization function.
[0005] However, due to the enormous search space of quantization functions, traditional methods typically optimize the discrete errors of activation values layer by layer to determine the optimal rounding strategy for each layer. While these methods reduce search complexity, they ignore the interdependencies of discrete errors between different layers, making it difficult to obtain a globally optimal quantization scheme, thus severely impacting the final performance of the model. In summary, effectively reducing the computational complexity and deployment threshold of LVLMs without sacrificing model performance is a critical issue that urgently needs to be addressed for the practical application of multimodal artificial intelligence systems. This also provides an important research direction for efficient model compression methods, especially quantization strategies that consider cross-layer collaborative optimization. Summary of the Invention
[0006] The present invention aims to at least partially solve one of the technical problems in the related art.
[0007] Therefore, the first objective of this invention is to propose a multimodal large model compression method based on sparse codebook quantization.
[0008] The second objective of this invention is to propose a multimodal large model compression device based on sparse codebook quantization.
[0009] The third objective of this invention is to provide an electronic device.
[0010] The fourth objective of this invention is to provide a computer-readable storage medium.
[0011] The fifth objective of this invention is to provide a computer program product.
[0012] To achieve the above objectives, a first aspect of the present invention proposes a multimodal large model compression method based on sparse codebook quantization, comprising: For any given large-scale vision-language model, optimize the visual encoder in the model to make its weight saliency distribution more concentrated; For each layer of weights in the optimized language model, the influence of each weight on the model output is evaluated through second-order information, and the number of codewords to be allocated to each weight group is dynamically determined based on the significance of the weights. After determining the number of codewords, the optimal sparse combination is selected from a large-scale codebook through a two-stage strategy of high-level candidate search and low-level subset refinement. By combining the optimal sparse combination with sparse coefficients, model quantization is completed, achieving the best balance between model compression and inference performance.
[0013] Optionally, let the model weights be... The Taylor expansion of its loss function is approximately:
[0014] in This indicates the weight changes after removing certain codewords; The error increment corresponding to the weight change is expressed as:
[0015] in It is the first Hessian matrix. A diagonal piece, It is the change in the sparsity coefficient.
[0016] Optionally, select a subset of codewords from the codebook that have the highest similarity to the current weight. Cosine similarity is used to measure the degree of matching:
[0017] in It is the first The reconstruction error of the next iteration.
[0018] Optionally, the codeword combination can be further optimized in the subset selected at the higher level, retaining the codewords with the greatest impact and removing redundant terms, and the sparsity coefficient can be updated iteratively. To minimize the reconstruction error: .
[0019] Optionally, the optimization objective is to minimize the activation entropy while also considering the model output error:
[0020] in It is a normalized Hessian distribution; The ultimate overall optimization goal is:
[0021] in This represents the autoregressive loss of the original language model.
[0022] To achieve the above objectives, a second aspect of the present invention provides a multimodal large model compression device based on sparse codebook quantization, comprising: The optimization module is used to optimize the visual encoder in any given large-scale vision-language model, making its weight saliency distribution more concentrated. The allocation module is used to evaluate the impact of each weight on the model output using second-order information for each layer of weights in the optimized language model, and dynamically determine the number of codewords to be allocated to each weight group based on the significance of the weights. The computation module is used to select the optimal sparse combination from a large-scale codebook after determining the number of codewords, through a two-stage strategy of high-level candidate search and low-level subset refinement. The compression module is used to combine the optimal sparse combination and sparse coefficients to complete model quantization, achieving the best balance between model compression and inference performance.
[0023] To achieve the above objectives, a third aspect of the present invention provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects.
[0024] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of the first aspects.
[0025] To achieve the above objectives, a fifth aspect of the present invention provides a computer program product that, when executed by a processor, implements the method described in any one of the first aspects.
[0026] The technical solutions provided by the embodiments of the present invention bring at least the following beneficial effects: This invention proposes a dynamic codeword allocation and hierarchical search method based on sparse coding. Without introducing additional training, it adaptively allocates the optimal sparse codeword combination for different weights, thereby maintaining the model's expressive power at extremely low bit depths and improving quantization compression efficiency and inference performance.
[0027] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0028] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating a multimodal large model compression method based on sparse codebook quantization provided in an embodiment of the present invention. Figure 2 This is an architecture diagram of a multimodal large model compression method based on sparse codebook quantization provided in an embodiment of the present invention; Figure 3 This is a comparison diagram between the multimodal large model compression method based on sparse codebook quantization provided in the embodiments of the present invention and traditional quantization methods; Figure 4 This is a schematic diagram of a multimodal large model compression device based on sparse codebook quantization provided in an embodiment of the present invention. Detailed Implementation
[0029] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.
[0030] To address the issue of severely degraded inference performance of existing large-scale vision-language multimodal models after low-bit quantization, this invention provides a large-scale multimodal model compression method based on sparse codebook quantization. Figure 1 This is a flowchart illustrating a multimodal large model compression method based on sparse codebook quantization provided in an embodiment of the present invention. Figure 2 This is an architecture diagram of a multimodal large model compression method based on sparse codebook quantization provided in an embodiment of the present invention; Figure 3 This diagram compares the multimodal large model compression method based on sparse codebook quantization provided in this invention with traditional quantization methods. Figure 1-3 As shown, the method includes the following steps: Step 101: For any given large-scale vision-language model, optimize the visual encoder in the model to make its weight saliency distribution more concentrated.
[0031] To concentrate the saliency distribution of the visual encoder weights in the visual-language model more in key regions, reduce redundant codewords, and improve overall inference efficiency while taking into account the quality of model output, this application proposes a saliency distribution optimization method for the visual encoder and constructs a corresponding overall optimization objective.
[0032] With the deep integration of large language models and computer vision, the performance of visual-language models (LVLMs) has attracted increasing attention, and the optimization of the visual encoder is a key factor in improving model efficiency. This application proposes a saliency distribution optimization method for the visual encoder, aiming to make the weight saliency distribution of the visual encoder more focused on the key regions of the image.
[0033] In the inference process of LVLM, if the visual encoder can accurately capture key regions, it can reduce the encoding of redundant visual information, thereby reducing the number of codewords required and significantly improving the overall inference efficiency. Specifically, in the embodiments of this application, the activation entropy is minimized while also taking into account the model's output error, thus constructing a loss function:
[0034] in It is a normalized Hessian distribution, and its role is to model the influence of each layer of the network on the final output error. Through this design, the significance distribution of the weights can be more reasonably used to reduce the error.
[0035] In this embodiment, after optimizing the saliency distribution of the visual encoder, it needs to be integrated into the overall optimization framework of the model. The final overall optimization objective is:
[0036] in, The autoregressive loss of the original language model ensures that the quality of the text generated by the model is not affected by the optimization of the visual part; the regularization term is to keep the weight optimization of the visual encoder stable and effective throughout the entire model parameter space.
[0037] In practical applications, such models can accurately capture key information in image understanding tasks and maintain good language quality in generation tasks, laying a solid foundation for the efficient application of LVLM in various vision-language scenarios (such as image and text retrieval, visual question answering, etc.).
[0038] In this embodiment, through a series of optimizations, the salient weights of the visual encoder are more concentrated, greatly reducing the use of redundant codewords and laying the foundation for subsequent dynamic codeword allocation based on weight salientity.
[0039] Step 102: For each layer of weights in the optimized language model, evaluate the degree of influence of each weight on the model output through second-order information, and dynamically determine the number of codewords to be allocated to each weight group based on the significance of the weights.
[0040] To improve quantization efficiency and reduce information loss, this application proposes a dynamic codeword allocation mechanism based on weight saliency, which evaluates the influence of weights on model output through second-order information (Hessian matrix) to optimize codeword allocation while maintaining low search complexity.
[0041] In the field of quantization and efficient inference of visual-language models, dynamic codeword allocation mechanism based on weight saliency has emerged. Its core goal is to improve quantization efficiency while minimizing information loss.
[0042] During model quantization, the rationality of codeword allocation directly affects the performance of the quantized model. If the same number of codewords is used for all ownership reorganizations, either the accuracy will plummet due to insufficient key weight codewords, or resources will be wasted due to excessive redundant weight codewords. Therefore, a mechanism is needed that can allocate codewords "on demand" according to the importance of weights.
[0043] In this embodiment, second-order information (Hessian matrix) is used to evaluate the influence of each weight on the model output. Simply put, the Hessian matrix reflects the curvature of the loss function as the weights change; the greater the curvature, the more critical the influence of that weight on the model output.
[0044] Specifically, the model weights The loss function is approximated by a Taylor expansion:
[0045] in This represents the change in weights after removing certain codewords. Since the model has converged after training, the first-order term of the loss function can be ignored. Therefore, the error increment corresponding to the weight change can be expressed as:
[0046] in It is the first Hessian matrix. A diagonal piece, It is the change in sparsity coefficients. By calculating the Hessian matrix and analyzing the importance of each weight, we can flexibly allocate the optimal number of codewords to different weight reorganizations, that is, allocate more codewords to key weight reorganizations that have a great impact on the model output, and allocate fewer codewords to redundant weight reorganizations that have a small impact.
[0047] In practical applications, LVLM using this mechanism can maintain high task performance after quantization while reducing computational and storage overhead during inference. It is very suitable for deployment on resource-constrained devices (such as mobile devices), providing a more efficient technical path for the industrialization of LVLM.
[0048] In this embodiment, the dynamic codeword allocation mechanism based on weight saliency can optimize codeword allocation as much as possible while maintaining low search complexity, thereby effectively reducing quantization error and laying the foundation for subsequent hierarchical codeword selection algorithms.
[0049] Step 103: After determining the number of codewords, the optimal sparse combination is selected from the large-scale codebook through a two-stage strategy of high-level candidate search and low-level subset refinement.
[0050] To efficiently search for the optimal sparse linear combination from a large-scale codebook and minimize discretization error, this application adopts a hierarchical strategy, which optimizes codeword selection through two stages: high-level candidate search and low-level subset refinement.
[0051] In the field of model quantization and sparse coding, the hierarchical codeword selection algorithm is a strategy for efficiently searching for the optimal sparse linear combination from a large-scale codebook, with the aim of minimizing discretization error.
[0052] The algorithm consists of two stages, and the function and principle of each stage are as follows: High-level candidate search phase: Select a subset of codewords from the codebook that have the highest similarity to the current weight. Cosine similarity is used to measure the matching degree between codewords and weights.
[0053] in It is the first The reconstruction error of the second iteration. This stage can select the most representative set of codewords, laying the foundation for subsequent fine optimization.
[0054] Low-level subset refinement stage: Further optimize codeword combinations within the subset selected from the high-level refinement stage, retaining the most influential codewords and removing redundant terms. This is achieved through iterative updates of the sparsity coefficients. To minimize the reconstruction error.
[0055]
[0056] This hierarchical design balances search efficiency and encoding accuracy, and has significant application value in scenarios such as model compression and efficient inference, enabling models to maintain performance while reducing storage and computational overhead.
[0057] In this embodiment, the process gradually constructs the optimal sparse coding combination, effectively reducing the search space and improving quantization accuracy, laying the foundation for achieving a balance between model compression and inference performance in the future.
[0058] Step 104: Combine the optimal sparse combination and sparse coefficients to complete model quantization, achieving the best balance between model compression and inference performance.
[0059] To create a scheme that can compress the model without accessing the original training data and with only a small number of calibration samples, model quantization is performed by combining the selected codewords and sparse coefficients, achieving the best balance between model compression and inference performance.
[0060] In this embodiment, the three technologies mentioned above—saliency distribution optimization for visual encoders, dynamic codeword allocation mechanism based on weighted saliency, and hierarchical codeword selection algorithm—are integrated to form a complete quantization scheme. This scheme is highly practical, requiring no access to the original training data and only a small number of calibration samples to complete model compression.
[0061] Specifically, firstly, by optimizing the saliency distribution of the visual encoder, the weights of the visual encoder are made more focused on key regions, laying the foundation for subsequent quantization.
[0062] Next, a dynamic codeword allocation mechanism based on weight significance is used to reasonably allocate the number of codewords for different weight groups according to the degree of influence of the weights on the model output, thereby improving quantization efficiency while reducing information loss.
[0063] Then, a hierarchical codeword selection algorithm is used to efficiently search for the optimal sparse linear combination from a large-scale codebook, further reducing discretization error.
[0064] Finally, the model is quantized by combining the selected codewords and sparse coefficients, achieving an optimal balance between model compression and inference performance.
[0065] In this embodiment, the optimal balance between model compression and inference performance is achieved by combining three technologies.
[0066] To achieve the above embodiments, the present invention also proposes a multimodal large model compression device based on sparse codebook quantization. Figure 4 This is a schematic diagram of a multimodal large model compression device based on sparse codebook quantization, provided as an embodiment of the present invention. Figure 4 As shown, the device includes: The optimization module 100 is used to optimize the visual encoder in any given large-scale vision-language model, so that its weight saliency distribution is more concentrated. The allocation module 200 is used to evaluate the influence of each weight on the model output through second-order information for each layer of weights in the optimized language model, and dynamically determine the number of codewords to be allocated to each weight group based on the significance of the weights. The computation module 300 is used to select the optimal sparse combination from a large-scale codebook after determining the number of codewords through a two-stage strategy of high-level candidate search and low-level subset refinement. Compression module 400 is used to combine the optimal sparse combination and sparse coefficients to complete model quantization, achieving the best balance between model compression and inference performance.
[0067] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0068] To implement the above embodiments, the present invention also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.
[0069] To implement the above embodiments, the present invention also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.
[0070] To implement the above embodiments, the present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.
[0071] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0072] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.
[0073] This invention is intended to provide implementation schemes for users to selectively prevent the use or access to personal information data. That is, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information can be de-identified to protect user privacy.
[0074] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0075] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0076] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.
[0077] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.
[0078] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0079] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0080] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0081] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.
[0082] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0083] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A multimodal large model compression method based on sparse codebook quantization, characterized in that, include: For any given large-scale vision-language model, optimize the visual encoder in the model to make its weight saliency distribution more concentrated; For each layer of weights in the optimized language model, the influence of each weight on the model output is evaluated through second-order information, and the number of codewords to be allocated to each weight group is dynamically determined based on the significance of the weights. After determining the number of codewords, the optimal sparse combination is selected from a large-scale codebook through a two-stage strategy of high-level candidate search and low-level subset refinement. By combining the optimal sparse combination with sparse coefficients, model quantization is completed, achieving the best balance between model compression and inference performance.
2. The method as described in claim 1, characterized in that, The method of evaluating the impact of each weight on the model output using second-order information for each layer of weights in the optimized language model, and dynamically determining the number of codewords to be allocated to each weight group based on the significance of the weights, also includes: Let the model weights be... The Taylor expansion of its loss function is approximately: in This indicates the weight changes after removing certain codewords; The error increment corresponding to the weight change is expressed as: in It is the first Hessian matrix. A diagonal piece, It is the change in the sparsity coefficient.
3. The method as described in claim 1, characterized in that, The high-level candidate search also includes: Select a subset of codewords from the codebook that have the highest similarity to the current weight. Cosine similarity is used to measure the degree of matching: in It is the first The reconstruction error of the next iteration.
4. The method as described in claim 1, characterized in that, The underlying subset refining also includes: Further optimize the codeword combinations in the subset selected from the high-level selection, retaining the codewords with the greatest impact and removing redundant terms, and iteratively update the sparsity coefficients. To minimize the reconstruction error: 。 5. The method as described in claim 1, characterized in that, The method of combining optimal sparse combination and sparse coefficients to complete model quantization, achieving the best balance between model compression and inference performance, also includes: The optimization objective is to minimize the activation entropy while also considering the model output error. in It is a normalized Hessian distribution; The ultimate overall optimization goal is: in This represents the autoregressive loss of the original language model.
6. A multimodal large model compression device based on sparse codebook quantization, characterized in that, include: The optimization module is used to optimize the visual encoder in any given large-scale vision-language model, making its weight saliency distribution more concentrated. The allocation module is used to evaluate the impact of each weight on the model output using second-order information for each layer of weights in the optimized language model, and dynamically determine the number of codewords to be allocated to each weight group based on the significance of the weights. The computation module is used to select the optimal sparse combination from a large-scale codebook after determining the number of codewords through a two-stage strategy of high-level candidate search and low-level subset refinement. The compression module is used to combine the optimal sparse combination and sparse coefficients to complete model quantization, achieving the best balance between model compression and inference performance.
7. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.
9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-5.