Model compression method, system and equipment for multi-modal large model

Abstract

The invention discloses a model compression method, system and device for a multi-modal large model, and relates to the technical field of computer application, and the method comprises the steps: obtaining a training set and a test set of an event data set; obtaining the original prediction precision of the corresponding specific event through the test set and the original model; obtaining a prediction precision range according to the original prediction precision and the landing scene of the specific event; pruning the original model; obtaining a to-be-evaluated intermediate model; determining a target intermediate model based on the specific event; determining a quantization parameter according to an event demand, converting the target intermediate model into a quantization model, and determining the model precision of the quantization model; and selecting the compressed multi-modal large model. According to the method, the technical problems of large deployment resource consumption, relatively high model parameter quantity and relatively complex compression process in the compression process of the traditional large model are solved, and the technical effects of reducing the deployment cost of the model and accelerating the reasoning speed of the model on the premise of not influencing the precision of the model are realized.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and in particular to a model compression method, system and device for a multimodal large model. Background Art

[0002] Multimodal large models have excellent performance in the field of urban event detection due to their powerful zero-sample generalization capabilities. However, for large models, the number of parameters is huge. The number of parameters of existing multimodal large models often reaches hundreds of millions to tens of billions. The deployment consumes a lot of resources and is costly, which limits the implementation of multimodal large models in urban business. At the same time, due to the needs of business scenarios, the accuracy value of the optimized model must be small compared to the original model, and the optimization of computing efficiency has business value. In this case, the implementation of large model compression technology not only means that data can be processed faster, but more importantly, it can be analyzed and responded to in real time, which is crucial to ensuring urban safety and improving emergency response efficiency. Therefore, under the premise of ensuring model accuracy, model compression of multimodal large models is indispensable from the perspective of cost and efficiency.

[0003] However, existing model compression methods have technical problems such as large deployment resource consumption and high number of model parameters during the compression process, which leads to a more complicated compression process and low compression efficiency. Summary of the invention

[0004] The embodiments of the present application provide a model compression method, system and device for a multimodal large model, which solves the technical problem that the model compression method in the prior art consumes a lot of resources during the compression process, has a high number of model parameters, and thus leads to a complex compression process and low compression efficiency. The technical effect of reducing deployment costs and accelerating model compression speed while ensuring model accuracy is achieved.

[0005] In a first aspect, an embodiment of the present application provides a model compression method for a multimodal large model, comprising: obtaining an event data set, the event data set comprising a training set and a test set; based on the test set and the original model, determining the original prediction accuracy of the specific event corresponding to the original model; determining the prediction accuracy range of the specific event according to the original prediction accuracy and the landing scenario of the specific event; pruning the original model to obtain an intermediate model to be evaluated; based on the specific event, performing an accuracy evaluation on the intermediate model to be evaluated, and determining a target intermediate model, the target intermediate model being the intermediate model to be evaluated whose model accuracy meets the prediction accuracy range; determining a quantization parameter according to the event requirement of the specific event, converting the target intermediate model into quantization models under multiple different data types based on the quantization parameter, and determining the model accuracy of the quantization model based on the test set; based on the event requirement, determining a target model from the multiple quantization models, the target model being the compressed multimodal large model.

[0006] In combination with the first aspect, in a possible implementation manner, the original prediction accuracy is within the prediction accuracy range, and the model accuracy of the quantization model is within the prediction accuracy range.

[0007] In combination with the first aspect, in a second possible implementation manner, pruning the original model to obtain the intermediate model to be evaluated includes: splitting the original model into a visual encoder, a linear mapping layer, and a large language model; obtaining all MLP layers of the decoding layer in the large language model, and obtaining the average activation value of each channel of the MLP layer; sorting the corresponding channels of the MLP layer from large to small based on the average activation value; pruning the sorted large language model to obtain the pruned original model, and determining the pruned original model as the intermediate model to be evaluated.

[0008] In combination with the second possible implementation manner of the first aspect, in a third possible implementation manner, pruning the sorted large language model to obtain a pruned original model, and determining the pruned original model as the intermediate model to be evaluated includes: setting a pruning rate, removing channels whose average activation value ratio is less than the pruning rate based on a ratio of the pruning rate, obtaining the pruned original model, and determining the pruned original model as the intermediate model to be evaluated.

[0009] In combination with the second possible implementation manner of the first aspect, in a fourth possible implementation manner, pruning the sorted large language model to obtain a pruned original model, and determining the pruned original model as the intermediate model to be evaluated, including: pruning the sorted large language model to obtain a model to be fine-tuned; using the training set to perform low-rank adaptive fine-tuning on the model to be fine-tuned to obtain the pruned original model, and determining the pruned original model as the intermediate model to be evaluated.

[0010] In combination with the first aspect, in a fifth possible implementation method, the accuracy of the intermediate model to be evaluated is evaluated based on the specific event to determine the target intermediate model, including: using the intermediate model to be evaluated to evaluate the specific event to obtain the model accuracy of the intermediate model to be evaluated; if the model accuracy of the intermediate model to be evaluated is within the prediction accuracy range, the intermediate model to be evaluated is determined as the target intermediate model; if the model accuracy of the intermediate model to be evaluated is outside the prediction accuracy range, the intermediate model to be evaluated is discarded.

[0011] In combination with the first aspect, in a sixth possible implementation, after obtaining the compressed multimodal large model, it also includes: performing an executable judgment step on the compressed multimodal large model; the executable judgment step includes: performing a model accuracy index test on the compressed multimodal large model based on the test set to obtain the final accuracy of the specific event in the compressed multimodal large model; if the final accuracy is within the prediction accuracy range, retaining the multimodal large model; if the final accuracy does not fall within the prediction accuracy range, discarding the multimodal large model.

[0012] In the second aspect, an embodiment of the present application provides a system for executing a model compression method for a multimodal large model, including: a setting module, used to set the original prediction accuracy of a specific event corresponding to the original model, and set the prediction accuracy range of the specific event; a pruning module, used to prune the original model, and perform accuracy evaluation and low-rank adaptive fine-tuning on the pruned original model to obtain a target intermediate model; a quantization module, used to convert the target intermediate model into a plurality of quantization models under different data types based on quantization parameters; a selection module, used to determine the target model from the plurality of quantization models based on event requirements, and obtain a compressed multimodal large model.

[0013] In a third aspect, an embodiment of the present application provides a device, comprising: a processor; a memory for storing processor executable instructions; when the processor executes the executable instructions, it implements the model compression method for a multimodal large model as described in the first aspect or any possible implementation method of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium includes a medium for storing a computer program or instructions. When the computer program or instructions are executed, the model compression method for a multimodal large model as described in the first aspect or any possible implementation method of the first aspect is implemented.

[0015] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0016] The embodiment of the present application provides a model compression method for a multimodal large model. By pruning, quantizing, and restoring the accuracy of the original model, the method effectively solves the technical problems of large resource consumption, high model parameter volume, and complex compression process in the compression process of the traditional large model, thereby achieving the technical effect of reducing the deployment cost of the model and accelerating the reasoning speed of the model without affecting the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a model compression method for a multimodal large model provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of the process of pruning the original model provided in an embodiment of the present application;

[0020] Figure 3 A system block diagram of a model compression system for a multimodal large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The embodiments of the present application provide a model compression method, system and device for a large multimodal model, which are used to reduce the number of model parameters and improve reasoning performance. Figure 1Flowchart of the model compression method for a multi-modal large model provided in the embodiment of the present application. Figure 1 As shown, it includes steps 101 to 107.

[0023] Step 101: Obtain an event dataset, and divide the event dataset into a training set and a test set.

[0024] Step 102: Based on the test set and the original model, determine the original prediction accuracy of the specific event corresponding to the original model.

[0025] Step 103: Determine the prediction accuracy range of the specific event based on the original prediction accuracy and the landing scenario of the specific event.

[0026] Step 104: Prune the original model to obtain an intermediate model to be evaluated.

[0027] Step 105: Based on the specific event, the accuracy of the intermediate model to be evaluated is evaluated, and the target intermediate model is determined. The target intermediate model is the intermediate model to be evaluated whose model accuracy meets the prediction accuracy range.

[0028] Step 106: Determine quantization parameters according to event requirements of a specific event, convert the target intermediate model into quantization models under multiple different data types based on the quantization parameters, and determine the model accuracy of the quantization model based on the test set.

[0029] Step 107: Based on event requirements, determine a target model from multiple quantitative models, where the target model is a compressed multimodal large model.

[0030] The embodiment of the present application provides a model compression method for a large multimodal model. By pruning, quantizing, and restoring the accuracy of the original model, the method effectively solves the technical problems of large resource consumption, high model parameter volume, and complex compression process in the compression process of traditional large models. The method further achieves the technical effect of reducing the deployment cost of the model and accelerating the reasoning speed of the model while minimizing or not affecting the accuracy of the model.

[0031] Specifically, the event data in the event dataset are obtained through open source databases, and these event data are processed and integrated to form an event dataset. The event dataset is divided into a training set and a test set.

[0032] The training set is used to fine-tune the compressed multimodal model, so that the multimodal model performs better in specific events. The test set is used to evaluate the original model and the compressed multimodal model, and to determine the original prediction accuracy of the specific event corresponding to the original model.

[0033] Specifically, the original model is tested for indicators using the data in the test set. For example, the indicator test content includes the detection of specific urban events such as fighting and drying clothes on the street. The accuracy, recall rate, F1-Score value (balanced F score, an indicator to measure the performance of a classification model, suitable for processing extremely unbalanced data sets) and other parameters of the corresponding events are calculated. This is to enhance the indicator test process of the test set for the original model and improve the indicator test results.

[0034] At the same time, the landing scenario of the specific event is evaluated, and based on the landing scenario and the original prediction accuracy of the specific event, the prediction accuracy range of the specific event is set. The prediction accuracy range is used to limit the model accuracy of the intermediate model to be evaluated after pruning and the final accuracy of the compressed multimodal large model. At the same time, in the embodiment of the present application, the original prediction accuracy, model accuracy, and the final accuracy of the compressed multimodal large model are all within the prediction accuracy range.

[0035] For example, a decrease in prediction accuracy of 0.5% to 1% is acceptable, and the decrease in model accuracy of the intermediate model to be evaluated compared with the original prediction accuracy of the original model before pruning should be within the range of 0.5% to 1%, otherwise the intermediate model to be evaluated is unusable; the decrease in model accuracy of the compressed multimodal large model on the test set compared with the model accuracy of the original model before compression should be within the range of 0.5% to 1%, otherwise the compressed multimodal large model is unusable.

[0036] Figure 2 1 is a flow chart of pruning the original model provided in the embodiment of the present application. In the embodiment of the present application, the pruning of the original model mentioned in step 104 is performed, such as Figure 2 As shown, the pruning process specifically includes steps 401 to 405.

[0037] Step 401: Split the original model into a visual encoder, a linear mapping layer, and a large language model.

[0038] Step 402: Obtain all MLP layers of the decoding layer in the large language model, and use the small sample value to infer the model to obtain the average activation value of each channel of the MLP layer.

[0039] Step 403: Sort the corresponding channels of the MLP layer from large to small based on the average activation value.

[0040] Step 404: Prune the sorted large language model to obtain a model to be fine-tuned.

[0041] Specifically, it is necessary to set the pruning rate of the multimodal large model proposed in this application, and remove channels whose average activation value ratio is less than the pruning rate based on the ratio of the pruning rate.

[0042] After obtaining the model to be fine-tuned, it also includes:

[0043] Step 405: Use part of the training set to perform low-rank adaptive fine-tuning on the model to be fine-tuned to obtain the pruned original model, and determine the pruned original model as the intermediate model to be evaluated.

[0044] Specifically, a part of the training data set is used to perform LoRA (low rank adaptation) fine-tuning on the model to be fine-tuned to restore the accuracy of the model to be fine-tuned. The main idea of ​​LoRA is to divide the parameters in the model to be fine-tuned into two parts: pre-trained weights and fine-tuning incremental weights. The pre-trained weights remain unchanged during the fine-tuning process, while the fine-tuning incremental weights are approximated by a low-rank matrix.

[0045] The learnable weight matrix in the model to be fine-tuned is represented as W, and the update value is ΔW. The calculation method of the update value is as follows:

[0046]

[0047] Among them, P and Q represent the parameters in the weight matrix W, R represents the real number set, which represents the set of all real numbers, d - represents the number of rows of the original weights in the weight matrix W, d + Represents the number of columns of the original weights in the weight matrix W, and d is the rank in the LoRA low-rank adaptive fine-tuning method. It should be noted that the value of rank d is much smaller than the number of rows d of the original weights - and the number of columns of original weights d + .

[0048] The previous calculation of the updated value can be expressed as:

[0049] f(x)=(W+ΔW)X+b=(WX+b)+(PQ)X.

[0050] Among them, f(x) represents the preceding calculation formula of the update value ΔW, X represents the input data to be processed in the large language model, and b represents the bias in the large language model. Training only P and Q reduces the overall training complexity and reduces the need for large-scale training data. In addition, the additional parameters P and Q can be reparameterized into the update value ΔW, which will not cause additional parameters in the final compressed multimodal large model.

[0051] For example, the LLaVA-7B multimodal large model is used as the original model mentioned in the embodiments of this application to explain the pruning process as follows:

[0052] From the structure diagram of the LLaVA multimodal large model, we can see that the LLaVA multimodal large model includes a visual encoder, a linear mapping layer, and a large language model. Since the parameters of the large language model are dominant. Especially in the 7B multimodal large model, the parameters of the large language model module account for 6.1B. The large language model is split into modules, and the LlamaDecoderLayer (calling the decoder layer) in the large language model is pruned.

[0053] Using 100 small samples and using the LLaVA multimodal model for inference, record the activation values ​​of SiLUActivation in all LlamaMLP [a key component in the LLaVA multimodal model, representing the multilayer perceptron layer (MLP) in the LLaVA multimodal model]. And take the average of these activation values ​​to get the average activation value. These average activation values ​​represent the importance of the corresponding channel.

[0054] At the same time, the 11008 channels corresponding to the activation function are grouped into groups of 128, and then sorted from large to small within the group according to the corresponding average activation values.

[0055] Set the pruning rate. According to the set pruning rate, remove the corresponding channels in each group, and adjust the number of channels of up_proj, gate_proj and down_proj in LlamaMLP. If the pruning rate is 20%, the number of channels after pruning is 6027. It should be noted that this pruning rate is the overall pruning rate in the LLaVA multimodal large model, which is applicable to the pruning process of channels in other groups.

[0056] After pruning is completed, it is necessary to use LoRA's fine-tuning method to train the pruned LLaVA multimodal large model (that is, the model to be fine-tuned proposed in the embodiment of the present application) to restore the accuracy of the pruned LLaVA multimodal large model.

[0057] Specifically, first, 5% of the training set is uniformly extracted from the event dataset used to train the original model as training data for restoring accuracy.

[0058] Then the LLaVA large multimodal model is loaded, and the LoRA method is used to train the pruned LLaVA large multimodal model for recovery accuracy. The rank d is fine-tuned to 128. After training to a certain number of iterations, the test set is used to evaluate the effect of the pruned LLaVA large multimodal model.

[0059] Finally, if the model accuracy of the pruned LLaVA multimodal large model meets the prediction accuracy range, stop training. Otherwise, adjust the training strategy, reset the pruning rate, and continue to prune the LLaVA multimodal large model until the model accuracy meets the prediction accuracy range.

[0060] In the embodiment of the present application, based on a specific event, the accuracy of the intermediate model to be evaluated is evaluated to determine the target intermediate model, which specifically includes:

[0061] The intermediate model to be evaluated is used to evaluate specific events to obtain the model accuracy of the intermediate model to be evaluated.

[0062] If the model accuracy of the intermediate model to be evaluated is within the prediction accuracy range, the intermediate model to be evaluated is determined as the target intermediate model.

[0063] If the model accuracy of the intermediate model to be evaluated is outside the prediction accuracy range, the intermediate model to be evaluated is discarded.

[0064] After discarding the intermediate model to be evaluated whose model accuracy is not within the prediction accuracy range, the original model is pruned again based on step 104 until the model accuracy of the intermediate model to be evaluated is within the prediction accuracy range.

[0065] After obtaining the target intermediate model, it is also necessary to determine the quantitative parameters based on the event requirements of the specific event.

[0066] It should be noted that the process of determining the quantization parameter in step 106 mentioned in the embodiment of the present application is as follows:

[0067] Select a data type of a mainstream storage model in the prior art (eg, 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, 8-bit integers and float16 floating point types).

[0068] This application uses GGUF (GPT-Generated Unified Format, a storage file format for large models) format to store the weights of multimodal large models.

[0069] In GGUF, there are different precision quantization parameter selections, and the quantization parameters mainly include:

[0070] q2_k: Certain tensors use a higher precision setting, while others use the 2-bit method to store tensors.

[0071] q3_k_l, q3_k_m, q3_k_s: These formats use different levels of precision on different tensors to achieve a balance between performance and efficiency.

[0072] q4_0: This is the most primitive quantization method, storing tensors with 4 bits of precision.

[0073] q4_1, q4_k_m, q4_k_s: These quantization formats provide different degrees of accuracy and inference speed, suitable for scenarios that need to balance resource usage.

[0074] q5_0, q5_1, q5_k_m, q5_k_s: These formats use more resources and have slower inference speed while ensuring higher accuracy.

[0075] q6_k and q8_0: These methods provide the highest accuracy but use the most resources.

[0076] According to event requirements and hardware resources, the above different quantization parameters are selected to convert the target intermediate model into quantization models under different data types.

[0077] The multimodal large model is converted into a model in GGUF format through the llama.cpp inference engine to achieve model quantization and obtain a quantized model. This conversion method does not require recalibration and retraining of the quantized model through small samples.

[0078] The llama.cpp inference engine is a specially designed inference engine for large models. It is lightweight, has minimal external dependencies, supports multiple platforms, and has efficient inference performance. It can speed up inference and reduce memory usage. Finally, it uses hybrid inference technology that combines CPU and GPU to support inference of large models that exceed the total VRAM capacity.

[0079] In the embodiment of the present application, after obtaining the compressed multimodal large model, the method further includes:

[0080] The executable judgment step is performed on the compressed multi-modal large model. The executable judgment step includes:

[0081] The model accuracy index of the compressed multimodal large model is tested based on the test set to obtain the final accuracy of the specific event in the compressed multimodal large model.

[0082] If the final accuracy is within the predicted accuracy range, the multimodal large model is retained.

[0083] If the final accuracy does not fall within the prediction accuracy range, the multimodal large model is discarded.

[0084] After discarding the multimodal large model whose final accuracy does not fall within the prediction accuracy range, reset the pruning rate and / or select the quantization parameter, and perform pruning and quantization processing on the original model based on steps 104 to 107 until the final accuracy of the compressed multimodal large model is within the prediction accuracy range.

[0085] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in this embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in sequence or in parallel according to the method shown in this embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment).

[0086] In an embodiment of the present application, a system for executing a model compression method for a multimodal large model is also provided. Specifically, the system block diagram is as follows: Figure 3 As shown, including:

[0087] The setting module is used to set the original prediction accuracy of the specific event corresponding to the original model and set the prediction accuracy range of the specific event.

[0088] The pruning module is used to prune the original model, and perform accuracy evaluation and low-rank adaptation fine-tuning on the pruned original model to obtain the target intermediate model.

[0089] The quantization module is used to convert the target intermediate model into quantization models of multiple different data types based on the quantization parameters.

[0090] The selection module is used to determine the target model from multiple quantitative models based on event requirements and obtain a compressed multimodal large model.

[0091] Some modules in the apparatus described in the present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0092] The devices or modules described in the above application embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described in various modules according to their functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0093] The methods, devices or modules described in this application can be implemented in the form of computer-readable program codes. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program codes (such as software or firmware) that can be executed by the (micro) processor, logic gates, switches, application-specific integrated circuits (English: Application Specific Integrated Circuit; Abbreviation: ASIC), programmable logic controllers and embedded microcontrollers. Examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included in it for implementing various functions can also be regarded as structures within the hardware component. Or even, the means for realizing various functions may be regarded as both a software module for realizing the method and a structure within a hardware component.

[0094] An embodiment of the present application also provides a device, comprising: a processor; a memory for storing processor executable instructions; when the processor executes the executable instructions, the model compression method for a multimodal large model as described in the embodiment of the present application is implemented.

[0095] The embodiment of the present application also provides a non-volatile computer-readable storage medium, on which a computer program or instruction is stored. When the computer program or instruction is executed, the model compression method for the multimodal large model as described in the embodiment of the present application is implemented.

[0096] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist independently, or two or more modules may be integrated into one module.

[0097] The above storage media include but are not limited to random access memory (RAM), read-only memory (ROM), cache, hard disk (HDD) or memory card. The memory can be used to store computer program instructions.

[0098] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, or it can be reflected in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0099] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0100] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A model compression method for a multimodal large model, characterized in that: include: Acquire an event dataset, where the event dataset includes a training set and a test set; Based on the test set and the original model, determining the original prediction accuracy of the specific event corresponding to the original model; Determining a prediction accuracy range for the specific event based on the original prediction accuracy and the landing scenario of the specific event; Pruning the original model to obtain an intermediate model to be evaluated; Based on the specific event, the accuracy of the intermediate model to be evaluated is evaluated to determine a target intermediate model, where the target intermediate model is the intermediate model to be evaluated whose model accuracy meets the prediction accuracy range; Determining a quantization parameter according to an event requirement of the specific event, converting the target intermediate model into quantization models under a plurality of different data types based on the quantization parameter, and determining a model accuracy of the quantization model based on the test set; Based on the event requirements, a target model is determined from the multiple quantization models, and the target model is the compressed multimodal large model.

2. The method according to claim 1, characterized in that The original prediction accuracy is within the prediction accuracy range, and the model accuracy of the quantization model is within the prediction accuracy range.

3. The method according to claim 1, characterized in that The pruning of the original model to obtain an intermediate model to be evaluated includes: Splitting the original model into a visual encoder, a linear mapping layer, and a large language model; Obtain all MLP layers of the decoding layer in the large language model, and obtain the average activation value of each channel of the MLP layer; Sort the corresponding channels of the MLP layer from large to small based on the average activation value; The sorted large language model is pruned to obtain a pruned original model, and the pruned original model is determined as the intermediate model to be evaluated.

4. The method according to claim 3, characterized in that The step of pruning the sorted large language model to obtain a pruned original model, and determining the pruned original model as the intermediate model to be evaluated includes: A pruning rate is set, and channels whose average activation value ratio is less than the pruning rate are removed based on the ratio of the pruning rate to obtain a pruned original model, and the pruned original model is determined as the intermediate model to be evaluated.

5. The method according to claim 3, characterized in that: Pruning the sorted large language model to obtain a pruned original model, and determining the pruned original model as the intermediate model to be evaluated, including: Pruning the sorted large language model to obtain a model to be fine-tuned; The model to be fine-tuned is subjected to low-rank adaptive fine-tuning using the training set to obtain the pruned original model, and the pruned original model is determined as the intermediate model to be evaluated.

6. The method according to claim 1, characterized in that The step of performing accuracy evaluation on the intermediate model to be evaluated based on the specific event to determine the target intermediate model includes: Using the intermediate model to be evaluated to evaluate the specific event, and obtaining the model accuracy of the intermediate model to be evaluated; If the model accuracy of the intermediate model to be evaluated is within the prediction accuracy range, determining the intermediate model to be evaluated as the target intermediate model; If the model accuracy of the intermediate model to be evaluated is outside the prediction accuracy range, the intermediate model to be evaluated is discarded.

7. The method according to claim 1, characterized in that After obtaining the compressed multimodal large model, it also includes: Performing an executability judgment step on the compressed multimodal large model; The executability determination step includes: Performing a model accuracy index test on the compressed multimodal large model based on the test set to obtain a final accuracy of the specific event in the compressed multimodal large model; If the final accuracy is within the prediction accuracy range, retaining the multimodal large model; If the final accuracy does not fall within the prediction accuracy range, the multimodal large model is discarded.

8. A system for executing a model compression method for a multimodal large model, characterized in that: include: A setting module, used to set the original prediction accuracy of a specific event corresponding to the original model and to set the prediction accuracy range of the specific event; The pruning module is used to prune the original model, and perform accuracy evaluation and low-rank adaptation fine-tuning on the pruned original model to obtain the target intermediate model; A quantization module, used for converting the target intermediate model into quantization models of multiple different data types based on quantization parameters; The selection module is used to determine the target model from the multiple quantization models based on event requirements to obtain a compressed multimodal large model.

9. A device for executing a model compression method for a multimodal large model, characterized in that: include: processor; a memory for storing processor-executable instructions; When the processor executes the executable instructions, it implements the model compression method for a multimodal large model as described in any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that: The device comprises a computer program or an instruction for storing the computer program or the instruction, which, when executed, enables the model compression method for a multimodal large model as claimed in any one of claims 1 to 7 to be implemented.

Images