Multi-modal large model pruning method and device, computer equipment and storage medium
By generating heating maps and calculating the thermal contribution of filters for sorting and pruning, the multimodal large model is optimized, solving the problem of efficient operation and deployment in resource-constrained environments, and achieving efficient operation and accuracy of the model.
Patent Information
- Application Number
- CN202411057369.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-19
AI Technical Summary
Multimodal large models are difficult to operate efficiently and be effectively deployed and maintained in power grids due to their large scale, high computational resource and storage requirements, and limited resources.
Heating maps are generated using a class activation mapping method based on enhanced features and gradients. The batch total thermal contribution of filters within a multimodal large model is calculated, and pruning is performed according to the ranking to optimize the model structure.
It reduces the number of model parameters and computational complexity, improves the model's operating efficiency and deployment and maintenance capabilities in resource-constrained environments, reduces training and inference costs, and enhances the model's transparency and prediction accuracy.
Smart Images

Figure CN118982058A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a large multimodal model, and more specifically to a large multimodal model pruning method, device, computer equipment and storage medium. Background Art
[0002] After the power grid enters the strong coupling mode, its operating data becomes more complex and diverse. Multi-source information such as load, weather, and equipment status are intertwined, resulting in a significant increase in the dimension and complexity of the data. In this case, the monitoring and regulation of the power system faces huge challenges. How to effectively extract useful information from huge and complex data, make accurate predictions and fault diagnosis, has become a key issue in power grid operation; the introduction of multimodal large models provides a new solution for power grid data processing. This model can simultaneously process structured and unstructured data, including multiple data types such as power grid load, equipment operation, weather, and images. By integrating these multi-source information, the multimodal large model can describe the operating status and changing trends of the power grid more comprehensively and accurately, especially in fault diagnosis, load forecasting and status assessment, the multimodal large model has shown strong advantages; however, as the model size increases, the demand for computing resources and storage space is also increasing. Multimodal large models are usually large in scale and require a lot of computing resources for training and inference, which exceeds the current computing power of the power grid monitoring system, resulting in high deployment and operation costs; moreover, the multimodal large model has a huge number of parameters and requires a lot of storage space to store the model itself and its related data, making it difficult to effectively deploy and maintain in an environment with limited resources.
[0003] Therefore, it is necessary to design a new method to prune the multimodal large models applied to power grid data so that the multimodal large models can be efficiently operated and effectively deployed and maintained in resource-limited environments. Summary of the invention
[0004] The purpose of the present invention is to overcome the defects of the prior art and provide a multimodal large model pruning method, device, computer equipment and storage medium.
[0005] To achieve the above object, the present invention adopts the following technical solution: a multimodal large model pruning method, comprising:
[0006] Obtaining multi-modal large models applied to power grid data and raw image data;
[0007] Inputting the raw image data into a multimodal macro model to obtain a feature map;
[0008] Generate a heating map according to the feature map using a class activation mapping method based on enhanced features and gradients;
[0009] Calculating the batch total heat contribution of all filters in the multimodal large model according to the heating map in combination with the original image data;
[0010] The total heat contributions of the batches are ranked, pruned according to the rankings, and the weights of the multimodal large model are updated to obtain a final multimodal large model.
[0011] A further technical solution is: generating a heating map according to the feature map using a class activation mapping method based on enhanced features and gradients, including:
[0012] Performing image classification on the feature map, and calculating the score value corresponding to the image category and the convolution feature map;
[0013] Calculating gradient difference information corresponding to the features of the feature map according to the score value and the convolution feature map;
[0014] Calculating mean information according to the convolution feature map, and performing weighted summation of the mean information and the original features of the feature map to obtain enhanced feature values;
[0015] A weighted combination is performed based on the enhanced eigenvalues and gradient difference information, and a heating map is generated through the ReLU activation function.
[0016] A further technical solution is: the step of calculating the gradient difference information corresponding to the feature of the feature map according to the score value and the convolution feature map includes:
[0017] use Calculate the gradient difference information corresponding to the features of the feature map, where: is the gradient difference information; y c is the score value; is a convolutional feature map, representing the data at the kth channel position coordinate (i, j) in the feature map.
[0018] A further technical solution is: calculating mean information according to the convolution feature map, and performing weighted summation of the mean information and the original features of the feature map to obtain enhanced feature values, including:
[0019] According to the convolution feature map, The feature map is enhanced to obtain the enhanced feature value, where: is the enhanced eigenvalue; is the convolution feature map; Z is the product of i and j, where i and j are coordinates.
[0020] A further technical solution is: calculating the batch total heat contribution of all filters in the multimodal large model according to the heating map combined with the original image data, including:
[0021] use Calculate the batch total heat contribution of all filters in the multimodal large model, where FHC k represents the thermal contribution of the k-th filter, which is determined by the gradient difference information in the heating map; batch represents the number of original image data processed in each batch; t represents the total thermal contribution of the current k-th filter batch.
[0022] A further technical solution is: the total heat contribution of the batch is sorted, pruning is performed according to the ranking, and the weight of the multimodal large model is updated to obtain the final multimodal large model, including:
[0023] Sorting the filters in a set order according to the total heat contribution of the batch to obtain a sorting result; wherein the set order includes an order from high to low or from low to high;
[0024] The corresponding filters in the sorting result are deleted according to the network pruning rate, and the weights of the multimodal large model are updated to obtain the final multimodal large model.
[0025] A further technical solution is: deleting the corresponding filter in the sorting result according to the network pruning rate, and updating the weight of the multimodal large model to obtain the final multimodal large model, including:
[0026] Determining the number of filters to be deleted according to the network pruning rate to obtain a target pruning quantity;
[0027] Deleting a low-ranked target reduction number of filters from the sorted results to obtain remaining filters;
[0028] The weights of the multimodal large model are updated according to the remaining filters to obtain the final multimodal large model.
[0029] The present invention also provides a multi-modal large model pruning device, comprising:
[0030] An acquisition unit, used for acquiring a multi-modal large model applied to power grid data and original image data;
[0031] A feature map generating unit, used for inputting the original image data into the multimodal large model to obtain a feature map;
[0032] A heating map generating unit, configured to generate a heating map according to the feature map by using a class activation mapping method based on enhanced features and gradients;
[0033] A batch total heat contribution calculation unit, used to calculate the batch total heat contribution of all filters in the multimodal large model according to the heating map in combination with the original image data;
[0034] A pruning unit is used to rank the total heat contribution of the batch, prune according to the ranking, and update the weight of the multimodal large model to obtain a final multimodal large model.
[0035] The present invention further provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.
[0036] The present invention also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements the above method when executed by a processor.
[0037] The beneficial effects of the present invention compared with the prior art are as follows: the present invention uses a large multimodal model of power grid data and original image data, utilizes a feature map and adopts a class activation mapping method based on enhanced features and gradients to generate a heating map, calculates the thermal contribution of the filter, optimizes the model based on sorting pruning, and implements pruning of the large multimodal model applied to power grid data, so that the large multimodal model can be efficiently operated and effectively deployed and maintained in an environment with limited resources.
[0038] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0040] Figure 1 A schematic diagram of an application scenario of a multimodal large model pruning method provided by an embodiment of the present invention;
[0041] Figure 2 A schematic diagram of a flow chart of a multimodal large model pruning method provided in an embodiment of the present invention;
[0042] Figure 3 A schematic diagram of a sub-process of a multi-modal large model pruning method provided in an embodiment of the present invention;
[0043] Figure 4 A schematic diagram of a sub-process of a multi-modal large model pruning method provided in an embodiment of the present invention;
[0044] Figure 5 A schematic diagram of a sub-process of a multi-modal large model pruning method provided in an embodiment of the present invention;
[0045] Figure 6 A schematic diagram of a flow chart of a class activation mapping method based on enhanced features and gradients provided in an embodiment of the present invention;
[0046] Figure 7 A comparison chart of the effects of Grad-CAM and EFG-CAM provided by an embodiment of the present invention on each layer of a neural network;
[0047] Figure 8 A schematic diagram of a competition mechanism within a filter bank provided by an embodiment of the present invention;
[0048] Fig. 9 Schematic diagram of the pruning process provided by an embodiment of the present invention
[0049] Fig.10 A schematic block diagram of a multi-modal large model pruning device provided in an embodiment of the present invention;
[0050] Fig.11 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0053] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0054] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0055] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of an application scenario of the multimodal large model pruning method provided in an embodiment of the present invention. Figure 2 A schematic flow chart of a multimodal large model pruning method provided in an embodiment of the present invention. The multimodal large model pruning method is applied to a server. The server interacts with a terminal for data, and obtains original image data through the terminal. The server uses a class activation mapping technology that strengthens features and gradients to realize visualization of each convolutional layer of a convolutional neural network. By injecting gradient information into the visualization module layer by layer, and using the gradient difference of the current feature channel as a weight, the effectiveness of the gradient information is enhanced. In addition, according to the mean information of the feature map channel, the important channels in the original feature map are enhanced, so that the enhanced feature channels can better retain the heated areas in the feature map, while suppressing the non-heated areas in the non-important channels; by sorting and screening the filters according to the total thermal contribution of the batch, the mechanism reduces the influence of a single data sample on the pruning decision, improves the stability and consistency of the pruning results, and ensures that the model can maintain good performance on a variety of data sets, thereby enhancing the stability and effect of the pruning operation.
[0056] Figure 2 is a flow chart of a multi-modal large model pruning method provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S150.
[0057] S110 , obtaining a multimodal large model applied to power grid data and original image data.
[0058] In this embodiment, the multimodal large model refers to a convolutional neural network model applied to power grid data; the original image data refers to the original image data of the power grid field.
[0059] S120, inputting the original image data into a multimodal large model to obtain a feature map.
[0060] In this embodiment, the feature map refers to the image generated after the original image data is input into the multimodal large model and passes through the filter.
[0061] S130, generating a heating map according to the feature map by using a class activation mapping method based on enhanced features and gradients.
[0062] In this embodiment, the heating map shows the density or intensity changes in the data through changes in color or brightness, which helps to understand the important features and decision-making basis of the model when processing data.
[0063] like Figure 6As shown in the figure, by injecting gradient information layer by layer in the visualization process of the convolutional neural network, the gradient difference is used as the weight of the current feature channel to enhance the effectiveness of the gradient information. In addition, the algorithm also enhances the important channels in the original feature map according to the mean information of the feature map channel, thereby ensuring that the enhanced feature channel better retains the key areas in the feature map while suppressing useless information in non-key channels.
[0064] In one embodiment, see Figure 3 , the above-mentioned step S130 may include steps S131 to S135.
[0065] S131, performing image classification on the feature map, and calculating the score value corresponding to the image category and the convolution feature map.
[0066] In this embodiment, gradient information can be used to calculate the gradients of each convolutional layer during back propagation. These gradients can help understand how the network generates activation maps or feature maps for a specific class based on the input image. Gradient information can be used in methods such as EFG-CAM (Enhanced Feature and Gradient Class Activation Mapping) to visualize the network's attention to and response areas for different features in the image.
[0067] The score value corresponding to the image category refers to the probability or confidence that the original image data belongs to each category; the convolutional feature map refers to the data at the kth channel position coordinate (i, j) in the feature map.
[0068] S132. Calculate the gradient difference information corresponding to the features of the feature map according to the score value and the convolution feature map.
[0069] In this embodiment, the gradient difference information refers to the gradient weight of the current feature channel.
[0070] Specifically, using Calculate the gradient difference information corresponding to the features of the feature map, where: is the gradient difference information; y c is the score value; is a convolutional feature map, representing the data at the kth channel position coordinate (i, j) in the feature map.
[0071] S133. Calculate mean information according to the convolution feature map, and perform weighted summation on the mean information and the original features of the feature map to obtain enhanced feature values.
[0072] In this embodiment, the enhanced feature value refers to the value obtained after the original feature in the feature map is enhanced. Among them, the mean information refers to the weighted average value in the convolution feature map, which is used to enhance the original feature.
[0073] Specifically, according to the convolution feature map, The feature map is enhanced to obtain the enhanced feature value, where: is the enhanced eigenvalue; is the convolution feature map; Z is the product of i and j, where i and j are coordinates.
[0074] S134, performing weighted combination according to the enhanced eigenvalues and gradient difference information, and generating a heating map through a ReLU activation function.
[0075] In this embodiment, the Calculate the batch total heat contribution of all filters in the multimodal large model, where FHC k represents the thermal contribution of the k-th filter, which is determined by the gradient difference information in the heating map; batch represents the number of original image data processed in each batch; t represents the total thermal contribution of the current k-th filter batch.
[0076] Compared with traditional Grad-CAM, shallow Grad-CAM has a problem that the heat distribution of the heating map may blur the relationship between the learned features and label information. Specifically, shallow Grad-CAM may be too scattered when generating the heating map, making it difficult to clearly show which features have the greatest impact on the classification label.
[0077] However, EFG-CAM, the class activation mapping method based on enhanced features and gradients, partially offsets the negative impact of the gradient vanishing phenomenon on the class activation map by enhancing the gradient weights. This means that in EFG-CAM, especially in the shallow layers of the network, the learning of edge features that play an important role in label classification can be observed. Specifically, EFG-CAM can more clearly highlight some edge features that are crucial for label classification in shallow networks.
[0078] This improvement can be achieved through detailed comparative analysis, such as Figure 7 As shown in the figure, we further demonstrate the advantages and improvements of EFG-CAM over traditional Grad-CAM when processing shallow features.
[0079] S140. Calculate the total batch thermal contribution of all filters in the multimodal large model according to the heating map in combination with the original image data.
[0080] In this embodiment, the total heat contribution of a batch refers to the total heat contribution value of all filters in a batch processing in a multimodal large model, which is calculated by combining the heating map with the original image data.
[0081] In order to improve the stability of thermal contribution, the traditional method uses only a small amount of original image data to calculate the thermal contribution to evaluate the performance of the filter, which may be affected by the data label attributes, resulting in unstable performance. However, it is costly to update the thermal contribution of the filter using a complete data set. Therefore, this embodiment adopts an intra-filter group competition mechanism to enhance the stability of the thermal contribution of the filter and reduce the pruning cost.
[0082] like Figure 8 As shown in Figure 1, in this competition mechanism, filters are organized into filter groups, which compete based on their thermal contributions on a certain batch of input image data. The filters that succeed in the competition will have their thermal contributions accumulated, while the filter groups that fail in the competition may be pruned. This competition process not only helps stabilize the performance evaluation of the filter, but also reduces the complexity and computational cost of the network model without sacrificing performance.
[0083] Specifically, using Calculate the batch total heat contribution of all filters in the multimodal large model, where FHC k represents the thermal contribution of the k-th filter, which is determined by the gradient difference information in the heating map; batch represents the number of original image data processed in each batch; t represents the total thermal contribution of the current k-th filter batch.
[0084] S150, sorting the total heat contributions of the batches, performing pruning according to the rankings, and updating the weights of the multimodal large model to obtain a final multimodal large model.
[0085] In one embodiment, see Figure 4 , the above-mentioned step S150 may include steps S151 to S152.
[0086] S151. Sort the filters in a set order according to the total heat contribution of the batch to obtain a sorting result; wherein the set order includes an order from high to low or an order from low to high.
[0087] In this embodiment, the filters in the current batch are sorted according to their thermal contributions in a set order (which can be from high to low or from low to high); they are arranged by calculating the contribution of each filter to the overall performance so that subsequent pruning operations can be performed more efficiently.
[0088] S152. Delete the corresponding filter in the sorting result according to the network pruning rate, and update the weight of the multimodal large model to obtain the final multimodal large model.
[0089] In this embodiment, first, the number of filters to be deleted is determined according to a preset network pruning rate to achieve a target pruning rate or pruning effect; lower-ranked filters are selected from the sorting results, and filters that match the target pruning number are deleted to ensure that filters that have a greater impact on performance are retained while reducing the complexity of the overall model; the remaining filters will have their weights recalculated to reflect the changes in the model structure after the deletion operation, ensuring that the model can still maintain efficient operating performance and accuracy after pruning.
[0090] In one embodiment, see Figure 5 , the above-mentioned step S152 may include steps S1521 to S1523.
[0091] S1521. Determine the number of filters to be deleted according to the network pruning rate to obtain a target pruning quantity.
[0092] In this embodiment, the target pruning quantity refers to the number of filters that need to be deleted during the pruning process.
[0093] S1522: Delete the filters with the target reduction quantity having low rankings from the sorting results to obtain the remaining filters.
[0094] In this embodiment, filters with lower rankings are selected and deleted from the sorting results, and the number of deleted filters is the target deletion number.
[0095] S1523. Update the weights of the multimodal large model according to the remaining filters to obtain a final multimodal large model.
[0096] In this embodiment, the remaining filters will recalculate and update weights according to their new structure after pruning to ensure that the model can still perform tasks effectively after pruning.
[0097] like Fig. 9 As shown in Figure 1, starting from the original benchmark model W, by calculating the thermal contribution of each filter and sorting them by importance, the filters with smaller contributions are eliminated. This can determine the key feature channels that affect network prediction and retain filters with good performance. Finally, the pruned network parameters are updated to the new small network W*, and the pruned network is pruned according to the given pruning rate P and the number of iterations Epoch. max Adjustments are made to achieve network compression and optimization; the input is the original benchmark model W; the network pruning rate P; the number of iterations Epoch max; The output is the compressed model W*, where the original benchmark model W is the multimodal large model; the compressed model W* is the final multimodal large model; specifically, for each epoch, use EFG-CAM to obtain the heating map of the output feature map; calculate the total batch thermal contribution of each filter; use the intra-group competition mechanism to perform cumulative calculation of thermal contributions; prune the filter groups that failed the competition, and update the weights of the entire network model so that the pruning rate is P.
[0098] By dynamically adjusting and optimizing the number and weights of filters, we can optimize and streamline large multimodal models to reduce computing and storage costs while maintaining performance. We also make the originally unexplainable black box model transparent, so that we can clearly understand the decision-making process of each convolutional layer, thereby guiding the pruning of the model.
[0099] Traditional large multimodal models process information from multiple data sources, so the number of parameters and computing requirements are huge, making it difficult to operate efficiently in resource-limited environments; the training and inference process of large multimodal models is time-consuming, affecting application scenarios with high real-time requirements; large multimodal models usually have strong black box characteristics, making their prediction results difficult to interpret, which is not conducive to the maintenance and decision-making of power systems; in power systems, the computing resources and storage space of many devices are limited, making it difficult to deploy complex large multimodal models; overly large models are prone to overfitting on training data and are difficult to cope with diverse power data. The method of this embodiment uses a precise pruning strategy to significantly reduce the number of model parameters and computational complexity, so that large multimodal models can run effectively on resource-constrained devices; by reducing redundant parameters, the training and inference speed of the model is accelerated to meet the needs of real-time monitoring and online prediction; through methods such as visual class activation mapping, the transparency of the model decision-making process is improved to help understand the behavior of the model and make more accurate maintenance and operation decisions; through effective pruning methods, the storage and computing requirements of the model are reduced, so that it can be easily deployed on edge devices and sensor nodes to achieve distributed intelligent monitoring; through pruning, the complexity of the model is reduced to prevent overfitting, and the generalization ability of the model when processing multimodal data is improved to ensure its reliability and accuracy in different application scenarios.
[0100] Therefore, the method of this embodiment not only optimizes the performance of the multimodal large model, but also significantly improves its application value in the power system, and solves the challenges of the multimodal large model in computing resources, training speed, interpretability and deployment of resource-constrained devices.
[0101] The above-mentioned multimodal large model pruning method uses the multimodal large model of power grid data and the original image data, uses the feature map to generate a heating map using a class activation mapping method based on enhanced features and gradients, and calculates the thermal contribution of the filter. It optimizes the model based on sorting pruning to achieve pruning of the multimodal large model applied to the power grid data, so that the multimodal large model can operate efficiently and be effectively deployed and maintained in an environment with limited resources.
[0102] Fig.10 is a schematic block diagram of a multi-modal large model pruning device 300 provided in an embodiment of the present invention. Fig.10 As shown, corresponding to the above multimodal large model pruning method, the present invention also provides a multimodal large model pruning device 300. The multimodal large model pruning device 300 includes a unit for executing the above multimodal large model pruning method, and the device can be configured in a server. Fig.10 The multimodal large model pruning device 300 includes an acquisition unit 301, a feature map generation unit 302, a heating map generation unit 303, a batch total heat contribution calculation unit 304 and a pruning unit 305.
[0103] An acquisition unit 301 is used to acquire a multimodal large model and original image data applied to power grid data; a feature map generation unit 302 is used to input the original image data into the multimodal large model to obtain a feature map; a heating map generation unit 303 is used to generate a heating map according to the feature map using a class activation mapping method based on enhanced features and gradients; a batch total heat contribution calculation unit 304 is used to calculate the batch total heat contribution of all filters in the multimodal large model according to the heating map combined with the original image data; a pruning unit 305 is used to sort the batch total heat contribution, perform pruning according to the ranking, and update the weight of the multimodal large model to obtain the final multimodal large model.
[0104] In one embodiment, the heating map generating unit 303 includes a classification subunit, a gradient difference calculating subunit, a strengthening subunit and a combining subunit.
[0105] A classification subunit is used to perform image classification on the feature map, and calculate the score value corresponding to the image category and the convolution feature map; a gradient difference calculation subunit is used to calculate the gradient difference information corresponding to the feature of the feature map according to the score value and the convolution feature map; an enhancement subunit is used to calculate the mean information according to the convolution feature map, and perform weighted summation of the mean information and the original features of the feature map to obtain the enhanced feature value; a combination subunit is used to perform weighted combination according to the enhanced feature value and the gradient difference information, and generate a heating map through the ReLU activation function.
[0106] In one embodiment, the gradient difference calculation subunit is used to adopt Calculate the gradient difference information corresponding to the features of the feature map, where: is the gradient difference information; y c is the score value; is a convolutional feature map, representing the data at the kth channel position coordinate (i, j) in the feature map.
[0107] In one embodiment, the strengthening subunit is used to adopt The feature map is enhanced to obtain the enhanced feature value, where: is the enhanced eigenvalue; is the convolution feature map; Z is the product of i and j, where i and j are coordinates.
[0108] In one embodiment, the batch total heat contribution calculation unit 304 is used to use Calculate the batch total heat contribution of all filters in the multimodal large model, where FHC k represents the thermal contribution of the k-th filter, which is determined by the gradient difference information in the heating map; batch represents the number of original image data processed in each batch; t represents the total thermal contribution of the current k-th filter batch.
[0109] In one embodiment, the pruning unit 305 includes a sorting subunit and a deleting subunit.
[0110] A sorting subunit is used to sort the filters in a set order according to the total heat contribution of the batch to obtain a sorting result; wherein the set order includes an order from high to low or an order from low to high; a deleting subunit is used to delete the corresponding filter in the sorting result according to the network pruning rate, and update the weight of the multimodal large model to obtain the final multimodal large model.
[0111] In one embodiment, the deleting subunit includes a number determination module, a filter deletion module and an update module.
[0112] A number determination module is used to determine the number of filters that need to be deleted according to the network pruning rate to obtain a target deletion number; a filter deletion module is used to delete low-ranked filters of the target deletion number from the sorting results to obtain remaining filters; an update module is used to update the weights of the multimodal large model according to the remaining filters to obtain a final multimodal large model.
[0113] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned multimodal large model pruning device 300 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.
[0114] The multi-modal large model pruning device 300 can be implemented in the form of a computer program. The computer program can be used in Fig.11 Runs on the computer device shown.
[0115] See also Fig.11 , Fig.11 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.
[0116] See also Fig.11 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .
[0117] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a multi-modal large model pruning method.
[0118] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .
[0119] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a multimodal large model pruning method.
[0120] The network interface 505 is used to communicate with other devices over the network. Fig.11 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0121] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:
[0122] Acquire a multimodal large model applied to power grid data and original image data; input the original image data into the multimodal large model to obtain a feature map; generate a heating map based on the feature map using a class activation mapping method based on enhanced features and gradients; calculate the batch total thermal contribution of all filters in the multimodal large model based on the heating map combined with the original image data; sort the batch total thermal contribution, perform pruning according to the ranking, and update the weight of the multimodal large model to obtain the final multimodal large model.
[0123] In one embodiment, when the processor 502 implements the step of generating a heating map according to the feature map using a class activation mapping method based on enhanced features and gradients, the processor 502 specifically implements the following steps:
[0124] The feature map is subjected to image classification, and the score value and the convolution feature map corresponding to the image category are calculated; the gradient difference information corresponding to the feature of the feature map is calculated according to the score value and the convolution feature map; the mean information is calculated according to the convolution feature map, and the mean information is weightedly summed with the original features of the feature map to obtain the enhanced feature value; the enhanced feature value and the gradient difference information are weightedly combined, and a heating map is generated through a ReLU activation function.
[0125] In one embodiment, when the processor 502 implements the step of calculating the gradient difference information corresponding to the feature of the feature map according to the score value and the convolution feature map, it specifically implements the following steps:
[0126] use Calculate the gradient difference information corresponding to the features of the feature map, where: is the gradient difference information; y c is the score value; is a convolutional feature map, representing the data at the kth channel position coordinate (i, j) in the feature map.
[0127] In one embodiment, when the processor 502 implements the step of calculating mean information according to the convolution feature map and performing weighted summation of the mean information and the original features of the feature map to obtain the enhanced feature value, the processor 502 specifically implements the following steps:
[0128] According to the convolution feature map, The feature map is enhanced to obtain the enhanced feature value, where: is the enhanced eigenvalue; is the convolution feature map; Z is the product of i and j, where i and j are coordinates.
[0129] In one embodiment, when the processor 502 implements the step of calculating the batch total heat contribution of all filters in the multimodal large model according to the heating map combined with the original image data, the processor 502 specifically implements the following steps:
[0130] use Calculate the batch total heat contribution of all filters in the multimodal large model, where FHC k represents the thermal contribution of the k-th filter, which is determined by the gradient difference information in the heating map; batch represents the number of original image data processed in each batch; t represents the total thermal contribution of the current k-th filter batch.
[0131] In one embodiment, when the processor 502 implements the step of sorting the total heat contribution of the batch, pruning according to the ranking, and updating the weight of the multimodal large model to obtain the final multimodal large model, the processor 502 specifically implements the following steps:
[0132] The filters are sorted in a set order according to the total thermal contribution of the batch to obtain a sorting result; wherein the set order includes an order from high to low or an order from low to high; the corresponding filters in the sorting result are deleted according to the network pruning rate, and the weights of the multimodal large model are updated to obtain the final multimodal large model.
[0133] In one embodiment, when the processor 502 implements the step of deleting the corresponding filter in the sorting result according to the network pruning rate and updating the weight of the multimodal large model to obtain the final multimodal large model, the processor 502 specifically implements the following steps:
[0134] The number of filters to be deleted is determined according to the network pruning rate to obtain a target pruning number; filters with low rankings and a target pruning number are deleted from the sorting results to obtain remaining filters; and the weights of the multimodal large model are updated according to the remaining filters to obtain a final multimodal large model.
[0135] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0136] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.
[0137] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:
[0138] Acquire a multimodal large model applied to power grid data and original image data; input the original image data into the multimodal large model to obtain a feature map; generate a heating map based on the feature map using a class activation mapping method based on enhanced features and gradients; calculate the batch total thermal contribution of all filters in the multimodal large model based on the heating map combined with the original image data; sort the batch total thermal contribution, perform pruning according to the ranking, and update the weight of the multimodal large model to obtain the final multimodal large model.
[0139] In one embodiment, when the processor executes the computer program to implement the step of generating a heating map according to the feature map using a class activation mapping method based on enhanced features and gradients, the processor specifically implements the following steps:
[0140] The feature map is subjected to image classification, and the score value and the convolution feature map corresponding to the image category are calculated; the gradient difference information corresponding to the feature of the feature map is calculated according to the score value and the convolution feature map; the mean information is calculated according to the convolution feature map, and the mean information is weightedly summed with the original features of the feature map to obtain the enhanced feature value; the enhanced feature value and the gradient difference information are weightedly combined, and a heating map is generated through a ReLU activation function.
[0141] In one embodiment, when the processor executes the computer program to implement the step of calculating the gradient difference information corresponding to the feature of the feature map according to the score value and the convolution feature map, the processor specifically implements the following steps:
[0142] use Calculate the gradient difference information corresponding to the features of the feature map, where: is the gradient difference information; y c is the score value; is a convolutional feature map, representing the data at the kth channel position coordinate (i, j) in the feature map.
[0143] In one embodiment, when the processor executes the computer program to implement the step of calculating mean information according to the convolution feature map and weighted summing the mean information with the original features of the feature map to obtain the enhanced feature value, the following steps are specifically implemented:
[0144] According to the convolution feature map, The feature map is enhanced to obtain the enhanced feature value, where: is the enhanced eigenvalue; is the convolution feature map; Z is the product of i and j, where i and j are coordinates.
[0145] In one embodiment, when the processor executes the computer program to implement the step of calculating the batch total thermal contribution of all filters in the multimodal large model according to the heating map combined with the original image data, the processor specifically implements the following steps:
[0146] use Calculate the batch total heat contribution of all filters in the multimodal large model, where FHC k represents the thermal contribution of the k-th filter, which is determined by the gradient difference information in the heating map; batch represents the number of original image data processed in each batch; t represents the total thermal contribution of the current k-th filter batch.
[0147] In one embodiment, when the processor executes the computer program to implement the steps of ranking the total heat contribution of the batch, pruning according to the ranking, and updating the weight of the multimodal large model to obtain the final multimodal large model, the following steps are specifically implemented:
[0148] Sorting the filters in a set order according to the total heat contribution of the batch to obtain a sorting result; wherein the set order includes an order from high to low or from low to high;
[0149] The corresponding filters in the sorting result are deleted according to the network pruning rate, and the weights of the multimodal large model are updated to obtain the final multimodal large model.
[0150] In one embodiment, when the processor executes the computer program to implement the step of deleting the corresponding filter in the sorting result according to the network pruning rate and updating the weight of the multimodal large model to obtain the final multimodal large model, the following steps are specifically implemented:
[0151] The number of filters to be deleted is determined according to the network pruning rate to obtain a target pruning number; filters with low rankings and a target pruning number are deleted from the sorting results to obtain remaining filters; and the weights of the multimodal large model are updated according to the remaining filters to obtain a final multimodal large model.
[0152] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.
[0153] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0154] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0155] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.
[0157] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A multimodal large model pruning method, characterized in that: include: Obtaining multi-modal large models applied to power grid data and raw image data; Inputting the raw image data into a multimodal macro model to obtain a feature map; Generate a heating map according to the feature map using a class activation mapping method based on enhanced features and gradients; Calculating the batch total heat contribution of all filters in the multimodal large model according to the heating map in combination with the original image data; The total heat contributions of the batches are ranked, pruned according to the rankings, and the weights of the multimodal large model are updated to obtain a final multimodal large model.
2. The multimodal large model pruning method according to claim 1, characterized in that: The step of generating a heating map according to the feature map by using a class activation mapping method based on enhanced features and gradients includes: Performing image classification on the feature map, and calculating the score value corresponding to the image category and the convolution feature map; Calculating gradient difference information corresponding to the features of the feature map according to the score value and the convolution feature map; Calculating mean information according to the convolution feature map, and performing weighted summation of the mean information and the original features of the feature map to obtain enhanced feature values; A weighted combination is performed based on the enhanced eigenvalues and gradient difference information, and a heating map is generated through the ReLU activation function.
3. The multimodal large model pruning method according to claim 2, characterized in that: The calculating the gradient difference information corresponding to the feature of the feature map according to the score value and the convolution feature map includes: use Calculate the gradient difference information corresponding to the features of the feature map, where: is the gradient difference information; y c is the score value; is a convolutional feature map, representing the data at the kth channel position coordinate (i, j) in the feature map.
4. The multimodal large model pruning method according to claim 3, characterized in that: The calculating mean information according to the convolution feature map and performing weighted summation of the mean information and the original features of the feature map to obtain enhanced feature values includes: According to the convolution feature map, The feature map is enhanced to obtain the enhanced feature value, where: is the enhanced eigenvalue; is the convolution feature map; Z is the product of i and j, where i and j are coordinates.
5. The multimodal large model pruning method according to claim 4, characterized in that: The step of calculating the batch total heat contribution of all filters in the multimodal large model according to the heating map in combination with the original image data includes: use Calculate the batch total heat contribution of all filters in the multimodal large model, where FHC k represents the thermal contribution of the k-th filter, which is determined by the gradient difference information in the heating map; batch represents the number of original image data processed in each batch; t represents the total thermal contribution of the current k-th filter batch.
6. The multimodal large model pruning method according to claim 1, characterized in that: The step of ranking the total heat contributions of the batches, performing pruning according to the rankings, and updating the weights of the multimodal large model to obtain a final multimodal large model includes: Sorting the filters in a set order according to the total heat contribution of the batch to obtain a sorting result; wherein the set order includes an order from high to low or from low to high; The corresponding filters in the sorting result are deleted according to the network pruning rate, and the weights of the multimodal large model are updated to obtain the final multimodal large model.
7. The multimodal large model pruning method according to claim 6, characterized in that: The step of deleting the corresponding filter in the sorting result according to the network pruning rate and updating the weight of the multimodal large model to obtain the final multimodal large model includes: Determining the number of filters to be deleted according to the network pruning rate to obtain a target pruning quantity; Deleting a low-ranked target reduction number of filters from the sorted results to obtain remaining filters; The weights of the multimodal large model are updated according to the remaining filters to obtain the final multimodal large model.
8. A multi-modal large model pruning device, characterized in that: include: An acquisition unit, used for acquiring a multi-modal large model applied to power grid data and original image data; A feature map generating unit, used for inputting the original image data into the multimodal large model to obtain a feature map; A heating map generating unit, configured to generate a heating map according to the feature map by using a class activation mapping method based on enhanced features and gradients; A batch total heat contribution calculation unit, used to calculate the batch total heat contribution of all filters in the multimodal large model according to the heating map in combination with the original image data; A pruning unit is used to rank the total heat contribution of the batch, prune according to the ranking, and update the weight of the multimodal large model to obtain a final multimodal large model.
9. A computer device, characterized in that: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.
10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
RGB-D multi-mode fusion person detection method based on asymmetric double-flow network
CN110956094A
Neural network filter pruning method based on batch feature heat map
CN112561054A
Class-based filter pruning method
CN113850373A
Filter pruning method based on output activation mapping, image classification system and edge device
CN117035044A
Construction material visual inventory method based on lightweight deep neural network
CN117197649A
Cited By
Elevator lever drum brake fault detection method, device and equipment
CN121456527A
Plastic architecture design and pruning method based on heterogeneous experts and related device
CN121581103A
