Multi-modal network pruning method and system for realizing target perception of mobile terminal

By defining fusion matching metrics and non-structural pruning methods in a multimodal perception network, the problem of difficulty in taking into account the speed and accuracy of multimodal perception networks when deploying on mobile devices is solved, and the balance of model performance is reduced Calculate costs, providing new technical support for the fields of autonomous driving and robotics.

CN118917375APending Publication Date: 2024-11-08NORTHWESTERN POLYTECHNICAL UNIV
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410920160.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

When deploying multimodal perception networks on resource-constrained mobile devices, they face the problem of difficulty in taking into account both speed and accuracy. The existing single-modal pruning methods are not effective when compressing multimodal fusion models because they do not Taking into account the multimodal fusion mechanism, the model performance declined after pruning.

Method used

The evaluation module defines the fusion matching metrics during multimodal information fusion, measures the matching correlation between multimodal fusion features and singlemodal fusion features before fusion, and performs non-structural pruning based on this standard, and controls the module Perform pruning operations within the preset number of iterations until the model converges, ensuring balanced model performance reduces computational costs.

Benefits of technology

Significantly reducing computing costs while maintaining or improving model performance, it provides the possibility for the efficient deployment of multimodal perception networks on edge devices, and promotes the development of fields such as autonomous driving and robotics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917375A_ABST
    Figure CN118917375A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-modal network pruning method and system for realizing mobile terminal target perception. The method comprises the following steps: defining a fusion matching measurement standard during multi-modal information fusion through an evaluation module; wherein the fusion matching measurement standard is used for measuring the matching correlation between the multi-modal fusion feature and the single-modal feature before fusion; performing non-structural pruning on a corresponding modal network layer in the initialized multi-modal perception model through a pruning module based on the fusion matching measurement standard; and controlling the evaluation module and the pruning module to perform pruning operation on the modal network layer within a preset number of iterations through a control module until the model converges. According to the method, unimportant weights can be effectively trimmed in the multi-modal sensing network, the features before and after fusion are balanced, and finally real-time and high-precision deployment of the multi-modal network at the mobile terminal is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a multimodal network optimization technology in the field of deep learning, and relates to but is not limited to a multimodal network pruning method and system for realizing mobile terminal target perception. Background Art

[0002] In autonomous driving and robotics, multimodal perception networks achieve a more comprehensive understanding of the environment by combining data from different sensors (such as cameras and lidar). However, these networks face the challenge of high computational cost when deployed on resource-constrained edge devices due to the exponential growth in the number of parameters.

[0003] Network pruning is an important method to alleviate the above problems, but existing single-modal pruning methods (SOTA) do not work well when compressing multimodal fusion models because they do not take into account the multimodal fusion mechanism, resulting in a decrease in model performance after pruning. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a multimodal network pruning method and system for realizing mobile terminal target perception, which at least solves the problem of difficulty in balancing speed and accuracy when a multimodal perception network is deployed on a mobile terminal with limited resources.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a multimodal network pruning method for realizing mobile terminal target perception, comprising:

[0007] The evaluation module is used to define a fusion matching metric when fusing multimodal information; wherein the fusion matching metric is used to measure the matching correlation between the multimodal fusion features and the single modal features before fusion; the pruning module is used to perform non-structural pruning on the corresponding modal network layer in the initialized multimodal perception model based on the fusion matching metric; the control module is used to control the evaluation module and the pruning module to perform pruning operations on the modal network layer within a preset number of iterations until the model converges.

[0008] In some possible embodiments, the fusion matching metric standard defined by the evaluation module during multimodal information fusion includes: extracting features using the backbone network of the corresponding modality in the multimodal perception model for the preprocessed original data of different modalities; converting the extracted unimodal features of different branches into a shared representation space for fusion to obtain the current fusion feature; using the cosine similarity function to calculate the matching correlation between the current fusion feature and the unimodal feature of each branch; and normalizing the matching correlation into a unified fusion matching metric standard.

[0009] In some possible embodiments, the pruning module performs non-structural pruning on the corresponding modal network layer in the initialized multimodal perception model based on the fusion matching metric, including: calculating a custom pruning loss based on the fusion matching metric; wherein the custom pruning loss is used to provide a pruning criterion by calculating the importance of the unimodal features of the different branches; based on the custom pruning loss, determining a weight-based pruning mask matrix; and performing non-structural pruning on the modal network layer based on the pruning mask matrix.

[0010] In some possible embodiments, the step of determining a weight-based pruning mask matrix based on the custom pruning loss includes: calculating the number of weights to be retained based on the pruning rate and the total number of weights in the network; multiplying the result of the derivative of the custom pruning loss with respect to the weights by the corresponding weights to determine the score of each weight; and calculating a pruning mask matrix that satisfies the current conditions and maximizes the total score of the retained weights.

[0011] In some possible embodiments, the control module controls the evaluation module and the pruning module to perform pruning operations on the modal network layer within a preset number of iterations until the model converges, including: during model fine-tuning, the control module controls the pruning module to traverse each modal network layer, and sets or removes non-important parameters according to the pruning mask matrix updated in the current round, until the number of cycles reaches the preset number of iterations and the model converges, thereby completing the pruning and fine-tuning of the multimodal perception model.

[0012] In some possible embodiments, the method further includes: performing task prediction after obtaining the current fusion feature in the multimodal fusion perception model; determining the target loss based on the prediction result and the label of the original data; using the target loss to perform backpropagation at the end of each batch of data, and updating the model weight matrix by the Adam optimizer.

[0013] In a second aspect, an embodiment of the present invention provides a multimodal network pruning system for realizing mobile terminal target perception, including an evaluation module, a pruning module and a control module, wherein:

[0014] The evaluation module is used to define a fusion matching metric when fusing multimodal information; wherein the fusion matching metric is used to measure the matching correlation between the multimodal fusion feature and the single modal feature before fusion;

[0015] The pruning module is used to perform non-structural pruning on the corresponding modality network layer in the initialized multimodal perception model based on the fusion matching metric;

[0016] The control module is used to control the evaluation module and the pruning module to perform pruning operations on the modal network layer within a preset number of iterations until the model converges.

[0017] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0018] In an embodiment of the present invention, a fusion matching metric is first proposed through an evaluation module for the unbalanced feature fusion phenomenon to measure the matching correlation between the multimodal fusion features and the single modal features before fusion, and then the pruning module uses the fusion matching metric to prune unimportant weights in the multimodal perception network and balance the features before and after fusion, thereby significantly reducing the computational cost while maintaining or improving the model performance. The present invention makes it possible to efficiently deploy multimodal perception networks on resource-constrained edge devices, provides a new perspective and method for pruning multimodal perception networks, and can further promote the development of autonomous driving, robotics and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0020] Figure 1 A flow chart of a multimodal network pruning method for realizing mobile terminal target perception provided by an embodiment of the present invention;

[0021] Figure 2 An overall block diagram of a multimodal perception network pruning method provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the composition structure of a multimodal network pruning system for realizing mobile terminal target perception provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0024] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0025] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present invention are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present invention described here can be implemented in an order other than that illustrated or described here.

[0026] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the embodiments of the present invention belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0027] In order to study the reasons for the decrease in model accuracy caused by the unimodal pruning method, the applicant visualized the features before and after fusion in the experiment and found that, for example, the fused features only contained very few distant and inaccurate semantic features of the LiDAR modality. Therefore, each unimodal feature before fusion contained redundant features that were irrelevant and unimportant to the fused features.

[0028] By analyzing the key fusion mechanism in multimodal perception networks, it is concluded that the information extracted by each single-modal backbone is more than that extracted after fusion of features, and the phenomenon of feature fusion imbalance (FFI) is proposed.

[0029] In order to solve this problem, the applicant believes that the pruning of multimodal perception networks should consider the importance of the fused features, rather than simply applying the unimodal pruning method. Therefore, it is necessary to rely on an effective metric to prune the features before fusion while still being beneficial to the fused features.

[0030] An embodiment of the present invention provides a multimodal network pruning method for realizing mobile terminal target perception, which is applied to electronic devices. The electronic devices include but are not limited to mobile phones, laptops, tablet computers and handheld Internet devices, multimedia devices, streaming media devices, mobile Internet devices, wearable devices or other types of electronic devices. The functions implemented by the method can be implemented by calling program codes by a processor in the electronic device. Of course, the program codes can be stored in a computer storage medium. It can be seen that the electronic device includes at least a processor and a storage medium. The processor can be used to process the multimodal network pruning process for realizing mobile terminal target perception, and the memory can be used to store the data required and the data generated in the multimodal network pruning process for realizing mobile terminal target perception.

[0031] Figure 1 A flow chart of a multimodal network pruning method for realizing mobile terminal target perception provided by an embodiment of the present invention is as follows: Figure 1 As shown, the method comprises at least the following steps:

[0032] Step S110, defining a fusion matching metric during multimodal information fusion through an evaluation module.

[0033] Here, the fusion matching metric is used to measure the matching correlation between the multimodal fusion features and the single modal features before fusion.

[0034] When multimodal information is fused, the features extracted from backbone networks of different modalities have different importance to the fused features. There is an imbalance phenomenon that the information extracted by each single-modal backbone is more than the information contained in the fused features. The evaluation module uses this imbalance phenomenon to quantitatively evaluate the matching correlation between the features before and after fusion, and calculates the fusion matching metric for feature extraction networks of different modalities.

[0035] By fusing matching metrics to analyze the importance of features from different modalities, the evaluation module can identify features that contribute less to the final task, thereby facilitating the subsequent pruning of unimportant weights, significantly reducing computational costs while maintaining or improving model performance.

[0036] Step S120, performing non-structural pruning on the corresponding modality network layer in the initialized multimodal perception model based on the fusion matching metric through a pruning module.

[0037] Here, the fusion matching metric is a quantitative indicator of the degree of feature matching correlation before and after fusion. Based on the fusion matching metric, a new custom pruning loss is introduced to measure the importance of features from different backbone networks before fusion and after fusion, and provide an importance indicator, i.e., a pruning criterion, for pruning. By balancing features before and after fusion, it facilitates the identification and pruning of unimportant features and their associated weights. In other words, the pruning module removes weights or neurons that do not contribute much to the performance of the model.

[0038] The accuracy of the model may be affected after pruning. In order to restore the accuracy of the model, a fine-tuning process is usually performed, that is, additional training is performed on the pruned network structure to optimize the value of the remaining weights. Through pruning and fine-tuning, the model is finally optimized.

[0039] This pruning strategy helps improve the efficiency and performance of the model while reducing the need for computing resources. By removing unimportant features, the model can focus more on information that contributes substantially to the task, which may improve generalization and reduce the risk of overfitting. In addition, pruning can also help improve the interpretability of the model, because a more streamlined network structure is easier for humans to understand and analyze.

[0040] Step S130, controlling the evaluation module and the pruning module through the control module to perform pruning operations on the modal network layer within a preset number of iterations until the model converges.

[0041] Here, the control module is responsible for coordinating the work of the evaluation module and the pruning module to ensure that they perform pruning operations on the complete multimodal network.

[0042] By looping through the fusion of multimodal features, evaluating the importance of features, pruning operations, and fine-tuning the model within a preset number of iterations, the aim is to improve the efficiency and performance of the model. Through pruning, the multimodal perception model can remove unimportant weights and reduce computational complexity while maintaining or even improving the accuracy of the model. This approach is particularly suitable for autonomous vehicles and other multimodal perception tasks that require processing large amounts of sensor data.

[0043] In an embodiment of the present invention, a fusion matching metric is first proposed through an evaluation module for the unbalanced feature fusion phenomenon to measure the matching correlation between the multimodal fusion features and the single modal features before fusion, and then the pruning module uses the fusion matching metric to prune unimportant weights in the multimodal perception network and balance the features before and after fusion, thereby significantly reducing the computational cost while maintaining or improving the model performance. The present invention makes it possible to efficiently deploy multimodal perception networks on resource-constrained edge devices, provides a new perspective and method for pruning multimodal perception networks, and can further promote the development of autonomous driving, robotics and other fields.

[0044] In some possible embodiments, the above step S110 "defining the fusion matching metric during multimodal information fusion by the evaluation module" can be further implemented by the following steps:

[0045] S111, extracting features from the preprocessed raw data of different modalities using the backbone network of the corresponding modality in the multimodal perception model;

[0046] S112, converting the extracted unimodal features of different branches into a shared representation space for fusion to obtain current fused features;

[0047] S113, using a cosine similarity function to respectively calculate the matching correlation between the current fusion feature and the single modal feature of each branch;

[0048] S114, normalizing the matching correlation into a unified fusion matching metric.

[0049] Here, the raw data of different modes usually come from different sensors, such as images collected by cameras, point cloud data collected by lidar, etc. The data is preprocessed, including denoising and normalization.

[0050] Different methods are used to extract features from different sensor data, such as using CNN (convolutional neural network) to extract features from images, or using point cloud processing network to extract features from point cloud data. Then the features extracted by different sensors are fused and all features are converted into a shared representation space called BEV (bird's eye view) to unify multimodal features.

[0051] Specifically, the current multimodal perception network is gradually developing towards a unified architecture to extract multimodal features in a shared bird's-eye view space. Therefore, the multimodal perception network can be regarded as consisting of the following three parts:

[0052] Feature extractor F of the LiDAR backbone network P Take point cloud information as input and output point cloud feature map Among them C P , H and W are the number of channels, height and width of the point cloud feature map respectively; the feature extractor F of the camera backbone network I Take image information as input and output image feature map Among them C I , H and W are the number of channels, height and width of the image feature map respectively;

[0053] The feature extractor F of the fusion model F Point cloud feature map and image feature map As input, output fusion feature map Among them C F , H and W are the number of channels, height and width of the fused feature map respectively.

[0054] The weights of the above three backbone networks or models are expressed as W = {W P , W I , W F}, where W P , W I and W F They are the weights of the LiDAR backbone network, the camera backbone network and the fusion model. Taking the camera backbone network as an example, Represents all its weights, where N i is the total number of parameters. Therefore, for the input x of the point cloud information P And the input x of image information I , we can get the fusion feature map

[0055] The matching correlation between the current fused features and the unimodal features before fusion is calculated through the cosine similarity function, and the resulting matrix is ​​normalized into a unified evaluation scalar, which successfully solves the key problem of how to maintain the beneficialness of the fused features during the pruning process.

[0056] In some possible embodiments, the above step S120 "performing non-structural pruning on the corresponding modality network layer in the initialized multimodal perception model based on the fusion matching metric through the pruning module" can be further implemented through the following steps:

[0057] S121, calculating a custom pruning loss based on the fusion matching metric; wherein the custom pruning loss is used to provide a pruning criterion by calculating the importance of the unimodal features of the different branches;

[0058] S122, determining a weight-based pruning mask matrix based on the custom pruning loss;

[0059] S123, performing non-structural pruning on the modality network layer based on the pruning mask matrix.

[0060] Here, in order to maintain accuracy as much as possible while pruning the model (Pruning), the present invention adopts the method of gradient-based weight pruning. Gradient-based weight pruning usually involves using indicators to evaluate the importance of weights, and then using optimization methods to prune those weights that are considered unnecessary in the model. Specifically, the number of weights that need to be retained is calculated based on the pruning rate and the total number of weights in the network, that is, the total number of 1s in the pruning mask matrix, and then the importance score of each weight is calculated. Finally, the mask matrix that satisfies the current conditions and makes the total score of the retained weights the highest is calculated, and the model is pruned and optimized accordingly.

[0061] The optimization process is shown in formulas (1) and (2):

[0062]

[0063] In the formula, represents the jth weight in the LiDAR backbone network, camera backbone network or fusion model, δ ij is an indicator if Will be pruned ij It is 0 if Will be retained ij Then it is 1, K represents the number of retained weights, as shown in formula (2), and K = ρ·∑ i∈{P,I,F} N i , ρ is the pruning rate, S index is used to measure the importance of weight, the score index S (equivalent to the weight score) can be designed in different ways. The present invention uses a gradient-based method, which is expressed as follows:

[0064]

[0065] In the formula, is specifically for the feedforward network F i The designed scalar loss function is the custom pruning loss, which depends on the pruning scenario.

[0066] On this basis, the applicant experimentally analyzed the visualization features of the single modality (before fusion) and after fusion. The observation results show that each single modality feature contains many parts that are redundant for the fusion feature, which is the feature fusion imbalance phenomenon mentioned above. This phenomenon emphasizes the difference between the excessive extraction of each single modality feature before fusion and the limited demand for fusion features. This imbalance may mislead the pruning method based on single modality attitude measures (such as weight size), resulting in a decrease in accuracy.

[0067] To solve the problem of unbalanced feature fusion, the present invention uses the fusion matching metric mentioned above to evaluate the matching correlation between each single modal feature before fusion and the fusion feature. Taking the camera backbone network as an example, the fusion matching metric can compare the camera backbone features. and fusion features It is expressed as follows:

[0068]

[0069] in, is a function that normalizes the first dimension, sim is a similarity function, and the embodiment of the present invention uses cosine similarity as the similarity function. The similarity function quantifies the matching correlation between features, and the normalization function is used to convert the matrix into a unified scalar for evaluating image features. The importance of fusion on different image channels. In addition, a temperature parameter τ is adopted to obtain better performance.

[0070] Similarly, the fusion matching metric is based on the LiDAR backbone features The application is shown as follows:

[0071]

[0072] Specifically, we use the effective quantization metric calculated using the fusion matching metric, and then introduce a new custom pruning loss It measures the importance of features from different backbone networks before fusion relative to the fused features and provides an importance indicator for pruning. Combining formulas (4) and (5), this custom pruning loss It can be expressed as follows:

[0073]

[0074] Among them, C k Represents the number of channels of the image feature map or point cloud feature map, represents the sum of the first dimension, when hour, is its i-th element. Therefore, a higher score is a reliable indicator of the importance of the fused feature.

[0075] The custom pruning loss designed in formula (6) Combining formulas (1), (2) and (3), the final pruning and scoring formula can be obtained as follows:

[0076]

[0077] Combined with the fusion matching metric, this paper proposes an innovative system framework, namely the multimodal fusion balanced pruning system, which balances features before and after fusion through customized pruning losses, facilitating the identification and pruning of unimportant features and their associated weights.

[0078] In some possible embodiments, the step of determining a weight-based pruning mask matrix based on the custom pruning loss includes: calculating the number of weights to be retained based on the pruning rate and the total number of weights in the network; multiplying the result of the derivative of the custom pruning loss with respect to the weights by the corresponding weights to determine the score of each weight; and calculating a pruning mask matrix that satisfies the current conditions and maximizes the total score of the retained weights.

[0079] Here, the initial values ​​of the pruning mask matrix are all 1 and the shape is consistent with the shape of the model weight matrix. First, the number of weights to be retained is calculated according to the pruning rate and the total number of weights in the network, that is, the total number of corresponding 1s in the pruning mask matrix, and then the importance score of each weight is calculated. Finally, the pruning mask matrix that satisfies the current conditions and makes the total score of the retained weights the highest is calculated, and the model can be pruned and optimized accordingly.

[0080] In some possible embodiments, the control module controls the evaluation module and the pruning module to perform pruning operations on the modal network layer within a preset number of iterations until the model converges, including: during model fine-tuning, the control module controls the pruning module to traverse each modal network layer, and sets or removes non-important parameters according to the pruning mask matrix updated in the current round, until the number of cycles reaches the preset number of iterations and the model converges, thereby completing the pruning and fine-tuning of the multimodal perception model.

[0081] Here, the weights of 1 in the pruning mask matrix are retained, and the weights of 0 are pruned. The control module coordinates the evaluation module and the pruning module, and performs data feature extraction, fusion, evaluation and pruning in a loop in n iterations. The evaluation module is called in the loop to update the pruning mask. At the end of each iteration, the pruning module is called to perform pruning operations according to the updated pruning mask, and the corresponding weights in the model weight matrix are reset to 0. The termination condition of pruning is that the model converges and reaches the preset maximum number of iterations n.

[0082] In summary, the working principle of the multi-modal fusion balanced pruning process proposed in the present invention is as follows:

[0083] S1. Prepared input data: pruning ratio ρ, number of iterations n, network model containing multimodal feature extractor and corresponding weights

[0084] S2. Initialize the binary mask μ, which has the same shape as the model weight matrix W and all initial values ​​are 1.

[0085] S3, traverse the network layers of LiDAR and Camera backbones, and perform n pruning iterations. Step S3 can be expanded into the following steps S31 to S36:

[0086] S31. Perform v operation on the pruning mask matrix μ and the model weight matrix W, that is, multiply the corresponding elements, and obtain W after pruning. μ value, which is the currently retained weight;

[0087] S32, calculate F P ,F I and F F Between Value, that is, the fusion matching metric result, where m∈{I,P};

[0088] S33, Substitution To calculate the custom pruning loss using formula (6) value;

[0089] S34, Substitution The weight score S is calculated by formula (3) k (equivalent to weight score), where k is the current iteration number and k≤n;

[0090] S35, according to Get the pruning rate ρ for this round k , and ρ k With weight score S k Multiply them together to get the weight score threshold h for this round of pruning k ;

[0091] S36, update μ, if h k ≤S k ,The corresponding weight binary value in the pruning mask matrix μ is 1, otherwise it is 0.

[0092] The control module iterates the above steps during model fine-tuning, uses the pruning module to traverse each modal network layer, and sets or removes unimportant parameters according to the non-structural pruning mask until the iteration ends and the model converges, thus completing the pruning and fine-tuning of the original model.

[0093] In some possible embodiments, the method further includes: performing task prediction after obtaining the current fusion feature in the multimodal fusion perception model; determining the target loss based on the prediction result and the label of the original data; using the target loss to perform backpropagation at the end of each batch of data, and updating the model weight matrix by the Adam optimizer.

[0094] Here, the current fused features are used to perform specific tasks, such as 3D object detection. Then the target loss is calculated, which can be obtained through a variety of loss functions, such as classification loss, bounding box regression loss, and difference loss. Among them, the target classification loss of the category distribution predicted by the model and the true category distribution of the original dataset is calculated by the cross entropy loss function, the bounding box regression loss of the target bounding box predicted by the model and the true target position of the original dataset is calculated by the L1 loss function, and the difference loss between the heat map predicted by the model and the true heat map of the original dataset is calculated by the binary cross entropy loss function. Backpropagation is performed at the end of each batch, and the model weights are updated using the Adam optimizer.

[0095] The multimodal network pruning method for realizing mobile terminal target perception is described below in conjunction with a specific embodiment. However, it should be noted that this specific embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0096] This specific embodiment uses the nuScenes dataset as an example to conduct experiments to verify the superiority and effectiveness of the method of the present invention. It has been experimentally verified that the application of the fusion balanced pruning method of the present invention makes the performance of the multimodal perception network exceed that of other existing single-modal pruning methods. This achievement makes it possible to efficiently deploy multimodal perception networks on resource-constrained edge devices, and provides a new perspective and method for pruning multimodal perception networks, which can further promote the development of autonomous driving, robotics and other fields.

[0097] This specific embodiment provides a fusion balanced pruning method for a multimodal perception network, and the specific implementation process includes the following stages:

[0098] 1. Build a multimodal perception network.

[0099] The present invention uses BEVfusion-mit and BEVfusion-pku for experimental evaluation, where the camera backbone network uses the Swin-Transformer (Swin-T) architecture, and the LiDAR backbone network uses the VoxelNet architecture. In addition, in order to prove the universality of the present invention, the BEVFusion-mit model with different backbone networks is further experimented, where the LiDAR backbone network is replaced by PointPillar and the Camera backbone network is replaced by ResNet.

[0100] Second, train and evaluate the multimodal perception network on a known dataset to observe the impact of model pruning on accuracy.

[0101] Here, the nuScenes dataset is used to conduct experiments on 3D object detection and BEV segmentation tasks. nuScenes is a large-scale outdoor dataset in which the 40,157 labeled samples contain a collection of six monocular camera images that can capture a panoramic 360-degree field of view. In addition, the dataset also contains 32-beam LiDAR scans, which further enriches its practicality and enables multi-angle data-driven research.

[0102] 3. According to the workflow of the multimodal fusion balanced pruning method provided by the present invention, set the corresponding hyperparameters.

[0103] For the baseline model training of 3D object detection, Adam is used as the optimizer, the learning rate is 1e-4, cosine annealing is used as the parameter scheduler, and the batch size is set to 2.

[0104] For the BEV segmentation task, Adam is also used as the optimizer with a learning rate of 1e-4. At the same time, a one-cycle learning rate scheduler is used and the batch size is set to 2.

[0105] The number of pruning iterations n in the multimodal network pruning method is set to 100.

[0106] 4. Fine-tune and prune the multimodal perception model under the multimodal network pruning framework, and repeat this process according to the number of iterations n until the model converges.

[0107] The process specifically includes the following steps S41 to S47:

[0108] S41, initialize a multimodal perception model, such as a BEVfusion model, and load its trained weights;

[0109] S42, the data preprocessing layer receives raw data from different sensors, such as point cloud data from a laser radar and image data from a camera, and performs necessary preprocessing, such as denoising and normalization;

[0110] S43, for different sensor data, the feature extraction layer uses corresponding feature extraction methods, such as a convolutional neural network (CNN) to extract features from an image, or a point cloud processing network to extract features from point cloud data;

[0111] S44, the feature fusion layer fuses the features extracted by each sensor. Fusion methods such as BEVFusion convert the features of all sensors into the shared representation space of BEV in order to unify multimodal features.

[0112] S45, the evaluation module combines the current fusion feature and the single features from different branches to calculate the fusion matching metric result according to the similarity function As shown in formulas (4) and (5), according to formula (6), substitute Calculating custom pruning loss Further, the gradient-based score index S is calculated according to formula (3), and finally substituted into formulas (1) and (2) to obtain formula (7), and the weight-based pruning mask matrix μ is calculated, that is, the non-structural pruning standard, such as: [[1,0,0,1...]...], where the shape of the pruning mask matrix is ​​the same as the weight matrix, 0 represents the corresponding weight to be pruned, and 1 represents the corresponding weight to be retained;

[0113] S46. After obtaining the fusion features, the model performs specific tasks, such as 3D target detection, and calculates the target classification loss based on the category distribution predicted by the model and the real category distribution of the nuscenes dataset, such as the cross entropy loss function, the bounding box regression loss between the target bounding box predicted by the model and the real target position of the nuscenes dataset, such as the L1 loss function, the difference loss between the heat map predicted by the model and the real heat map of the nuscenes dataset, etc., and performs back propagation at the end of each batch, and the Adam optimizer updates the model weights;

[0114] S47, the control module continuously loops the above process in n pruning iterations, and calls the pruning module to perform pruning on the model at the end of each iteration, that is, the weights in the corresponding weight matrix are reset to 0 according to the pruning mask updated in this round. The termination condition of the model pruning is that the model converges and the number of cycles reaches the preset maximum number of iterations n, and finally the pruning and fine-tuning of the model is achieved.

[0115] Figure 2 The overall block diagram of the fusion balanced pruning method for a multimodal perception network provided by the present invention. Specifically, the camera branch processes the image data to obtain image features, the radar branch processes the point cloud data to obtain point cloud features, and then the fusion module processes the BEV fusion features to obtain the BEV fusion features. At this time, the evaluation module evaluates the correlation between the fusion features and the image features, and the fusion features and the point cloud features according to the fusion matching metric standard, and obtains the fusion matching metric result ( Figure 2 The pruning module calculates the custom pruning loss ( Figure 2 The weight scores are further calculated to finally obtain the pruning mask matrix. According to this mask matrix, the feature extraction networks of the two single-modal branches of camera and radar are pruned respectively to complete the pruning operation of this iteration until the iteration ends and the model converges.

[0116] The present invention provides a fusion balanced pruning method and system for multimodal perception networks. Based on the exploration of the field of multimodal pruning, it focuses on perception tasks and introduces a new fusion balanced pruning system framework. As an innovative method for pruning in the multimodal field, this method solves the problem of feature fusion imbalance caused by the unique fusion mechanism of the multimodal perception network. A new fusion matching metric is used to evaluate the matching correlation between each single modal feature and the fusion feature before fusion, so as to achieve effective pruning of the model. Through rigorous experimental evaluation, the multimodal fusion balanced pruning method proposed in the present invention achieves high pruning performance while achieving real-time performance. In the field of multimodal network optimization, its performance surpasses the existing most advanced single modal pruning methods.

[0117] Based on the foregoing embodiments, an embodiment of the present invention further provides a multimodal network pruning system for realizing mobile terminal target perception. The modules included in the system and the units included in the modules can be implemented by a processor in an electronic device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0118] Figure 3 A schematic diagram of the composition structure of a multimodal network pruning system for realizing mobile terminal target perception provided by an embodiment of the present invention is as follows: Figure 3 As shown, the system 300 includes an evaluation module 310, a pruning module 320 and a control module 330, wherein:

[0119] The evaluation module 310 is used to define a fusion matching metric when multimodal information is fused; wherein the fusion matching metric is used to measure the matching correlation between the multimodal fusion feature and the single modal feature before fusion;

[0120] The pruning module 320 is used to perform non-structural pruning on the corresponding modality network layer in the initialized multimodal perception model based on the fusion matching metric;

[0121] The control module 330 is used to control the evaluation module and the pruning module to perform pruning operations on the modal network layer within a preset number of iterations until the model converges.

[0122] In some possible embodiments, the evaluation module 310 includes: a feature extraction unit, used to extract features from the preprocessed original data of different modalities using the backbone network of the corresponding modality in the multimodal perception model; a feature fusion unit, used to convert the extracted unimodal features of different branches into a shared representation space for fusion to obtain the current fused feature; a similarity calculation unit, used to calculate the matching correlation between the current fused feature and the unimodal feature of each branch using a cosine similarity function; and a normalization unit, used to normalize the matching correlation into a unified fusion matching metric.

[0123] In some possible embodiments, the pruning module 320 includes: a loss calculation unit, used to calculate a custom pruning loss based on the fusion matching metric; wherein the custom pruning loss is used to provide a pruning criterion by calculating the importance of the unimodal features of the different branches; a mask determination unit, used to determine a weight-based pruning mask matrix based on the custom pruning loss; and a pruning unit, used to perform non-structural pruning on the modal network layer based on the pruning mask matrix.

[0124] In some possible embodiments, the mask determination unit is specifically used to calculate the number of weights that need to be retained based on the pruning rate and the total number of weights in the network; multiply the result of the derivative of the custom pruning loss with the corresponding weight by point multiplication to determine the score of each weight; and calculate the pruning mask matrix that satisfies the current conditions and makes the total score of the retained weights the highest.

[0125] In some possible embodiments, the control module 330 is specifically used to control the pruning module to traverse each modal network layer through the control module during model fine-tuning, and to set or remove non-important parameters according to the pruning mask matrix updated in the current round until the number of loops reaches the preset number of iterations and the model converges, thereby completing the pruning and fine-tuning of the multimodal perception model.

[0126] In some possible embodiments, the system also includes a model weight updating module, which is used to perform task prediction after obtaining the current fusion feature in the multimodal fusion perception model; determine the target loss based on the prediction result and the label of the original data; use the target loss to perform backpropagation at the end of each batch of data, and update the model weight matrix by the Adam optimizer.

[0127] It should be noted that the description of the above system embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the system embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0128] It should be noted that in the embodiments of the present invention, if the above-mentioned multimodal network pruning method for realizing mobile terminal target perception is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0129] Correspondingly, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any of the multimodal network pruning methods for realizing mobile terminal target perception described in the above embodiments are implemented. Correspondingly, an embodiment of the present invention also provides a computer program product, and when the computer program product is executed by a processor of an electronic device, it is used to implement the steps in any of the multimodal network pruning methods for realizing mobile terminal target perception described in the above embodiments.

[0130] It should be noted that the description of the above storage medium and computer program product is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the storage medium and device embodiments of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0131] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present invention, the size of the serial number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. The serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0132] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0133] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0134] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention. In addition, the functional units in the embodiments of the present invention may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0135] The methods or features disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0136] The above is only an embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal network pruning method for realizing mobile terminal target perception, characterized in that: include: Defining a fusion matching metric during multimodal information fusion through an evaluation module; wherein the fusion matching metric is used to measure the matching correlation between the multimodal fusion feature and the single modal feature before fusion; Based on the fusion matching metric, the pruning module performs non-structural pruning on the corresponding modality network layer in the initialized multimodal perception model; The control module controls the evaluation module and the pruning module to perform pruning operations on the modal network layer within a preset number of iterations until the model converges.

2. The method according to claim 1, characterized in that The fusion matching metric standard when multimodal information fusion is defined by the evaluation module includes: For the preprocessed raw data of different modalities, the backbone network of the corresponding modality in the multimodal perception model is used to extract features respectively; The extracted unimodal features of different branches are converted into a shared representation space for fusion to obtain the current fusion features; Using a cosine similarity function, respectively calculating the matching correlation between the current fusion feature and the single modal feature of each branch; The matching correlations are normalized into a unified fusion matching metric.

3. The method according to claim 1, characterized in that The pruning module performs non-structural pruning on the corresponding modality network layer in the initialized multimodal perception model based on the fusion matching metric, including: Calculating a custom pruning loss based on the fusion matching metric; wherein the custom pruning loss is used to provide a pruning criterion by calculating the importance of the unimodal features of the different branches; Determining a weight-based pruning mask matrix based on the custom pruning loss; The modality network layer is subjected to unstructured pruning based on the pruning mask matrix.

4. The method according to claim 3, characterized in that: Determining a weight-based pruning mask matrix based on the custom pruning loss includes: Calculate the number of weights that need to be retained based on the pruning rate and the total number of weights in the network; Perform a dot multiplication of the result of the derivative of the custom pruning loss with respect to the weight and the corresponding weight to determine the score of each weight; Calculate the pruning mask matrix that satisfies the current conditions and maximizes the total score of the retained weights.

5. The method according to claim 3 or 4, wherein: The controlling module controls the evaluation module and the pruning module to perform pruning operations on the modal network layer within a preset number of iterations until the model converges, including: During model fine-tuning, the control module controls the pruning module to traverse each modal network layer, and sets non-important parameters to zero or removes them according to the pruning mask matrix updated in the current round, until the number of cycles reaches the preset number of iterations and the model converges, thereby completing the pruning and fine-tuning of the multimodal perception model.

6. The method according to claim 2, characterized in that The method further comprises: After obtaining the current fusion feature in the multimodal fusion perception model, performing task prediction; Determining a target loss based on the prediction result and the label of the original data; The target loss is used to perform backpropagation at the end of each batch of data, and the model weight matrix is ​​updated by the Adam optimizer.

7. A multimodal network pruning system for realizing mobile terminal target perception, characterized in that: It includes evaluation module, pruning module and control module, among which: The evaluation module is used to define a fusion matching metric when fusing multimodal information; wherein the fusion matching metric is used to measure the matching correlation between the multimodal fusion feature and the single modal feature before fusion; The pruning module is used to perform non-structural pruning on the corresponding modality network layer in the initialized multimodal perception model based on the fusion matching metric; The control module is used to control the evaluation module and the pruning module to perform pruning operations on the modal network layer within a preset number of iterations until the model converges.

Citation Information

Cited By

  • Multi-hardware mixed large model training method, system and related device

    CN119539012A

  • A method, system and related device for multi-hardware hybrid large model training

    CN119539012B

  • Network model lightweight method, device, equipment, medium and program product

    CN119886233A

  • Intelligent agent lightweight deployment method and computing power elasticity distribution method

    CN120407044A

  • System optimization method and device, electronic equipment, storage medium and program

    CN120803877A