Multimodal structured sparse inference optimization method and system for resource-constrained end-side device

Abstract

The application discloses a multi-modal structured sparse inference optimization method and system for resource-constrained end-side devices, and relates to the technical field of cloud computing and artificial intelligence scheduling. The method comprises the following steps: identifying the modal attributes of each network layer in a multi-modal model, combining the static weight distribution characteristics and dynamic activation response characteristics, assigning a structured sparsity to each network layer, and performing structured pruning to obtain a compressed multi-modal model; extracting multi-layer visual hidden features from the compressed multi-modal model and performing splicing and fusion to generate enhanced visual representations; calculating the initial importance scores of each visual Token in the enhanced visual representations and performing smoothing enhancement to generate structure continuity perception scores; forming a sparse visual Token sequence based on the structure continuity perception scores, inputting the sparse visual Token sequence and a text prompt into the compressed multi-modal model for inference to generate an inference result. In this way, efficient and stable inference of the multi-modal model in a resource-constrained environment is realized.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of cloud computing and artificial intelligence scheduling technology, and in particular to a multimodal structured sparse inference optimization method and system for resource-constrained end-side devices. Background Technology

[0002] Deploying large multimodal models on resource-constrained edge devices currently faces the following three major challenges:

[0003] First, the visual modality, text modality, and cross-modal fusion module in a multimodal model have significant differences in activation characteristics and parameter redundancy. Existing unified pruning methods usually use a fixed sparsity rate and ignore the sensitivity differences between different modalities, which makes key semantic information susceptible to loss and makes it difficult to maintain cross-modal semantic alignment capability while reducing model size.

[0004] Second, visual semantic representation in multimodal models typically relies on the output of the last layer of the visual encoder, neglecting the complementary relationships between shallow texture information, mid-level structural information, and deep semantic information. Existing single-layer feature projection methods struggle to fully exploit cross-layer semantic associations, resulting in insufficient representational capabilities for complex visual understanding tasks.

[0005] Third, current mainstream dynamic token selection methods are usually based on Top-K attention scores for hard selection, which can easily randomly discard spatially adjacent visual tokens, leading to visual structure fragmentation and local semantic breaks, making it difficult for the model to maintain stable fine-grained visual perception capabilities under high sparsity conditions. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a multimodal structured sparse inference optimization method and system for resource-constrained edge devices. First, a modality-aware pruning mechanism is used to perform differentiated structural compression on the visual, text, and cross-modal fusion modules to reduce model parameter size and memory access overhead. Second, a cross-layer integrated fusion module is designed to jointly model multi-scale semantic features at different levels of the visual encoder, thereby enhancing visual semantic expression capabilities. Finally, a structured token adaptive selection mechanism is introduced to achieve structure-preserving dynamic sparsity through neighborhood continuity perception, reducing the computational burden of visual tokens while maintaining the semantic integrity of the visual space, thus achieving efficient and stable inference of multimodal models in resource-constrained environments.

[0007] In a first aspect, embodiments of the present invention provide a multimodal structured sparse inference optimization method for resource-constrained edge devices, comprising:

[0008] Identify the modal attributes of different network layers in a multimodal model, and statistically analyze the static weight distribution characteristics and dynamic activation response characteristics of each network layer.

[0009] Based on the modal properties of each network layer, combined with its static weight distribution characteristics and dynamic activation response characteristics, asymmetric structured sparsity is adaptively allocated to different network layers, and structured pruning is performed to obtain a compressed multimodal model.

[0010] Multi-layer visual hidden features are extracted from the compressed multimodal model, and the multi-layer visual hidden features are aligned and spliced ​​and fused along the channel dimension. An enhanced visual representation is generated through a nonlinear mapping network.

[0011] Calculate the initial importance score of each visual token in the enhanced visual representation, and smooth the initial importance score according to the spatial neighborhood relationship between visual tokens to generate a structural continuity perception score.

[0012] Based on the structural continuity perception score, key visual tokens are selected to form a sparse visual token sequence.

[0013] The system receives text prompts input by the user, and inputs the sparse visual token sequence and the text prompts together into the language decoding module of the compressed multimodal model for inference, generating inference results.

[0014] As a preferred implementation, the modal attributes of different network layers in the multimodal model are identified, and the static weight distribution characteristics and dynamic activation response characteristics of each network layer are statistically analyzed, including:

[0015] Determine the modal attributes of each network layer based on the keywords in the name of each network layer;

[0016] The activation tensors output by each network layer are collected through forward propagation during the offline calibration phase, and the average absolute activation magnitude and activation sparsity of the activation tensors are statistically analyzed as dynamic activation response characteristics of the network layer.

[0017] Calculate the entropy, kurtosis, and Frobenius norm of the weights of each network layer as the static weight distribution characteristics of that network layer.

[0018] In a preferred embodiment, based on the modal attributes of each network layer, combined with its static weight distribution characteristics and dynamic activation response characteristics, an asymmetric structured sparsity rate is adaptively assigned to different network layers, and structured pruning is performed to obtain a compressed multimodal model, including:

[0019] The pruning intervals for each network layer are determined based on the modal attributes of each network layer.

[0020] The static weight distribution characteristics and dynamic activation response characteristics of each network layer are input into the dual-domain fusion scoring function for scoring, and the asymmetric structured sparsity is adaptively assigned to each network layer within the pruning interval corresponding to each network layer based on the scoring results.

[0021] Structured pruning is performed based on the asymmetric structured sparsity of each network layer to obtain a compressed multimodal model.

[0022] In a preferred embodiment, the static weight distribution characteristics and dynamic activation response characteristics of each network layer are input into a dual-domain fusion scoring function for scoring. Based on the scoring results, an asymmetric structured sparsity rate is adaptively assigned to each network layer within the corresponding pruning interval, including:

[0023] The static weight distribution characteristics of each network layer are input into the dual-domain fusion scoring function to calculate the weight-side adjustment score of the network layer.

[0024] The dynamic activation response features of each network layer are input into the dual-domain fusion scoring function to calculate the activation-side scissorability score of the network layer.

[0025] A comprehensive evaluation score is obtained by weighting and fusing the weight-side adjustment score and the activation-side scissorability score of each network layer.

[0026] The comprehensive evaluation score of each network layer is mapped to the corresponding pruning interval of each network layer, and the adaptive structured sparsity of each network layer is calculated.

[0027] In a preferred embodiment, structured pruning is performed based on the asymmetric structured sparsity of each network layer to obtain a compressed multimodal model, including:

[0028] Based on the asymmetric structured sparsity of each network layer, determine the corresponding quantile threshold for the absolute value distribution of all weights in that network layer.

[0029] Set the weights in the network layer whose absolute values ​​are less than the quantile threshold to zero, and retain the remaining weights;

[0030] By traversing all network layers and performing pruning operations, a compressed multimodal model is obtained.

[0031] In a preferred embodiment, multi-layer visual hidden features are extracted from the compressed multimodal model, and the multi-layer visual hidden features are aligned and spliced ​​and fused along the channel dimension. An enhanced visual representation is then generated through a nonlinear mapping network, including:

[0032] Extract multi-layer visual hidden features from the visual encoder of the compressed multimodal model;

[0033] Remove the classification tokens from the visual hidden features of each layer to ensure that the features of each layer are consistent in the sequence length dimension;

[0034] The aligned visual hidden features of each layer are stitched together along the channel dimension to obtain multi-scale fused features;

[0035] The multi-scale fused features are input into a multilayer perceptron and sequentially subjected to linear transformation, activation function, and another linear transformation to generate an enhanced visual representation that matches the input dimension of the language decoding module.

[0036] In a preferred embodiment, an initial importance score is calculated for each visual token in the enhanced visual representation, and the initial importance score is smoothed and enhanced based on the spatial neighborhood relationship between visual tokens to generate a structural continuity perception score, including:

[0037] For each visual token in the enhanced visual representation, its activation intensity in the channel dimension is calculated as the initial importance score for that visual token.

[0038] Based on the neighborhood index relationship of each visual token in the spatial flattening order, the initial importance score of each visual token and the initial importance scores of other tokens in its neighborhood are weighted and aggregated to obtain the smoothing enhancement score of the visual token.

[0039] The initial importance score and the smoothing enhancement score are weighted and fused to obtain the structural continuity perception score of each visual token.

[0040] In a preferred embodiment, key visual tokens are selected based on the structural continuity perception score to form a sparse visual token sequence, including:

[0041] The visual tokens are sorted according to the structural continuity perception score.

[0042] Retain a preset number of visual tokens that are ranked at the top as key visual tokens;

[0043] Features corresponding to the key visual tokens are extracted from the enhanced visual representation to form a sparse visual token sequence.

[0044] In a preferred embodiment, the sparsed visual token sequence and the text prompt are input together into the language decoding module of the compressed multimodal model for inference, generating an inference result, including:

[0045] The sparse visual token sequence and the text prompt are concatenated along the sequence dimension to form a multimodal input sequence;

[0046] The multimodal input sequence is input into the language decoding module of the compressed multimodal model, and autoregressive inference is performed to generate inference results word by word.

[0047] Secondly, embodiments of the present invention also provide a multimodal structured sparse inference optimization system for resource-constrained edge devices, used to execute the multimodal structured sparse inference optimization method for resource-constrained edge devices described in any of the above embodiments, the system comprising:

[0048] The modal analysis module is used to identify the modal attributes of different network layers in a multimodal model, and to statistically analyze the static weight distribution characteristics and dynamic activation response characteristics of each network layer.

[0049] The differential pruning module is used to adaptively allocate asymmetric structured sparsity to different network layers based on their modal attributes, combined with their static weight distribution characteristics and dynamic activation response characteristics, and to perform structured pruning to obtain a compressed multimodal model.

[0050] The cross-layer feature fusion module is used to extract multi-layer visual hidden features from the compressed multimodal model, align the multi-layer visual hidden features and splice and fuse them along the channel dimension, and generate enhanced visual representations through a nonlinear mapping network.

[0051] The structure-aware scoring module is used to calculate the initial importance score of each visual token in the enhanced visual representation, and to smooth and enhance the initial importance score according to the spatial neighborhood relationship between visual tokens to generate a structure continuity perception score.

[0052] The Token sparsification module is used to select key visual tokens based on the structural continuity perception score to form a sparse visual token sequence.

[0053] The inference execution module is used to receive text prompts input by the user, input the sparse visual token sequence and the text prompts together into the language decoding module of the compressed multimodal model for inference, and generate inference results.

[0054] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0055] One or more processors;

[0056] Storage device for storing one or more programs;

[0057] When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal structured sparse inference optimization method for resource-constrained end-side devices as described in any embodiment of the present invention.

[0058] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal structured sparse inference optimization method for resource-constrained end-side devices as described in any embodiment of the present invention.

[0059] In summary, the present invention achieves the following beneficial effects:

[0060] (1) This invention constructs a modality-aware pruning mechanism to identify modal attributes of each layer of the model, and combines static weight distribution and dynamic activation statistics to adaptively allocate asymmetric sparsity rate, thereby achieving differentiated structural compression of visual, text, and fusion modules. This method can effectively maintain cross-modal semantic consistency while significantly reducing the model parameter size and memory access overhead, solving the problems of low compression efficiency and severe semantic degradation of traditional unified pruning strategies.

[0061] (2) This invention constructs a cross-layer integrated fusion module to extract multi-scale hidden features at different levels from the visual encoder, and performs joint splicing and nonlinear mapping along the channel dimension to achieve collaborative modeling of low-level detailed information and high-level abstract semantics. This mechanism significantly enhances the semantic density and hierarchical expressive power of visual features without increasing the number of tokens in the input language model, effectively solving the problem of insufficient semantic utilization in traditional single-layer projection methods.

[0062] (3) This invention designs a structured token adaptive selection mechanism. By constructing a neighborhood continuity perception scoring model, the importance of visual tokens is smoothly enhanced, and dynamic selection is completed while ensuring structural continuity. This mechanism can maintain spatial semantic integrity while reducing the visual computational burden, and significantly improve the stability and fine-grained visual understanding ability of the model under highly sparse inference conditions. Attached Figure Description

[0063] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0064] Figure 1 This is a flowchart of a multimodal structured sparse inference optimization method for resource-constrained edge devices provided in an embodiment of the present invention;

[0065] Figure 2 This is a diagram of the overall architecture of MACS provided in this embodiment of the invention;

[0066] Figure 3 This is a comparative diagram of the performance of various representative sparse inference methods provided in the embodiments of the present invention;

[0067] Figure 4 This is the motivation diagram for the structure-aware token sparsity of the visual language model provided in the embodiments of the present invention;

[0068] Figure 5 This is a schematic diagram of modality-sensing pruning provided in an embodiment of the present invention;

[0069] Figure 6 This is a qualitative visualization diagram of structured label selection under different sparsity levels provided in the embodiments of the present invention;

[0070] Figure 7 This is a schematic diagram of a multimodal structured sparse inference optimization system for resource-constrained edge devices provided in an embodiment of the present invention;

[0071] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0072] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0073] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as being processed sequentially, many of these operations (or steps) may be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations may be rearranged. The process may be terminated when its operation is completed, but may also have additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0074] Example 1

[0075] like Figure 4As shown, in resource-constrained edge scenarios, traditional Top-K dynamic token sparsification methods tend to randomly discard spatially relevant key visual tokens, leading to visual structure disintegration and multimodal inference failure. In contrast, by explicitly maintaining visual structural continuity during sparsification, the model can maintain stable visual understanding capabilities even under high sparsity settings and achieve significant improvements in inference efficiency. This invention proposes a multimodal structured sparse inference optimization method for resource-constrained edge devices. First, a modality-aware pruning mechanism is used to perform differentiated structural compression on visual, text, and cross-modal fusion modules to reduce model parameter size and memory access overhead. Second, a cross-layer integrated fusion module is designed to jointly model multi-scale semantic features at different levels of the visual encoder, thereby enhancing visual semantic expression capabilities. Finally, a structured token adaptive selection mechanism is introduced to achieve structure-preserving dynamic sparsification through neighborhood continuity awareness, reducing the computational burden of visual tokens while maintaining the integrity of visual spatial semantics, thus achieving efficient and stable inference of multimodal models in resource-constrained environments.

[0076] like Figure 1 As shown, Embodiment 1 of the present invention provides a flowchart of a multimodal structured sparse inference optimization method 100 for resource-constrained edge devices. The method 100 specifically includes the following steps:

[0077] Step S110: Identify the modal attributes of different network layers in the multimodal model, and statistically analyze the static weight distribution characteristics and dynamic activation response characteristics of each network layer.

[0078] Specifically, by parsing the keywords in the names of each network layer of the multimodal model, each network layer is classified into visual modality, text modality, or multimodal fusion category. For example, vision_tower and clip correspond to visual modality; mm_projector and fusion correspond to multimodal fusion category; and others are classified as text modality by default.

[0079] After completing modality classification, the activation tensors output by each network layer are collected through forward propagation during the offline calibration phase. The mean absolute activation magnitude and activation sparsity of these activation tensors are then calculated as dynamic activation response features of the network layer. Specifically, for each network layer to be pruned, its output activation tensor Y is recorded during forward propagation without changing the model parameters, and the mean absolute activation magnitude (mean_abs) and activation sparsity are accumulated. The mean absolute activation magnitude reflects the average activation amplitude of the layer on the calibration input; a larger magnitude generally indicates that the layer is more active and important in the current task. Activation sparsity reflects the proportion of activation values ​​close to zero. Higher sparsity suggests potentially higher redundancy in the layer.

[0080] The mean absolute activation magnitude (mean_abs) is obtained using the following formula:

[0081] ;

[0082] in, It is the total number of elements in the tensor. It is an extremely small sparsity threshold used to determine "near zero", and its value is set to... .

[0083] Simultaneously, the static distribution characteristics of the weights of each network layer are calculated offline, including the entropy of the weights. kurtosis and Frobenius norm These static features, together with the aforementioned dynamic features, form the basis for subsequent pruning decisions.

[0084] Step S120: Based on the modal attributes of each network layer, combined with its static weight distribution characteristics and dynamic activation response characteristics, adaptively allocate asymmetric structured sparsity to different network layers, and perform structured pruning to obtain the compressed multimodal model.

[0085] In a preferred embodiment, firstly, based on the modal attributes of each network layer in step S110 above, a corresponding pruning interval [b,m] is determined for each network layer, where b is the minimum pruning rate allowed for that modality and m is the maximum pruning rate allowed for that modality. The pruning intervals for different modalities are different from each other, thereby achieving differentiated processing for different semantic sensitivities.

[0086] Next, the static weight distribution characteristics and dynamic activation response characteristics of each network layer are input into the dual-domain fusion scoring function for scoring, and the asymmetric structured sparsity is adaptively assigned to each network layer within the pruning interval corresponding to each network layer based on the scoring results.

[0087] Finally, structured pruning is performed based on the asymmetric structured sparsity of each network layer to obtain the compressed multimodal model.

[0088] The static weight distribution characteristics and dynamic activation response characteristics of each network layer are input into a dual-domain fusion scoring function for scoring. Based on the scoring results, an asymmetric structured sparsity is adaptively assigned to each network layer within its corresponding pruning interval, including:

[0089] First, the static weight distribution features of each network layer are input into the dual-domain fusion scoring function, and the entropy in the static weight distribution features is utilized. kurtosis and Frobenius norm Calculate the weight-side adjustment score of this network layer. The specific formula is as follows:

[0090] ;

[0091] in, Normalization is an operation used to normalize entropy. kurtosis and Frobenius norm Perform scaling.

[0092] Next, the dynamic activation response features of each network layer are input into the dual-domain fusion scoring function, and the activation-side scissability score of the network layer is calculated using the mean absolute activation amplitude (mean_abs) and activation sparsity (sparsity) from the dynamic activation response features. :

[0093] ;

[0094] in, As a sensitivity control parameter, in this embodiment =0.1.

[0095] Next, the scores are adjusted according to the weights of each network layer. and activation side scissorability score The scores are weighted and combined to obtain a comprehensive evaluation score. :

[0096] ;

[0097] in, The weighting coefficient for the activation side can be adjusted flexibly, for example, by taking a value of 0.5.

[0098] Finally, the comprehensive evaluation scores of each network layer will be calculated. Mapping to the pruning interval [b,m] corresponding to each network layer, the adaptive structured sparsity of each network layer is calculated. :

[0099] ;

[0100] in, This is the modal sensitivity factor.

[0101] Structured pruning is performed based on the asymmetric structured sparsity of each network layer to obtain a compressed multimodal model, including:

[0102] Based on the asymmetric structured sparsity of each network layer Determine all weights of the network layer The corresponding quantile threshold of the absolute value distribution The network layer whose absolute value is less than the quantile threshold. The weights are reset to zero, while the remaining weights are retained. Specifically:

[0103] ;

[0104] ;

[0105] in, This is the initial weight matrix of the network layer. This is the weight matrix after pruning. For element-wise multiplication, Quantile threshold, It represents the quantile.

[0106] Finally, the pruning operation is performed by traversing all network layers to obtain the compressed multimodal model.

[0107] Step S130: Extract multi-layer visual hidden features from the compressed multimodal model, align the multi-layer visual hidden features and splice and fuse them along the channel dimension, and generate enhanced visual representations through a nonlinear mapping network.

[0108] In a preferred embodiment, at least two hidden layer features of different depths are extracted from the visual encoder (e.g., CLIP ViT-L / 14) of the compressed multimodal model. In this embodiment, the output features of layers 7, 15, and 23 are preferably extracted, denoted as h7, h... 15 and h 23 Among them, the 7th layer features focus on shallow texture information, the 15th layer features focus on medium-level structural information, and the 23rd layer features focus on deep semantic information.

[0109] Because the CLIP ViT encoder typically includes an additional classification token ([CLS]Token) in its intermediate layer output, its sequence length is usually [missing information]. The token has been removed from the output of the 23rd level of the main branch, and the sequence length is... Therefore, to maintain consistent feature dimensions, this embodiment removes the [CLS]Token from each intermediate layer feature through a slicing operation, thereby ensuring that all feature maps to be fused maintain consistent dimensions, and that the aligned feature maps all have the same dimensions. ,in B represents the batch size, and N represents the number of visual tokens.

[0110] After the alignment operation is complete, the three selected feature maps will be... , and Along the channel dimension To splice them together:

[0111]

[0112] This operation concatenates features from three levels along the channel dimension to obtain a fused feature map. This tensor preserves the structure of all tokens in the spatial dimension, while gathering complementary semantic information of low, medium and high-level features in the channel dimension.

[0113] In a preferred embodiment, the stitched multi-scale fused features are input into a two-layer multilayer perceptron (MLP), and sequentially undergo linear transformation, GELU activation function, and a second linear transformation to output an enhanced visual representation that matches the input dimension of the language decoding module. The specific formula is as follows:

[0114] ;

[0115] Right now:

[0116] ;

[0117] in, This is a nonlinear mapping network used to perform cross-layer feature fusion, and its structure consists of two fully connected layers plus an activation function. For linear transformation layers, GELU is a Gaussian error linear unit. This is the fused feature map.

[0118] To ensure the three feature layers h7 and h8 are trained in the early stages of training 15 and h 23 To ensure equal treatment, rather than allowing high-level semantics (layer 23) to dominate, this invention addresses the first linear layer (initial weight matrix) of the MLP. A special initialization strategy was designed: this strategy will initialize the weight matrix. The input dimension 3072 is logically divided into 3 blocks: Each block is initialized independently using Xavier uniform initialization, and then the weight values ​​of each block are multiplied by a scaling factor. ,in It refers to the number of fusion layers and the final weight matrix. for:

[0119] , ;

[0120] in, This indicates that initial values ​​are randomly sampled from a uniform distribution, the upper and lower bounds of which are determined by the number of input neurons and the number of output neurons in the current layer. Indicates will The input dimension is logically divided into three sub-blocks.

[0121] Through the above initialization, the features of the three levels enter the fusion module with approximately equal contributions at the start of training, enabling the model to learn the fusion weights from a balanced state, thereby avoiding the dominance of one level and improving the fusion effect of multi-scale features.

[0122] Through the above operations, this embodiment significantly enhances the semantic density and hierarchical expressive ability of visual representation without increasing the number of tokens input to the language decoder.

[0123] Step S140: Calculate the initial importance score of each visual token in the enhanced visual representation, and smooth the initial importance score according to the spatial neighborhood relationship between visual tokens to generate a structural continuity perception score.

[0124] In a preferred embodiment, the enhanced visual representation generated in step S130 is considered as a feature sequence composed of multiple visual tokens. First, the activation intensity of each token in the channel dimension is calculated as an initial importance score, specifically using the following calculation formula:

[0125] , ;

[0126] in: This represents the initial importance score of each q tokens across the channel dimension. For channel index, It represents the total number of channels contained in the token. Let q be the feature value of each token on the c-th channel. Let B be the score tensor for all tokens, B be the batch size, and V be the number of visual tokens per image.

[0127] Because visual tokens in CLIP ViT are flattened from left to right and top to bottom, adjacent indices mostly correspond to spatially adjacent patches in the image, making them highly semantically related. If only based on... Top-K selection may result in fragmented and discontinuous tokens in the selected space, thus disrupting the local structure. Therefore, to address this issue, TAS introduces a neighborhood smoothing mechanism, applying a linear decay kernel to the score sequence of each sample. Perform 1D convolution smoothing, where the linear decay kernel The definition is as follows:

[0128] ;

[0129] Next, for each token, its initial score sequence is smoothed using one-dimensional convolution to obtain the neighborhood-enhanced score. :

[0130] ;

[0131] in: Used to control the neighborhood radius It is an offset index relative to the central token. The rate at which the control weight decays determines the weight of the central token relative to its neighbors. For index The initial importance score of the token.

[0132] Finally, the above initial importance scores will be used... Score after neighborhood enhancement Weighted fusion is performed to obtain the final structural continuity perception score. :

[0133] , ;

[0134] in, The fusion coefficient is... The value of determines the degree to which the model depends on structural continuity. When When the value is high, the model emphasizes the structural continuity of the token; conversely, it focuses more on the strong response of a single token. Preferably, in this embodiment... It is 0.5.

[0135] At this point, each visual token obtains a structural continuity perception score that integrates its own activation intensity and spatial neighborhood information, providing a basis for subsequent selection of structure-preserving tokens.

[0136] Through the above operations, this embodiment can effectively avoid the problem of randomly discarding important tokens that are spatially adjacent in the traditional Top-K method, and maintain the continuity and integrity of the visual structure under high sparsity conditions.

[0137] Step S150: Select key visual tokens based on the structural continuity perception score to form a sparse visual token sequence.

[0138] In a preferred embodiment, for each sample, the structural continuity perception scores of the N visual tokens obtained in step S140 are sorted in descending order from high to low, and the corresponding token indices are recorded. The set of indices corresponding to the top K scores is retained.

[0139] , ;

[0140] in, It is the highest scorer A set of indices for each token. The model uses these indices... From the original Collect the corresponding token features to form a sparse visual representation. .this This will be used as the downstream input of the model, reducing the computational complexity of subsequent modules.

[0141] Based on the obtained index set The feature vectors at the corresponding positions are extracted from the enhanced visual representation generated in step S130 above, and a sparse visual token sequence is formed. The sparse visual token sequence is used as the visual input part of the subsequent language decoding module.

[0142] Through the above operations, this embodiment achieves a high proportion of token compression while maintaining the continuity of the visual structure, laying a data foundation for efficient inference on resource-constrained edge devices.

[0143] Step S160: Receive text prompts input by the user, and input the sparse visual token sequence and the text prompts together into the language decoding module of the compressed multimodal model for inference to generate inference results.

[0144] In a preferred embodiment, the system receives text prompts input by the user through the input interface of the terminal device (such as a keyboard, voice recognition module, etc.), for example: Please describe the objects in the picture and their positional relationships. The text prompts are input to the text embedding layer of the compressed multimodal model and converted into the corresponding text token embedding sequence.

[0145] Next, the text token embedding sequence and the sparse visual token sequence generated in step S150 above are concatenated along the sequence dimension to form a unified multimodal input sequence.

[0146] Finally, the multimodal input sequence is fed into the language decoding module of the compressed multimodal model, such as MobileLLaMA-2.7B-Chat. This decoding module uses an autoregressive generation method to predict output tokens one by one. The series of outputs generated by the decoding module is detoxified and converted into natural language text, which is then returned to the user as the final inference result. For example, given the prompt "Please describe the objects in the picture and their positional relationships," it might generate answers such as "There is a cat on the left side of the sofa and a vase in the middle of the table in the picture."

[0147] Through the above steps, this embodiment significantly reduces inference computation and memory overhead while maintaining high-precision multimodal understanding capabilities by utilizing structure-preserving sparse visual sequences and cross-layer enhanced visual representations. It is suitable for resource-constrained edge scenarios such as mobile phones and embedded devices.

[0148] Based on the above embodiments, the present invention achieves the following beneficial effects:

[0149] (1) This invention constructs a modality-aware pruning mechanism to identify modal attributes of each layer of the model, and combines static weight distribution and dynamic activation statistics to adaptively allocate asymmetric sparsity rate, thereby achieving differentiated structural compression of visual, text, and fusion modules. This method can effectively maintain cross-modal semantic consistency while significantly reducing the model parameter size and memory access overhead, solving the problems of low compression efficiency and severe semantic degradation of traditional unified pruning strategies.

[0150] (2) This invention constructs a cross-layer integrated fusion module to extract multi-scale hidden features at different levels from the visual encoder, and performs joint splicing and nonlinear mapping along the channel dimension to achieve collaborative modeling of low-level detailed information and high-level abstract semantics. This mechanism significantly enhances the semantic density and hierarchical expressive power of visual features without increasing the number of tokens in the input language model, effectively solving the problem of insufficient semantic utilization in traditional single-layer projection methods.

[0151] (3) This invention designs a structured token adaptive selection mechanism. By constructing a neighborhood continuity perception scoring model, the importance of visual tokens is smoothly enhanced, and dynamic selection is completed while ensuring structural continuity. This mechanism can maintain spatial semantic integrity while reducing the visual computational burden, and significantly improve the stability and fine-grained visual understanding ability of the model under highly sparse inference conditions.

[0152] Example 2

[0153] To verify the effectiveness of the proposed Multimodal Structured Sparse Inference Optimization Method (MACS) for resource-constrained edge devices, this embodiment conducts a systematic evaluation on several mainstream multimodal visual language understanding benchmark datasets, including MME Perception, GQA, ScienceQA-IMG, TextVQA, and MMBench-dev.

[0154] The experimental setup is as follows: The overall architecture of MACS is as follows: Figure 2 As shown, the architecture comprises three core components: edge modality-aware pruning, sparse cross-layer stitching and fusion, and structured token adaptation selection. This MACS framework is integrated into the MobileVLMV2-3B model, with the visual encoder using CLIPViT-L / 14 and the language model using MobileLLaMA-2.7B-Chat. All images are scaled to 336×336 resolution, and the visual token budget is set to k=64. To simulate realistic resource-constrained edge scenarios, a batch size of 1 and a greedy decoding strategy are employed. Comparison methods include models such as LLaVA-1.5, Qwen-VL, and MoE-LLaVA, as well as efficient inference frameworks such as ToMe, FastV, PDrop, and SparseVLM.

[0155] Overall Performance Comparison: As shown in Table 1 below, when the number of visual tokens is compressed to k=96, MACS achieves performance comparable to the full visual input model on multiple benchmarks. It reaches 68.9% on ScienceQA-IMG, outperforming LLaVA-1.5 (66.8%) and some larger-scale Qwen-VL series models. On GQA, TextVQA, MME Perception, and MMBench-dev, MACS maintains stable performance with an average accuracy of 63.5%, approaching that of comparative models with larger parameter scales, validating its ability to effectively preserve key visual information while significantly reducing computational overhead.

[0156] Table 1

[0157]

[0158] Comparison of sparsification mechanisms: such as Figure 3 As shown, on TextVQA, GQA, and MMBench, as the visual token budget decreases from 144 to 64, the performance degradation of MACS is significantly smaller than that of methods such as ToMe and FastV. In particular, it still maintains a high accuracy when k=64, which proves the stability of the structure-preserving dynamic token sparsity mechanism proposed in this invention under extreme compression conditions.

[0159] Efficiency Comparison: As shown in Table 2, compared to the full-token baseline (N=144), MACS reduces FLOPs by 26.4% at k=96, accelerates prefill latency by 1.36 times, and saves 23.1% of GPU memory; at k=64, FLOPs are reduced by 43.8%, the speedup is 1.78 times, and GPU memory is saved by 38.5%. Meanwhile, intermediate activation usage decreases from 0.39GB to 0.24GB, indicating that MACS significantly alleviates the computational and memory access bottlenecks of the language model's self-attention module.

[0160] Table 2

[0161]

[0162] Ablation study comparison: As shown in Table 3, after removing the STAS module, FLOPs increased from 0.629T to 1.178T, and peak memory usage increased to 6.49GB, indicating that STAS is key to achieving efficient compression. After removing the CLIF module, the MMBEnch accuracy decreased by 0.43 percentage points, proving that cross-layer fusion effectively compensates for semantic loss without increasing computational cost. Although removing the MAP module slightly reduced FLOPs, it could not stabilize the activation distribution, and memory behavior was uncontrollable when deployed on the edge. However, after introducing MAP, the model maintained a lower peak memory usage (6.31GB).

[0163] Table 3

[0164]

[0165] Impact of Token Budget: As shown in Table 4, reducing k=144 to k=64 only decreases the average accuracy by 0.47 percentage points, while increasing the speedup to 1.78 times; further compression leads to a significant performance decline. Therefore, k=64 is the recommended default configuration, achieving a good balance between accuracy and efficiency.

[0166] Table 4

[0167]

[0168] The core design of MAP (Modality Aware Pruning) lies in its modality-sensitive adaptive sparsity allocation mechanism. As the underlying optimization foundation of the MACS framework, MAP is responsible for de-redundant parameter processing of the visual encoder and language model before inference. This invention observes that visual, textual, and fused modal components in a multimodal model exhibit different activation characteristics and semantic redundancy when processing information. Therefore, MAP abandons the traditional indiscriminate global pruning strategy and, by identifying modal attributes at different levels, customizes an asymmetric sparsity scheme for them. The specific implementation process of MAP is as follows... Figure 5As shown, the process mainly includes three steps: (1) Modal layer grouping: by parsing the keywords in the network layer names, each layer is classified into visual, text, or multimodal fusion categories; (2) Dual-domain importance estimation: the static weight distribution characteristics and dynamic activation response characteristics of each layer are statistically analyzed to construct a dual-domain fusion scoring function; (3) Adaptive sparsity allocation: the comprehensive score is mapped to the pruning interval corresponding to each modality, an exclusive adaptive structured sparsity rate is calculated for each layer, and amplitude thresholding pruning is performed. This process realizes differentiated compression of visual, text, and cross-modal fusion modules, effectively reducing the model parameter scale and memory access overhead.

[0169] To intuitively understand the impact of different visual token budgets on performance as shown in Table 4, this embodiment... Figure 6 The paper presents a qualitative visualization of the selected visual tokens at different sparsity levels. From left to right, it shows the distribution of key visual tokens retained by the STAS module as the visual token budget k gradually decreases from 144, 96, and 64. Observations show that when k=144, the model retains almost all visual information, but computational redundancy is high; when k=96, STAS begins to filter out background noise while basically maintaining the object outline and local structure; when k=64, although the number of tokens is halved, the model can still stably capture discriminative structures such as object edges and key interaction regions, while the traditional Top-K method shows obvious structural fragmentation at this sparsity. This visualization further verifies that the structural continuity perception scoring mechanism proposed in this invention can effectively maintain visual semantic integrity under high sparsity conditions.

[0170] In summary, this invention achieves efficient and stable inference of multimodal models on resource-constrained edge devices through three major mechanisms: modality-aware differential pruning, cross-layer multi-scale visual enhancement, and structure-preserving token sparsification. Compared with existing methods, it has significant performance-efficiency advantages.

[0171] Example 3

[0172] Figure 7 This is a schematic diagram of the structure of a multimodal structured sparse inference optimization system 700 for resource-constrained edge devices provided in Embodiment 3 of the present invention, as shown below. Figure 7 As shown, the system includes:

[0173] The modal analysis module 710 is used to identify the modal attributes of different network layers in a multimodal model and to statistically analyze the static weight distribution characteristics and dynamic activation response characteristics of each network layer.

[0174] The differential pruning module 720 is used to adaptively allocate asymmetric structured sparsity to different network layers based on the modal attributes of each network layer, combined with their static weight distribution characteristics and dynamic activation response characteristics, and perform structured pruning to obtain a compressed multimodal model.

[0175] The cross-layer feature fusion module 730 is used to extract multi-layer visual hidden features from the compressed multimodal model, align the multi-layer visual hidden features and splice and fuse them along the channel dimension, and generate enhanced visual representations through a nonlinear mapping network.

[0176] The structure perception scoring module 740 is used to calculate the initial importance score of each visual token in the enhanced visual representation, and to smooth and enhance the initial importance score according to the spatial neighborhood relationship between visual tokens to generate a structure continuity perception score.

[0177] The Token sparsification module 750 is used to select key visual tokens based on the structural continuity perception score to form a sparse visual token sequence.

[0178] The inference execution module 760 is used to receive text prompts input by the user, input the sparse visual token sequence and the text prompts together into the language decoding module of the compressed multimodal model for inference, and generate inference results.

[0179] The multimodal structured sparse inference optimization system for resource-constrained edge devices provided in this embodiment of the invention can execute the multimodal structured sparse inference optimization method for resource-constrained edge devices provided in any of the embodiments of the invention above. It has the corresponding functions and beneficial effects of executing the multimodal structured sparse inference optimization method for resource-constrained edge devices. For detailed process, please refer to the relevant operations of the multimodal structured sparse inference optimization method for resource-constrained edge devices in the foregoing embodiments.

[0180] Example 4

[0181] Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, and may also represent various forms of mobile devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0182] like Figure 8As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0183] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0184] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 executes a multimodal structured sparse inference optimization method for resource-constrained edge devices as described above.

[0185] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0186] The above embodiments are merely illustrative examples and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multimodal structured sparse reasoning optimization method for resource-constrained edge devices, characterized in that, include: Identify the modal attributes of different network layers in a multimodal model, and statistically analyze the static weight distribution characteristics and dynamic activation response characteristics of each network layer, including: The modal attributes of each network layer are determined based on the keywords in the name of each network layer; the activation tensors output by each network layer are collected through forward propagation in the offline calibration stage, and the average absolute activation amplitude and activation sparsity of the activation tensors are statistically analyzed as the dynamic activation response characteristics of the network layer; the entropy, kurtosis and Frobenius norm of the weights of each network layer are calculated as the static weight distribution characteristics of the network layer. Based on the modal attributes of each network layer, combined with its static weight distribution characteristics and dynamic activation response characteristics, an asymmetric structured sparsity rate is adaptively assigned to different network layers, and structured pruning is performed to obtain a compressed multimodal model, including: The pruning intervals for each network layer are determined based on its modal attributes. The static weight distribution features and dynamic activation response features of each network layer are input into a dual-domain fusion scoring function for scoring. Based on the scoring results, an asymmetric structured sparsity rate is adaptively assigned to each network layer within its corresponding pruning interval. Structured pruning is performed based on the asymmetric structured sparsity rate of each network layer to obtain a compressed multimodal model. Multi-layer visual hidden features are extracted from the compressed multimodal model, and these features are aligned and spliced ​​along the channel dimension. An enhanced visual representation is then generated through a nonlinear mapping network. Calculate the initial importance score of each visual token in the enhanced visual representation, and smooth the initial importance score according to the spatial neighborhood relationship between visual tokens to generate a structural continuity perception score, including: For each visual token in the enhanced visual representation, its activation intensity in the channel dimension is calculated as the initial importance score of the visual token; based on the neighborhood index relationship of each visual token in the spatial flattening order, the initial importance score of each visual token and the initial importance scores of other tokens in its neighborhood are weighted and aggregated to obtain the smooth enhancement score of the visual token; the initial importance score and the smooth enhancement score are weighted and fused to obtain the structural continuity perception score of each visual token. Based on the structural continuity perception score, key visual tokens are selected to form a sparse visual token sequence. The system receives text prompts input by the user, and inputs the sparse visual token sequence and the text prompts together into the language decoding module of the compressed multimodal model for inference, generating inference results.

2. The multimodal structured sparse reasoning optimization method for resource-constrained edge devices according to claim 1, characterized in that, The static weight distribution characteristics and dynamic activation response characteristics of each network layer are input into a dual-domain fusion scoring function for scoring. Based on the scoring results, an asymmetric structured sparsity is adaptively assigned to each network layer within its corresponding pruning interval, including: The static weight distribution characteristics of each network layer are input into the dual-domain fusion scoring function to calculate the weight-side adjustment score of the network layer. The dynamic activation response features of each network layer are input into the dual-domain fusion scoring function to calculate the activation-side scissorability score of the network layer. The comprehensive evaluation score is obtained by weighting and fusing the weight-side adjustment score and the activation-side scissorability score of each network layer. The comprehensive evaluation score of each network layer is mapped to the corresponding pruning interval of each network layer, and the asymmetric structured sparsity is adaptively assigned to each network layer.

3. The multimodal structured sparse reasoning optimization method for resource-constrained edge devices according to claim 2, characterized in that, Structured pruning is performed based on the asymmetric structured sparsity of each network layer to obtain a compressed multimodal model, including: Based on the asymmetric structured sparsity of each network layer, determine the corresponding quantile threshold for the absolute value distribution of all weights in that network layer. In this network layer, the weights whose absolute values ​​are less than the quantile threshold are reset to zero, while the remaining weights are retained. By traversing all network layers and performing pruning operations, a compressed multimodal model is obtained.

4. The multimodal structured sparse reasoning optimization method for resource-constrained edge devices according to claim 1, characterized in that, Multi-layer visual hidden features are extracted from the compressed multimodal model, and these features are aligned and then spliced ​​and fused along the channel dimension. An enhanced visual representation is then generated using a nonlinear mapping network, including: Extract multi-layer visual hidden features from the visual encoder of the compressed multimodal model; Remove the classification tokens from the visual hidden features of each layer to ensure that the features of each layer are consistent in the sequence length dimension; The aligned visual hidden features of each layer are stitched together along the channel dimension to obtain multi-scale fused features; The multi-scale fused features are input into a multilayer perceptron and sequentially subjected to linear transformation, activation function, and another linear transformation to generate an enhanced visual representation that matches the input dimension of the language decoding module.

5. The multimodal structured sparse reasoning optimization method for resource-constrained edge devices according to claim 1, characterized in that, Based on the structural continuity perception score, key visual tokens are selected to form a sparse visual token sequence, including: The visual tokens are sorted according to the structural continuity perception score. Retain a preset number of visual tokens that are ranked at the top as key visual tokens; Features corresponding to the key visual tokens are extracted from the enhanced visual representation to form a sparse visual token sequence.

6. The multimodal structured sparse inference optimization method for resource-constrained edge devices according to claim 5, characterized in that, The sparsed visual token sequence and the text prompt are input together into the language decoding module of the compressed multimodal model for inference, generating inference results, including: The sparse visual token sequence and the text prompt are concatenated along the sequence dimension to form a multimodal input sequence; The multimodal input sequence is input into the language decoding module of the compressed multimodal model, and autoregressive inference is performed to generate inference results word by word.

7. A multimodal structured sparse inference optimization system for resource-constrained edge devices, characterized in that, The system is used to execute the multimodal structured sparse inference optimization method for resource-constrained edge devices according to any one of claims 1 to 6, the system comprising: The modal analysis module is used to identify the modal attributes of different network layers in a multimodal model, and to statistically analyze the static weight distribution characteristics and dynamic activation response characteristics of each network layer. The differential pruning module is used to adaptively allocate asymmetric structured sparsity to different network layers based on their modal attributes, combined with their static weight distribution characteristics and dynamic activation response characteristics, and to perform structured pruning to obtain a compressed multimodal model. The cross-layer feature fusion module is used to extract multi-layer visual hidden features from the compressed multimodal model, align the multi-layer visual hidden features and splice and fuse them along the channel dimension, and generate enhanced visual representations through a nonlinear mapping network. The structure-aware scoring module is used to calculate the initial importance score of each visual token in the enhanced visual representation, and to smooth and enhance the initial importance score according to the spatial neighborhood relationship between visual tokens to generate a structure continuity perception score. The Token sparsification module is used to select key visual tokens based on the structural continuity perception score to form a sparse visual token sequence. The inference execution module is used to receive text prompts input by the user, input the sparse visual token sequence and the text prompts together into the language decoding module of the compressed multimodal model for inference, and generate inference results.

Images