ViT optimization method combining adaptive hybrid expert and lightweight attention

Abstract

The invention belongs to the technical field of computer vision, and discloses a ViT optimization method combining adaptive hybrid experts and lightweight attention, which comprises the following steps: acquiring a data set comprising an input image, and dividing the input image into a plurality of image blocks; performing linear embedding and position coding on the plurality of image blocks to form a feature sequence, and inputting the feature sequence into a ViT backbone network formed by connecting a plurality of encoder layers in series; in each encoder layer, input features are processed through a lightweight attention module and a self-adaptive hybrid expert module in sequence; wherein the lightweight attention module dynamically selects and activates part of attention heads through a gating network, and the activated attention heads share a query projection matrix and a key projection matrix; the problems that in the prior art, visual converter model parameters are redundant, the calculation efficiency is low, and parameter simplification is difficult to balance and feature diversity is difficult to maintain in the model optimization process are effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a ViT optimization method that combines adaptive hybrid experts and lightweight attention. Background Technology

[0002] In recent years, ViT has leveraged self-attention mechanisms to model global image features, gradually replacing Convolutional Neural Networks (CNNs) as the mainstream architecture in tasks such as image classification, object detection, and semantic segmentation. However, ViT models typically have some shortcomings: their parameter count is often quite large, especially in the Multi-Head Self-Attention (MHA) module, where each attention head needs to independently maintain the query (Q), key (K), and value (V) projection matrix, directly leading to significant parameter redundancy. Simultaneously, all attention heads are fully activated in every layer, failing to consider the differences in importance between different attention heads, thus resulting in unnecessary waste of computational resources. Furthermore, the standard Feed-Forward Network (FFN) in the model uses a fixed-width structure, failing to distinguish the differences in feature abstraction capabilities between different layers, ultimately leading to parameter redundancy in some deep structures.

[0003] The Mixture of Experts (MoE) mechanism offers a sparse activation approach to address partial redundancy issues. By selecting a subset of experts for each input image block (token) during inference, the model effectively improves its expressive power with a slight increase in computational cost. Current expert routing networks still suffer from several core problems, with the commonly used Top-k approach... The routing mechanism forces the selection of a fixed number of experts, which not only results in some invalid experts passively participating in the calculation, but also fails to match the actual needs of different input samples for the number of experts. Meanwhile, the routing threshold, as a fixed hyperparameter, is difficult to adaptively adjust according to the training process and data distribution, directly limiting the model's convergence efficiency and generalization ability. Furthermore, the unified expert dimension design adopted by most methods, due to insufficient consideration of the differentiated needs of feature representation at different levels, may simultaneously lead to insufficient shallow feature representation and redundant deep structures. On the other hand, optimization studies on attention mechanisms show that the contributions of different attention heads to task performance are not balanced. Existing works have reduced the computational cost of attention through pruning, low-rank decomposition, or weight sharing, but these methods either destroy structural integrity or sacrifice the expressive differences between attention heads, making it difficult to achieve a balance between "reducing redundancy" and "maintaining diversity." Therefore, existing technologies suffer from redundant visual transformer model parameters, low computational efficiency, and difficulty in balancing parameter simplification and maintaining feature diversity during model optimization. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a ViT optimization method that combines adaptive hybrid experts and lightweight attention, thereby solving the problems of redundant visual transformer model parameters, low computational efficiency, and difficulty in balancing parameter simplification and feature diversity during model optimization in existing technologies.

[0005] The objective of this invention can be achieved through the following technical solutions: A ViT optimization method combining adaptive hybrid experts and lightweight attention includes the following steps: Obtain the dataset including the input image and divide the input image into multiple image patches; Multiple image patches are linearly embedded and positionally encoded to form a feature sequence, which is then input into the ViT backbone network, which consists of multiple encoder layers connected in series. In each encoder layer, the input features are processed sequentially through a lightweight attention module and an adaptive hybrid expert module; The lightweight attention module dynamically selects and activates some attention heads through a gating network. The activated attention heads share the query projection matrix and the key projection matrix, and a differentiated transformation matrix is ​​introduced to maintain the diversity among attention heads, outputting an attention-enhanced feature sequence. The adaptive hybrid expert module receives the attention-enhanced feature sequence and dynamically selects and activates some experts from a preset expert pool for each feature position in the feature sequence based on a preset learnable threshold. It also performs differentiated configuration of the capacity of each expert according to the network depth and outputs the expert-enhanced feature sequence. The expert-enhanced feature sequence output by the last encoder layer is used as the output result of the ViT backbone network. Based on the output of the ViT backbone network and combined with attention head selection and expert selection information, a total loss function is constructed that includes visual task loss, attention head load balancing loss and expert load balancing loss. The parameters of the ViT backbone network are optimized by optimizing the total loss function.

[0006] Furthermore, the lightweight attention module processes the input feature sequence, specifically including the following steps: Based on the feature sequence of the input lightweight attention module, a score for each attention head is generated through a gating network; Based on the scores of the attention heads, select the highest-scoring attention heads from all attention heads according to a preset ratio and activate them; The activated attention heads share the query projection matrix and the key projection matrix, and each activated attention head is independently configured with a value projection matrix and a differentiated transformation matrix. Based on the query projection matrix, key projection matrix, and differentiated transformation matrix, the query matrix and key matrix of each activated attention head are calculated respectively, and the value matrix corresponding to each attention head is calculated based on the value projection matrix. Based on the query matrix, key matrix, and value matrix of the activated attention head, scaled dot product attention calculation is used to obtain the output of each activated attention head; The outputs of each activated attention head are weighted and concatenated according to learnable coefficients, and then output through the output matrix to obtain the attention-enhanced feature sequence.

[0007] Furthermore, based on the query projection matrix, key projection matrix, and differentiated transformation matrix, the query matrix and key matrix for each activated attention head are calculated, specifically including the following steps: Shared query projection matrix Bond projection matrix After introducing a differentiated transformation matrix Calculate the first i The query matrix and key matrix of each activated attention head are expressed as follows: in, This represents the query feature sequence after transformation by the shared query projection matrix; The diagram shows the key feature sequence after transformation by the shared key projection matrix; Represented as the first i A differentiated transformation matrix for each activated attention head configuration.

[0008] Furthermore, the adaptive hybrid expert module receives and processes the attention-enhanced feature sequence, specifically including the following steps: For the received attention-enhanced feature sequence, the routing network calculates the scores of each expert in the corresponding preset expert pool for each feature position in the attention-enhanced feature sequence; A preset learnable threshold is set, and experts with scores greater than or equal to the learnable threshold are selected for each feature position in the attention-enhanced feature sequence to obtain the set of activated experts for each feature position. If any feature location is activated, the expert in the expert set E i Greater than the preset maximum value E max Then, in the activated expert set at the corresponding position, only the one with the highest score is retained. E max One expert; The scores of each expert in the activated expert set at each feature location are normalized to obtain the routing probability of each expert at each feature location. Based on the routing probability, the output of the feedforward network of the expert corresponding to each feature location is weighted and summed to obtain the output of each feature location; The outputs of each feature location are combined to form an expert-enhanced feature sequence.

[0009] Furthermore, the scores of each expert in the activated expert set at each feature location are normalized to obtain the routing probability of each expert at each feature location. The specific calculation formula is as follows: in, Represents image blocks i Experts corresponding to feature locations e The routing probability; Represents image blocks i For experts e The score; Represents image blocks i For experts k The score; Represents an image block i For experts e Score The value of the natural exponential function; Represents an image block i Activated expertsk Score The value of the natural exponential function; Represents image blocks i The final set of activated experts corresponding to the feature locations; The specific calculation formula for the expert's feedforward network output is as follows: in, Represents image blocks i Corresponding activated expert group experts e The output; Experts e The second-level linear transformation weight matrix; Represents the ReLU nonlinear activation function; expert e The first-level linear transformation weight matrix; Represents image blocks i Input feature vector Based on the routing probability, the weighted summation of the feedforward network outputs of the expert corresponding to each feature location is performed to obtain the output of each feature location, as shown in the following expression: in, Represents image blocks i Output corresponding to the feature location.

[0010] Furthermore, based on the output of the ViT backbone network and combined with attention head selection and expert selection information, a total loss function is constructed that includes visual task loss, attention head load balancing loss, and expert load balancing loss. This specifically includes the following steps: Calculate the visual task loss based on the output of the ViT backbone network. ; Based on the attention head selection information during the lightweight attention module processing, the attention head load balancing loss is calculated. ; Based on the expert selection information during the adaptive hybrid expert module processing, the expert load balancing loss is calculated. ; visual task loss Attention head load balancing loss and expert load balancing losses We perform weighted combination to construct the total loss function. L The specific expression is as follows: in, and Expert load balancing losses and attention head load balancing loss The weighting coefficients.

[0011] Furthermore, calculate the attention head load balancing loss. The specific expression is as follows: in, H Indicates the total number of attention points; Indicates attention head h Frequency of normalization usage in the current training batch; N This represents the total number of feature locations in the current training batch; when the feature location... i Assigned to attention head h hour, Otherwise 0 ; Indicates attention head h The average selection probability in the current training batch; Indicates location features i Assign attention to head h of softmax Gating probability; Calculate expert load balancing losses The specific expression is as follows: in, E Indicates the total number of experts; Experts e Frequency of normalization usage in the current training batch; when feature location i Assigned to experts e hour, Otherwise 0 ; Indicates location features i Assigned to experts e of softmax Gating probability; Computational vision task loss The specific expression is as follows: in, Indicates the training batch size; The model represents the first b The predicted probability distribution of each sample; Indicates the first b The true class label of each sample.

[0012] The beneficial effects of this invention are: This invention significantly improves the parameter efficiency and computational performance of the Vision Transformer (ViT) through the synergistic cooperation of a lightweight attention module and an adaptive hybrid expert module. The lightweight attention module employs a gating network to dynamically select and activate some attention heads, sharing query projection matrices and key projection matrices while introducing differentiated transformation matrices. This significantly reduces the number of parameters while ensuring rich feature representation. The adaptive hybrid expert module, based on a preset learnable threshold, achieves adaptive expert selection and differentiates the capacity of each expert according to network depth. This avoids the rigid limitations of traditional fixed routing and effectively reduces computational redundancy. A total loss function is constructed, incorporating visual task loss, attention head load balancing loss, and expert load balancing loss. This effectively prevents experts or attention heads from being idle or overloaded for extended periods, improving model training stability and generalization performance. This effectively solves the problems of parameter redundancy, low computational efficiency, and difficulty in balancing parameter simplification and feature diversity during model optimization in existing vision transformers. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of the ViT model of the present invention performing image classification; Figure 2 This is a schematic diagram of the ViT model structure of the present invention; Figure 3 This is a schematic diagram of the lightweight attention module processing of the present invention; Figure 4 This is a schematic diagram of the adaptive hybrid expert module processing procedure of the present invention; Figure 5 This is a schematic diagram comparing experimental results in an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] like Figures 1 to 5 As shown, a ViT optimization method combining adaptive hybrid experts and lightweight attention includes the following steps: Obtain the dataset including the input image and divide the input image into multiple image patches; Multiple image patches are linearly embedded and positionally encoded to form a feature sequence, which is then input into the ViT backbone network, which consists of multiple encoder layers connected in series. In each encoder layer, the input features are processed sequentially through a lightweight attention module and an adaptive hybrid expert module; The lightweight attention module dynamically selects and activates some attention heads through a gating network. The activated attention heads share the query projection matrix and the key projection matrix, and a differentiated transformation matrix is ​​introduced to maintain the diversity among attention heads, outputting an attention-enhanced feature sequence. The adaptive hybrid expert module receives the attention-enhanced feature sequence and dynamically selects and activates some experts from a preset expert pool for each feature position in the feature sequence based on a preset learnable threshold. It also performs differentiated configuration of the capacity of each expert according to the network depth and outputs the expert-enhanced feature sequence. The expert-enhanced feature sequence output by the last encoder layer is used as the output result of the ViT backbone network. Based on the output of the ViT backbone network and combined with attention head selection and expert selection information, a total loss function is constructed that includes visual task loss, attention head load balancing loss and expert load balancing loss. The parameters of the ViT backbone network are optimized by optimizing the total loss function. It should be noted that the dataset is divided into training set, validation set and test set in a ratio of 7:1:2; Preferably, the input image (resolution 224×224 pixels) is divided into 16×16 image blocks (the block size can be adjusted according to hardware resources, and it must be ensured that the image size is divisible by the block size). Each image patch is mapped to a feature vector through a linear embedding layer, with an embedding dimension d=96; Preferably, the ViT backbone network uses 12 encoder layers connected in series; wherein, a single encoder layer performs feature transformation in the order of "lightweight attention module → residual connection and layer normalization → adaptive hybrid expert module → residual connection and layer normalization"; In this application, the adaptive hybrid expert module completely replaces the feedforward network in the traditional ViT to achieve efficient computation of sparse activation; Preferably, the gated network adopts a structure of "linear layer (96→32) + GELU activation + linear layer (32→12)".

[0017] This invention significantly improves the parameter efficiency and computational performance of the Vision Transformer (ViT) through the synergistic cooperation of a lightweight attention module and an adaptive hybrid expert module. The lightweight attention module employs a gating network to dynamically select and activate some attention heads, sharing query projection matrices and key projection matrices while introducing differentiated transformation matrices. This significantly reduces the number of parameters while ensuring rich feature representation. The adaptive hybrid expert module, based on a preset learnable threshold, achieves adaptive expert selection and differentiates the capacity of each expert according to network depth. This avoids the rigid limitations of traditional fixed routing and effectively reduces computational redundancy. A total loss function is constructed, incorporating visual task loss, attention head load balancing loss, and expert load balancing loss. This effectively prevents experts or attention heads from being idle or overloaded for extended periods, improving model training stability and generalization performance. This effectively solves the problems of parameter redundancy, low computational efficiency, and difficulty in balancing parameter simplification and feature diversity during model optimization in existing vision transformers.

[0018] The lightweight attention module processes the input feature sequence, specifically including the following steps: Based on the feature sequence of the input lightweight attention module, a score for each attention head is generated through a gating network; Based on the scores of the attention heads, select the highest-scoring attention heads from all attention heads according to a preset ratio and activate them; The activated attention heads share the query projection matrix and the key projection matrix, and each activated attention head is independently configured with a value projection matrix and a differentiated transformation matrix. Based on the query projection matrix, key projection matrix, and differentiated transformation matrix, the query matrix and key matrix of each activated attention head are calculated respectively, and the value matrix corresponding to each attention head is calculated based on the value projection matrix. Based on the query matrix, key matrix, and value matrix of the activated attention head, scaled dot product attention calculation is used to obtain the output of each activated attention head; The outputs of each activated attention head are weighted and concatenated according to learnable coefficients, and then output through the output matrix to obtain the attention-enhanced feature sequence. Preferably, the preset proportion of attention points selected is 75%.

[0019] Based on the query projection matrix, key projection matrix, and differentiated transformation matrix, the query matrix and key matrix for each activated attention head are calculated, specifically including the following steps: Shared query projection matrix Bond projection matrix After introducing a differentiated transformation matrix Calculate the first iThe query matrix and key matrix of each activated attention head are expressed as follows: in, This represents the query feature sequence after transformation by the shared query projection matrix; The diagram shows the key feature sequence after transformation by the shared key projection matrix; Represented as the first i A differentiated transformation matrix for each activated attention head configuration; The output of the activated attention head is obtained by using scaled dot product attention calculation, as shown in the following expression: in, Indicates the first i The output of an activated attention head; The outputs of each activated attention head are weighted and concatenated according to learnable coefficients, and then output through the output matrix to obtain the attention-enhanced feature sequence. The specific expression is as follows: in, This represents an attention-enhanced feature sequence; Indicates the output matrix; This indicates splicing / joining. Indicates the activated expert set; Choose an indicator function for the head level, if attention head ,but Otherwise 0 ; The adaptive hybrid expert module receives and processes attention-enhanced feature sequences, specifically including the following steps: For the received attention-enhanced feature sequence, the routing network calculates the scores of each expert in the corresponding preset expert pool for each feature position in the attention-enhanced feature sequence; A preset learnable threshold is set, and experts with scores greater than or equal to the learnable threshold are selected for each feature position in the attention-enhanced feature sequence to obtain the set of activated experts for each feature position. If any feature location is activated, the expert in the expert set E i Greater than the preset maximum value E max Then, in the activated expert set at the corresponding position, only the one with the highest score is retained. E max One expert; The scores of each expert in the activated expert set at each feature location are normalized to obtain the routing probability of each expert at each feature location. Based on the routing probability, the output of the feedforward network of the expert corresponding to each feature location is weighted and summed to obtain the output of each feature location; The outputs of each feature location are combined to form an expert-enhanced feature sequence.

[0020] A preset learnable threshold θ is set, and θ is constrained to the interval (0,1) by the Sigmoid function: in, This represents the Sigmoid activation function. This represents the learnable parameters preset by the model; Preferably, the total number of experts in the expert pool is 4; E max Set to 2; The scores of each expert in the activated expert set at each feature location are normalized to obtain the routing probability of each expert at each feature location. The specific calculation formula is as follows: in, Represents image blocks i Experts corresponding to feature locations e The routing probability; Represents image blocks i For experts e The score; Represents image blocks i For experts k The score; Represents an image block i For experts e Score The value of the natural exponential function; Represents an image block i Activated experts k Score The value of the natural exponential function; Represents image blocks i The final set of activated experts corresponding to the feature locations; The specific calculation formula for the expert's feedforward network output is as follows: in, Represents image blocks i Corresponding activated expert group experts e The output; Experts e The second-level linear transformation weight matrix; Represents the ReLU nonlinear activation function; expert e The first-level linear transformation weight matrix; Represents image blocks i The input feature vector; Based on the routing probability, the weighted summation of the feedforward network outputs of the expert corresponding to each feature location is performed to obtain the output of each feature location, as shown in the following expression: in, Represents image blocks i Output corresponding to the feature location; To further improve the rationality of expert dimension selection, this invention compares the representation differences of different layers before and after hybrid expert replacement based on the Centralized Kernel Alignment (CKA) metric. The similarity of the CKA of the hybrid expert layer output of layer l with the baseline layer output of the data-efficient image transformer model is considered. Less than the similarity corresponding to the original FFN structure Then update the expert hidden layer dimension. The specific expression is as follows: Where the scaling factor λ is 0.75, and the minimum dimension is... Experimental results show that performing this dimensionality reduction on layers 7 to 12 (from 768 to 576) can reduce the number of parameters by about 25% without decreasing classification accuracy.

[0021] Based on the output of the ViT backbone network and combined with attention head selection and expert selection information, a total loss function is constructed, which includes visual task loss, attention head load balancing loss, and expert load balancing loss. The specific steps are as follows: Calculate the visual task loss based on the output of the ViT backbone network. ; Based on the attention head selection information during the lightweight attention module processing, the attention head load balancing loss is calculated. ; Based on the expert selection information during the adaptive hybrid expert module processing, the expert load balancing loss is calculated. ; visual task loss Attention head load balancing loss and expert load balancing losses We perform weighted combination to construct the total loss function. L The specific expression is as follows: in, and Expert load balancing losses and attention head load balancing loss Weighting coefficients; Preferably, the values ​​of α and β are both 0.2.

[0022] Calculate the attention head load balancing loss The specific expression is as follows: in, H Indicates the total number of attention points; Indicates attention head h Frequency of normalization usage in the current training batch; N This represents the total number of feature locations in the current training batch; when the feature location... i Assigned to attention head h hour, Otherwise 0 ; Indicates attention head h The average selection probability in the current training batch; Indicates location features i Assign attention to head h of softmax Gating probability; Calculate expert load balancing losses The specific expression is as follows: in, E Indicates the total number of experts; Experts e Frequency of normalization usage in the current training batch; when feature location i Assigned to experts e hour, Otherwise 0 ; Indicates location features i Assigned to experts e of softmax Gating probability; Computational vision task loss The specific expression is as follows: in, Indicates the training batch size; The model represents the first b The predicted probability distribution of each sample; Indicates the first b The true class label of each sample.

[0023] The specific process for classifying input images in the test set after combining the training set and validation set, and optimizing ViT using the ViT optimization method combining adaptive hybrid expert and lightweight attention as proposed in this application, is as follows: Figure 1 As shown, the input image is first segmented and embedded, and then its position is encoded. Then, it is input into a lightweight attention module for dynamic attention head selection and shared projection of the selected head. After residual connection and normalization, the generated attention-enhanced feature sequence is input into an adaptive hybrid expert module for dynamic routing threshold filtering and differential dimension configuration. Finally, after residual connection and normalization, the final linear classification head is output to produce the classification result.

[0024] To verify the effectiveness of the method, this embodiment conducted classification and ablation experiments on multiple datasets such as ImageNet-1K, Aircraft, and Cars, compared the performance of the method of the present invention with some classic or advanced methods, and analyzed the impact of various technical details of the present invention on the overall method effect. In this embodiment, ImageNet-1K is used as the image classification benchmark, which includes 1.28 million training images, 50,000 validation images and 100,000 test images, covering 1,000 categories. It is based on WordNet hierarchical annotation and is widely used to evaluate large-scale image classification models. The transfer learning experiments in this embodiment cover the CIFAR-10, CIFAR-100, Flowers, Aircraft, Cars, and Pets datasets, all of which are widely used for transfer learning and image classification tasks. The experiment used Top-1 accuracy as the primary evaluation metric; the model was based on the ViT-Tiny architecture, combined with the parametrically efficient ViT (PE-ViT) framework (referred to as PE-ViT-Tiny), and the number of attention heads increased from 3 to 12; The model is first pre-trained on ImageNet-1K using a masked autoencoder (MAE) and then fine-tuned on downstream datasets; Regarding the experimental environment: All experiments in this embodiment were conducted on a computer with Ubuntu 20.04 LTS operating system installed. The training hardware platform used 10 RTX 3090 GPUs, CUDA version 12.1, and PyTorch version 2.0.

[0025] Experimental results: (1) Layer representation similarity analysis Layer representation analysis was used to guide the selection of intermediate dimensions for expert layers. Using ViT-Tiny as the baseline and DeiT-Tiny as the reference model, the inter-layer representation similarity of different layers was examined. The similarity comparison results are as follows: Figure 5 As shown, MoE-MAE-Tiny replaces the MLP layer of MAE-Tiny with a MoE layer; dark areas indicate high similarity, and light areas indicate low similarity. Table 1: Impact of Expert Dimensions on Accuracy The introduction of MoE significantly improves the alignment of shallow representations, but has limited improvement on deep representations, indicating that deep experts contribute little to representation learning and may have parameter redundancy. The results in Table 1 verify that the reduction of shallow dimension leads to a significant decrease in accuracy (-0.9%), while moderate dimensionality reduction of deep layers (e.g., from 768 to 576) does not affect performance, confirming the redundancy of deep experts.

[0026] (2) Evaluation results on the ImageNet dataset Comparing the PE-ViT-Tiny and D-MAE-Tiny baselines (both based on the ViT-Tiny architecture) on ImageNet-1K, PE-ViT-Tiny not only improves accuracy but also reduces convergence time by 25%, as shown in Table 2. Despite the large number of total parameters, the sparse MoE layer achieves efficient expert specialization and accelerates convergence. Table 2: Model Performance Comparison (Parameters, Accuracy, Training Time) As shown in Table 3, PE-ViT-Tiny achieves an accuracy of 79.8% with 95M parameters, surpassing ViT-B (87M, 78.6%) and MoE-ViT-B (128M, 77.9%), demonstrating a significant advantage in parameter efficiency. Table 3: Comparison of ImageNet-1K Top-1 Accuracy (3) Transfer learning assessment Generalization ability was evaluated on six datasets, comparing PE-ViT-Tiny with supervised / self-supervised baselines using the ViT-Tiny backbone: As shown in Table 4, PE-ViT-Tiny achieves the highest accuracy on the CIFAR-10, CIFAR-100, Aircraft, and Cars datasets, especially significantly outperforming DeiT-Tiny on Aircraft (+8.7%) and Cars (+3.2%), demonstrating its cross-task adaptability. In the table, bold indicates the best result for the metric, underline indicates the second best result for the metric, and - indicates that the authors of the model did not conduct experiments on this dataset. Table 4: Evaluation of Transfer Learning in Image Classification Task (4) Ablation test As shown in Table 5, the performance of traditional attention and SSA is compared under different projection dimensions (Dk). SSA maintains 79.8% accuracy when Dk=96, and the number of parameters is reduced by 40.4% (from 0.57M to 0.34M), which verifies its parameter efficiency. Table 5: Comparison of Traditional Attention and SSA under Different Projection Dimensions Dk As shown in Table 6, removing the adaptive hybrid expert module resulted in a 1.5% decrease in accuracy, verifying its enhancement effect on feature representation. Table 6: Adaptive Hybrid Expert Module Ablation Experiment As shown in Table 7, it is verified that SSA can reduce the attention module parameters from 0.57M to 0.34M while maintaining accuracy, and the redundancy is significantly reduced.

[0027] Table 7: SSA module ablation experiment In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0028] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A ViT optimization method combining adaptive hybrid experts and lightweight attention, characterized in that, Includes the following steps: Obtain the dataset including the input image and divide the input image into multiple image patches; Multiple image patches are linearly embedded and positionally encoded to form a feature sequence, which is then input into the ViT backbone network, which consists of multiple encoder layers connected in series. In each encoder layer, the input features are processed sequentially through a lightweight attention module and an adaptive hybrid expert module; The lightweight attention module dynamically selects and activates some attention heads through a gating network. The activated attention heads share the query projection matrix and the key projection matrix, and a differentiated transformation matrix is ​​introduced to maintain the diversity among attention heads, outputting an attention-enhanced feature sequence. The adaptive hybrid expert module receives the attention-enhanced feature sequence and dynamically selects and activates some experts from a preset expert pool for each feature position in the feature sequence based on a preset learnable threshold. It also performs differentiated configuration of the capacity of each expert according to the network depth and outputs the expert-enhanced feature sequence. The expert-enhanced feature sequence output by the last encoder layer is used as the output result of the ViT backbone network. Based on the output of the ViT backbone network and combined with attention head selection and expert selection information, a total loss function is constructed that includes visual task loss, attention head load balancing loss and expert load balancing loss. The parameters of the ViT backbone network are optimized by optimizing the total loss function.

2. The ViT optimization method combining adaptive hybrid experts and lightweight attention as described in claim 1, characterized in that, The lightweight attention module processes the input feature sequence, specifically including the following steps: Based on the feature sequence of the input lightweight attention module, a score for each attention head is generated through a gating network; Based on the scores of the attention heads, select the highest-scoring attention heads from all attention heads according to a preset ratio and activate them; The activated attention heads share the query projection matrix and the key projection matrix, and each activated attention head is independently configured with a value projection matrix and a differentiated transformation matrix. Based on the query projection matrix, key projection matrix, and differentiated transformation matrix, the query matrix and key matrix of each activated attention head are calculated respectively, and the value matrix corresponding to each attention head is calculated based on the value projection matrix. Based on the query matrix, key matrix, and value matrix of the activated attention head, scaled dot product attention calculation is used to obtain the output of each activated attention head; The outputs of each activated attention head are weighted and concatenated according to learnable coefficients, and then output through the output matrix to obtain the attention-enhanced feature sequence.

3. The ViT optimization method combining adaptive hybrid experts and lightweight attention as described in claim 2, characterized in that, Based on the query projection matrix, key projection matrix, and differentiated transformation matrix, the query matrix and key matrix for each activated attention head are calculated, specifically including the following steps: Shared query projection matrix Bond projection matrix After introducing a differentiated transformation matrix Calculate the first i The query matrix and key matrix of each activated attention head are expressed as follows: in, This represents the query feature sequence after transformation by the shared query projection matrix; The diagram shows the key feature sequence after transformation by the shared key projection matrix; Represented as the first i A differentiated transformation matrix for each activated attention head configuration.

4. The ViT optimization method combining adaptive hybrid experts and lightweight attention as described in claim 1, characterized in that, The adaptive hybrid expert module receives and processes attention-enhanced feature sequences, specifically including the following steps: For the received attention-enhanced feature sequence, the routing network calculates the scores of each expert in the corresponding preset expert pool for each feature position in the attention-enhanced feature sequence; A preset learnable threshold is set, and experts with scores greater than or equal to the learnable threshold are selected for each feature position in the attention-enhanced feature sequence to obtain the set of activated experts for each feature position. If any feature location is activated, the expert in the expert set E i Greater than the preset maximum value E max Then, in the activated expert set at the corresponding position, only the one with the highest score is retained. E max One expert; The scores of each expert in the activated expert set at each feature location are normalized to obtain the routing probability of each expert at each feature location. Based on the routing probability, the output of the feedforward network of the expert corresponding to each feature location is weighted and summed to obtain the output of each feature location; The outputs of each feature location are combined to form an expert-enhanced feature sequence.

5. The ViT optimization method combining adaptive hybrid experts and lightweight attention as described in claim 4, characterized in that, The scores of each expert in the activated expert set at each feature location are normalized to obtain the routing probability of each expert at each feature location. The specific calculation formula is as follows: in, Represents image blocks i Experts corresponding to feature locations e The routing probability; Represents image blocks i For experts e The score; Represents image blocks i For experts k The score; Represents an image block i For experts e Score The value of the natural exponential function; Represents an image block i Activated experts k Score The value of the natural exponential function; Represents image blocks i The final set of activated experts corresponding to the feature locations; The specific calculation formula for the expert's feedforward network output is as follows: in, Represents image blocks i Corresponding activated expert group experts e The output; Experts e The second-level linear transformation weight matrix; Represents the ReLU nonlinear activation function; expert e The first-level linear transformation weight matrix; Represents image blocks i The input feature vector; Based on the routing probability, the weighted summation of the feedforward network outputs of the expert corresponding to each feature location is performed to obtain the output of each feature location, as shown in the following expression: in, Represents image blocks i Output corresponding to the feature location.

6. The ViT optimization method combining adaptive hybrid experts and lightweight attention as described in claim 1, characterized in that, Based on the output of the ViT backbone network and combined with attention head selection and expert selection information, a total loss function is constructed, which includes visual task loss, attention head load balancing loss, and expert load balancing loss. The specific steps are as follows: Calculate the visual task loss based on the output of the ViT backbone network. ; Based on the attention head selection information during the lightweight attention module processing, the attention head load balancing loss is calculated. ; Based on the expert selection information during the adaptive hybrid expert module processing, the expert load balancing loss is calculated. ; visual task loss Attention head load balancing loss and expert load balancing losses We perform weighted combination to construct the total loss function. L The specific expression is as follows: in, and Expert load balancing losses and attention head load balancing loss The weighting coefficients.

7. The ViT optimization method combining adaptive hybrid experts and lightweight attention as described in claim 6, characterized in that, Calculate the attention head load balancing loss The specific expression is as follows: in, H Indicates the total number of attention points; Indicates attention head h Frequency of normalization usage in the current training batch; N This represents the total number of feature locations in the current training batch; when the feature location... i Assigned to attention head h hour, Otherwise 0 ; Indicates attention head h The average selection probability in the current training batch; Indicates location features i Assign attention to head h of softmax Gating probability; Calculate expert load balancing losses The specific expression is as follows: in, E Indicates the total number of experts; Experts e Frequency of normalization usage in the current training batch; when feature location i Assigned to experts e hour, Otherwise 0 ; Indicates location features i Assigned to experts e of softmax Gating probability; Computational vision task loss The specific expression is as follows: in, Indicates the training batch size; The model represents the first b The predicted probability distribution of each sample; Indicates the first b The true class label of each sample.

Images