A visual transformer model evaluation method based on cumulative attention map and Jaccard similarity coefficient

Abstract

The present invention discloses a method for evaluating visual transformer models based on cumulative attention maps and Jaccard similarity coefficients. The method comprises the following steps: initialization; training a reference visual transformer model based on full attention; training multiple alternative visual transformer models based on grouped attention; recording the attention maps of key layers in the visual transformer model during forward propagation of a small amount of calibration data; determining the cumulative attention maps of key layers in the visual transformer model; determining the Jaccard similarity coefficient; and selecting the optimal visual transformer model based on the Jaccard similarity coefficient. The present invention can improve the difference index among grouped attention models with minimal differences in final performance, making it easier to find a more superior and interpretable model.

Description

Technical Field

[0001] The present invention belongs to the field of visual perception technology, and in particular relates to a visual transformer model evaluation method based on cumulative attention maps and Jaccard similarity coefficients. Background Art

[0002] In the field of visual perception, Transformer-based vision backbone networks have achieved remarkable success in recent years. Since the introduction of the Vision Transformer (ViT), the Transformer architecture has successfully introduced the self-attention mechanism used in natural language processing to computer vision tasks. In the Vision Transformer architecture, an image is first divided into a series of fixed-size image patches. After information is extracted from these patches, image tokens are formed. These image tokens are then treated as sequential data and input into the Transformer, thereby modeling global image information. This innovation enables the Transformer to perform well in tasks such as image classification, object detection, and semantic segmentation.

[0003] However, its significant computational and memory footprint has hindered its further development. Specifically, the transformer's self-attention mechanism requires computing the relationships between all image patches, resulting in a computational complexity that is proportional to the square of the number of image annotations. When processing high-resolution images, the number of annotations can be enormous, leading to an exponential increase in computational and storage requirements. This poses significant challenges in practical applications, especially on resource-constrained devices such as mobile devices and embedded systems.

[0004] Therefore, many researchers have introduced grouped attention as an alternative to full attention, and have proposed numerous visual transformer models based on grouped attention. The core idea of ​​these models is to divide all image landmarks into several groups and compute self-attention only within each group, thereby reducing computational complexity and significantly reducing overhead. Grouped attention not only reduces computational complexity but also captures local features, similar to the local receptive field in convolutional neural networks. This approach has demonstrated a good balance between performance and efficiency in practical applications. While these proposed methods also incorporate other improvements, such as new position representations (such as relative position encoding and coordinate embedding), additional convolutional layers for feature extraction, adjustments to normalization layers, and optimization of activation functions, the key difference between these models lies in how they group image landmarks. For example, the Swin Transformer uses a sliding window mechanism to restrict self-attention computation to a local window and gradually integrates global information through cross-window connections. The CSwin Transformer, on the other hand, partitions the feature map into multiple cross-shaped windows, computing self-attention separately in the horizontal and vertical directions. The cross-shaped design enables the model to capture longer-range dependencies within each layer.

[0005] However, the final model performance is very similar between different models. For example, in the image classification task on the ImageNet dataset (a large-scale benchmark dataset for computer vision tasks), all other conditions being equal, the first-class classification accuracy of different grouped attention models differed by only around 1%. This small performance difference may not be noticeable in real-world applications, and may even be statistically insignificant. This makes choosing which grouped attention method to use in practice difficult.

[0006] This phenomenon may be due to the fact that these models are very similar in overall architecture, differing only in the way they group and label images. Furthermore, factors such as training strategy, hyperparameter settings, and data preprocessing may have a greater impact on model performance than the grouping strategy itself. Therefore, relying solely on final metrics such as first-class accuracy to evaluate a model is insufficient to fully reflect its actual performance and application value. Summary of the Invention

[0007] The purpose of the present invention is to provide a visual transformer model evaluation method based on cumulative attention map and Jaccard similarity coefficient to solve the above technical problems.

[0008] To achieve the purpose of the present invention, the present invention provides a visual transformer model evaluation method based on cumulative attention map and Jaccard similarity coefficient, comprising the following steps:

[0009] Step 1: Select the specific visual perception task, dataset, model architecture settings, and complete the initialization settings;

[0010] Step 2: Using the visual perception task, dataset, and model architecture, train a reference full-attention-based visual transformer model.

[0011] Step 3: Using the visual perception task, dataset, and model architecture settings, train multiple alternative grouped attention-based visual transformer models;

[0012] Step 4: Select a small amount of calibration data from the dataset, input the small amount of calibration data into the full-attention-based visual transformer model and the grouped-attention-based visual transformer model, and record the attention map of the key layer in the visual transformer model during the forward propagation of the small amount of calibration data;

[0013] Step 5: determining a cumulative attention map of the key layer in the visual transformer model according to the attention map of the key layer in the visual transformer model;

[0014] Step 6: Compare the cumulative attention map of each candidate group-attention-based visual transformer model with the cumulative attention map of the corresponding position of the visual transformer model to determine the Jaccard similarity coefficient;

[0015] Step 7: Select the optimal visual converter model according to the Jaccard similarity coefficient.

[0016] Compared with the existing technology, the significant progress of the present invention lies in: 1) The present invention proposes a visual transformer model evaluation method based on cumulative attention maps and Jaccard similarity coefficient, which provides a new perspective for model selection in addition to the final performance index. This method not only considers the overall performance of the model, but also conducts a detailed analysis of the model from the perspective of the attention mechanism, thereby providing a more comprehensive basis for model selection; 2) The present invention utilizes the characteristics of the attention mechanism in the visual transformer model to intuitively display the attention distribution within the model through the cumulative attention map. This attention map-based evaluation method makes the model behavior more transparent and easy to understand. In addition, the Jaccard similarity coefficient, as an intuitive metric, further enhances the interpretability of the evaluation results; 3) In the existing visual transformer model, when only relying on final performance indicators such as precision and recall for comparison, the differences between models may be very small, making it difficult to distinguish between good and bad. However, by calculating the Jaccard similarity coefficient between cumulative attention maps, the present invention can significantly reflect the differences in the attention mechanisms of different models. This difference is more obvious than traditional performance indicators, thereby providing a more intuitive and reliable basis for model selection.

[0017] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 is a flow chart of the present invention;

[0020] Figure 2 It is a line graph of the Jaccard similarity coefficient of the cumulative attention graphs of the three visual transformers based on grouped attention and the visual transformer model based on complete attention in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0022] The present invention is a method for evaluating a visual transformer model based on cumulative attention graph and Jaccard similarity coefficient, combined with Figure 1 , including the following steps:

[0023] Step 1: Select the specific visual perception task, dataset, model architecture settings, and complete the initialization settings;

[0024] The initialization settings specifically include:

[0025] Step 1-1: Select a specific visual perception task and dataset as the basis for subsequent comparison; in this example, the image classification task and the ImageNet dataset are selected.

[0026] Step 1-2: Select the model architecture settings. Since various models will be compared in various forms later, the model architecture design needs to be unified. In the full-attention-based visual transformer model and the group-attention-based visual transformer model, the number of stages and model blocks is equal; the number of image tags within the corresponding position model blocks is equal; in the group-attention-based visual transformer model, the number of groups within the corresponding position model blocks is equal. Referring to the common designs of various visual transformers, in this embodiment, the image size for training and inference is 224 × 224, the number of image tags in the four stages is 56 × 56, 28 × 28, 14 × 14, and 7 × 7, respectively, and the group size for group attention is 49. Considering representativeness and diversity, this embodiment selects three classic visual transformer models: the sliding window transformer, the shuffle transformer, and the cross-domain transformer.

[0027] Step 2: Using the visual perception task, data set, and model architecture settings, train a reference full-attention-based visual transformer model; according to the initialization settings, using the visual perception task, data set, and model architecture settings, select appropriate training strategies and hyperparameter settings to train the full-attention-based visual transformer model to convergence.

[0028] Step 3: Utilizing the visual perception task, dataset, and model architecture, multiple candidate grouped attention-based visual transformer models are trained. Based on the initialization settings, the multiple candidate grouped attention-based visual transformer models are trained to convergence using the visual perception task, dataset, and model architecture, maintaining the same training settings as the full attention-based visual transformer. Experimental conditions are standardized to fairly evaluate the impact of the grouped attention mechanism on model performance. This example trains a sliding window transformer, a shuffled transformer, and a cross-domain transformer to convergence.

[0029] Step 4: Select a small amount of calibration data from the dataset, input the small amount of calibration data into the full-attention-based visual transformer model and the grouped-attention-based visual transformer model, and record the attention map of the key layer in the visual transformer model during the forward propagation of the small amount of calibration data;

[0030] Step 4-1, selecting a small amount of calibration data from the dataset. The calibration data is selected from a small amount of training set data or validation set data in the dataset. In this embodiment, 512 is selected.

[0031] Step 4-2, input the small amount of calibration data into the visual transformer model, and record the attention map of the key layer in the visual transformer model during the forward propagation of the small amount of calibration data. The attention map exists in the calculation process of self-attention, which shows how the model allocates attention when processing input data, that is, the degree of attention of each image tag to other image tags; specifically, in the self-attention mechanism, when the image tag is input, a query (Query, Q), key (Key, K) and value (Value, V) vector are generated, and the attention map assigns weights to the calculation of the value V vector. Attention map of the layer for:

[0032] ;

[0033] in, is the dimension of the key vector, used for scaling to prevent the value from being too large; the elements of the attention map Indicates in In the layer, Image tag pair The higher the value, the more important it is.

[0034] In this embodiment, the layers in the third stage are selected, and the third stage has a large number of layers, 18 layers in total.

[0035] Step 5: determining a cumulative attention map of the key layer in the visual transformer model according to the attention map of the key layer in the visual transformer model;

[0036] The attention map in the previous step is an incomplete representation of the mutual attention relationship between the image tags in the visual transformer model. To more accurately represent the mutual attention relationship between the image tags in the visual transformer model, the calculation of the cumulative attention map should be used. Based on the attention map, in the In the layer, Image tag pair The cumulative attention of image tags is as follows, where k represents the index of all possible intermediate image tags in the current layer l:

[0037] ;

[0038] In addition, the residual connection transfers the information of the image label from the previous layer to the corresponding position of the next layer, so it is necessary to add the cumulative attention map of the current layer to the Add the identity matrix I, and finally the entire cumulative attention map is calculated using matrix multiplication:

[0039] ;

[0040] in, is the cumulative attention map of layer l−1.

[0041] Step 6: Compare the cumulative attention map of each candidate group-attention-based visual transformer model with the cumulative attention map of the corresponding position of the visual transformer model to determine the Jaccard similarity coefficient;

[0042] The Jaccard similarity coefficient is defined as the ratio of the size of the intersection of two sets to the size of the union, and the value range is between 0 and 1. The larger the value, the more similar the two sets are. and the second attention map The similarity between them is expressed as:

[0043] ;

[0044] The Jaccard similarity coefficient of the cumulative attention graphs of the corresponding layers of the full attention-based visual transformer model and the grouped attention-based visual transformer model is calculated. The specific steps are as follows:

[0045] Obtaining Attention Maps: Extracting attention maps of corresponding layers from two different visual transformer models;

[0046] Calculate the cumulative attention map: For each layer, summarize or average the results of all attention heads of the layer to obtain the cumulative attention map of the layer;

[0047] Compute intersection and union: For each pair of cumulative attention maps (from the same layer of the two models), compute their intersection and union;

[0048] Calculate the Jaccard similarity coefficient: Use the above formula , calculate the Jaccard similarity coefficient for each pair of cumulative attention maps.

[0049] Step 7, select the optimal visual transformer model according to the Jaccard similarity coefficient: compare the Jaccard similarity coefficients of each visual transformer model based on grouped attention with the visual transformer model based on complete attention, and reveal which visual transformer model's attention mechanism is closer to the complete visual transformer model; since such comparison is performed in each visual transformer model, no threshold is set, and the evaluation is performed by comparing the visual transformer models with each other. Figure 2 In the comparison of similarity coefficients of each layer in the third stage, the similarity coefficients of the cross-domain transformer and the shuffled transformer are much higher than those of the sliding window transformer. This also shows that the attention forms of the cross-domain transformer and the shuffled transformer are closer to the visual transformer model based on complete attention. Therefore, they are more suitable visual transformer models under this evaluation standard.

[0050] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0051] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A visual transformer model evaluation method based on cumulative attention map and Jaccard similarity coefficient, characterized in that: The following steps are involved: Step 1: Select the specific visual perception task, dataset, model architecture settings, and complete the initialization settings; Step 2: Using the visual perception task, dataset, and model architecture, train a reference full-attention-based visual transformer model. Step 3: Using the visual perception task, dataset, and model architecture settings, train multiple alternative grouped attention-based visual transformer models; Step 4: Select a small amount of calibration data from the dataset, input the small amount of calibration data into the full-attention-based visual transformer model and the grouped-attention-based visual transformer model, and record the attention map of the key layer in the visual transformer model during the forward propagation of the small amount of calibration data; Step 5: determining a cumulative attention map of the key layer in the visual transformer model according to the attention map of the key layer in the visual transformer model; Step 6: Compare the cumulative attention map of each candidate group-attention-based visual transformer model with the cumulative attention map of the corresponding position of the visual transformer model to determine the Jaccard similarity coefficient; The Jaccard similarity coefficient is defined as the ratio of the size of the intersection of two sets to the size of the union, and the value range is between 0 and 1. The larger the value, the more similar the two sets are. and the second attention map The similarity between them is expressed as: ； The Jaccard similarity coefficient of the cumulative attention graphs of the corresponding layers of the full attention-based visual transformer model and the grouped attention-based visual transformer model is calculated as follows: Obtaining Attention Maps: Extracting attention maps of corresponding layers from two different visual transformer models; Calculate the cumulative attention map: For each layer, summarize or average the results of all attention heads of the layer to obtain the cumulative attention map of the layer; Compute intersection and union: For each pair of accumulated attention maps, calculate their intersection and union; Calculate the Jaccard similarity coefficient: Use the above formula , calculate the Jaccard similarity coefficient of each pair of cumulative attention maps; Step 7: Select the optimal visual converter model according to the Jaccard similarity coefficient.

2. The method for evaluating a visual transformer model based on cumulative attention graph and Jaccard similarity coefficient according to claim 1, wherein: The initialization settings of step 1 specifically include: Step 1-1: Select a specific visual perception task and dataset; Step 1-2: Select the model architecture settings and unify the model architecture design; in the full attention-based visual transformer model and the group attention-based visual transformer model, the number of stages and model blocks are equal; the number of image tags in the corresponding position model blocks is equal; in the group attention-based visual transformer model, the number of groups in the corresponding position model blocks is equal.

3. The method for evaluating a visual transformer model based on cumulative attention graph and Jaccard similarity coefficient according to claim 1, wherein: The step 2 specifically includes: according to the initialization settings, using the visual perception task, data set, and model architecture settings, selecting appropriate training strategies and hyperparameter settings, and training the full attention-based visual transformer model to convergence.

4. The method for evaluating a visual transformer model based on cumulative attention graph and Jaccard similarity coefficient according to claim 3, wherein: The specific steps of step 3 are as follows: according to the initialization settings, using the visual perception task, data set, and model architecture settings, maintaining the same training settings as the full attention-based visual transformer, and training multiple alternative grouped attention-based visual transformer models until convergence.

5. The method for evaluating a visual transformer model based on cumulative attention graph and Jaccard similarity coefficient according to claim 1, wherein: The step 4 specifically includes: Step 4-1, selecting a small amount of calibration data from the dataset, wherein the calibration data is selected from a small amount of training set data or validation set data in the dataset; Step 4-2, input the small amount of calibration data into the visual transformer model, record the attention map of the key layer in the visual transformer model during the forward propagation of the small amount of calibration data, and the attention map exists in the calculation process of self-attention; specifically, in the self-attention mechanism, when the image tag is input, the query Q, key K and value V vector are generated, and the attention map assigns weights to the calculation of the value V vector. Attention map of the layer for: ； in, is the dimension of the key vector, used for scaling to prevent the value from being too large; the elements of the attention map Indicates in In the layer, Image tag pair The higher the value, the more important it is.

6. The method for evaluating a visual transformer model based on cumulative attention graph and Jaccard similarity coefficient according to claim 5, characterized in that: The step 5 specifically includes: Based on the attention map, In the layer, Image tag pair The cumulative attention of image tags is as follows, where k represents the index of all possible intermediate image tags in the current layer l: ； Cumulative attention map to the current layer Add the identity matrix I, and finally the entire cumulative attention map is calculated using matrix multiplication: ； in, is the cumulative attention map of layer l−1.

7. The method for evaluating a visual transformer model based on cumulative attention graph and Jaccard similarity coefficient according to claim 1, wherein: The specific steps of step 7 are as follows: comparing the Jaccard similarity coefficients of the visual transformer models based on grouped attention with the visual transformer models based on complete attention, and determining whether the attention mechanism of one of the visual transformer models is closer to the complete visual transformer model; since such comparison is performed in each visual transformer model, no threshold is set, and the results are evaluated by comparing each visual transformer model with each other.

Images