Multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory biasing

Abstract

The application relates to the field of artificial intelligence and feature enhancement technology, and provides a multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias, which comprises the following steps: acquiring a visual token sequence; calculating the weight of multi-head attention and single-head output; constructing a trajectory vector containing at least attention intensity, active attention intensity, entropy, change rate, divergence, historical gating, CLS bidirectional attention, CLS rollout accumulation and spatial neighborhood mean by using cross-layer attention; layer-in standardization; projecting into a bias injection current token representation; combining the bias representation and the trajectory to generate a layer-by-layer, head-by-head, token-by-token and channel-by-channel gating; writing into a cache, and obtaining an effective gating through strength coefficient conversion, and then modulating channel by channel before splicing; splicing and modulating output and projecting. The method can inhibit redundant propagation, retain token sequences and enhance foreground representation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and feature enhancement technology, and in particular to a multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias. Background Technology

[0002] The VisionTransformer (ViT) divides the input image into fixed-size non-overlapping image patches, transforms them into patch tokens through linear projection, and adds a classification token (CLS token) to the beginning of the sequence. After superimposing positional encoding, the resulting token sequence is input into the Transformer encoder. Variants (such as DeiT) introduce strategies such as distillation training to improve image recognition performance.

[0003] The core of the visual Transformer is a multi-head self-attention mechanism. For the h-th attention head in the l-th layer:

[0004]

[0005]

[0006]

[0007] ; in, This represents the normalized output of the layer before the attention module in layer l. , and The first Layer The query, key, and value projection matrix of each attention head. Represents the attention weight matrix. This represents the single-head attention output, where d is the single-head feature dimension.

[0008] The above calculations are performed independently in each network layer and each attention head, forming a layer-by-layer propagation representation of patchtoken and CLStoken. The attention weights of CLStoken on patchtoken can, to some extent, reflect the model's tendency to select information from different image regions when performing global semantic modeling.

[0009] In visual Transformers, background regions, repetitive texture regions, or patch tokens with low semantic information density may consistently receive high attention weights across multiple network layers, but their contribution to the final classification, detection, or segmentation results is limited, resulting in a redundant propagation phenomenon of "high attention, low contribution." This redundant propagation consumes attention computational resources and may interfere with the feature representation of key foreground region patch tokens.

[0010] To reduce the computational cost of visual Transformers or suppress the propagation of invalid information, traditional solutions mainly optimize in three directions: Structural compression includes hard pruning methods such as patchtoken pruning (e.g., PatchMerger, DynamicViT) and attention head pruning (e.g., headpruning methods based on attention importance scores).

[0011] Computational sparsity includes methods such as Spatial Reduction Attention (SRA) and Axial Attention, which reduce computational complexity by limiting the range of attention connections between patch tokens.

[0012] Dynamic modulation: including soft weighting methods based on gating, such as token-level attention gating, which calculates token-level gating values ​​based on the token representation or attention weights of the current layer.

[0013] However, the above solution still has the following shortcomings: Traditional attention gating generates gating values ​​based solely on information from the current layer, making it difficult to identify long-term redundancy patterns formed by the continuous propagation of patch tokens across multiple network layers.

[0014] Traditional patchtoken pruning uses a hard deletion method, which may delete patchtokens with low current scores but potential contributions to subsequent layers, resulting in irreversible information loss.

[0015] Traditional headpruning removes the entire attention head, which is coarse-grained and makes it difficult to distinguish the differences in the propagation state of different patch tokens or different feature channels within the same attention head.

[0016] Traditional spatial attention sparsification alters the connection structure between patch tokens, but it cannot determine whether a patch token remains in a state of high attention and low contribution for an extended period during cross-layer propagation.

[0017] Traditional solutions lack systematic modeling of the attention propagation trajectory of patch tokens across layers and heads, and cannot utilize the unique CLS semantic convergence relationship and two-dimensional patch space neighborhood structure for redundant discrimination. Summary of the Invention

[0018] The purpose of this invention is to solve at least one technical problem in the background art and to provide a multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias.

[0019] To achieve the above objectives, this invention provides a multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias, applied to visual Transformer networks, comprising: Obtain the visual token sequence corresponding to the input image, wherein the visual token sequence contains a classification token and multiple image patch tokens; Multi-head attention calculation is performed on the visual token sequence through multiple network layers of the visual Transformer network to obtain the attention weight matrix of each attention head in each network layer and the single-head attention output of each attention head; For the target token in the visual token sequence, a cross-layer visual propagation trajectory vector of the target token is constructed using the attention weight matrix of each attention head in the current network layer and the historical network layer. The cross-layer visual propagation trajectory vector includes at least the average window attention intensity, the average window active attention intensity, the current layer attention entropy, the cross-layer attention intensity change rate, the multi-head divergence degree, the historical gating mean, the attention intensity from the classification token to the target token, the attention intensity from the target token to the classification token, the cumulative intensity of cross-layer classification attention propagation, and the mean attention intensity based on the image patch token spatial neighborhood. The cross-layer visual propagation trajectory vectors of all visual tokens within the same network layer are standardized according to the feature dimension to obtain normalized trajectory vectors. By using a learnable projection matrix and a bias vector, the normalized trajectory vector is mapped to the model hidden dimension to obtain a trajectory projection bias vector. The trajectory projection bias vector is then added to the target token's token representation in the current network layer to obtain a biased token representation. Using at least the biased token representation and the normalized trajectory vector as input, the original gating vectors are generated per network layer, per attention head, per token, and per channel. The original gating vector is written into the gating history cache, and the attention weight matrix of the current network layer is written into the attention history cache for use by subsequent network layers when constructing the cross-layer visual propagation trajectory vector; A gating strength coefficient is introduced to transform the original gating vector to obtain an effective gating vector; After the single-head attention output of each attention head and before the multi-head attention concatenation, the effective gating vector is multiplied channel by channel with the single-head attention output of the corresponding attention head to obtain the modulated single-head attention output. The modulated single-head attention outputs of all attention heads in the same network layer are concatenated, and the input token representation of the next network layer is obtained through the output projection matrix.

[0020] According to one aspect of the present invention, the average attention intensity of the window is the average intensity of the target token being followed by all query tokens under each network layer and each attention head within a preset historical window as a key token; the average active attention intensity of the window is the average intensity of the target token following all key tokens under each network layer and each attention head within the preset historical window as a query token; and the current layer attention entropy is the average value of the attention distribution entropy of the target token under each attention head of the current network layer.

[0021] According to one aspect of the invention, the cross-layer attention intensity variation rate is the absolute value of the difference between the average attention intensity of the window in the current network layer and the average attention intensity of the window in the adjacent previous network layer; the multi-head divergence is the variance of the total attention intensity of the target token for different attention heads in the current network layer.

[0022] According to one aspect of the present invention, when constructing the cross-layer visual propagation trajectory vector, obtaining the historical gating mean includes: Read the original gating values ​​of all visual tokens in each attention head and each channel from the pre-set number of historical network layers before the current network layer from the gating history cache, and extract the original gating value of the target token; After averaging the original gating values ​​of the target token along the channel dimension, the average values ​​are then averaged along the historical network layer dimension and the attention head dimension to obtain the historical gating mean of the target token. When the gating history cache is empty, the historical gating average is set to a preset constant.

[0023] According to one aspect of the present invention, obtaining the cumulative intensity of the cross-layer classification attention propagation includes: The average attention matrix is ​​obtained by averaging the attention weight matrices of all attention heads in the current network layer. The average attention matrix is ​​fused with an identity matrix of the same dimension using equal weights, and the fusion result is normalized by row to obtain the residual normalized attention matrix. Based on the residual normalized attention matrix, the cross-layer attention propagation cumulative matrix up to the current network layer is calculated recursively. Extract the element values ​​corresponding to the category token row and the target token column from the cross-layer attention propagation accumulation matrix, and use them as the cross-layer classification attention propagation accumulation intensity.

[0024] According to one aspect of the present invention, the acquisition of the mean attention intensity based on the spatial neighborhood of image patch tokens includes: Determine the spatial neighborhood of the target token in a two-dimensional image block token grid, wherein the spatial neighborhood is centered on the image block token corresponding to the target token; Calculate the arithmetic mean of the window average attention intensity of each neighborhood token within the spatial neighborhood to obtain the mean attention intensity of the spatial neighborhood based on the image patch token; For the classified token, the average attention intensity of its neighborhood based on the image patch token space is set to be equal to the average attention intensity of its window.

[0025] According to one aspect of the present invention, standardizing the cross-layer visual propagation trajectory vector includes: For the same sample and the same network layer, calculate the mean and standard deviation of the trajectory features for each dimension across all visual tokens; The mean and standard deviation are used to standardize the trajectory features in each dimension to obtain the normalized trajectory vector.

[0026] According to one aspect of the present invention, a gating strength coefficient is introduced to transform the original gating vector, comprising: The difference between the gating strength coefficient and the original gating vector is multiplied by a vector of the same dimension and whose elements are all 1, to obtain the first product. Multiply the gating strength coefficient by the original gating vector to obtain the second product; Add the first product to the second product to obtain the effective gating vector; The gating strength coefficient ranges from 0 to 1. When the gating strength coefficient is 0, the effective gating vector is a vector with the same dimension as the original gating vector and all elements are 1. When the gating strength coefficient is 1, the effective gating vector is equal to the original gating vector.

[0027] According to one aspect of the invention, the method further includes: When the amount of data in the attention history cache or the gated history cache exceeds the preset window depth, the earliest written historical data is removed.

[0028] To achieve the above objectives, the present invention also provides a multi-head dynamic soft gating system based on cross-layer visual attention propagation trajectory bias, applied to visual Transformer networks, comprising: The token sequence acquisition module acquires the visual token sequence corresponding to the input image. The visual token sequence contains a category token and multiple image block tokens. The attention calculation module performs multi-head attention calculation on the visual token sequence through multiple network layers of the visual Transformer network to obtain the attention weight matrix of each attention head in each network layer and the single-head attention output of each attention head. The trajectory construction module, for the target token in the visual token sequence, constructs a cross-layer visual propagation trajectory vector of the target token using the attention weight matrix of each attention head in the current network layer and the historical network layer; wherein, the cross-layer visual propagation trajectory vector includes at least the average window attention intensity, the average window active attention intensity, the current layer attention entropy, the cross-layer attention intensity change rate, the multi-head divergence degree, the historical gating mean, the attention intensity from the classification token to the target token, the attention intensity from the target token to the classification token, the cumulative intensity of cross-layer classification attention propagation, and the mean attention intensity based on the image patch token spatial neighborhood; The normalization module normalizes the cross-layer visual propagation trajectory vectors of all visual tokens within the same network layer according to the feature dimension to obtain normalized trajectory vectors. The bias injection module maps the normalized trajectory vector to the model hidden dimension using a learnable projection matrix and a bias vector to obtain a trajectory projection bias vector. The trajectory projection bias vector is then added to the target token's token representation in the current network layer to obtain the biased token representation. The gating generation module takes at least the biased token representation and the normalized trajectory vector as input to generate original gating vectors per network layer, per attention head, per token, and per channel. The cache update module writes the original gate vector into the gate history cache and writes the attention weight matrix of the current network layer into the attention history cache for use by subsequent network layers when constructing the cross-layer visual propagation trajectory vector. The effective gating module introduces a gating strength coefficient to transform the original gating vector to obtain an effective gating vector; The soft modulation module performs channel-by-channel multiplication of the effective gating vector with the single-head attention output of the corresponding attention head after the single-head attention output of each attention head and before the multi-head attention splicing, to obtain the modulated single-head attention output. The output projection module concatenates the modulated single-head attention outputs of all attention heads in the same network layer, and obtains the input token representation of the next network layer through the output projection matrix.

[0029] To achieve the above objectives, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias as described above.

[0030] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias as described above.

[0031] According to the present invention, the cross-layer visual propagation trajectory vector records the attention status of the target token in multiple network layers and multiple attention heads. At the same time, it integrates visual priors such as the semantic convergence relationship of classification tokens, the accumulation of cross-layer attention propagation, and the spatial neighborhood structure of image patches. This enables the visual attention signal to be characterized in the composite propagation pattern in the image space and network depth direction. As a result, the gating network can learn to recognize visual redundancy propagation patterns that are difficult to perceive by traditional methods (such as the propagation pattern that background patch tokens and repeated texture patch tokens may continue to receive high attention in multiple layers but have relatively limited contribution to task discrimination). This allows the gating module to make correction decisions based on the visual propagation history rather than just the current layer state, thereby suppressing the continuous propagation of redundant information between layers.

[0032] This invention projects the normalized trajectory vector as a bias and incorporates the current token representation to obtain a biased token representation. This allows low-dimensional trajectory information to directly alter the state of the high-dimensional token received by the gating network, forming a biased trajectory path. Simultaneously, the original normalized trajectory vector still participates in gating generation as an explicit feature, creating a dual-path trajectory input. This mechanism enhances the influence of low-dimensional trajectory information on gating decisions, prevents trajectory features from being overwhelmed by the high-dimensional token representation in the concatenated input, and improves the gating's sensitivity to visual propagation states.

[0033] This invention uses network layer embedding and attention head embedding as gating auxiliary inputs, enabling the gating network to distinguish the functional states of different network depths and different attention heads, thereby learning gating strategies that are adapted to the layer position and attention head type, and improving the adaptability of gating modulation in the multi-layer multi-head structure of the visual Transformer.

[0034] This invention introduces a gating strength coefficient to perform a residual transformation on the original gating vector. When the gating strength coefficient is 0, the effective gating vector is a vector with the same dimension as the original gating vector and all elements being 1, without modulating the single-head output. When the gating strength coefficient is 1, the effective gating vector is equal to the original gating vector, fully enabling dynamic modulation. This residual gating mechanism, by retaining part of the identity propagation path, can reduce the risk of over-suppression caused by random initialization of the gating value in the early stages of training, and maintain the stability of gradient backpropagation. In one embodiment, the gating strength coefficient can be set to a fixed value; in another embodiment, it can be gradually increased as the training progresses to gradually enhance the gating modulation effect.

[0035] This invention applies channel-wise soft modulation after each attention head completes its single-head attention output and before multi-head stitching. This ensures that each attention head independently adjusts its output intensity based on the cross-layer visual propagation trajectory of the same target token, preserving the functional differentiation of the multi-head mechanism and avoiding confusion between the representations of different heads caused by uniform modulation. At the same time, it utilizes the continuous values ​​of the original gating in the (0,1) interval and converts them into effective gating through the gating intensity coefficient. It does not directly delete any patch tokens, maintains the correspondence between the token sequence structure and the two-dimensional image grid, and reduces the risk of irreversible information loss caused by hard pruning.

[0036] This invention writes the original gating vector of the current layer into the gating history cache and the attention weight matrix into the attention history cache, enabling subsequent network layers to read historical gating states and participate in trajectory construction, forming a cross-layer closed-loop adjustment. The gating decision of the current layer thus affects the trajectory features of subsequent layers, and the gating of subsequent layers is adjusted based on the updated trajectory, making the gating strategy have cross-layer continuity rather than independent single-layer judgments.

[0037] This invention introduces cross-layer visual propagation trajectory bias on the basis of existing layer-by-layer visual attention computation. Through dual-path trajectory input and residual effective gating, it achieves layer-by-layer, head-by-head, token-by-token, and channel-by-channel adaptive soft modulation. While preserving the complete visual token sequence, it suppresses redundant propagation and enhances the feature representation of the target region patch token. Attached Figure Description

[0038] Figure 1 The flowchart illustrates a multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias according to an embodiment of the present invention. Detailed Implementation

[0039] The invention will now be discussed with reference to exemplary embodiments. It should be understood that the described embodiments are merely intended to enable those skilled in the art to better understand and thus implement the invention, and are not intended to imply any limitation on the scope of the invention.

[0040] As used herein, the term "comprising" and its variations are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "at least partially based on". The terms "one embodiment" and "an embodiment" are to be interpreted as "at least one embodiment".

[0041] Figure 1 The flowchart schematically illustrates a multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias according to an embodiment of the present invention. Figure 1 As shown, in this embodiment, a multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias is applied to a visual Transformer network, including: Obtain the visual token sequence corresponding to the input image. The visual token sequence contains a classification token and multiple image patch tokens. Multi-head attention calculation is performed on the visual token sequence through multiple network layers of the visual Transformer network to obtain the attention weight matrix of each attention head in each network layer and the single-head attention output of each attention head. For a target token in a visual token sequence, a cross-layer visual propagation trajectory vector of the target token is constructed using the attention weight matrix of each attention head in the current and historical network layers. The cross-layer visual propagation trajectory vector includes at least the average window attention intensity, the average window active attention intensity, the current layer attention entropy, the cross-layer attention intensity change rate, the multi-head divergence degree, the historical gating mean, the attention intensity from the classification token to the target token, the attention intensity from the target token to the classification token, the cumulative intensity of cross-layer classification attention propagation, and the mean attention intensity based on the spatial neighborhood of the image patch token. The cross-layer visual propagation trajectory vectors of all visual tokens within the same network layer are standardized according to the feature dimension to obtain normalized trajectory vectors. By using a learnable projection matrix and a bias vector, the normalized trajectory vector is mapped to the hidden dimension of the model to obtain the trajectory projection bias vector. The trajectory projection bias vector is then added to the token representation of the target token in the current network layer to obtain the biased token representation. At least the biased token representation and normalized trajectory vector are used as input to generate the original gating vectors per network layer, per attention head, per token, and per channel; The original gating vector is written to the gating history cache, and the attention weight matrix of the current network layer is written to the attention history cache for use by subsequent network layers when constructing cross-layer visual propagation trajectory vectors; By introducing a gating strength coefficient, the original gating vector is transformed to obtain an effective gating vector; After the single-head attention output of each attention head and before the multi-head attention concatenation, the effective gating vector is multiplied channel by channel with the single-head attention output of the corresponding attention head to obtain the modulated single-head attention output. The modulated single-head attention outputs of all attention heads in the same network layer are concatenated, and the input token representation of the next network layer is obtained through the output projection matrix.

[0042] In this embodiment, the cross-layer visual propagation trajectory vector of the present invention records the attention status of the target token in multiple network layers and multiple attention heads. At the same time, it integrates visual priors such as the semantic convergence relationship of classification tokens, cross-layer attention propagation accumulation, and image patch spatial neighborhood structure. It can characterize the composite propagation pattern of visual attention signals in the image space and network depth direction, so that the gating network can learn to recognize visual redundancy propagation patterns that are difficult to perceive by traditional methods (such as the propagation pattern that background patch tokens and repeated texture patch tokens may continue to receive high attention in multiple layers but have relatively limited contribution to task discrimination). This allows the gating module to make correction decisions based on the visual propagation history rather than just the current layer state, and suppress the continuous propagation of redundant information between layers.

[0043] This invention projects the normalized trajectory vector as a bias and incorporates the current token representation to obtain a biased token representation. This allows low-dimensional trajectory information to directly alter the state of the high-dimensional token received by the gating network, forming a biased trajectory path. Simultaneously, the original normalized trajectory vector still participates in gating generation as an explicit feature, creating a dual-path trajectory input. This mechanism enhances the influence of low-dimensional trajectory information on gating decisions, prevents trajectory features from being overwhelmed by the high-dimensional token representation in the concatenated input, and improves the gating's sensitivity to visual propagation states.

[0044] This invention uses network layer embedding and attention head embedding as gating auxiliary inputs, enabling the gating network to distinguish the functional states of different network depths and different attention heads, thereby learning gating strategies that are adapted to the layer position and attention head type, and improving the adaptability of gating modulation in the multi-layer multi-head structure of the visual Transformer.

[0045] This invention introduces a gating strength coefficient to perform a residual transformation on the original gating vector. When the gating strength coefficient is 0, the effective gating vector is a vector with the same dimension as the original gating vector and all elements being 1, without modulating the single-head output. When the gating strength coefficient is 1, the effective gating vector is equal to the original gating vector, fully enabling dynamic modulation. This residual gating mechanism, by retaining part of the identity propagation path, can reduce the risk of over-suppression caused by random initialization of the gating value in the early stages of training, and maintain the stability of gradient backpropagation. In one embodiment, the gating strength coefficient can be set to a fixed value; in another embodiment, it can be gradually increased as the training progresses to gradually enhance the gating modulation effect.

[0046] This invention applies channel-wise soft modulation after each attention head completes its single-head attention output and before multi-head stitching, ensuring that each attention head independently adjusts its output intensity based on the cross-layer visual propagation trajectory of the same target token. This preserves the functional differentiation of the multi-head mechanism and avoids confusion between the representations of different heads caused by uniform modulation. At the same time, it utilizes the continuous values ​​of the original gating in the (0,1) interval and converts them into effective gating through the gating intensity coefficient. It does not directly delete any patch tokens, maintains the correspondence between the token sequence structure and the two-dimensional image grid, and reduces the risk of irreversible information loss caused by hard pruning.

[0047] This invention writes the original gating vector of the current layer into the gating history cache and the attention weight matrix into the attention history cache, enabling subsequent network layers to read historical gating states and participate in trajectory construction, forming a cross-layer closed-loop adjustment. The gating decision of the current layer thus affects the trajectory features of subsequent layers, and the gating of subsequent layers is adjusted based on the updated trajectory, making the gating strategy have cross-layer continuity rather than independent single-layer judgments.

[0048] This invention introduces cross-layer visual propagation trajectory bias on the basis of existing layer-by-layer visual attention calculation. Through dual-path trajectory input and residual effective gating, it achieves layer-by-layer, head-by-head, token-by-token, and channel-by-channel adaptive soft modulation. While preserving the complete visual token sequence, it suppresses redundant propagation and enhances the feature expression of the target region patch token.

[0049] Furthermore, according to one embodiment of the present invention, the average attention intensity of the window is the average intensity of the target token being paid attention to by all query tokens under each network layer and each attention head within the preset historical window as a key token; the average active attention intensity of the window is the average intensity of the target token being paid attention to by all key tokens under each network layer and each attention head within the preset historical window as a query token; and the current layer attention entropy is the average value of the attention distribution entropy of the target token under each attention head of the current network layer.

[0050] In this implementation, the average window attention intensity characterizes the cumulative degree of attention the target token receives from other tokens from the perspective of propagation history; the average window active attention intensity characterizes the cumulative state of the target token actively aggregating information from the perspective of information output; and the current layer attention entropy characterizes the degree of attention dispersion of the target token in the current layer from the perspective of distribution concentration. These three metrics describe the visual propagation state of the target token from the perspectives of passive reception, active transmission, and distribution pattern, respectively, providing multi-dimensional criteria for subsequent gating decisions.

[0051] Furthermore, according to one embodiment of the present invention, the cross-layer attention intensity change rate is the absolute value of the difference between the average attention intensity of the window of the current network layer and the average attention intensity of the window of the adjacent previous network layer; when there is no adjacent previous network layer, the cross-layer attention intensity change rate is set to a preset value; the multi-head divergence degree is the variance of the total attention intensity of different attention heads of the current network layer on the target token.

[0052] In this implementation, the cross-layer attention intensity change rate quantifies the variation of the cumulative attention state of the target token between adjacent network layers, capturing the dynamic evolution trend of the visual attention propagation process. This change rate complements the cumulative attention intensity: the cumulative value reflects the cumulative state of propagation, while the change rate reflects the state transition of propagation. When the change rate approaches zero, it indicates that the patch token is in a continuous and stable propagation mode, and the gating module can maintain or fine-tune the current gating strategy accordingly. When the absolute value of the change rate increases, it indicates that the propagation state of the token has turned around, possibly corresponding to a change in the semantics of the region, and the gating module can trigger an adaptive adjustment of the gating value accordingly. Thus, the cross-layer attention intensity change rate introduces the ability to perceive propagation trends into the gating decision, enabling the gating value to not only respond to the static evaluation of the current propagation state but also to perceive the dynamic changes in propagation in a timely manner. It works in conjunction with the cumulative statistics to take into account both the static evaluation of the propagation state and the timely perception of dynamic changes, improving the tracking accuracy of gating modulation on the evolution of attention propagation patterns.

[0053] Furthermore, according to one embodiment of the present invention, when constructing the cross-layer visual propagation trajectory vector, the acquisition of the historical gating mean includes: Read the original gating values ​​of all visual tokens in each attention head and each channel from the pre-set number of historical network layers before the current network layer from the gating history cache, and extract the original gating value of the target token; After averaging the original gating values ​​of the target token along the channel dimension, we then average them along the historical network layer dimension and the attention head dimension to obtain the historical gating mean of the target token. When the gating history cache is empty, the historical gating average is set to a preset constant; in one implementation, the preset constant is 1.

[0054] In this implementation, the historical gating mean feeds back the gating decisions of previous layers to the current trajectory, allowing the current gating to reference the retention and suppression states of the historical layers. If a patch token has been suppressed for a long time in the historical layers, its mean will be low in the current layer trajectory, and the gating network can maintain or strengthen the suppression strategy accordingly. If the historical gating fluctuates greatly, it indicates that the token's propagation state is unstable, and the gating network can learn a smoother modulation strategy accordingly. When the historical cache is empty (such as in the first layer of the network), a preset constant provides a neutral initial value to avoid computational anomalies caused by null values.

[0055] Furthermore, according to one embodiment of the present invention, obtaining the cumulative intensity of cross-layer classification attention propagation includes: The average attention matrix is ​​obtained by averaging the attention weight matrices of all attention heads in the current network layer. The average attention matrix is ​​fused with an identity matrix of the same dimension using equal weights, and the fusion result is normalized by row to obtain the residual normalized attention matrix. Based on the residual normalized attention matrix, the cross-layer attention propagation cumulative matrix up to the current network layer is calculated recursively; Extract the element values ​​corresponding to the classification token row and the target token column from the cross-layer attention propagation accumulation matrix, and use them as the cross-layer classification attention propagation accumulation strength.

[0056] In this implementation, the cumulative intensity of cross-layer classification attention propagation is recursively accumulated by incorporating the attention matrix of the residual path, describing the propagation relationship formed from the classification token through multiple Transformer layers. This feature integrates the global convergence process of classification semantics into the trajectory, enabling the gating network to perceive the indirect correlation strength between the target patch token and the final classification decision, exerting stronger suppression on background patches far from the classification semantic convergence center, and preserving foreground patches close to the semantic center.

[0057] Furthermore, according to one embodiment of the present invention, obtaining the average attention intensity based on the spatial neighborhood of image patch tokens includes: Determine the spatial neighborhood of the target token in the two-dimensional image block token grid, with the image block token corresponding to the target token as the center; Calculate the arithmetic mean of the window average attention intensity of each neighborhood token within the spatial neighborhood to obtain the average attention intensity of the spatial neighborhood based on the image patch token; For a classification token, the mean attention intensity of its neighborhood based on the image patch token space is set to be equal to the average attention intensity of its window.

[0058] In this implementation, the mean attention intensity of the spatial neighborhood of an image patch token is used to aggregate the spatial neighborhood propagation state of the target patch into local contextual features, leveraging the spatial continuity prior of the visual image. For patches located in a uniform background region, their neighborhoods typically exhibit similar attention patterns, and this mean provides a local consistency constraint. For patches located at the target boundary, the differences between the inside and outside of the neighborhood are significant, and this mean assists in identifying boundary transition regions. Since classification tokens have no spatial location correspondence, their global attention intensity is directly used as a substitute.

[0059] Furthermore, according to one embodiment of the present invention, standardizing the cross-layer visual propagation trajectory vector includes: For the same sample and the same network layer, calculate the mean and standard deviation of the trajectory features for each dimension across all visual tokens; The mean and standard deviation are used to standardize the trajectory features of each dimension to obtain the normalized trajectory vector.

[0060] In this implementation, standardization is performed independently on all visual tokens in a single sample and single layer, eliminating differences in the dimensions and numerical ranges of different trajectory features, thus making each dimension comparable in the gating network. This sample-level standardization avoids interference from batch-to-batch data distribution drift on gating decisions and adapts to the characteristics of large differences in image content and unstable batch statistics in visual tasks.

[0061] Furthermore, according to one embodiment of the present invention, a gating strength coefficient is introduced to transform the original gating vector, including: The difference between the first and the gating strength coefficients is multiplied by a vector of the same dimension as the original gating vector, with all elements being 1, to obtain the first product; Multiply the gating strength coefficient by the original gating vector to obtain the second product; Add the first product to the second product to obtain the effective gating vector; The gating strength coefficient ranges from 0 to 1. When the gating strength coefficient is 0, the effective gating vector is a vector with the same dimension as the original gating vector and all elements are 1. When the gating strength coefficient is 1, the effective gating vector is equal to the original gating vector.

[0062] In this embodiment, the gating strength coefficient controls the degree of intervention in dynamic modulation. The gating strength coefficient can be set to a fixed value or scheduled according to the training process. In one embodiment, the gating strength coefficient is set to a fixed value less than 1 to retain some identity propagation paths and avoid excessive suppression caused by random gating initialization; in another embodiment, the gating strength coefficient can be gradually increased as training progresses, allowing the gating network to gradually assume the modulation responsibility. This gradual strategy balances training stability and final modulation effect, enabling the visual Transformer to gain redundancy propagation suppression capability while maintaining its original learning ability.

[0063] Furthermore, according to one embodiment of the present invention, the method further includes: When the amount of data in the attention history cache or gated history cache exceeds the preset window depth, the earliest written historical data is removed.

[0064] In this implementation, a preset window depth limits the amount of historical data stored in the cache, controlling computational overhead and memory usage. When the network layer count is shallow, the cache contains all historical layers; when the network layer count exceeds the window depth, only the most recent layers are retained, allowing trajectory features to focus on recent propagation trends rather than excessively distant past events, thus adapting to the functional differentiation between early and late layers in deep visual Transformer networks.

[0065] Furthermore, to achieve the above objectives, the present invention also provides a multi-head dynamic soft gating system based on cross-layer visual attention propagation trajectory bias, applied to a visual Transformer network, comprising: The token sequence acquisition module obtains the visual token sequence corresponding to the input image. The visual token sequence contains a category token and multiple image block tokens. The attention calculation module performs multi-head attention calculation on the visual token sequence through multiple network layers of the visual Transformer network, and obtains the attention weight matrix of each attention head in each network layer and the single-head attention output of each attention head. The trajectory construction module constructs a cross-layer visual propagation trajectory vector for the target token in the visual token sequence, using the attention weight matrix of each attention head in the current and historical network layers. The cross-layer visual propagation trajectory vector includes at least the average window attention intensity, the average window active attention intensity, the current layer attention entropy, the cross-layer attention intensity change rate, the multi-head divergence degree, the historical gating mean, the attention intensity from the classification token to the target token, the attention intensity from the target token to the classification token, the cumulative intensity of cross-layer classification attention propagation, and the mean attention intensity based on the image patch token spatial neighborhood. The normalization module standardizes the cross-layer visual propagation trajectory vectors of all visual tokens within the same network layer according to the feature dimension to obtain normalized trajectory vectors. The bias injection module maps the normalized trajectory vector to the hidden dimension of the model through the learnable projection matrix and bias vector to obtain the trajectory projection bias vector. The trajectory projection bias vector is then added to the target token's token representation in the current network layer to obtain the biased token representation. The gating generation module takes at least the biased token representation and the normalized trajectory vector as input to generate the original gating vectors per network layer, per attention head, per token, and per channel. The cache update module writes the original gate vector into the gate history cache and the attention weight matrix of the current network layer into the attention history cache, for use by subsequent network layers when constructing cross-layer visual propagation trajectory vectors; The effective gating module introduces a gating strength coefficient to transform the original gating vector, resulting in an effective gating vector; The soft modulation module performs channel-by-channel multiplication of the effective gating vector with the single-head attention output of the corresponding attention head after the single-head attention output of each attention head and before the multi-head attention splicing, to obtain the modulated single-head attention output. The output projection module concatenates the modulated single-head attention outputs of all attention heads in the same network layer, and obtains the input token representation of the next network layer through the output projection matrix.

[0066] The multi-head dynamic soft gating system based on cross-layer visual attention propagation trajectory bias according to the present invention can realize the multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias. The specific process steps are as described above and will not be repeated here.

[0067] Furthermore, to achieve the above objectives, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias as described above.

[0068] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias as described above.

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely one preferred embodiment of the invention and are only used to explain the invention. They do not limit the scope of protection of the invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0070] Example 1

[0071] This embodiment provides a multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias, applied to visual Transformer image recognition tasks. The method uses DeiT (Data-efficient ImageTransformer) as the backbone network and performs image classification training and evaluation on the CIFAR100 dataset, specifically including: (1) Obtain the input token sequence; The input image is divided into multiple image patches. A linear mapping is performed on each image patch to obtain a patch token. A CLS token (classification token) is added before the patch token sequence, and a positional code is superimposed to form the initial visual token sequence.

[0072] in, This represents the initial visual token sequence; Represents the CLS token; Indicates the first One patch token; This represents the total number of tokens, including the CLS token. This represents the hidden dimension of the model.

[0073] For batch input, a batch dimension can be added before the tensor mentioned above. To keep the formula concise, the batch subscript is omitted in subsequent formulas.

[0074] (2) Calculate the multi-head attention of the current layer; In the In the layered network, the input token is represented as For a visual Transformer using a Pre-LN structure, the attention input is calculated first:

[0075] in, Indicates the first Layer normalization before the layer attention module.

[0076] Pre-LN refers to performing layer normalization before multi-head attention computation, describing the order of operations within the Visual Transformer. Pre-LN does not mean the backbone network parameters are frozen; whether the backbone network parameters are updated depends on whether parameter gradients are enabled during the training phase. The DeiT backbone used in this embodiment is a Pre-LN structure, and in the comparative experiments, the pre-trained backbone parameters were frozen.

[0077] For the Each attention point, calculate:

[0078] Further results were obtained:

[0079] in, Indicates the first Layer Attention weight matrix for each attention head; and These represent the query matrix and key matrix corresponding to the attention head, respectively; superscript Indicates the network layer number; subscript Indicates the attention head number; Represents the transpose of the key matrix; The feature dimension of a single attention head; This is the scaling factor; This means performing a normalized exponentiation operation along the key token dimension, so that the sum of the attention weights for each row corresponding to each query token is 1.

[0080]

[0081] in, Indicates the first Layer The single-head output matrix of the attention head before gating; This represents the attention weight matrix of the attention head; Value matrix representation; matrix multiplication This indicates that the value vectors are weighted and aggregated according to their attention weights. The shape is ,in For the number of tokens, It is a single-head feature dimension.

[0082] Mode The Softmax function is calculated along the key token dimension; therefore, for each query token, the sum of its attention weights pointing to all key tokens is 1. From the equation... Obtain the single-head attention output before multi-head splicing.

[0083] (3) Constructing cross-layer visual attention propagation trajectories; 1) Historical Window Let the historical cache depth be In the first When constructing the trajectory for a layer, use the current layer's attention matrix and at most the cached data. A historical layer attention matrix. The set of layers participating in the statistics is denoted as:

[0084] in, Indicates the first The set of layer numbers used when constructing the propagation trajectory; Indicates the current network layer number; Indicates the maximum number of historical layers to retain; This represents the theoretical starting point of the historical window. Used to prevent the window's starting point from being less than the 0th layer; Indicates the distance from the start of the window to the current layer. Continuous layer numbering.

[0085] Therefore, trajectory statistics can cover at most There are historical and current layers. The first or shallow layer is insufficient. When considering historical layers, use the layers that are actually available.

[0086] 2) Six basic characteristics of communication For the Layer Each token is used to construct the following basic propagation characteristics.

[0087] First, the average attention intensity of the window:

[0088] in, Indicates the first Layer The average attention intensity per window for each token; Indicates the target keytoken number; This represents the set of layers that participate in the statistics; This indicates the number of layers contained in the set; Indicates the number of attention heads; represents the layer number within the historical window; represents the attention head number; represents the query token number; represents the total number of tokens; represents the -th layer and the -th attention head, the attention weight of token pointing to token ; the three summations perform aggregation over layers, attention heads and query tokens respectively; the coefficient is used for averaging across the layer dimension and the attention head dimension. Second, the window-average active attention intensity:

[0089] wherein, represents the window-average active attention intensity of the -th layer and the -th token; in represents the query token number; represents the number of the attended key token; represents the attention weight of token pointing to token ; the three summations perform aggregation over layers, attention heads and all key tokens respectively.

[0090] The window-average active attention intensity formula counts the total attention allocated by the target token as a query token to other tokens, which is used to describe its active information aggregation state.

[0091] Third, the attention entropy of the current layer:

[0092] wherein, represents the average attention entropy of the -th layer and the -th query token; represents the number of attention heads; represents the attention head number; represents the key token number; represents the total number of tokens; represents the attention weight of token pointing to token ; represents natural logarithm; The sign indicates a small positive number that prevents the logarithm of zero from being taken; the negative sign makes the entropy value non-negative. This represents the average entropy of each attention head.

[0093] A higher entropy indicates that attention is more dispersed, while a lower entropy indicates that attention is concentrated on a few tokens.

[0094] Fourth, the rate of change of intensity of concern across layers:

[0095] in, Indicates the first Layer The cross-layer strength change rate of each token is monitored; This indicates the average attention intensity of the current layer window; This indicates the average attention intensity of the window corresponding to the previous layer; Indicates taking the absolute value; when At that time, there is no previous layer, therefore let This formula describes whether the cross-layer propagation state of the target token has changed significantly by comparing the average attention intensity of the windows in the current layer and the previous layer.

[0096] Fifth, the degree of divergence between bullish and bearish positions at the current level:

[0097] in, Indicates the first Layer The variance of the attention intensity of each token across different attention heads; Indicates the numbering along the attention head. Calculate the variance; Indicates the query token number; Indicates the total number of tokens; Represents token Pointing to the target token Attention weights; internal summation yields a single attention head pair for the target token. The total received strength. External variance is used to measure the difference in received strength among different attention heads. A larger value indicates a greater difference in the received strength of different attention heads for the target token; a smaller value indicates that the judgments of the attention heads are more consistent.

[0098] Sixth, historical gating average:

[0099] in, Indicates the first Layer Historical gating average of each token; This represents the set of historical gated layers that participated in the statistics before the current layer; Indicates the number of historical gating layers; Indicates the number of attention heads; Indicates the historical layer number; Indicates the attention head number; Indicates the single-head channel number; Indicates the first Layer Height, number The token, the The original gating values ​​for each channel; This indicates that the average is taken along the channel dimension; when If empty, meaning there are no available historical gating records before the current layer, then... This default value indicates that the token's propagation state is preserved by default in the absence of historical evidence of suppression.

[0100] The historical gating average is used to make the current layer gating refer to the previous layer's retention or suppression of the token.

[0101] 3) Four visually enhanced communication features In addition to the six basic propagation features, the following visual enhancement features are further constructed.

[0102] First, average the attention heads of the current layer:

[0103] in, Indicates the first The attention matrix obtained by averaging all attention heads over the layer; Indicates the first Attention matrix of one attention head; Indicates the number of attention heads; Indicates the attention head number; Indicates to The arithmetic mean of the attention points is calculated.

[0104] First, CLS-to-token attention:

[0105] in, Indicates the first Layer CLS-to-token attention features; index 0 represents the CLS token; index Indicates the target token; This indicates that the CLS token, when used as a query, points to the target token. Attention weights.

[0106] Mode Indicates that the CLS token is for the first The direct attention intensity of a token is used to describe the relationship between the target token and the current classification semantic convergence process.

[0107] Second, token-to-CLS attention:

[0108] in, Indicates the first Layered token-to-CLS attention features; index This indicates that the target token is used as the query; index 0 indicates that the CLS token is used as the key. Indicates the target token Attention weights pointing to the CLS token.

[0109] Mode Indicates the first The attention intensity of each token pointing to the CLS token is used to describe the direct interaction between that token and the global semantic token.

[0110] Third, cross-layer CLS attention rollout: First, add the identity matrix to preserve the residual propagation path, and then normalize:

[0111] in, This represents the current layer attention matrix after adding the residual propagation path and completing row normalization; Indicates and Identity matrices of the same order; dividing by 2 indicates equal-weighted fusion of attention paths and unit residual paths; This indicates row-wise normalization, making the sum of the elements in each row equal to 1.

[0112] Then recursively calculate the cross-layer propagation matrix:

[0113] in, Indicates as of the date The cross-layer attention rollout matrix of the layer; Represents the residual normalized attention matrix of layer 0; when Use it as the initial rollout; when hour, This indicates the attention to the normalized residuals of the current layer. This indicates the cumulative rollout of the previous layer; multiplying the two matrices is used to accumulate the propagation path between the current layer and the historical layers.

[0114] No. The CLS rollout characteristics of each token are:

[0115] in, Indicates the first Layer CLS rollout characteristics of each token; This represents the cross-layer rollout matrix obtained from the cross-layer propagation matrix expression; row index 0 indicates starting from the CLS token; column index... Indicates propagation to the target token ; This indicates that the CLS token has reached the target token after cumulative propagation up to the current layer. The intensity.

[0116] Mode It is used to describe the propagation relationship that originates from the CLS token and accumulates through multiple Transformer layers.

[0117] Fourth, the mean of neighborhood propagation in the patch space: Let the first The neighborhood of each patch token in the two-dimensional patch grid is ,but

[0118] in, Indicates the first Layer The spatial neighborhood propagation mean of each patch token; Represents token The neighborhood set in a two-dimensional patch grid; Indicates the number of patches in the neighborhood; Indicates the neighboring patch number; Represents the neighborhood token The average attention intensity of the window is calculated; the summation is then divided by the number of neighborhoods to obtain the arithmetic mean.

[0119] In this embodiment, For target patch centered Neighborhood. Boundary locations are handled by copying boundary values ​​or other padding methods. For CLS tokens, this can be done by... .

[0120] 4) Ten-dimensional visual propagation trajectory vector By piecing together the above features in a fixed order, we obtain the first... Layer Visual propagation trajectory vectors of each token:

[0121] in, Indicates the first Layer A ten-dimensional visual propagation trajectory vector of each token; This indicates concatenation based on feature dimensions; , , , , , These represent the intensity of attention received, the intensity of active attention, the attention entropy, the rate of change across layers, the degree of divergence among bullish positions, and the historical gating mean, respectively. , , and These represent CLS-to-token, token-to-CLS, CLS rollout, and spatial neighborhood features, respectively; therefore .

[0122] In this embodiment, The above sequence remains consistent throughout the trajectory construction, normalization, trajectory projection, and gating generation processes.

[0123] (4) Trajectory feature normalization; To reduce the differences in the dimensions and numerical ranges of different features, the trajectory features of each dimension of each token within the same sample and the same network layer are standardized.

[0124] No. The mean value of the tokens for the dimensional trajectory features is:

[0125] in, Indicates the first Layer The mean of the dimensional trajectory feature across all tokens; This represents the trajectory feature dimension number, with a value ranging from 0 to 9; Indicates the total number of tokens; Indicates the token number; Indicates the first In the trajectory vector of the th token Dimensional features; This represents the arithmetic average of all tokens.

[0126] No. The standard deviation of the token for the dimensional trajectory feature is:

[0127] in, Indicates the first Layer The overall standard deviation of the dimensional trajectory feature across all tokens; The term represents the mean of this feature; the squared term represents the squared deviation of each token feature value from the mean. It represents the square root.

[0128] The normalized trajectory vector is

[0129] in, Represents the standardized first Layer The first token 3D trajectory features; Indicates original features; and These represent the mean and standard deviation of the corresponding feature dimensions, respectively. Represents a small positive number; This is used to prevent division by zero when the standard deviation is too small or zero.

[0130] This embodiment does not require explicit deletion or marking of redundant tokens through a fixed threshold. The gating generation module, based on... The redundancy propagation pattern is jointly learned with the current token representation. This results in an implicit, continuous redundancy propagation discrimination result, which is reflected in the gating value.

[0131] For example, if a patch token has a high level of attention, a low rate of change, a weak cumulative propagation relationship with CLS, and its neighborhood exhibits a repetitive propagation pattern, then the gating network can learn to reduce the gating value of that patch token on a specific attention head and a specific channel, without having to delete the token from the sequence.

[0132] (5) Visual trajectory bias and dynamic gating generation; 1) Trajectory bias Let the first The first layer of input token representation Each token is Through learnable projection matrices and bias vector Map the normalized trajectory vectors to the model's hidden dimensions:

[0133] in, Indicates the first Layer The trajectory projection bias vector of each token; This represents the learnable projection matrix that maps the ten-dimensional trajectory to the model's hidden dimensions, with the shape of... ; This represents the normalized ten-dimensional trajectory vector; This represents the learnable bias vector of the projection layer, with dimension . ; This represents the hidden dimension of the model.

[0134] The projection result is added as a trajectory bias to the current token representation:

[0135] in, The gating after adding trajectory bias is represented by a token; Indicates the first The first layer of input Each token represents a unit; This represents the trajectory projection bias; addition is the element-wise addition of vectors of the same dimension, both of which have the same dimension. .

[0136] Mode This allows the propagation trajectory to not only participate in the splicing as a low-dimensional additional feature, but also directly change the token state received by the gating network, thereby enhancing the influence of trajectory information on gating decisions.

[0137] 2) Gated input Let the first Layer embedding is , No. The head embedding of each attention head is The input to the gating network is uniformly defined as follows:

[0138] in, Indicates the first Layer The first attention point is targeted at the first The gating network input vector of each token; Indicates feature splicing; This represents the token after trajectory offset. Represents the normalized trajectory vector; Indicates the current network layer embedding; Indicates the current attention head embedding; , and These represent the layer, header, and token number, respectively.

[0139] Therefore, the current gating input only includes the following four parts: the current token representation after trajectory bias. Normalized ten-dimensional visual propagation trajectory The current Transformer layer is embedded. Current attention head embedding .

[0140] The query matrix, key matrix, value matrix, and individual attention scores are not directly used as input to the gating network. The attention weights are first converted into cross-layer visual propagation trajectories before participating in gating generation.

[0141] 3) Original gate value The gating generation module uses a multilayer perceptron:

[0142] in, This represents the hidden layer output of the gated multilayer perceptron; and These represent the learnable weight matrix and bias vector of the first linear layer, respectively. Indicates gated input; Represents the activation function of the Gaussian error linear unit; , and These represent the layer, attention header, and token number, respectively.

[0143]

[0144] in, Indicates the first Layer The first one in the attention. The original channel-level gating vector of each token; and These represent the learnable weight matrix and bias vector of the second linear layer, respectively. Indicates the output of the hidden layer; Each channel's gate value is mapped to a value between 0 and 1; the output dimension is a single-head dimension. .

[0145] in: ; For a single attention head, there is a channel dimension; different gating values ​​can be obtained for each layer, each attention head, each token, and each single-head channel.

[0146] Therefore, the granularity of the complete gate tensor is

[0147] (6) Single-head attention output soft modulation; 1) Effective gating of residuals To avoid excessive suppression of attention output in the early stages of training, a gating strength coefficient is introduced. Convert the original gating into a valid gating:

[0148] in, Represents the effective gated vector with retention coefficients; Represents the original gate vector; Indicates the gate strength coefficient; This represents a unit vector with the same dimension as the gate vector and all its elements being 1; Indicates the portion that is permanently retained; This indicates that the gating mechanism can be learned.

[0149] when When the effective gate is always 1, no modulation is generated on the single-head output; when When, the effective gating is equal to the original gating; when At that time, a portion of the identity propagation path is retained.

[0150] 2) Attention-head gating The formula for calculating the single-head output matrix is ​​used to obtain the first... The first one in the attention. Single-head output of each token Execute before concatenating multiple attention heads:

[0151] in, This represents the output vector of a single-headed token after gating; Indicates the first Layer The first one in the attention. The output of each token before gating; Indicates a valid gating; This represents the channel-wise multiplication of two vectors of the same dimension; the dimension of each vector is 1. .

[0152] in, This indicates channel-by-channel multiplication.

[0153] This process does not delete any tokens, nor does it directly delete the entire attention head. Even if the original gating of a channel is close to zero, in The time still retains the right to be Residual information from the decision.

[0154] 3) Multi-head splicing and layer output splice the single-head outputs of all doors controlled on the same layer along the channel dimension:

[0155] in, Indicates the first Multi-head output is formed by splicing single-head outputs after all layers are under control; Indicates the first The output matrix after gating by an attention head; Indicates the number of attention heads; This indicates that the numbers are arranged from 1 to... The concatenation is performed along the feature channel dimension; the concatenated feature dimension is... .

[0156] Then, the projection is output through attention:

[0157] in, Indicates the first Output projection results of the multi-head attention module; This indicates a multi-head splicing output after gating; Indicates the first Layer attention outputs the projection matrix; matrix multiplication concatenates the resulting matrix. 1D feature mapping back to model hidden dimension .

[0158] Then, following the original structure of the visual Transformer, residual connections, layer normalization, and feedforward network calculations are performed to obtain the input for the next layer:

[0159] in, Indicates the first The layer's input token representation; Indicates the first Layer input; This represents the projection of attention output; This represents the residual connection, layer normalization, and feedforward network processing after the attention output projection in the visual Transformer.

[0160] This represents the residual connections, layer normalization, and feedforward network processing after the attention output projection in the original visual Transformer. Gating changes the output of a single head fed into the multi-head splicing, but does not change the basic computation order of the subsequent backbone structure.

[0161] (7) Historical cache update; Complete the first After layer computation, the current attention matrix and the original gate values ​​are written to the history cache:

[0162] in, This indicates an attention history cache; This indicates that the left-side cache is updated with the result from the right side; This indicates that data in the current layer is appended and old data that exceeds the window length is removed; Indicates the first Layer from the 1st to the 2nd The total attention matrix of each attention head; This indicates the number of attention heads.

[0163]

[0164] in, Indicates the gating history cache; Indicates the first The original gating values ​​of all attention heads, all tokens, and all channels of the layer are cached and updated for subsequent network layers to build historical gating features.

[0165] The cache retains only a preset number of historical layers. The gating results of the current layer can then be used to calculate the historical gating features of subsequent layers, forming a continuous adjustment across layers.

[0166] Those skilled in the art will recognize that the modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0167] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and equipment can be referred to the corresponding process in the foregoing method implementation, and will not be repeated here.

[0168] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0169] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs.

[0170] In addition, the functional modules in the embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0171] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to conventional solutions, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the sending / receiving methods of various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0172] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this application.

[0173] It should be understood that the sequence number of each step in the invention and its embodiments does not absolutely imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

Claims

1. A multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias, applied to visual Transformer networks, characterized in that... include: Obtain the visual token sequence corresponding to the input image, wherein the visual token sequence contains a classification token and multiple image patch tokens; Multi-head attention calculation is performed on the visual token sequence through multiple network layers of the visual Transformer network to obtain the attention weight matrix of each attention head in each network layer and the single-head attention output of each attention head; For the target token in the visual token sequence, a cross-layer visual propagation trajectory vector of the target token is constructed using the attention weight matrix of each attention head in the current network layer and the historical network layer. The cross-layer visual propagation trajectory vector includes at least the average window attention intensity, the average window active attention intensity, the current layer attention entropy, the cross-layer attention intensity change rate, the multi-head divergence degree, the historical gating mean, the attention intensity from the classification token to the target token, the attention intensity from the target token to the classification token, the cumulative intensity of cross-layer classification attention propagation, and the mean attention intensity based on the image patch token spatial neighborhood. The cross-layer visual propagation trajectory vectors of all visual tokens within the same network layer are standardized according to the feature dimension to obtain normalized trajectory vectors. By using a learnable projection matrix and a bias vector, the normalized trajectory vector is mapped to the model hidden dimension to obtain a trajectory projection bias vector. The trajectory projection bias vector is then added to the target token's token representation in the current network layer to obtain a biased token representation. Using at least the biased token representation and the normalized trajectory vector as input, the original gating vectors are generated per network layer, per attention head, per token, and per channel. The original gating vector is written into the gating history cache, and the attention weight matrix of the current network layer is written into the attention history cache for use by subsequent network layers when constructing the cross-layer visual propagation trajectory vector; A gating strength coefficient is introduced to transform the original gating vector to obtain an effective gating vector; After the single-head attention output of each attention head and before the multi-head attention concatenation, the effective gating vector is multiplied channel by channel with the single-head attention output of the corresponding attention head to obtain the modulated single-head attention output. The modulated single-head attention outputs of all attention heads in the same network layer are concatenated, and the input token representation of the next network layer is obtained through the output projection matrix.

2. The multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias according to claim 1, characterized in that, The average attention intensity of the window is the average intensity of attention paid by all query tokens to the target token as a key token in each network layer and each attention head within the preset historical window; the average active attention intensity of the window is the average intensity of attention paid by the target token as a query token to all key tokens in each network layer and each attention head within the preset historical window; the current layer attention entropy is the average value of the attention distribution entropy of the target token under each attention head in the current network layer.

3. The multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias according to claim 1, characterized in that, The cross-layer attention intensity change rate is the absolute value of the difference between the average attention intensity of the window in the current network layer and the average attention intensity of the window in the adjacent previous network layer; the multi-head divergence is the variance of the total attention intensity of the target token by different attention heads in the current network layer.

4. The multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias according to claim 1, characterized in that, When constructing the cross-layer visual propagation trajectory vector, the acquisition of the historical gating mean includes: Read the original gating values ​​of all visual tokens in each attention head and each channel from the pre-set number of historical network layers before the current network layer from the gating history cache, and extract the original gating value of the target token; After averaging the original gating values ​​of the target token along the channel dimension, the average values ​​are then averaged along the historical network layer dimension and the attention head dimension to obtain the historical gating mean of the target token. When the gating history cache is empty, the historical gating average is set to a preset constant.

5. The multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias according to claim 1, characterized in that, The acquisition of the cumulative intensity of cross-layer classification attention propagation includes: The average attention matrix is ​​obtained by averaging the attention weight matrices of all attention heads in the current network layer. The average attention matrix is ​​fused with an identity matrix of the same dimension using equal weights, and the fusion result is normalized by row to obtain the residual normalized attention matrix. Based on the residual normalized attention matrix, the cross-layer attention propagation cumulative matrix up to the current network layer is calculated recursively. Extract the element values ​​corresponding to the category token row and the target token column from the cross-layer attention propagation accumulation matrix, and use them as the cross-layer classification attention propagation accumulation intensity.

6. The multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias according to claim 1, characterized in that, The acquisition of the average attention intensity based on the neighborhood of the image block token space includes: Determine the spatial neighborhood of the target token in a two-dimensional image block token grid, wherein the spatial neighborhood is centered on the image block token corresponding to the target token; Calculate the arithmetic mean of the window average attention intensity of each neighborhood token within the spatial neighborhood to obtain the mean attention intensity of the spatial neighborhood based on the image patch token; For the classified token, the average attention intensity of its neighborhood based on the image patch token space is set to be equal to the average attention intensity of its window.

7. The multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias according to claim 1, characterized in that, Standardizing the cross-layer visual propagation trajectory vector includes: For the same sample and the same network layer, calculate the mean and standard deviation of the trajectory features for each dimension across all visual tokens; The mean and standard deviation are used to standardize the trajectory features in each dimension to obtain the normalized trajectory vector.

8. The multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias according to claim 1, characterized in that, The original gated vector is transformed by introducing a gate strength coefficient, including: The difference between the gating strength coefficient and the original gating vector is multiplied by a vector of the same dimension and whose elements are all 1, to obtain the first product. Multiply the gating strength coefficient by the original gating vector to obtain the second product; Add the first product to the second product to obtain the effective gating vector; The gating strength coefficient ranges from 0 to 1. When the gating strength coefficient is 0, the effective gating vector is a vector with the same dimension as the original gating vector and all elements are 1. When the gating strength coefficient is 1, the effective gating vector is equal to the original gating vector.

9. The multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias according to claim 1, characterized in that, The method further includes: When the amount of data in the attention history cache or the gated history cache exceeds the preset window depth, the earliest written historical data is removed.

10. A multi-head dynamic soft gating system based on cross-layer visual attention propagation trajectory bias, applied to visual Transformer networks, characterized in that... include: The token sequence acquisition module acquires the visual token sequence corresponding to the input image. The visual token sequence contains a category token and multiple image block tokens. The attention calculation module performs multi-head attention calculation on the visual token sequence through multiple network layers of the visual Transformer network to obtain the attention weight matrix of each attention head in each network layer and the single-head attention output of each attention head. The trajectory construction module, for the target token in the visual token sequence, constructs a cross-layer visual propagation trajectory vector of the target token using the attention weight matrix of each attention head in the current network layer and the historical network layer; wherein, the cross-layer visual propagation trajectory vector includes at least the average window attention intensity, the average window active attention intensity, the current layer attention entropy, the cross-layer attention intensity change rate, the multi-head divergence degree, the historical gating mean, the attention intensity from the classification token to the target token, the attention intensity from the target token to the classification token, the cumulative intensity of cross-layer classification attention propagation, and the mean attention intensity based on the image patch token spatial neighborhood; The normalization module normalizes the cross-layer visual propagation trajectory vectors of all visual tokens within the same network layer according to the feature dimension to obtain normalized trajectory vectors. The bias injection module maps the normalized trajectory vector to the model hidden dimension using a learnable projection matrix and a bias vector to obtain a trajectory projection bias vector. The trajectory projection bias vector is then added to the target token's token representation in the current network layer to obtain the biased token representation. The gating generation module takes at least the biased token representation and the normalized trajectory vector as input to generate original gating vectors per network layer, per attention head, per token, and per channel. The cache update module writes the original gate vector into the gate history cache and writes the attention weight matrix of the current network layer into the attention history cache for use by subsequent network layers when constructing the cross-layer visual propagation trajectory vector. The effective gating module introduces a gating strength coefficient to transform the original gating vector to obtain an effective gating vector; The soft modulation module performs channel-by-channel multiplication of the effective gating vector with the single-head attention output of the corresponding attention head after the single-head attention output of each attention head and before the multi-head attention splicing, to obtain the modulated single-head attention output. The output projection module concatenates the modulated single-head attention outputs of all attention heads in the same network layer, and obtains the input token representation of the next network layer through the output projection matrix.

11. An electronic device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the multi-head dynamic soft gating method based on cross-layer visual attention propagation trajectory bias as described in any one of claims 1-9.

Images