Plug-and-play feedforward neural network module and visual Transform model

Abstract

The plug-and-play feedforward neural network module adopts an asymmetric factorization bilinear operation model and comprises a first branch circuit, a second branch circuit and a full connection layer, the first branch circuit and the second branch circuit are arranged in parallel, the full connection layer is connected with the first branch circuit, the second branch circuit is connected with the full connection layer, and the full connection layer is connected with the first branch circuit and the second branch circuit. The outputs of the first branch and the second branch are multiplied element by element and then are used as the input of a full connection layer; the first branch comprises a first branch channel mapping layer, a first branch space mapping layer and an activation function layer which are sequentially arranged from input to output, and the second branch comprises a second branch channel mapping layer and a second branch space mapping layer which are sequentially arranged from input to output; the visual Transform model comprises an input embedding layer, a multi-head attention module and a feedforward neural network module, and the feedforward neural network module adopts the plug-and-play feedforward neural network module. According to the method, better balance between efficiency and effect can be achieved, and stronger performance can be achieved in a visual task.

Description

Technical Field

[0001] The present invention relates to neural networks and image recognition technology, and belongs to the fields of machine learning and deep learning. More specifically, it relates to a plug-and-play feedforward neural network module and a visual Transformer model. Background Art

[0002] Image classification is a cornerstone problem in the field of modern computer vision. The performance and efficiency of deep neural network models in image classification tasks usually directly reflect the model's ability to extract image features. In recent years, the outstanding performance of Transformer in natural language processing has attracted great attention from researchers in the field of computer vision. Compared with traditional convolutional neural networks, the visual Transformer architecture has a larger receptive field and can capture the global information of image data at the input of the network. Therefore, the visual Transformer architecture exhibits stronger feature extraction capabilities and shows higher performance in tasks such as image classification and object detection. However, the mainstream visual Transformer architecture still has a series of shortcomings such as high training difficulty and high model complexity. In the current field of computer vision, mainstream research focuses on improving the attention mechanism of Transformer. Researchers hope to maintain good performance while reducing its quadratic complexity.

[0003] The mainstream visual Transformer architecture is usually composed of multiple isomorphic Transformer blocks stacked together. Each Transformer block contains a self-attention module (used to model the spatial correlation of feature maps) and a feedforward neural network (used to model the correlation between channels). In the self-attention module and the feedforward neural network, a large number of fully connected layers are used to process features, introducing a large number of learnable parameters, which greatly increases the spatial complexity of the model. At the same time, in the process of feature extraction of image data by the self-attention module, the time complexity is proportional to the square of the image size, which also makes the traditional visual Transformer model difficult to control the amount of calculation when processing high-resolution images. It can be seen that although the visual Transformer architecture has strong feature extraction capabilities, it also brings large time complexity and spatial complexity. Although a large number of studies have improved the spatial mixing module of the visual Transformer, few studies have focused on its channel mixing module. How to improve the capabilities of the visual Transformer channel mixing module and interact feature information more effectively is still a major challenge.

[0004] In order to improve the channel mixing module of the current visual Transformer, the current main technical routes are as follows:

[0005] The first technical route is inspired by the success of GLU (Gated Linear Unit) in natural language processing tasks, and uses GLU in the form of gated MLP (Multi-layer Perceptron) as the FFN (Feedforward Neural Network, also Channel Mixing Module) paradigm. This method can improve the performance of the model and enhance the robustness of the model by using the double-branch point multiplication method without increasing the number of parameters. However, due to the characteristics of the double branch, the hidden dimension of this method is lower than that of the normal method, which will limit its performance in channel mixing to a certain extent.

[0006] The second technical route is mainly focused on reducing the parameters and computational complexity of FFN. SCHEME designed a block diagonal MLP, which greatly reduced the parameters and computational complexity of the model, while adding a covariance attention method to maintain the performance of the model. IMLP reduces the parameters and computational complexity of FFN by designing an AGELU activation function to share parameters, while keeping the performance of the model unchanged or even unchanged.

[0007] The third technical route is to add additional modules to FFN to improve its performance. For example, LocalViT adds 3x3 depth-separable convolution to FFN to inject locality into FFN to improve the performance of the model. StripMLP is influenced by ConvNextV2 and introduces GRN and 7x7 depth-separable convolution into the channel mixing module.

[0008] In general, the above methods show that channel mixing can be enhanced by increasing the computational burden, or it can be simplified at the expense of performance improvement. However, existing work is still trying to solve the paradox of performance and complexity trade-off, lacking the extraction of feature information itself, and unable to achieve the best trade-off between model performance and efficiency. Summary of the invention

[0009] The purpose of the present invention is to overcome the deficiencies in the prior art and propose a plug-and-play feedforward neural network module and a visual Transformer model, which adopts an asymmetric factorized bilinear operation (AFBO) as a plug-and-play channel mixing module to replace the FFN of the visual Transformer class architecture. Compared with the traditional channel mixing method, AFBO can achieve a better trade-off between efficiency and effect and exert more powerful performance in visual tasks.

[0010] The purpose of the present invention can be achieved through the following technical solutions.

[0011] The plug-and-play feedforward neural network module of the present invention adopts an asymmetric factorized bilinear operation model, wherein the asymmetric factorized bilinear operation model includes a first branch, a second branch and a fully connected layer, wherein the first branch and the second branch are arranged in parallel, and the output of the first branch and the output of the second branch are multiplied element by element as the input of the fully connected layer, and the output of the fully connected layer is the output of the entire feedforward neural network module; wherein the first branch includes a first branch channel mapping layer, a first branch space mapping layer, and an activation function layer which are arranged in sequence from the input to the output direction, and the second branch includes a second branch channel mapping layer and a second branch space mapping layer which are arranged in sequence from the input to the output direction.

[0012] Furthermore, both the first branch spatial mapping layer and the second branch spatial mapping layer adopt depthwise separable convolutional layers.

[0013] Furthermore, the first branch channel mapping layer adopts an overlapping cyclic channel mapping strategy or a grouped cross channel mapping strategy, and the second branch channel mapping layer adopts an overlapping cyclic channel mapping strategy or a grouped cross channel mapping strategy.

[0014] Furthermore, the specific process of the group cross-channel mapping strategy is as follows: the input data is divided into g1 groups according to the channel direction, each group passes through g1 shared fully connected layers in sequence to obtain 2g1 groups of channel-mapped tensors, and finally the 2g1 groups of tensors are spliced ​​according to the channel direction to obtain the final output result.

[0015] Furthermore, the specific process of the overlapping cyclic channel mapping strategy is as follows: the input data is divided into g2 groups according to the channel direction, and each two adjacent groups pass through a fully connected layer together until each group passes through channel mapping twice, obtaining g2 tensors, which are spliced ​​according to the channel direction to obtain the final output result.

[0016] Furthermore, the asymmetric factorized bilinear operation model is expressed as follows:

[0017]

[0018] In the formula, X represents the model input, Y represents the model output, σ(·) represents the activation function, represents a point-by-point convolution with a kernel size of 1×1, represents a convolution operation with a kernel size of K×K, ⊙ represents element-by-element multiplication, Respectively represent the macroscopic weights of the first branch channel mapping and the second branch channel mapping, They represent the macroscopic bias terms of the first branch channel mapping and the second branch channel mapping, respectively. Respectively represent the weights of the first branch space mapping and the second branch space mapping, They represent the bias terms of the first branch spatial mapping and the second branch spatial mapping respectively, and W3 and b3 represent the weight and bias term of the fully connected layer at the output end of the model respectively.

[0019] The purpose of the present invention can also be achieved through the following technical solutions.

[0020] The visual Transformer model of the present invention includes an input embedding layer, a multi-head attention module, and a feedforward neural network module arranged in sequence from input to output, and the feedforward neural network module adopts any one of the above-mentioned plug-and-play feedforward neural network modules.

[0021] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0022] The plug-and-play feedforward neural network module proposed in the present invention adopts an asymmetric factorized bilinear operation (AFBO) model, which can replace the FFN of the Transformer-like architecture. Compared with the traditional channel mixing method, AFBO can achieve a better trade-off between efficiency and effect and exert more powerful performance in visual tasks.

[0023] The AFBO of the present invention calculates second-order statistical information through spatial channel factorized bilinear operations instead of simple linear projection in FFN, thereby enhancing the feature abstraction ability of the visual Transformer by modeling the second-order correlation between the labeled features. In order to solve the problem of computational complexity, the AFBO of the present invention proposes two structured sparsity channel mapping strategies (i.e., overlapping cyclic channel mapping strategy and grouped cross-channel mapping strategy) to perform bilinear mapping on channel features in groups, while reducing memory and computational costs, and realizing information interaction between groups to ensure performance. Finally, the asymmetric structured sparsity channel mapping designed by the present invention further helps the AFBO of the present invention to more effectively deal with the paradox of performance and complexity trade-offs. The present invention can be flexibly integrated with the existing similar visual Transformer architecture. The present invention has conducted experiments on several visual Transformers in various tasks. The results show that the AFBO of the present invention is superior to similar methods and achieves a better trade-off between efficiency and effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is the architecture diagram of the visual Transformer model.

[0025] Figure 2 This is the architecture diagram of the existing feedforward neural network module.

[0026] Figure 3 This is a diagram of the feedforward neural network module architecture of the present invention.

[0027] Figure 4 It is a schematic diagram of the group cross channel mapping strategy (GCCM strategy) in the present invention.

[0028] Figure 5 Schematic diagram of the overlapping cyclic channel mapping strategy (OCCM strategy) in the present invention.

[0029] Figure 6 Schematic diagram of the macro matrix of the channel mapping strategy; wherein, (a) is a schematic diagram of the macro matrix of the traditional fully connected mapping, (b) is a schematic diagram of the macro matrix of the traditional group mapping, (c) is a schematic diagram of the matrix of the group cross channel mapping strategy in the present invention, and (d) is a schematic diagram of the matrix of the overlapping cyclic channel mapping strategy in the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings.

[0031] Figure 1 This is the architecture diagram of the classic visual Transformer model. The visual Transformer model mainly includes an input embedding layer, a multi-head attention module, and a feedforward neural network module, which are arranged in sequence from input to output. The output data of the input embedding layer is used as the input data of the multi-head attention module after adding position coding. The output data of the multi-head attention module and the data after adding position coding are subjected to the first residual link and normalization processing. The data after the first residual link and normalization processing are used as the input data of the feedforward neural network module. The output data of the feedforward neural network module and the data after the first residual link and normalization processing are subjected to the second residual link and normalization processing to obtain the final output result. Among them, the multi-head attention module is the spatial mixing module, and the feedforward neural network module is the channel mixing module.

[0032] In the architecture of the classic visual Transformer model, the existing architecture of the feedforward neural network module (FFN module) is as follows: Figure 2 As shown, it is composed of a traditional first-order MLP module, which mainly includes a first fully connected layer, an activation function layer, and a second fully connected layer arranged in sequence from input to output.

[0033] The present invention proposes a plug-and-play feedforward neural network module to replace Figure 2The feedforward neural network module shown. The plug-and-play feedforward neural network module of the present invention adopts an asymmetric factorized bilinear operation (AFBO) model to calculate second-order statistics for feature extraction through spatial channel decomposition bilinear operation (SCFBO), instead of the first linear projection in FFN. SCFBO first approximates the second-order statistics through rank-1 decomposition and further decomposes the mapping operation into spatial mapping (SM) and channel mapping (CM).

[0034] Figure 3 The plug-and-play feedforward neural network module architecture diagram proposed in the present invention mainly includes a first branch, a second branch and a fully connected layer. The first branch and the second branch are arranged in parallel. The output of the first branch and the output of the second branch are multiplied element by element as the input of the fully connected layer. The output of the fully connected layer is the output of the entire feedforward neural network module. Among them, the first branch includes a first branch channel mapping layer CM1, a first branch space mapping layer SM1, and an activation function layer arranged in sequence from input to output, and the second branch includes a second branch channel mapping layer CM2 and a second branch space mapping layer SM2 arranged in sequence from input to output.

[0035] Preferably, both the first branch spatial mapping layer SM1 and the second branch spatial mapping layer SM2 may adopt depthwise separable convolutional layers.

[0036] First, for the classic visual Transformer model, the original FFN involving two fully connected (linear) layers with activation function is used for channel mixing.

[0037]

[0038] In the formula, X represents the model input, Y represents the model output, represents a point-by-point convolution with a kernel size of 1×1, σ(·) is an activation function (such as GeLU, etc.), W1 and W2 represent the weights of the first fully connected layer and the second fully connected layer, respectively, and b1 and b2 represent the bias terms of the first fully connected layer and the second fully connected layer, respectively.

[0039] As can be seen from formula (1), FFN mainly uses two linear projections for feature abstraction, which does not fully consider the rich information contained in the token features. However, the feature abstraction ability has a great impact on the performance of ViT's asymmetric decomposition bilinear operation. A naive solution to enhance the abstraction ability is to increase the hidden dimension, but it also brings more computational cost. Therefore, reasonably exploring the information in the token features is a potential solution to improve ViT.

[0040] Spatial channel decomposition Bilinear operation Previous studies have shown that proper modeling of the second-order statistics in the features is more effective in improving the model than simple linear projection. Inspired by this, our AFBO approximates the calculation of the second-order statistics through rank-1 decomposition (i.e., rank 1 decomposition). Our AFBO replaces the first linear projection in FFN (i.e., Figure 2 The first fully connected layer in

[0041]

[0042] Where W3 and b3 represent the weight and bias of the fully connected layer at the output of the model, respectively, and ⊙ represents element-by-element multiplication. Represent the weight and bias of the first spatial mapping, respectively. They represent the weight and bias of the second spatial mapping respectively, where the size of the spatial mapping convolution kernel is K.

[0043] In order to improve the computational efficiency, a spatial channel factorization bilinear operation (SCFBO) is further proposed, which converts the convolution Decomposed into convolution along the spatial dimension and convolution along the channel dimension. The expression of the asymmetric factorized bilinear operation model (AFBO model) is as follows:

[0044]

[0045] In the formula, X represents the model input, Y represents the model output, σ(·) represents the activation function, represents a point-by-point convolution with a kernel size of 1×1, represents a convolution operation with a kernel size of K×K, ⊙ represents element-by-element multiplication, Respectively represent the macroscopic weights of the first branch channel mapping and the second branch channel mapping, They represent the macroscopic bias terms of the first branch channel mapping and the second branch channel mapping, respectively. Respectively represent the weights of the first branch space mapping and the second branch space mapping, They represent the bias terms of the first branch spatial mapping and the second branch spatial mapping respectively, and W3 and b3 represent the weight and bias term of the fully connected layer at the output end of the model respectively.

[0046] The SCFBO introduced in formula (3) allows our AFBO to be implemented via a two-branch structure with unshared parameters, where each branch involves a depthwise convolution followed by a pointwise convolution.

[0047] Considering that the hidden dimension plays a crucial role in the performance of ViT, we also adopt a large expansion ratio (expansion ratio M of the fully connected layer, e.g., M=4) in channel mapping to ensure performance, which leads to higher computational complexity. Therefore, we further propose two structured sparse channel mapping strategies to reduce the memory and computational cost of SCFBO while ensuring performance.

[0048] In order to improve computational efficiency, the AFBO model is constructed in an asymmetric manner by two structured sparsity channel mapping strategies (i.e., overlapping cyclic channel mapping strategy and grouped cross channel mapping strategy). Specifically, the first branch channel mapping layer CM1 adopts an overlapping cyclic channel mapping strategy or a grouped cross channel mapping strategy, and the second branch channel mapping layer CM2 adopts an overlapping cyclic channel mapping strategy or a grouped cross channel mapping strategy. The mapping strategies adopted by the first branch channel mapping layer CM1 and the second branch channel mapping layer CM2 may be the same or different.

[0049] ① Group cross channel mapping strategy, referred to as "GCCM strategy", such as Figure 4 The specific process shown is as follows: after the input data enters the module, it is divided into g1 groups according to the channel direction. Each group passes through g1 shared fully connected layers in sequence to obtain 2g1 groups of channel-mapped tensors. Finally, the 2g1 groups of tensors are spliced ​​according to the channel direction to obtain the final output result.

[0050] ② Overlapping Circular Channel Mapping Strategy, referred to as "OCCM Strategy", such as Figure 5 The specific process shown is as follows: after the input data enters the module, it is divided into g2 groups according to the channel direction. Each two adjacent groups pass through a fully connected layer together until each group passes through two channel mappings to obtain g2 tensors, which are then spliced ​​in the channel direction to obtain the final output result.

[0051] As a common strategy, group channel mapping is used to reduce the computational complexity of deep CNNs. It can be regarded as adding a block diagonal sparsity constraint, that is,

[0052]

[0053] Where W . c The weight of channel mapping, G is the number of overlapping cyclic channel mapping groups. The computational complexity of block channel mapping in formula (4) is the original Although the model is efficient, the performance of block-based channel mapping is poor due to the lack of information interaction between different groups.

[0056] To ensure efficiency and effectiveness, we propose two new channel mapping strategies for structured sparsity, namely, grouped cross channel mapping (GCCM) and overlapped cyclic channel mapping (OCCM). Figure 6 As shown in Figure 1, our GCCM divides the input and output into G and 2G non-overlapping groups, respectively. Then, the GCCM uses parameter-sharing linear mappings for the outputs of the i-th group and the i+G-th group (i∈[1,G]) by using different input groups. In this way, the inputs of different groups are fed into the same linear mapping to achieve information interaction between the groups. Specifically, the mapping matrix of our GCCM can be written as:

[0057]

[0058] In the formula, is the macro weight of the group cross-channel mapping, W c (G) is the channel mapping weight of the Gth group.

[0059] The GCCM in formula (5) is only the original In addition, our OCCM proposes to use overlapping inputs to achieve information interaction between groups in a cyclic manner, so the mapping matrix of OCCM can be written as:

[0060]

[0061] In the formula, represents the channel mapping weight of the left half of group G, Represents the channel mapping weight of the right half of group G.

[0062] Among them, our OCCM sets the overlap range to half of the group feature dimension. From Equations (5) and (6), we can see that our GCCM and OCCM can effectively achieve information interaction between groups by introducing appropriate structural sparsity constraints in channel mapping. Note that our AFBO can control the trade-off between performance and complexity by adopting different numbers of groups and overlap rates.

[0063] Based on the proposed GCCM and OCCM, the present invention can implement SCFBO in different ways, thereby generating three AFBO variants, namely SCFBO-GC, SCFBO-OC and asymmetric SCFBO (AFBO).

[0064] SCFBO-GC: Using GCCM as the channel mapping of the present invention, that is, the first branch channel mapping layer CM1 and the second branch channel mapping layer CM2 of the present invention both adopt the GCCM strategy, thereby constructing a SCFBO-GC variant. In particular, for d-dimensional input, the number of parameters of SCFBO-GC is where g1 and M are the number of groups and the expansion ratio of the hidden dimension of GCCM, respectively. Correspondingly, the computational cost (FLOPs) of SCFBO-GC is

[0065] SCFBO-OC: The channel mapping of the two branches in SCFBO is replaced by OCCM, that is, the first branch channel mapping layer CM1 and the second branch channel mapping layer CM2 in the present invention both adopt the OCCM strategy, thereby constructing a SCFBO-OC variant. In particular, for d-dimensional input, the number of parameters of SCFBO-OC is The computational cost (FLOPs) of SCFBO-OC is where g2 is the number of groups used in OCCM.

[0066] Asymmetric SCFBO (AFBO): As a hybrid solution, we can use GCCM and OCCM to replace the left branch and right branch of SCFBO respectively, that is, the first branch channel mapping layer CM1 adopts the GCCM strategy, and the second branch channel mapping layer CM2 adopts the OCCM strategy, which leads to asymmetric SCFBO, namely AFBO. Specifically, the number of parameters of AFBO is The computational cost is Where g1 is the number of groups used in GCCM and g2 is the number of groups used in OCCM.

[0067] Computational Complexity Analysis To further analyze the performance and complexity of the three AFBO variants, we compare them with the original FFN and GLU. From a theoretical perspective, all of our AFBO variants have similar or lower computational complexity than FFN and GLU, where we discard the computational cost of DW convolution in spatial mapping. Our SCFBO-GC achieves similar performance to FFN on ImageNet-1K, but with lower computational complexity. SCFBO-OC improves FFN and GLU by 3.4% and 2.6%, respectively, with 0.8M additional parameters. Finally, our AFBO brings 2.4% and 1.6% gains over FFN and GLU, respectively, with similar parameters and FLOPs. Obviously, our AFBO with asymmetric channel mapping achieves a better performance and complexity trade-off, and it is used for all experiments in this work.

[0068] The principle of the present invention is further described in detail below in conjunction with specific embodiments. In this example, the plug-and-play feedforward neural network module (i.e., AFBO module) proposed in the present invention is evaluated on the source dataset image classification (ImageNet-1K, ImageNet-C, ImageNet-A, and ImageNet-R) and target detection (COCO-17) benchmarks.

[0069] The present invention regards the AFBO module as a plug-and-play module for replacing the original MLP in different visual Transformer and MLP-Mixer variants, so the training strategy is exactly the same as the baseline model. Based on the above analysis of module complexity, this embodiment selects the variant AFBO for experimentation because AFBO provides the best balance between efficiency and performance. Typically, g1 and g2, which represent the number of groups of GCCM and OCCM, are set to 2 and 3, respectively, to be consistent with the original FFN. For the spatial mapping module, we use 3x3 depth convolution in both branches.

[0070] In addition to the experiments on ImageNet, we further evaluate our method on COCO-17 using Mask R-CNN and RetinaNet, where Poolformer-S12 and Swin-T are used as backbone models. We implement all detectors using the MMDetection toolkit with default settings. Specifically, the shorter edge of the input image is resized to 800. All models are then optimized using SGD with weight decay of 1e-4, momentum of 0.9, and a mini-batch size of 16 (8 GPUs, 2 images per GPU). The learning rate is initialized to 0.01 and is reduced by 10 times after the 8th and 11th epochs, respectively. All programs are implemented by PyTorch and run on 8 A6000 GPUs and 8 RTX 3090 GPUs.

[0071] Although the functions and working processes of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific functions and working processes. The above-mentioned specific implementation methods are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all within the protection of the present invention.

Claims

1. A plug-and-play feedforward neural network module, It is characterized in that The feedforward neural network module adopts an asymmetric factorized bilinear operation model, which includes a first branch, a second branch and a fully connected layer. The first branch and the second branch are arranged in parallel, and the output of the first branch and the output of the second branch are multiplied element by element as the input of the fully connected layer, and the output of the fully connected layer is the output of the entire feedforward neural network module; wherein the first branch includes a first branch channel mapping layer (CM1), a first branch space mapping layer (SM1), and an activation function layer arranged in sequence from input to output, and the second branch includes a second branch channel mapping layer (CM2) and a second branch space mapping layer (SM2) arranged in sequence from input to output.

2. The plug-and-play feedforward neural network module according to claim 1, It is characterized in that The first branch spatial mapping layer (SM1) and the second branch spatial mapping layer (SM2) both adopt depthwise separable convolutional layers.

3. The plug-and-play feedforward neural network module according to claim 1, It is characterized in that The first branch channel mapping layer (CM1) adopts an overlapping cyclic channel mapping strategy or a grouped cross channel mapping strategy, and the second branch channel mapping layer (CM2) adopts an overlapping cyclic channel mapping strategy or a grouped cross channel mapping strategy.

4. The plug-and-play feedforward neural network module according to claim 3, It is characterized in that The specific process of the group cross channel mapping strategy is as follows: the input data is divided into g according to the channel direction. 1 groups, each group passes through g in order 1 shared fully connected layers, resulting in 2g 1 The group is mapped by channels, and finally 2g is mapped according to the channel direction. 1 The group tensors are concatenated to obtain the final output result; The specific process of the overlapping loop channel mapping strategy is as follows: the input data is divided into g according to the channel direction. 2 Each adjacent group passes through a fully connected layer together until each group passes through two channel mappings to obtain g 2 Tensors are concatenated along the channel direction to obtain the final output result.

5. The plug-and-play feedforward neural network module according to claim 1, It is characterized in that The asymmetric factorized bilinear operation model is expressed as follows: In the formula, X represents the model input, Y represents the model output, σ(·) represents the activation function, represents a point-by-point convolution with a kernel size of 1×1, represents a convolution operation with a kernel size of K×K, ⊙ represents element-by-element multiplication, Respectively represent the macroscopic weights of the first branch channel mapping and the second branch channel mapping, They represent the macroscopic bias terms of the first branch channel mapping and the second branch channel mapping, respectively. Respectively represent the weights of the first branch space mapping and the second branch space mapping, They represent the bias items of the first branch space mapping and the second branch space mapping, respectively, and W 3 、b 3 They represent the weights and bias terms of the fully connected layer at the output of the model respectively.

6. A visual Transformer model comprising a plug-and-play feedforward neural network module according to any one of claims 1 to 5, the visual Transformer model comprising an input embedding layer, a multi-head attention module, and a feedforward neural network module arranged in sequence from input to output, It is characterized in that The feedforward neural network module adopts the plug-and-play feedforward neural network module described in any one of claims 1 to 5.

Images