A lightweight visual global perception method based on RSFVIT

Abstract

The application discloses a lightweight visual global perception method based on RSFVIT, and relates to the technical field of visual perception models, and comprises the following steps: constructing an RSFVIT structure comprising a three-level network, and completing a training stage of the constructed RSFVIT structure based on reparameterization position coding; performing weight conversion and reparameterization on a frequency domain branch and a space domain branch to obtain a standard space domain convolution kernel; performing channel optimization on the RSFVIT structure through split mixed self-attention; introducing a learnable idle channel reparameterization feedforward network, dynamically weighting and distributing feature channels through a learnable mask vector in the training stage, and thresholding the mask vector to convert it into a binary mask in the inference stage; through split mixed self-attention, through the design of retaining full-resolution Query, and through more fine over-parameterization control on idle channels, the precision upper limit of the model is improved under the premise of not increasing the inference cost.

Description

Technical Field

[0001] This invention relates to the field of visual perception model technology, specifically a lightweight global visual perception method based on RSFVIT. Background Technology

[0002] Convolutional neural networks (CNNs) have always faced bottlenecks in capturing global long-range dependencies due to the limitations of their local receptive fields. To overcome this limitation, visual transformers have achieved superior global modeling capabilities by introducing self-attention mechanisms, significantly improving the performance boundary of computer vision tasks. However, the computational complexity of self-attention mechanisms increases quadratically with feature map resolution, posing a significant deployment challenge for standard visual transformers on resource-constrained mobile devices. To achieve efficient inference on mobile devices, researchers have proposed various optimization paths. Early RepVGG improved the accuracy of pure convolutional architectures while maintaining high speed through structural reparameterization techniques. Subsequently, hybrid architectures such as MobileViT attempted to embed Transformer operators into lightweight CNNs, which reduced the number of parameters, but the inference latency on actual hardware remained unsatisfactory.

[0003] Existing lightweight solutions still face three major conflicts: First, the perception conflict, namely, the acquisition of a large receptive field often comes at the cost of memory access efficiency; second, the redundancy conflict, the full-channel attention mechanism brings high performance but also has serious feature redundancy; and finally, the resource utilization conflict, for very small models, static allocation of idle channels often leads to further reduction in accuracy and waste of computing resources. To address this, we present a lightweight visual global perception method based on RSFVIT. Summary of the Invention

[0004] The purpose of this invention is to provide a lightweight visual global perception method based on RSFVIT.

[0005] The objective of this invention can be achieved through the following technical solution: a lightweight visual global perception method based on RSFVIT, comprising:

[0006] Step S1: Construct an RSFVIT structure including a three-level network, and complete the training phase of the constructed RSFVIT structure based on reparameterized positional encoding;

[0007] Step S2: Perform weight transformation and reparameterization on the frequency domain branch and the spatial domain branch to obtain the standard spatial domain convolution kernel;

[0008] Step S3: Perform channel optimization on the RSFVIT structure through split hybrid self-attention, wherein split hybrid self-attention includes channel splitting strategy, space reduction strategy and gated attention strategy;

[0009] Step S4: Introduce a learnable idle channel reparameterized feedforward network. During the training phase, the feature channels are dynamically weighted using a learnable mask vector. During the inference phase, the mask vector is thresholded and converted into a binary mask.

[0010] Furthermore, the process of step S1 is as follows:

[0011] Step S101: Construct a feature extraction branch, which includes a spatial domain branch and a frequency domain branch;

[0012] Step S102: Extract features from the input features using the constructed feature extraction branches;

[0013] Step S103: Based on frequency-spatial duality reparameterized position coding, the extracted features are dynamically fused to obtain the corresponding position sensing features.

[0014] Furthermore, the process of step S102 is as follows:

[0015] The spatial domain branch is used to extract the spatial domain features of the input features, and the frequency domain branch is used to extract the frequency domain features of the input features.

[0016] The input features are denoted as feature X;

[0017] In the spatial domain branch, multi-branch convolutional combinations are used to process the feature X, obtaining the corresponding local detail features, which are then used as the output of the spatial domain branch. The output of the spatial domain branch is denoted as... ,in:

[0018] ;

[0019] Wherein, SDB(.) represents a parallel structure containing main convolution, identity mapping and average pooling operators;

[0020] In the frequency domain branch, the input feature X is globally coupled through Fourier transform and inverse Fourier transform to obtain the output of the frequency domain branch, which is denoted as . ,in:

[0021] ;

[0022] in, Represents the frequency domain weights. Represents the inverse transform operator. This represents a transformation operator.

[0023] Furthermore, the process of step S2 is as follows:

[0024] Step S201: Combine the multi-branch convolutions in the spatial branch and perform weight fusion to obtain the corresponding spatial equivalent convolution kernel;

[0025] Step S202: Map the frequency domain weights of the frequency domain branch to the spatial domain and extract the corresponding equivalent frequency domain convolution kernels;

[0026] Step S203: The linear additivity of the convolution operator is used to statically reparameterize and fuse the weights in the spatial and frequency domains.

[0027] Furthermore, the process of step S202 is as follows:

[0028] The spatial global convolution kernel obtained after inverse Fourier transform of the frequency domain weights is denoted as . ;

[0029] The input feature X is then transformed in the frequency domain using the convolution theorem, and the transformation result is denoted as F(X), where the convolution theorem is:

[0030] ;

[0031] in, The result is the inverse Fourier transform of the frequency domain weights;

[0032] The frequency domain weights are then mapped to the spatial domain, as follows:

[0033] ;

[0034] ;

[0035] The obtained spatial global convolution kernel is truncated to obtain the equivalent frequency domain convolution kernel corresponding to the frequency domain weights, denoted as . ,in:

[0036] ;

[0037] in This indicates the truncation operator.

[0038] Furthermore, the process of step S3 is as follows:

[0039] Step S301: Split the input features using a channel splitting strategy to obtain active and passive branches;

[0040] Step S302: Compress the search space of the attention matrix using a space reduction strategy;

[0041] Step S303: Set up a gating attention strategy to enable deep feature fusion between active and passive branches.

[0042] Furthermore, the process of step S4 is as follows:

[0043] Let the mask vector be M. Then, the process of dynamically weighting the feature channels during the training phase can be represented as:

[0044] ;

[0045] in, This represents the output after fusion. Indicates active branch output. Indicates passive branch output. Here, M is a learnable bias term, and M is a learnable mask vector. The input features of the portion of the channels that are assigned to the "idle branch";

[0046] Then, the output of the learnable idle channel reparameterized feedforward network is obtained, denoted as... ,in:

[0047] ;

[0048] in, This represents the output projection weights, and BN(.) represents the batch normalization operation.

[0049] In the reasoning stage, through... Thresholding is performed to convert it into a binary mask, and the corresponding equivalent inference weights and equivalent biases are obtained, denoted as follows: and ,in:

[0050] ;

[0051] ;

[0052] in, Indicates the input projection weights. To output the projected weights, represent Represents a diagonal matrix; This indicates the bias of the input projection layer. This indicates the bias of the output projection layer.

[0053] Compared with the prior art, the beneficial effects of the present invention are:

[0054] 1. By splitting the self-attention to a small subset of channels ( ) applies complex self-attention computation for global modeling, while most channels ( As an identity mapping direct bypass transmission, it directly reduces the memory access cost caused by Query / Key / Value projection by 75%, while avoiding the loss of high-frequency information in deep networks by preserving the original passive branch;

[0055] 2. By retaining the full-resolution query design to enhance operator capacity, the model can still generate high-fidelity feature maps while significantly reducing computational load, thus mitigating the loss of spatial information in deep networks.

[0056] 3. By implementing more precise overparameterized control over idle channels during the training phase, the optimization path is effectively improved and the training loss is reduced. Ultimately, LIP-RepFFN can completely collapse the learning content accumulated during the training phase into the backbone weights before inference through mathematical transformations. This allows for higher model accuracy and feature learning capabilities than the original static reparameterization without increasing inference costs. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0058] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0059] like Figure 1 As shown, a lightweight visual global perception method based on RSFVIT includes:

[0060] Step S1: Construct an RSFVIT structure including a three-level network, and complete the training phase of the constructed RSFVIT structure based on reparameterized positional encoding. The specific process is as follows:

[0061] Step S101: Construct feature extraction branches; it should be noted that the feature extraction branches include spatial domain branches and frequency domain branches;

[0062] Step S102: Extract features from the input features using the constructed feature extraction branches; it should be noted that the spatial domain branch is used to extract the spatial domain features of the input features, and the frequency domain branch is used to extract the frequency domain features of the input features.

[0063] Specifically:

[0064] The input features are denoted as feature X;

[0065] In the spatial domain branch, multi-branch convolutional combinations are used to process the feature X, obtaining the corresponding local detail features, which are then used as the output of the spatial domain branch. The output of the spatial domain branch is denoted as... ,in:

[0066] ;

[0067] Wherein, SDB(.) represents a parallel structure containing operators such as main convolution, identity mapping and average pooling;

[0068] In the frequency domain branch, the input feature X is globally coupled through Fourier transform and inverse Fourier transform to obtain the output of the frequency domain branch, which is denoted as . ,in:

[0069] ;

[0070] in, Represents the frequency domain weights. Represents the inverse transform operator. Represents the transformation operator;

[0071] Step S103: Based on frequency-spatial duality reparameterized position coding, the extracted features are dynamically fused to obtain the corresponding position sensing features;

[0072] It should be noted that the frequency-spatial dual reparameterized position coding is denoted as SF-RepCPE. During the training phase, SF-RepCPE dynamically fuses the extracted features using residual connections to obtain enhanced position-sensing features, denoted as... ,in:

[0073] ;

[0074] in It is a learnable frequency domain scaling factor used to automatically adjust the contribution ratio of global frequency domain information in different channels during training.

[0075] Step S2: Perform weight transformation and reparameterization on the frequency domain branch and the spatial domain branch to obtain the standard spatial domain convolution kernel;

[0076] Step S201: Combine the multi-branch convolutions in the spatial branch and perform weight fusion to obtain the corresponding spatial equivalent convolution kernel;

[0077] Specifically:

[0078] In the multi-branch convolution combination, the main convolution is denoted as W1, the pointwise convolution as W2, the concatenated sequence weight as W3, the pooling equivalent weight as W4, and the identity mapping weight as W5.

[0079] By leveraging the linear additivity of convolution operators, multi-branch convolution combinations in the spatial domain are fused to obtain the corresponding spatial equivalent convolution kernel, denoted as . ,in:

[0080] .

[0081] Step S202: Map the frequency domain weights of the frequency domain branch to the spatial domain and extract the corresponding equivalent spatial domain convolution kernels;

[0082] Specifically:

[0083] The spatial global convolution kernel obtained after inverse Fourier transform of the frequency domain weights is denoted as . ;

[0084] The input feature X is then transformed in the frequency domain using the convolution theorem, and the transformation result is denoted as F(X), where the convolution theorem is:

[0085] ;

[0086] in, The result is the inverse Fourier transform of the frequency domain weights;

[0087] The frequency domain weights are then mapped to the spatial domain, as follows:

[0088] ;

[0089] ;

[0090] The obtained spatial global convolution kernel is truncated to obtain the equivalent frequency domain convolution kernel corresponding to the frequency domain weights, denoted as . ,in:

[0091] ;

[0092] in Indicates the truncation operator;

[0093] Step S203: The linear additivity of the convolution operator is used to statically reparameterize and fuse the weights in the spatial and frequency domains;

[0094] Specifically:

[0095] The standard spatial convolution kernel obtained by static reparameterization fusion is denoted as . ,in:

[0096] .

[0097] Step S3: Perform channel optimization on the RSFVIT structure through split hybrid self-attention, wherein split hybrid self-attention includes channel splitting strategy, space reduction strategy and gated attention strategy;

[0098] Step S301: Split the input features using a channel splitting strategy to obtain active and passive branches;

[0099] Specifically:

[0100] Each network block within the RSFVIT structure is labeled, denoted as i, where i = 1, 2, ..., n;

[0101] Let the input features to the network block labeled i be denoted as . ;

[0102] Then the input features to be input Split along the channel dimension to obtain the corresponding active and passive branches, denoted as follows: and ,Right now:

[0103]

[0104] This design allows for the use of a small portion of channels ( ) applies complex self-attention computation for global modeling, while most channels ( This is a direct bypass transmission of the identity mapping. It directly reduces the memory access cost caused by Query / Key / Value projection by 75%, while avoiding the loss of high-frequency information in deep networks by preserving the original passive branch;

[0105] Step S302: Compress the search space of the attention matrix using a space reduction strategy;

[0106] Specifically: By performing a linear projection on the input features, we obtain the corresponding keys and values ​​for the input features, which can be represented as follows:

[0107] ;

[0108] ;

[0109] in Indicates spatial sampling, A shared projection layer representing keys and values. The projection layer representing the resolution. Indicates full resolution;

[0110] While spatial reduction of features can significantly reduce computational complexity, we enhance operator capacity by preserving the full-resolution query design. This allows the model to generate high-fidelity feature maps while substantially reducing computation, mitigating the loss of spatial information in deep networks. It is worth noting that, due to… Maintaining full-resolution scale ensures that the output sequence length is naturally aligned with the input, thereby reducing... While reducing computational overhead, it retains fine-grained spatial location information; this space-saving design has two advantages: firstly, it significantly reduces computational costs, because... The sequence length of Value was reduced. times, of which To reduce the sampling rate, space reduction can decrease FLOPs and video memory usage. Secondly, space-reducing attention can preserve the global receptive field, thus preserving the query's resolution by compressing only the key / value pair, resulting in minimal latency overhead.

[0111] It should be noted that the dimensional evolution of the space reduction strategy follows the formula:

[0112] ;

[0113] Where N represents the sequence length of the query, M represents the sequence length of the key and value, and d represents the feature dimension. This represents matrix multiplication.

[0114] Step S303: Set up a gating attention strategy to enable deep feature fusion between active and passive branches;

[0115] Specifically:

[0116] The input feature X is processed through a linear layer to generate a corresponding hybrid representation containing gating information, specifically:

[0117] ;

[0118] Where G is the gating signal. For linear projection operations, For channel splitting operation;

[0119] It should be noted that the gating signal pass Function activation, generation and Dimensionally consistent weighted graph:

[0120] ;

[0121] in, For activation function, For the characteristics of sub-control branches, Represents the modulated value vector;

[0122] By proposing a gated modulation strategy, high-value semantic features are given higher weights in the nonlinear space, thus preventing key object information from being overwhelmed by noise. This design, while purifying feature representation, does not significantly increase computational burden due to sharing the projection computation of the linear layer.

[0123] Let the output of the final active branch be denoted as ,in:

[0124] ;

[0125] in Represents the normalized exponential function, Represents the transpose of the key matrix;

[0126] The output obtained after deeply fusing the active and passive branches is denoted as Y, where:

[0127] ;

[0128] in This represents the learnable projective weight matrix. This indicates a channel splicing operation.

[0129] Step S4: Introduce a learnable idle channel reparameterized feedforward network. During the training phase, the feature channels are dynamically weighted using a learnable mask vector. During the inference phase, the mask vector is thresholded and converted into a binary mask.

[0130] Specifically:

[0131] Let the mask vector be M. Then, the process of dynamically weighting the feature channels during the training phase can be represented as:

[0132] ;

[0133] in, This represents the output after fusion. Indicates active branch output. Indicates passive branch output. Here, M is a learnable bias term, and M is a learnable mask vector. The input features of the portion of the channels that are assigned to the "idle branch";

[0134] Then, the output of the learnable idle channel reparameterized feedforward network is obtained, denoted as... ,in:

[0135] ;

[0136] in, This represents the output projection weights, and BN(.) represents the batch normalization operation.

[0137] In the reasoning stage, through... Thresholding is performed to convert it into a binary mask, and the corresponding equivalent inference weights and equivalent biases are obtained, denoted as follows: and ,in:

[0138] ;

[0139] ;

[0140] in, Indicates the input projection weights. To output the projected weights, represent Represents a diagonal matrix; This indicates the bias of the input projection layer. Indicates the bias of the output projection layer;

[0141] By implementing finer-grained overparameterized control over idle channels during the training phase, the optimization path is effectively improved and the training loss is reduced. Ultimately, LIP-RepFFN can completely collapse the learning content accumulated during the training phase into the backbone weights before inference through mathematical transformations. This allows for higher model accuracy and feature learning capabilities than the original static reparameterization without increasing inference costs.

[0142] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications or equivalent substitutions made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A lightweight visual global perception method based on RSFVIT, characterized in that, include: Step S1: Construct an RSFVIT structure including a three-level network, and complete the training phase of the constructed RSFVIT structure based on reparameterized positional encoding; Step S2: Perform weight transformation and reparameterization on the frequency domain branch and the spatial domain branch to obtain the standard spatial domain convolution kernel; Step S3: Perform channel optimization on the RSFVIT structure through split hybrid self-attention, wherein split hybrid self-attention includes channel splitting strategy, space reduction strategy and gated attention strategy; Step S4: Introduce a learnable idle channel reparameterized feedforward network. During the training phase, the feature channels are dynamically weighted using a learnable mask vector. During the inference phase, the mask vector is thresholded and converted into a binary mask. The process of step S3 is as follows: Step S301: Split the input features using a channel splitting strategy to obtain active and passive branches; Step S302: Compress the search space of the attention matrix using a space reduction strategy; Step S303: Set up a gating attention strategy to enable deep feature fusion between active and passive branches; The process of step S4 is as follows: Let the mask vector be M. Then, the process of dynamically weighting the feature channels during the training phase can be represented as: ； in, This represents the output after fusion. Indicates active branch output. Indicates passive branch output. Here, M is a learnable bias term, and M is a learnable mask vector. The input features of the channels that are assigned to the "idle branch"; Then, the output of the learnable idle channel reparameterized feedforward network is obtained, denoted as... ,in: ； in, This represents the output projection weights, and BN(.) represents the batch normalization operation.

2. The lightweight visual global perception method based on RSFVIT according to claim 1, characterized in that, The process of step S1 is as follows: Step S101: Construct a feature extraction branch, which includes a spatial domain branch and a frequency domain branch; Step S102: Extract features from the input features using the constructed feature extraction branches; Step S103: Based on frequency-spatial duality reparameterized position coding, the extracted features are dynamically fused to obtain the corresponding position sensing features.

3. A lightweight visual global perception method based on RSFVIT according to claim 2, characterized in that, The spatial branch is used to extract the spatial features of the input features, and the frequency branch is used to extract the frequency features of the input features.

4. A lightweight visual global perception method based on RSFVIT according to claim 2, characterized in that, The process of step S102 is as follows: The input features are denoted as feature X; In the spatial domain branch, multi-branch convolutional combinations are used to process the feature X, obtaining the corresponding local detail features, which are then used as the output of the spatial domain branch. The output of the spatial domain branch is denoted as... ,in: ； Wherein, SDB(.) represents a parallel structure containing main convolution, identity mapping and average pooling operators; In the frequency domain branch, the input feature X is globally coupled through Fourier transform and inverse Fourier transform to obtain the output of the frequency domain branch, which is denoted as . ,in: ； in, Represents frequency domain weights, Represents the inverse transform operator. This represents a transformation operator.

5. A lightweight visual global perception method based on RSFVIT according to claim 4, characterized in that, The process of step S2 is as follows: Step S201: Combine the multi-branch convolutions in the spatial branch and perform weight fusion to obtain the corresponding spatial equivalent convolution kernel; Step S202: Map the frequency domain weights of the frequency domain branch to the spatial domain and extract the corresponding equivalent frequency domain convolution kernels; Step S203: The linear additivity of the convolution operator is used to statically reparameterize and fuse the weights in the spatial and frequency domains.

6. A lightweight visual global perception method based on RSFVIT according to claim 5, characterized in that, The process of step S202 is as follows: The spatial global convolution kernel obtained after inverse Fourier transform of the frequency domain weights is denoted as . ; The input feature X is then transformed in the frequency domain using the convolution theorem, and the transformation result is denoted as F(X), where the convolution theorem is: ； in, The result is the inverse Fourier transform of the frequency domain weights; The frequency domain weights are then mapped to the spatial domain, as follows: ； ； The obtained spatial global convolution kernel is truncated to obtain the equivalent frequency domain convolution kernel corresponding to the frequency domain weights, denoted as . ,in: ； in This indicates the truncation operator.

7. A lightweight visual global perception method based on RSFVIT according to claim 6, characterized in that, Step S4 also includes: In the reasoning stage, through... Thresholding is performed to convert it into a binary mask, and the corresponding equivalent inference weights and equivalent biases are obtained, denoted as follows: and ,in: ； ； in, Indicates the input projection weights. To output the projected weights, representing Represents a diagonal matrix; This indicates the bias of the input projection layer. This indicates the bias of the output projection layer.

Images