A training deployment decoupled attention aided convolution system and method

Abstract

The application relates to a training-deployment decoupled attention auxiliary convolution system and method, and belongs to the technical field of computer vision, deep learning and intelligent perception, and particularly relates to a training-deployment decoupled attention auxiliary convolution system and method. The application relates to a training-deployment decoupled attention auxiliary convolution system and method. The system comprises a main convolution branch, at least one additional convolution branch, a residual convolution branch and an auxiliary attention module; in the training stage, the main convolution branch, the additional convolution branch and the residual convolution branch process input features respectively to obtain calibrated output features; in the deployment stage, the auxiliary attention module is removed, and the main convolution branch, the additional convolution branch and the residual convolution branch are folded into a single equivalent convolution according to a structure reparameterization method, and the single equivalent convolution is used to perform inference in the deployment stage.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, deep learning, and intelligent perception, and specifically to an attention-assisted convolution system and method with decoupled training and deployment. Background Technology

[0002] In existing deep learning vision models, various attention mechanisms, such as channel attention, spatial attention, or joint channel-space attention, are typically introduced into convolutional modules to enhance feature representation capabilities. These modules can enhance salient features and suppress background noise during the training phase or throughout the entire inference path, thereby improving the recognition accuracy of visual tasks such as object detection, image classification, and instance segmentation.

[0003] Meanwhile, to meet practical deployment needs, a new convolutional design approach that decouples training and deployment has emerged in recent years. This type of method typically employs a multi-branch convolutional structure during the training phase to enhance the network's optimization and feature learning capabilities; during the deployment phase, the multi-branch structure is folded into a single-branch convolution through structural reparameterization, thereby reducing inference latency and runtime overhead, achieving a network structure design that is complex during the training phase but simple during the deployment phase.

[0004] Against this backdrop, Chen et al., in their paper "X-volution: On the Unification of Convolution and Self-attention," proposed constructing a multi-branch module with both convolutional and approximate self-attention branches during the training phase. This allows the network to simultaneously acquire local convolutional modeling capabilities and non-local context modeling capabilities. After training, structural reparameterization transforms the multi-branch structure into a single convolutional form, achieving a decoupled design between training enhancement and efficient deployment. This research focuses on the unified operator expression of convolution and self-attention, enhancing the network's feature representation capabilities through multi-branching, and reducing computational overhead during deployment through reparameterization.

[0005] Furthermore, Huang et al. proposed the Attention-alike Structural Re-parameterization (ASR) method in their paper "Stripe Observation Guided Inference Cost-free Attention Mechanism." Based on the observation that channel attention responses tend to a constant vector during training (Stripe Observation), this method folds the attention module during training into the convolution parameters, allowing the attention enhancement effect to be preserved in the deployment phase with zero additional inference overhead. This research provides a way to efficiently combine training enhancement with deployment, but its enhancement capability mainly relies on the assumption of constant attention response.

[0006] However, existing technologies still have the following shortcomings when combining attention mechanisms with training and deployment decoupled convolutional structures: (1) Attention mechanisms typically rely on online participation in inference. Most existing attention modules directly participate in feature enhancement as components of the complete forward path, and their performance improvement usually depends on the continued existence of attention branches during the inference stage. Once the attention branches are removed during the deployment stage to meet real-time requirements, the performance gains obtained during the training stage are often difficult to retain.

[0007] (2) Existing deployable convolutional structures mainly address the "structure folding" problem, but do not solve the "enhancement gain transfer" problem. Many training and deployment decoupling methods focus on how to convert the multi-branch structure in the training stage into a single convolutional structure in the deployment stage. However, there is a lack of specific design for how to effectively transfer and deposit the representation gains brought by the additional enhancement branches in the training stage, especially the attention enhancement branches, into the final deployment convolution.

[0008] (3) Directly retaining the attention mechanism will weaken deployment efficiency. If the attention module is retained during the deployment phase, it will introduce additional computation, memory access and dynamic inference overhead, which is not conducive to application scenarios with high requirements for inference efficiency, such as edge devices, embedded devices and real-time detection.

[0009] (4) Directly removing the attention mechanism can easily lead to performance degradation. If the training phase relies on strong dynamic attention enhancement, and the corresponding enhancement branch is directly deleted in the deployment phase, the main convolution will have difficulty fully inheriting the effective feature preferences obtained during training, which can easily lead to a break between the training effect and the deployment effect, resulting in a significant decrease in the final deployment accuracy. Summary of the Invention

[0010] To address the problems in existing decoupled training and deployment networks, such as attention enhancement typically relying on online inference paths, difficulty in effectively transferring training-stage enhancement branches to the final deployment convolution through structural reparameterization, and the increased deployment overhead of retaining the attention mechanism while directly removing it easily leads to degraded deployment performance, this invention aims to propose an attention-assisted convolution system and method for decoupled training and deployment. This system transfers the enhancement benefits from the training stage to the final single-convolution deployment structure.

[0011] The system includes: a main convolutional branch, at least one additional convolutional branch, a residual convolutional branch, and an auxiliary attention module; The main convolution branch is used to acquire input features and perform convolutional feature extraction on the input features; At least one additional convolutional branch, set in parallel with the main convolutional branch, is used to perform auxiliary feature extraction on the input features; The residual convolution branch is set in parallel with the main convolution branch and is used to retain input feature information and participate in feature fusion; An auxiliary attention module is placed after the main convolutional branch, the additional convolutional branch, and the residual convolutional branch; During the training phase, the main convolutional branch, the additional convolutional branch, and the residual convolutional branch process the input features respectively, and then the outputs of each branch are aggregated with the input features to obtain aggregated features; The auxiliary attention module generates channel recalibration results based on aggregated features, and uses the channel recalibration results to reweight the aggregated features to obtain calibrated output features; Based on the calibrated output features, calculate the training loss and update the training parameters of the main convolutional branch; During the deployment phase, the auxiliary attention module is removed, and the main convolutional branch, the additional convolutional branch, and the residual convolutional branch are folded into a single equivalent convolution according to the structure reparameterization method. The inference of the deployment phase is then performed using the single equivalent convolution.

[0012] Furthermore, the input features can be any one or more of images, text, and videos.

[0013] Furthermore, the auxiliary attention module generates channel recalibration results based on aggregated features, specifically as follows: The aggregated features are subjected to channel transformation to obtain transformed features; The transformation features are divided according to the channel dimension. One channel group; Global average pooling is performed on each channel group to obtain the group-level channel description vector corresponding to each channel group; The channel description vectors of each group are input into a weight generation unit that includes at least two layers of learnable mappings, and nonlinear activation and normalization mapping are performed sequentially to obtain the soft calibrator submask corresponding to each channel group. Each soft calibration sub-mask is concatenated along the channel dimension to obtain a global channel calibration mask corresponding to the number of transform feature channels, which serves as the channel recalibration result.

[0014] Furthermore, the aggregated features are reweighted using the channel recalibration results, specifically as follows: Multiply the global channel calibration mask by the learnable scaling factor and add it to the value 1 to obtain the residual soft calibration coefficient. The residual soft calibration coefficients and the transformation features are multiplied element-wise according to the corresponding channels and corresponding spatial positions to obtain the soft recalibration features; The soft recalibration feature is nonlinearly activated to obtain the calibrated output feature.

[0015] Furthermore, the training loss is calculated based on the calibrated output features, and the training parameters of the main convolutional branch are updated, specifically as follows: The calibrated output features are input into the task processing network corresponding to the target task to obtain the task prediction result; The task loss is calculated based on the task prediction result and the corresponding ground truth label. The task loss includes at least one of classification loss, location regression loss and confidence loss. Backpropagation is performed based on the task loss to calculate the gradients of the trainable parameters in the main convolutional branch, the additional convolutional branch, and the auxiliary attention module, and the corresponding trainable parameters are updated using the gradient optimization algorithm.

[0016] Furthermore, the structure reparameterization method specifically involves: converting the parameters to be fused from the main convolutional branch, the additional convolutional branch, and the residual convolutional branch into corresponding equivalent convolutional parameters; unifying the equivalent convolutional parameters of the main convolutional branch, the additional convolutional branch, and the residual convolutional branch into equivalent convolutional representations with the same kernel size; and fusing the unified equivalent convolutional parameters to obtain the deployment convolutional kernel parameters and deployment bias parameters for the deployment stage.

[0017] A training and deployment decoupled attention-assisted convolution method, the method being implemented based on the above system, and the method comprising the following steps: S1. Obtain the input features and input the input features into the main convolutional branch, at least one additional convolutional branch and the residual convolutional branch respectively to obtain the corresponding branch output features; S2. Aggregate the main convolutional branch output features, the additional convolutional branch output features, and the residual convolutional branch output features to obtain aggregated features; S3. Perform channel transformation on the aggregated features and divide them into n channel groups along the channel dimension; perform global average pooling and learnable mapping on each channel group to generate corresponding soft calibration sub-masks; concatenate each soft calibration sub-mask along the channel dimension to obtain the global channel calibration mask. S4. Generate residual soft calibration coefficients using the global channel calibration mask and learnable scaling coefficients. Multiply the residual soft calibration coefficients element-wise with the transformed aggregated features and process the resulting soft recalibrated features using an activation function to form the calibrated output features. S5. Obtain the task prediction result based on the calibrated output features, calculate the task loss based on the task prediction result and the real annotation, and perform backpropagation based on the task loss to update the trainable parameters of the main convolutional branch, the additional convolutional branch and the auxiliary attention module. S6. After training is completed, the auxiliary attention module is removed, and the main convolution branch, the additional convolution branch and the residual branch are converted into equivalent convolution representations with a uniform convolution kernel size. The convolution kernel parameters and bias parameters of each equivalent convolution representation are fused to obtain a single equivalent convolution. The inference of the deployment phase is performed using the single equivalent convolution.

[0018] The beneficial effects of this invention are as follows: (1) In the training phase, the present invention significantly enhances the feature learning ability of the network through multi-branch convolutional structure and auxiliary attention module. In the deployment phase, the multi-branch structure is folded into a single convolutional operation to realize the effective transfer of training enhancement benefits to the deployment structure, while maintaining efficient computation in the inference phase, thus taking into account both model performance and deployment efficiency.

[0019] (2) Unlike traditional attention modules that must be continuously involved in the inference phase to play a role, this invention, through the guided design in the training phase, allows the auxiliary attention branch to be completely removed in the deployment phase, while still retaining the performance improvement gained in the training phase, reducing deployment complexity and computational overhead.

[0020] (3) The present invention focuses on solving the problem that the enhancement gains in the training phase are difficult to inherit in the deployment phase, so that the auxiliary training module not only serves the complete forward path, but also effectively shapes the parameter preferences of the backbone convolution, realizes the stable transfer of training gains, and thus improves the expressiveness and task accuracy of the final deployment structure.

[0021] (4) Only a single equivalent convolution is retained during the deployment phase. The inference computation graph is simple and lightweight, significantly reducing the amount of computation and memory access overhead. It is suitable for application in embedded platforms, edge devices and real-time detection systems, meeting the requirements of efficient and low-latency inference. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the system structure described in this invention. Detailed Implementation

[0023] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Example 1 This embodiment provides an attention-assisted convolutional system with decoupled training and deployment. The structure of the system is as follows: Figure 1As shown, the system includes: a GSCConv convolution module and an auxiliary attention module located after the GSCConv convolution module. The auxiliary attention module is specifically a Group Soft Channel Recalibration (GSCR) module. The GSCConv convolutional module includes a main convolutional branch, at least one additional convolutional branch, a residual branch, and a feature aggregation unit. The main convolutional branch, additional convolutional branch, and residual branch are configured in parallel and each receives the same input feature. .

[0025] The input features It can be an image, a video frame, or a feature tensor extracted by a preceding neural network, and its dimension can be represented as... ,in, Indicates batch size. Indicates the number of channels. and These represent the height and width of the feature map, respectively.

[0026] The main convolutional branch is used to process the input features. Local spatial feature extraction is performed. In this embodiment, the main convolutional branch includes one... Convolutional module, the input features After that After the convolutional module, the output features of the main convolutional branch are obtained. , is represented as:

[0027] in, Indicating the main convolution branch Convolution operation.

[0028] The at least one additional convolutional branch is configured in parallel with the main convolutional branch to apply trainable parameters independent of the main convolutional branch to the input features. Perform auxiliary feature extraction.

[0029] In this embodiment, the additional convolutional branch includes one Additional convolution branches and one Additional convolution branches.

[0030] The The additional convolutional branch includes one The convolution module is used to extract local spatial features from the input features through a parameter path different from that of the main convolution branch, thus obtaining the output features of the first additional branch. , is represented as:

[0031] in, Indicating the first additional convolution branch Convolution operation.

[0032] The The additional convolutional branch includes one The convolutional module is used to perform channel mapping and inter-channel information exchange on the input features to obtain the output features of the second additional branch. , is represented as:

[0033] in, Indicating the second additional convolution branch Convolution operation.

[0034] The main convolutional branch and Although the additional convolution branches all use Both are convolutional modules, but their kernel parameters are independent, thus enabling the two... Convolutional branches can learn different feature responses during training.

[0035] In other embodiments, the number of additional convolutional branches is not limited to the number shown in this embodiment, and can be set to one or more according to the number of input channels, model size, and task requirements; the additional convolutional branches can employ... convolution, Convolution or other convolutional structures that can be converted into equivalent convolutions during the deployment phase.

[0036] The residual branch is set up in parallel with the main convolution branch and each additional convolution branch to retain the original information in the input features and pass it to the feature aggregation unit.

[0037] In this embodiment, the residual branch preferably adopts the identity mapping method, that is, the input features Directly used as the output feature of the residual branch Represented as:

[0038] When the number of channels or spatial dimensions of the input features are inconsistent with those of the output features of other convolutional branches, the residual branch can also be set... Projective convolution or other linear mapping units are used to complete feature dimension matching. The residual branch output features are then represented as follows:

[0039] The feature aggregation unit is positioned after the main convolutional branch, each additional convolutional branch, and the residual branch. It is used to add the output features of the main convolutional branch, each additional convolutional branch, and the residual branch element-wise to obtain the aggregated features. .

[0040] In this embodiment, aggregation feature Represented as:

[0041] After substituting the expressions of each branch, aggregate the features. This can be further expressed as:

[0042] Since each branch output has the same spatial size and number of channels before element-wise addition, the aggregation feature... Can contain two at the same time Local spatial information extracted by convolution branches The convolution branch extracts channel mapping information, while the residual branch preserves the original input information.

[0043] The GSCR module, located after the feature aggregation unit, is used to process the aggregated features. Perform group-level soft channel recalibration.

[0044] The GSCR module includes a channel transformation unit, a channel grouping unit, at least one group-level weight generation and soft recalibration unit, a feature splicing unit, and an activation unit.

[0045] The channel conversion unit includes one Convolutional modules are used to aggregate features. Perform channel transformation to obtain transformation characteristics. , is represented as:

[0046] in, This indicates the channel conversion unit in the GSCR module. Channel transformation operation.

[0047] The Convolutional modules map and fuse information from different channels to obtain transformed features while maintaining the feature space size. Transformation features The dimension is represented as:

[0048] in, This indicates the number of feature channels after channel transformation. Indicates batch size. express The height dimension of the feature map is the number of pixels / grids in the vertical direction; W represents the width dimension of the feature map, i.e., the number of pixels / grids in the horizontal direction. In implementations that maintain a consistent number of input and output channels, we have:

[0049] The channel grouping unit is used to transform features along the channel dimension. Divided into A group of channels is represented as:

[0050] in, Indicates the first One channel group, , Indicates the number of channel groups. In this embodiment, The value of is 4.

[0051] When each channel group contains the same number of channels, the first... Channel group characteristics The dimension is represented as:

[0052] At this time, the number of channels Number of channels that can be grouped Divisible, that is:

[0053] In other implementations, when the number of channels cannot be divided evenly by the number of groups, a non-equivalent channel grouping method with different numbers of channels in each group can also be used.

[0054] Each channel group corresponds to a set of group-level weight generation and soft recalibration units. These group-level weight generation and soft recalibration units are used to generate soft calibration weights based on the characteristic responses of the corresponding channel group. . Specifically, for the first Channel group characteristics First, extract the group-level channel statistics for the channel group. Preferably, global average pooling is used to obtain the group-level channel description vector. , is represented as: in, This indicates a global average pooling operation.

[0055] For channel groups Middle Each channel, and its group-level channel description value can be expressed as:

[0056] in, Indicates the first In the channel group, the first The spatial location of each channel eigenvalues ​​at that location and They represent Feature map height and feature map width, and These represent spatial position indices in the height and width directions, respectively.

[0057] Subsequently, the group-level channel description vector Input the learnable weight generation network to obtain the first The soft calibration submask (weight) corresponding to each channel group. .

[0058] In a preferred embodiment, the learnable weight generation network includes a first learnable mapping layer, a nonlinear activation function, a second learnable mapping layer, and a sigmoid normalization function. Soft calibrator mask. Represented as:

[0059] in, Indicates the first The weight parameters of the first learnable mapping layer corresponding to each channel group This represents the bias parameters of the first learnable mapping layer. This represents the weight parameters of the second learnable mapping layer. This represents the bias parameters of the second learnable mapping layer. This represents a non-linear activation function, preferably the ReLU function. This represents the Sigmoid function. In this embodiment, , , and The value of is not a fixed hyperparameter, but a learnable parameter obtained through network training. Its initial value can be set according to the conventional initialization method of neural networks, such as random initialization of weights and setting the bias to zero; it is automatically updated during training based on backpropagation of task loss, and its final value is determined by the training data and the optimization process. The ReLU activation function is expressed as:

[0060] The Sigmoid function is represented as:

[0061] The soft calibration weights are then processed using the Sigmoid function. Mapped to to The continuous range of values, that is:

[0062] The soft calibrator mask These are continuous weights used to adjust the feature response intensity of each channel in the corresponding channel group, rather than rigidly turning each channel on or off. The soft calibration weights corresponding to each channel group are concatenated along the channel dimension to obtain the overall channel calibration mask M, represented as:

[0063] in, This indicates that a splicing operation is performed along the channel dimension. Indicates the number of channel groups.

[0064] The overall channel calibration mask Channel number and transformation characteristics The number of channels is the same, that is: ; When the overall channel calibration mask The space dimensions are At that time, extend it along the height and width directions to match the transformation features. With the same spatial dimensions, the expanded overall channel calibration mask is obtained:

[0065] Preferably, a residual soft recalibration method is adopted, in which the residual soft calibration coefficients are multiplied element-wise with the transform features according to the corresponding channels and corresponding spatial positions to generate soft recalibration features. , is represented as:

[0066] in, Represents the residual soft calibration coefficient. The number of channels and When the number of channels is the same, , This indicates element-wise multiplication. This indicates a learnable scaling parameter. These are not fixed hyperparameters, but rather learnable parameters obtained through network training. Their initial values ​​can be set using conventional neural network initialization methods, such as random weight initialization and zeroing out biases. During training, they are automatically updated based on backpropagation of the task loss, and their final values ​​are determined by the training data and the optimization process. A value of 1 indicates that the features to be transformed are retained. The basic characteristic response.

[0067] Employing a residual-based soft recalibration method avoids excessive suppression of original features due to excessively small soft calibration weights, and makes the adjustment of features by group-level weights smoother. The recalibrated overall features (soft recalibrated features) are described below. The input activation unit undergoes nonlinear activation to obtain the output features during the training phase. .

[0068] In this embodiment, the activation unit preferably uses the SiLU nonlinear activation function, and the output features Represented as:

[0069] The SiLU nonlinear activation function is expressed as:

[0070] In other implementations, ReLU, Leaky ReLU, GELU, or other nonlinear activation functions may also be used.

[0071] The output features of the training phase The results are fed into the subsequent task network to obtain task predictions. , is represented as:

[0072] in, Indicates the subsequent task network, This represents the trainable parameters of the task network. It's not a fixed network name, but rather a general term for subsequent task networks / task heads. For example, in object detection tasks, it can be understood as the detection head; in classification tasks, it can be understood as the classification head; and in segmentation tasks, it can be understood as the segmentation head.

[0073] in, This indicates a subsequent task network set after the convolution module of this invention, used to generate prediction results for the corresponding task based on the output feature Y; This refers to the trainable parameters in the subsequent task network, including convolutional layer weights, bias parameters, normalization layer parameters, and task prediction layer parameters.

[0074] Taking object detection as an example, This can be a target detection head, which receives output features. And generate target category prediction results, target location regression results, and target confidence prediction results; at this time, This includes the convolution kernel parameters and bias parameters used in the detection head for classification prediction, bounding box regression, and confidence prediction.

[0075] Taking image classification tasks as an example, It can be a classification network consisting of a global pooling layer and a classifier, wherein the classification network receives output features. And generate category probability prediction results; at this time, This includes the weights and bias parameters of the fully connected layers in the classifier.

[0076] Taking semantic segmentation tasks as an example, This can be a segmentation decoding head, which receives the output feature Y and generates pixel-level category prediction results; at this time, This includes trainable parameters for the convolutional layers, upsampling layers, and pixel classification layers in the segmentation decoder.

[0077] Let the parameter set consist of all trainable parameters of the main convolutional branch, each additional convolutional branch, and the GSCR module. According to the training loss Performing backpropagation, the parameter gradient is expressed as:

[0078] The training loss The training loss can be determined based on the specific task type; for example, in image classification tasks, the training loss can be cross-entropy loss; in object detection tasks, the training loss can include classification loss, bounding box regression loss, and object confidence loss; in semantic segmentation tasks, the training loss can include pixel-level cross-entropy loss, Dice loss, or a combination of both. Therefore, the training loss... It can be represented as:

[0079] in, (·) represents the loss function corresponding to the target task, and P represents the true label, true detection box, true segmentation mask or other task supervision information.

[0080] The parameters are updated using a gradient optimization algorithm, as follows:

[0081] in, This indicates the current training iteration number. Represents the set of parameters for the current iteration. , Indicates the learning rate. Indicates to Find the gradient. Preferably, in this embodiment, the initial learning rate The value is 0.01; in other embodiments, the learning rate can also be set to other values ​​according to the training task and optimizer, for example... to The values ​​can be within a certain range, or a dynamic learning rate can be used that varies with the number of training iterations.

[0082] Because the outputs of the main convolutional branch and the additional convolutional branch together form the aggregated feature. And the aggregation feature After GSCR group-level soft channel recalibration, it is used in the training loss. Therefore, the soft calibration weights generated by GSCR can influence the update of parameters of each convolutional branch through the backpropagation process, enabling each convolutional branch to gradually learn the channel response preferences that are beneficial to the target task.

[0083] Figure 1 The training phase guidance path in the text represents the parameter guidance relationship implemented based on the training loss and backpropagation, and does not mean that additional feedback calculation paths are still retained in the deployment phase.

[0084] After training, the deployment phase begins. During deployment, the GSCR module, which was only used for feature guidance during training, is removed, and the main convolutional branch, all additional convolutional branches, and residual branches in GSCConv are transformed into a single equivalent module through structural reparameterization. convolution.

[0085] Let the GSCConv be in The kernel parameters and bias parameters of each convolutional branch are as follows: and In this embodiment ,in, Indicates the main convolution branch. This represents the first additional convolutional branch immediately following the main convolutional branch. Additional convolution branches). This represents the second additional convolutional branch immediately following the first additional convolutional branch. Additional convolution branches). This represents the residual branch. When the convolutional branch in GSCConv includes convolutional layers and batch normalization layers, the parameters of the batch normalization layers are first fused into the corresponding convolutional layers.

[0086] Let the scaling parameter, translation parameter, mean, variance, and numerical stability parameter corresponding to the batch normalization layer be as follows: , , and The equivalent convolution kernel after fusion Represented as: = ,

[0087] Equivalent bias after fusion Represented as: , ; For the main convolution branch and Additional convolution branches, whose convolution kernels are directly used as... Equivalent convolution kernel.

[0088] for Add convolution branches and use zero-padding to... Convolution kernel expansion to The equivalent convolution kernel is represented as:

[0089] in, Indicates will convolution kernel Placed in The operation of filling the center of the convolution kernel with zeros and filling the rest of the kernel with zeros.

[0090] The equivalent of each branch The convolution kernels are added element-wise to obtain the convolution kernel parameters for the deployment phase. , is represented as:

[0091] The bias parameters corresponding to each branch are summed to obtain the bias parameters for the deployment phase. , is represented as:

[0092] Therefore, single reparameterization during the deployment phase Convolution output Represented as:

[0093] in, This represents the convolution operation.

[0094] Final output features of the deployment phase Represented as:

[0095] Through the above training and deployment decoupling method, GSCR only performs group-level soft channel recalibration on aggregated features during the training phase, and guides each convolutional branch to learn effective channel feature preferences through backpropagation; during the deployment phase, there is no need to retain the multi-branch computational structure of GSCR and the training phase, and only a single reparameterization is used. Convolution and activation functions complete the forward inference.

[0096] In this embodiment, the channel recalibration result generated by the auxiliary attention module (CSGR) affects the task loss through the calibrated output features, and forms training guidance for the parameter update of the main convolutional branch through backpropagation, so that the main convolutional branch learns the channel feature preferences enhanced by group-level soft recalibration.

[0097] Example 2 This embodiment provides an attention-assisted convolution method with decoupled training and deployment, which is implemented based on the system described in Embodiment 1.

[0098] The method includes the following steps: S1. Obtain the input features and input the input features into the main convolutional branch, at least one additional convolutional branch and the residual convolutional branch respectively to obtain the corresponding branch output features; S2. Aggregate the main convolutional branch output features, the additional convolutional branch output features, and the residual convolutional branch output features to obtain aggregated features; S3. Input the aggregated features into the auxiliary attention module, and generate channel recalibration results based on the aggregated features; The auxiliary attention module does not serve as part of the permanent inference path, but rather as a guiding mechanism during the training phase. By reweighting, recalibrating, or soft-modulating the aggregated features, it prompts the main convolutional branch to learn feature preferences that are more beneficial to the final task.

[0099] More specifically, the auxiliary attention module preferably adopts a lightweight channel recalibration method rather than a computationally intensive and complex dynamic space selection method, thereby providing a smoother and more stable guidance for the main convolution learning.

[0100] S4. Reweight the aggregated features using the channel recalibration results to form calibrated output features; S5. Calculate the training loss based on the calibrated output features and update the training parameters of the main convolutional branch; S6. After training is completed, the deployment phase begins. This invention does not retain the auxiliary attention module from the training phase, but instead performs the following processing: 1. Fold multiple convolutional branches and residual branches during the training phase into a single equivalent convolution according to the reparameterization rule; 2. Remove the auxiliary attention branch used during the training phase; 3. Use the single equivalent convolution as the only convolutional inference unit in the deployment model.

[0101] This allows the model to have inference complexity comparable to a regular single convolutional structure when deployed, while retaining as many training phase enhancement benefits as possible.

[0102] Example 3 This embodiment further limits embodiment 1, and provides an alternative to embodiment 1, specifically: 1. The multi-branch convolutional structure during the training phase is not limited to the branches described in Example 1, but can also use other combinations of reparameterizable convolutional branches.

[0103] 2. The auxiliary attention module is not limited to a certain implementation method. As long as it can guide the learning of the main convolution during the training phase and be removed during the deployment phase, it is acceptable.

[0104] 3. The auxiliary attention module can be implemented using methods such as channel recalibration, group-level weight modeling, soft gating, and residual modulation.

[0105] 4. This invention is not only applicable to object detection, but can also be extended to visual tasks such as image classification, semantic segmentation, and object tracking that require a balance between training capability and deployment efficiency.

[0106] 5. This invention can be deployed on GPUs, edge computing devices, embedded chips, or other hardware platforms with neural network inference capabilities.

[0107] Example 4 This embodiment is based on experimental verification using the DETFLY UAV small target detection dataset: To verify the effectiveness of the proposed training-deployment decoupled attention-assisted convolutional structure (CSGR) in small target detection tasks, this embodiment uses the publicly available DETFLY UAV target detection dataset for experiments. This dataset is used for UAV target detection scenarios, where the target scale is small and the background interference is strong, which can effectively verify the detection performance and deployment consistency of the present invention under conditions of small targets, weakly textured targets, and complex backgrounds.

[0108] The experimental platform in this embodiment includes the Ubuntu operating system, Python 3.10.19, PyTorch 2.5.1+cu121, and Ultralytics 8.3.138. During training, two NVIDIA RTX4090 graphics cards were used, and the input image size was set to 640×640. Commonly used evaluation metrics in object detection were employed to assess model performance, including mAP@0.5, mAP@0.5:0.95, mAP@0.75, and FPS. Specifically, mAP@0.5:0.95 serves as the primary evaluation metric for overall detection accuracy, mAP@0.75 measures object localization accuracy, and FPS evaluates the model's inference real-time performance.

[0109] The DETFLY dataset is divided into training, validation, and test sets. The training set contains 9071 images, the validation set contains 1132 images, and the test set contains 1155 images. To adapt to the small target detection scenario for UAVs, all images are converted to grayscale for training and testing. Targets with an area less than 0.1% of the total image area are defined as extremely small targets, and targets with an area less than 1% are defined as small targets. Statistical results show that in the training set, extremely small targets account for 65.58% and small targets account for 98.94%; in the validation set, extremely small targets account for 65.28% and small targets account for 98.94%; and in the test set, extremely small targets account for 65.19% and small targets account for 98.79%. Therefore, the DETFLY dataset is a typical dataset for UAV extremely small target detection and is suitable for verifying the effectiveness of this invention in small target detection tasks.

[0110] To verify the effectiveness of the Group Soft Channel Recalibration (GSCR) module, this embodiment constructs three structures for comparison: GSCConv, GSCConv+CBAM, and GSCConv+GSCR. GSCConv represents a model using only the decoupled convolutional structure for training and deployment; GSCConv+CBAM represents the GSCConv structure incorporating the traditional Channel Spatial Joint Attention (CBAM) module; and GSCConv+GSCR represents the proposed Group Soft Channel Recalibration (GSCR) module as an auxiliary attention module during the training phase. This comparison allows for the examination of the performance differences between traditional attention modules and the proposed GSCR module during the training and deployment phases.

[0111] The experimental results during the training phase are shown in the table below: Table 1. Comparison of detection performance of different structures during the training phase

[0112] As shown in Table 1, after introducing attention enhancement during the training phase, both GSCConv+CBAM and GSCConv+GSCR achieved higher detection accuracy compared to the basic GSCConv. Specifically, GSCConv+GSCR achieved mAP@0.5:0.95 of 0.592 and mAP@0.75 of 0.685, indicating that the GSCR module can enhance the model's feature representation and localization capabilities for small targets during the training phase.

[0113] The experimental results during the deployment phase are shown in the table below: Table 2 Comparison of detection performance of different structures during the deployment phase

[0114] As shown in Table 2, although GSCConv+CBAM achieves high accuracy during the training phase, its detection performance declines significantly after deployment. Its mAP@0.5 decreases from 0.929 to 0.819, a decrease of 0.110; mAP@0.5:0.95 decreases from 0.594 to 0.462, a decrease of 0.132; and mAP@0.75 decreases from 0.683 to 0.481. This indicates that the performance improvement of the traditional attention module largely depends on the continuous participation of the attention branch during the inference phase. When the attention module is removed or not retained during the deployment phase, the enhancement gains obtained during the training phase are difficult to be stably transferred to the final deployment structure, which can easily lead to a degradation in deployment performance.

[0115] In contrast, GSCConv+GSCR maintained mAP@0.5 of 0.924, mAP@0.5:0.95 of 0.592, and mAP@0.75 of 0.691 during the deployment phase, while achieving an FPS of 183.31. These results demonstrate that the GSCR module guides the training of aggregated features through group-level soft-channel recalibration during the training phase, allowing the feature enhancement gains generated during training to be stably transferred to the final reparameterized single-convolutional deployment structure. Furthermore, even removing the GSCR module during deployment does not cause significant loss in detection performance; instead, it maintains a high inference speed.

[0116] Furthermore, GSCConv+GSCR was deployed on a Jetson AGX Orin embedded platform, and inference acceleration was performed using TensorRTFP16 to verify the practical deployment capability of this invention on edge devices. The experimental results are shown in the table below: Table 3. Deployment results of TensorRT FP16 on Jetson AGX Orin platform

[0117] As shown in Table 3, on the Jetson AGX Orin platform, the model of this invention using GSCConv+GSCR achieves an mAP@0.5 of 0.921 and an mAP@0.5:0.95 of 0.590, while maintaining a real-time inference speed of 125.00 FPS. Compared with YOLO11n, this invention shows significant improvements in both mAP@0.5 and mAP@0.5:0.95; compared with small target frames, this invention further improves detection accuracy while maintaining real-time inference capability. These results demonstrate that this invention not only improves the detection accuracy of small targets on UAVs but is also suitable for deployment scenarios with limited computing resources, such as embedded platforms, edge devices, and real-time detection systems.

[0118] In summary, this embodiment verifies the consistency advantages of the GSCR module and GSCConv structure proposed in this invention during both the training and deployment phases. While traditional attention modules can improve the detection accuracy of the complete forward path during training, their removal during deployment can easily lead to significant performance degradation. In contrast, this invention, through a group soft-channel recalibration mechanism during training, allows the enhancement gains from the auxiliary attention module to be stably transferred to the final single-convolutional deployment structure. This balances small target detection accuracy and real-time inference efficiency without adding extra attention computation paths during deployment.

Claims

1. A training-deployment decoupled attention-assisted convolutional system, characterized in that, The system includes: a main convolutional branch, at least one additional convolutional branch, a residual convolutional branch, and an auxiliary attention module; The main convolution branch is used to acquire input features and perform convolutional feature extraction on the input features; At least one additional convolutional branch, set in parallel with the main convolutional branch, is used to perform auxiliary feature extraction on the input features; The residual convolution branch is set in parallel with the main convolution branch and is used to retain input feature information and participate in feature fusion; An auxiliary attention module is placed after the main convolutional branch, the additional convolutional branch, and the residual convolutional branch; During the training phase, the main convolutional branch, the additional convolutional branch, and the residual convolutional branch process the input features respectively, and then the outputs of each branch are aggregated with the input features to obtain aggregated features; The auxiliary attention module generates channel recalibration results based on aggregated features, and uses the channel recalibration results to reweight the aggregated features to obtain calibrated output features; Based on the calibrated output features, calculate the training loss and update the training parameters of the main convolutional branch; During the deployment phase, the auxiliary attention module is removed, and the main convolutional branch, the additional convolutional branch, and the residual convolutional branch are folded into a single equivalent convolution according to the structure reparameterization method. The inference of the deployment phase is then performed using the single equivalent convolution.

2. The attention-assisted convolutional system with decoupled training and deployment according to claim 1, characterized in that, The input features are any one or more of images, text, and videos.

3. The attention-assisted convolutional system with decoupled training and deployment according to claim 2, characterized in that, The auxiliary attention module generates channel recalibration results based on aggregated features, specifically: The aggregated features are subjected to channel transformation to obtain transformed features; The transformation features are divided according to the channel dimension. One channel group; Global average pooling is performed on each channel group to obtain the group-level channel description vector corresponding to each channel group; The description vectors of each channel group are input into a weight generation unit that includes at least two layers of learnable mappings, and nonlinear activation and normalization mapping are performed sequentially to obtain the soft calibrator submask corresponding to each channel group. Each soft calibration sub-mask is concatenated along the channel dimension to obtain a global channel calibration mask corresponding to the number of transform feature channels, which serves as the channel recalibration result.

4. The attention-assisted convolutional system with decoupled training and deployment according to claim 3, characterized in that, The aggregated features are reweighted using the channel recalibration results, specifically as follows: Multiply the global channel calibration mask by the learnable scaling factor and add it to the value 1 to obtain the residual soft calibration coefficient. The residual soft calibration coefficients are multiplied element-wise with the transformation features according to the corresponding channels and corresponding spatial positions to obtain the soft recalibration features; The soft recalibration feature is nonlinearly activated to obtain the calibrated output feature.

5. The attention-assisted convolutional system with decoupled training and deployment according to claim 4, characterized in that, The training loss is calculated based on the calibrated output features, and the training parameters of the main convolutional branch are updated accordingly: The calibrated output features are input into the task processing network corresponding to the target task to obtain the task prediction result; The task loss is calculated based on the task prediction result and the corresponding ground truth label. The task loss includes at least one of classification loss, location regression loss and confidence loss. Backpropagation is performed based on the task loss to calculate the gradients of the trainable parameters in the main convolutional branch, the additional convolutional branch, and the auxiliary attention module, and the corresponding trainable parameters are updated using the gradient optimization algorithm.

6. The attention-assisted convolutional system with decoupled training and deployment according to claim 5, characterized in that, The structure reparameterization method specifically involves: converting the parameters to be fused from the main convolutional branch, the additional convolutional branch, and the residual convolutional branch into corresponding equivalent convolutional parameters; unifying the equivalent convolutional parameters of the main convolutional branch, the additional convolutional branch, and the residual convolutional branch into equivalent convolutional representations with the same kernel size; and fusing the unified equivalent convolutional parameters to obtain the deployment convolutional kernel parameters and deployment bias parameters for the deployment stage.

7. A training-deployment decoupled attention-assisted convolution method, characterized in that, The method is implemented based on the system according to any one of claims 1 to 6, and the method includes the following steps: S1. Obtain the input features and input the input features into the main convolutional branch, at least one additional convolutional branch and the residual convolutional branch respectively to obtain the corresponding branch output features; S2. Aggregate the main convolutional branch output features, the additional convolutional branch output features, and the residual convolutional branch output features to obtain aggregated features; S3. Perform channel transformation on the aggregated features and divide them into n channel groups along the channel dimension; perform global average pooling and learnable mapping on each channel group to generate the corresponding soft calibration submask; The soft calibration sub-masks are concatenated along the channel dimension to obtain the global channel calibration mask; S4. Generate residual soft calibration coefficients using the global channel calibration mask and learnable scaling coefficients. Multiply the residual soft calibration coefficients element-wise with the transformed aggregated features and process the resulting soft recalibrated features using an activation function to form the calibrated output features. S5. Obtain the task prediction result based on the calibrated output features, calculate the task loss based on the task prediction result and the real annotation, and perform backpropagation based on the task loss to update the trainable parameters of the main convolutional branch, the additional convolutional branch and the auxiliary attention module. S6. After training is completed, the auxiliary attention module is removed, and the main convolution branch, the additional convolution branch and the residual branch are converted into equivalent convolution representations with a uniform convolution kernel size. The convolution kernel parameters and bias parameters of each equivalent convolution representation are fused to obtain a single equivalent convolution. The inference of the deployment phase is performed using the single equivalent convolution.

Images