A soft threshold attention mechanism for computer vision

Abstract

This invention relates to the field of deep learning-based computer vision technology, specifically to a soft-thresholding attention mechanism for computer vision. Drawing inspiration from deep residual shrinking networks, it replaces the SENet soft-thresholding method with a CA attention mechanism that simultaneously considers channel and positional information. A novel soft-thresholding attention module is proposed: it integrates the soft-thresholding function from traditional signal denoising algorithms with the CA attention mechanism. The CA attention mechanism determines the thresholds required for two sets of soft-thresholding functions in both the vertical and horizontal directions, and then performs soft-thresholding processing separately, weighting the input feature maps. This approach is better suited for detection tasks in computer vision and can be easily integrated into existing neural network architectures. By setting unimportant features to zero, it reduces interference from noise or redundant information, enhances the neural network's ability to extract important features, and ultimately improves the network's prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology based on deep learning, and in particular to a soft threshold attention mechanism applied to computer vision. Background Art

[0002] In the field of deep learning, the quality of samples in the data set directly affects the training effect of the model. However, there will inevitably be some noise or redundant information in the collected samples. Broadly speaking, information in the sample that is irrelevant to the current classification task can also be understood as noise or redundant information. These noise or redundant information will interfere with the final result. Therefore, reducing the noise or redundant information contained in the sample can improve the accuracy of model prediction to a certain extent.

[0003] Soft thresholding is the core step of many noise reduction methods. The key to its application is to determine a threshold value. According to this threshold value, the useful information in the input data is converted into a positive or negative value close to 0, and the noise information is converted into 0 features. In this way, the purpose of reducing noise can be achieved. It is possible to consider introducing soft thresholding into the field of deep learning to reduce the interference of noise or redundant information. Whether the threshold is appropriate has a direct impact on the effect of soft thresholding, but soft thresholding often requires a large amount of prior knowledge to determine the appropriate threshold. In the field of deep learning, each sample contains different noise. Therefore, to introduce the soft thresholding method into the field of deep learning, a method for determining the threshold that is adaptive to different samples is required, which is a challenging problem.

[0004] The attention mechanism has been a major research hotspot in recent years and has been widely used in various fields of deep learning, such as natural language processing, speech recognition, and computer vision. In particular, the attention mechanism has become one of the basic structures of natural language processing algorithms. The attention mechanism actually allows the machine to judge the areas that need attention in the data by imitating the human perception method. After continuous training and iteration, the model assigns corresponding weights to different parts of a large amount of input data, so that it can extract the information in the input data that the model considers important and unimportant for the current task, and process the important data information so that the model can quickly calculate more accurate results without bringing a large amount of calculation. Therefore, the attention mechanism has a relatively broad research value and application prospects.

[0005] After continuous training iterations, the attention mechanism can accurately notice the important and unimportant features in different input data, and each sample can calculate its own unique set of weights. Using the attention mechanism, according to the situation of each sample, the appropriate threshold can be automatically set for each sample, so that soft thresholding can be introduced into the field of deep learning. Through soft thresholding, unimportant information is set to zero and important information is retained to improve the accuracy of model prediction.

[0006] At present, the deep residual shrinkage network uses the SENet attention mechanism to set an adaptive threshold for each feature map channel. In the existing technology (paper: Zhao M, Zhong S, Fu X, et al. Deep residual shrinkage networks for fault diagnosis [J]. IEEE Transactions on Industrial Informatics, 2019, 16 (7): 4681-4690.), the soft thresholding method is introduced into the field of deep learning and has achieved good results. The deep residual shrinkage network is proposed to solve the noise problem in the task of "mechanical transmission system fault diagnosis based on vibration signals". The SENet attention mechanism used only focuses on channel information and does not focus on position information. For general visual tasks, the image contains less noise, and the model uses a combination of soft thresholding and attention mechanism. It is not meaningful. For some visual tasks with more noise, soft thresholding can be used to reduce the impact of noise. Position information is very important information in image data. Therefore, we need an attention mechanism that considers both channel and position information to replace SENet combined with the soft thresholding method, and design a new soft thresholding attention module to further reduce the impact of noise or redundant information. The deep residual shrinkage network integrates the designed module into the Resnet residual block, which cannot be conveniently applied to other places. Therefore, the designed soft threshold attention module also needs to meet the plug-and-play requirements and can be easily integrated into the existing neural network architecture. Summary of the invention

[0007] In view of the above problems, the present invention provides a soft threshold attention mechanism applied to computer vision. Drawing on the idea of ​​deep residual shrinkage network, the CoordinateAttention mechanism and soft threshold function that simultaneously consider channel and position information are integrated, and a plug-and-play soft threshold attention module is proposed, which can be better applied to detection tasks in the field of computer vision and can be easily integrated into the existing neural network architecture. By setting unimportant features to zero and retaining important features, the interference of noise or redundant information is reduced, and the ability of neural network to extract important features is enhanced, thereby achieving the purpose of improving network prediction accuracy.

[0008] The technical solution of the present invention to solve the technical problem is:

[0009] A soft threshold attention mechanism applied to computer vision, comprising the following steps:

[0010] (1) Obtain the absolute values ​​of all features in the input feature map and use them as input for the next step, expressed as:

[0011] X Abs =|X| (3)

[0012] Among them, X is the input feature map, X Abs The feature map after taking the absolute value of the input feature map;

[0013] (2) Use two spatial size pooling kernels (1, W) and (H, 1) to pool the feature map X in the vertical and horizontal directions respectively. Abs Pooling is performed to obtain two feature maps z after pooling in the vertical and horizontal directions h and z w , the output of the c-th channel with a height of H and the output of the c-th channel with a width of W are expressed as:

[0014]

[0015]

[0016] Among them, h means processing the feature map in the vertical direction, representing all rows of the feature map; w means processing the feature map in the horizontal direction, representing all columns of the feature map;

[0017] is the output of the cth channel with a height of H, W is the width of the feature map, x c (h,i) is the feature of the i-th column of the c-th channel of the feature map;

[0018] is the output of the cth channel with a width of W, H is the height of the feature map, xc (j,w) is the feature of the jth row of the cth channel of the feature map;

[0019] each It constitutes the feature map z h ,each It constitutes the feature map z w , the two feature maps z h and z w , the shapes are (B,C,H,1), (B,C,1,W), B is the batch_size, C is the number of channels, H is the height of the feature map, and W is the width of the feature map;

[0020] The feature map obtained by performing global average pooling on each channel of the feature map in the vertical and horizontal directions can well preserve the position information and make the extracted features more accurate.

[0021] (3) The z obtained in step (2) w Flip to (B,C,W,1), and then with z h The fused feature map is concatenated and fused along the third dimension, with a shape of (B, C, W + H, 1). After fusion, a shared 1×1 convolution operation F1 and h_swish activation function δ are used to process the fused feature map to obtain f∈R C / r×(H+W) , the specific shape is (B, C / r, W+H, 1), r is the downsampling parameter, which is used to control the parameter amount, and the formula for obtaining f is:

[0022] f=δ(F1([z h ,z w ])) (6)

[0023] (4) Split f along the third dimension into two tensors f h ∈R C / r×H and f w ∈R C / r×w , and f w Flip to (B, C / r, 1, W), and then use two convolution kernels with a size of 1×1. h and F w f h With f w Transform the original input channel number C to get the output, and use the Sigmoid function σ to normalize the output to between 0 and 1 to get the normalized output g h and g w , the shapes are (B,C,H,1) and (B,C,1,W) respectively:

[0024] g h =σ(F h (f h)) (7)

[0025] g w =σ(F w (f w )) (8)

[0026] (5) The two vertical and horizontal feature maps z obtained in step (2) are h and z w , respectively, with the normalized output g in step (4) h and g w Multiply them together to get two sets of thresholds τ h and τ w , respectively expressed as:

[0027] τ h =z h ·g h (9)

[0028] τ w =z w ·g w (10)

[0029] Among them, τ h is the threshold in the vertical direction, τ w is the threshold in the horizontal direction;

[0030] The threshold is calculated by taking the absolute value and multiplying the output of the global average pooling by a set of outputs between 0 and 1. This ensures that the threshold is positive and not too large.

[0031] (6) Using the thresholds τ in both directions obtained in step (5) h and τ w , perform soft thresholding on the input feature map X, and obtain the soft thresholding output y in the vertical and horizontal directions h and w , the specific soft thresholding process is as follows:

[0032] y h =sign(X)*max((|X|-τ h ), 0) (11)

[0033] y w =sign(X)*max((|X|-τ w ), 0) (12)

[0034] Among them, X is the feature map input to this module, max is the maximum value function, and sign is the sign function.

[0035] (7) The soft threshold output y in both vertical and horizontal directions obtained in step (6) is h and w Add it to the input feature map X in sequence to get the final output Y of this module:

[0036] Y=X+y h +y w (13)

[0037] The effects provided in the content of the invention are only the effects of the embodiments, not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0038] The soft threshold attention module designed by the present invention integrates the Coordinate Attention mechanism and the soft thresholding function: Steps 1 to 5 use the CA attention mechanism to automatically notice the important and unimportant feature information, and automatically determine two sets of thresholds for each channel of the feature map in the vertical and horizontal directions; Step 6 uses the principle of soft thresholding noise reduction, and performs soft thresholding processing in the vertical and horizontal directions of the feature map according to the determined two sets of thresholds to reduce the noise and redundant information in the feature map; Step 7 appends the results in the two directions to the input feature map in turn to complete the processing of the entire module. Compared with the combination of SENet and soft thresholding that only considers channel information, this module also considers position information, is more accurate in locating noise, and can more accurately reduce noise or redundant information, thereby improving the detection accuracy of the model in multi-noise visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the structure of the CA attention mechanism;

[0040] Figure 2 Schematic diagram of the structure of the soft threshold attention module of the present invention; DETAILED DESCRIPTION

[0041] In order to clearly illustrate the technical features of the present solution, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The disclosure below provides many different embodiments or examples for realizing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below. In addition, the present invention may repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed. It should be noted that the components illustrated in the accompanying drawings are not necessarily drawn to scale. The present invention omits the description of known components and processing techniques and processes to avoid unnecessary limitations on the present invention.

[0042] Explanation of prior art terms:

[0043] The expression of the soft thresholding function is as follows:

[0044]

[0045] Where x is the input, y is the output, and τ is the threshold. The input x with an absolute value lower than the threshold τ is set to 0, and the input x with an absolute value greater than the threshold τ is shrunk toward 0. The threshold τ is a pre-set parameter. Whether the threshold τ is appropriate has a direct impact on the effect of soft thresholding. Generally, the following conditions need to be met:

[0046] (1) Must be a positive number

[0047] (2) The maximum absolute value of the input data must be greater than the threshold, otherwise the output is all 0

[0048] It can be seen that soft thresholding can set the features in the range of [-τ,τ] to zero, and the value of the threshold τ is adaptively adjusted in the neural network. Therefore, soft thresholding is a more flexible way to delete redundant information.

[0049] The derivative of the soft thresholding output y with respect to the input x is as follows:

[0050]

[0051] From equations (1) and (2), we can see that soft thresholding is a nonlinear transformation, and the derivative of the output y with respect to the input x is 0 or 1, which is similar to the ReLU activation function. It can reduce the risk of gradient explosion and gradient disappearance to a certain extent, and is conducive to back propagation during model training. Introducing the soft threshold function into the field of deep learning to reduce the noise in the sample is a direction worth studying.

[0052] Coordinate Attention Mechanism:

[0053] The CA (Coordinate Attention) attention mechanism embeds position information into channel attention, which can extract long-dependent spatial position information without a large amount of calculation, and is lightweight and efficient. The CA attention mechanism is shown in the figure below. When calculating channel attention, it does not simply use global average pooling, but aggregates features in both horizontal and vertical directions. While calculating channel attention, it retains the long-dependency information of the entire space. Finally, the attention map in two directions is obtained, which can accurately locate the position of the area of ​​interest, thus embedding position information in the channel attention. Compared with the SENet attention mechanism, which uses global average pooling to only consider information between channels, the CA attention mechanism also considers position information, and can more accurately extract important feature information.

[0054] The present invention draws on the idea of ​​deep residual shrinkage network, uses CA attention mechanism (Coordinate Attention) that considers both channel and position information to replace SENet and soft thresholding method, and proposes a new soft thresholding attention module: the soft thresholding function in the traditional signal denoising algorithm is integrated with CA attention mechanism, and the CA attention mechanism is used to determine the thresholds required for two sets of soft thresholding functions in the vertical and horizontal directions, and then soft thresholding is performed separately and the input feature map is weighted. It is a mechanism to improve the accuracy of the model by reducing the influence of noise or redundant information in the input data; it can be better applied to detection tasks in the field of computer vision, and can be easily integrated into the existing neural network architecture, by setting unimportant features to zero and retaining important features, to reduce the interference of noise or redundant information, and enhance the ability of the neural network to extract important features, so as to achieve the purpose of improving the prediction accuracy of the network. The structure is as follows Figure 2 shown.

[0055] The soft threshold attention module is as follows: first, the CA attention mechanism is used to generate weights in the horizontal and vertical directions for the input data, and the generated weights are used to automatically set appropriate thresholds for the input data in the horizontal and vertical directions respectively, and then the input data is soft-thresholded in the two directions according to the two sets of thresholds, and finally the results in the horizontal and vertical directions obtained by the soft thresholding processing are sequentially appended to the original input data to obtain the final output.

[0056] In the soft threshold attention module, for the input feature map X = [x1, x2, ..., x c ]∈R C×H×W , calculate an output feature map Y = [y1, y2, ..., y c ]∈R C×H×W , where C, H and W are the number of channels, height and width of the feature map respectively.

[0057] The technical solution of the present invention to solve the technical problem is:

[0058] A soft threshold attention mechanism applied to computer vision, comprising the following steps:

[0059] (1) Obtain the absolute values ​​of all features in the input feature map and use them as input for the next step, expressed as:

[0060] X Abs =|X| (3)

[0061] Among them, X is the input feature map, X Abs The feature map after taking the absolute value of the input feature map;

[0062] (2) Use two spatial size pooling kernels (1, W) and (H, 1) to pool the feature map X in the vertical and horizontal directions respectively. Abs Pooling is performed to obtain two feature maps z after pooling in the vertical and horizontal directions h and z w , corresponding to Figure 2 In XGAP and YGAP, the output of the c-th channel with height H and width W are expressed as:

[0063]

[0064]

[0065] Among them, h means processing the feature map in the vertical direction, representing all rows of the feature map; w means processing the feature map in the horizontal direction, representing all columns of the feature map;

[0066] is the output of the cth channel with a height of H, W is the width of the feature map, x c (h,i) is the feature of the i-th column of the c-th channel of the feature map;

[0067] is the output of the cth channel with a width of W, H is the height of the feature map, x c (j,w) is the feature of the jth row of the cth channel of the feature map;

[0068] each It constitutes the feature map z h ,each It constitutes the feature map z w , the two feature maps z h and z w , the shapes are (B,C,H,1), (B,C,1,W), B is the batch_size, C is the number of channels, H is the height of the feature map, and W is the width of the feature map;

[0069] The feature map obtained by performing global average pooling on each channel of the feature map in the vertical and horizontal directions can well preserve the position information and make the extracted features more accurate.

[0070] (3) The z obtained in step (2) w Flip to (B,C,W,1), and then with z h The fused feature map is concatenated and fused along the third dimension, with a shape of (B, C, W + H, 1). After fusion, a shared 1×1 convolution operation F1 and h_swish activation function δ are used to process the fused feature map to obtain f∈RC / r×(H+W) , the specific shape is (B, C / r, W+H, 1), r is the downsampling parameter, which is used to control the parameter amount, and the formula for obtaining f is:

[0071] f=δ(F1([z h ,z w ])) (6)

[0072] (4) Split f along the third dimension into two tensors f h ∈R C / r×H and f w ∈R C / r×w , and f w Flip to (B, C / r, 1, W), and then use two convolution kernels with a size of 1×1. h and F w f h With f w Transform the original input channel number C to get the output, and use the Sigmoid function σ to normalize the output to between 0 and 1 to get the normalized output g h and g w , the shapes are (B,C,H,1) and (B,C,1,W) respectively:

[0073] g h =σ(F h (f h )) (7)

[0074] g w =σ(F w (f w )) (8)

[0075] (5) The two vertical and horizontal feature maps z obtained in step (2) are h and z w , respectively, with the normalized output g in step (4) h and g w Multiply them together to get two sets of thresholds τ h and τ w , respectively expressed as:

[0076] τ h =z h ·g h (9)

[0077] τ w =z w ·g w (10)

[0078] Among them, τ h is the threshold in the vertical direction, τ 3 is the threshold in the horizontal direction;

[0079] The threshold is calculated by taking the absolute value and multiplying the output of the global average pooling by a set of outputs between 0 and 1. This ensures that the threshold is positive and not too large.

[0080] (6) Using the thresholds τ in both directions obtained in step (5) h and τ w , perform soft thresholding on the input feature map X, and obtain the soft thresholding output y in the vertical and horizontal directions h and w , the specific soft thresholding process is as follows:

[0081] y h =sign(X)*max((|X|-τ h ), 0) (11)

[0082] y w =sign(X)*max((|X|-τ w ), 0) (12)

[0083] Among them, X is the feature map input to this module, max is the maximum value function, and sign is the sign function.

[0084] (7) The soft threshold output y in both vertical and horizontal directions obtained in step (6) is h and w Add it to the input feature map X in sequence to get the final output Y of this module:

[0085] Y=X+y h +y w (13)

[0086] Example:

[0087] Resnet50+FPN is used as the feature extraction network of Faster RCNN, and the soft threshold attention mechanism proposed in the present invention is added to both ends of Resnet50 respectively. The PASCAL VOC 2012 data set is used to train the model and evaluate its effect. Pascal VOC2012 is one of the benchmark data in the field of computer vision, which is widely used in experiments and model effect evaluation of tasks such as target detection and image segmentation. There are 20 categories in total, with 5717 pictures in the training set and 5823 pictures in the test set. The pre-trained weights of Resnet50+FPN are used for transfer learning, and the model is iterated 42870 times in the training set to obtain a converged target detection model, and then the test set is used for effect evaluation. The model with the CA attention mechanism has a mAP of 74.9% in the test set, and the model with the attention mechanism proposed in the present invention has a mAP of 75.6% in the test set. It can be seen from the experiment that the use of the soft threshold attention mechanism proposed in the present invention will improve the accuracy of model prediction in the detection tasks in the field of computer vision.

[0088] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A soft threshold attention mechanism for computer vision, comprising the following steps: (1) Obtain the absolute values ​​of all features in the input feature map and use them as input for the next step, expressed as: X Abs =|X| (3) Among them, X is the input feature map, X Abs The feature map after taking the absolute value of the input feature map; (2) Use two spatial size pooling kernels (1, W) and (H, 1) to pool the feature map X in the vertical and horizontal directions respectively. Abs Pooling is performed to obtain two feature maps z after pooling in the vertical and horizontal directions h and z w , the output of the c-th channel with a height of H and the output of the c-th channel with a width of W are expressed as: Among them, h means processing the feature map in the vertical direction, representing all rows of the feature map; w means processing the feature map in the horizontal direction, representing all columns of the feature map; is the output of the cth channel with a height of H, W is the width of the feature map, x c (h,i) is the feature of the i-th column of the c-th channel of the feature map; is the output of the cth channel with a width of W, H is the height of the feature map, x c (j,w) is the feature of the jth row of the cth channel of the feature map; each It constitutes the feature map z h ,each It constitutes the feature map z w , the two feature maps z h and z w , the shapes are (B,C,H,1), (B,C,1,W), B is the batch_size, C is the number of channels, H is the height of the feature map, and W is the width of the feature map; (3) The z obtained in step (2) w Flip to (B,C,W,1), and then with z h The fused feature map is concatenated and fused along the third dimension, with a shape of (B, C, W + H, 1). After fusion, a shared 1×1 convolution operation F1 and h_swish activation function δ are used to process the fused feature map to obtain f∈R C / r×(H+W) , the specific shape is (B,C / r,W+H,1), where r is the downsampling parameter used to control the parameter amount; The formula for obtaining f is: f=δ(F1([z h ,z w ])) (6) (4) Split f along the third dimension into two tensors f h ∈R C / r×H and f w ∈R C / r×w , and f w Flip to (B, C / r, 1, W), and then use two convolution kernels with a size of 1×1. h and F w f h With f w Transform the original input channel number C to get the output, and use the Sigmoid function σ to normalize the output to between 0 and 1 to get the normalized output g h and g w , the shapes are (B,C,H,1) and (B,C,1,W) respectively: g h =σ(F h (f h )) (7) g w =σ(F w (f w )) (8) (5) The two vertical and horizontal feature maps z obtained in step (2) are h and z w , respectively, with the normalized output g in step (4) h and g w Multiply them together to get two sets of thresholds τ h and τ w , respectively expressed as: τ h =from h ·g h (9) τ w =from w ·g w (10) Among them, τ h is the threshold in the vertical direction, τ w is the threshold in the horizontal direction; (6) Using the thresholds τ in both directions obtained in step (5) h and τ w , perform soft thresholding on the input feature map X, and obtain the soft thresholding output y in the vertical and horizontal directions h and w , the specific soft thresholding process is as follows: y h =sign(X)*max((|X|-τ h ),0) (11) y w =sign(X)*max((|X|-τ w ),0) (12) Among them, X is the feature map input to this module, max is the maximum value function, and sign is the sign function; (7) The soft threshold output y in both vertical and horizontal directions obtained in step (6) is h and w Add it to the input feature map X in sequence to get the final output Y of this module: Y=X+y h +y w (13) Among them, X is the input feature map, y h is the soft threshold output in the vertical direction, y w It is the soft thresholding output in the horizontal direction.

Images