An object recognition method based on spatial and channel weighted sum and dynamic asymmetric convolution

Abstract

This invention discloses an object recognition method based on spatial and channel weighting and dynamic asymmetric convolution. The method reconstructs the squared convolution using a proposed asymmetric convolution group, strengthening the intrinsic connection between local and global information. This allows the network to acquire features from three different convolutional regions, enhancing the model's learning ability. Specifically, the asymmetric convolution group reconstructs the backbone of the squared convolution, increasing the convolution's capture of local salient features. This local salient information helps the model better obtain the contextual semantic information of object features, improving the model's object recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the field of image recognition. Specifically, it relates to a convolutional neural network model based on spatial information analysis and a feature channel weighting method. Background Technology

[0002] Images are the visual foundation for human perception, allowing us to obtain crucial information from the outside world. Therefore, enabling machines to automatically perform image recognition and classification is of great significance. In the field of image processing, efficient feature extraction algorithms are a major research direction, and advancements in feature extraction technology contribute to improved image classification performance. Deep learning algorithms have made groundbreaking progress in image classification. By establishing and simulating the hierarchical structure of the human brain, they extract features from externally input data such as sound, images, and text, obtaining low-dimensional and high-dimensional feature representations, thus yielding features with stronger representational capabilities. Therefore, deep learning has broad application potential in image classification, especially convolutional neural networks (CNNs). CNNs are a widely used deep learning model, with the advantage of directly convolving image pixels to extract image features. Furthermore, the weight-sharing property and pooling layers of CNNs reduce the number of parameters to be trained, simplifying the network model and improving training efficiency. However, currently popular deep networks all share the same static inference paradigm: once training is complete, the network structure and parameters remain unchanged during the testing phase, performing the same feature extraction on different images. This, to some extent, limits the model's representational ability, inference efficiency, and interpretability.

[0003] Existing research focuses on improving the basic convolutional layers in convolutional neural networks, proposing different types of convolutional improvement methods, such as content-adaptive convolution and convolution reparameterization. These methods are characterized by increasing the complexity of convolutions to improve model performance without changing the network's depth. Convolution reparameterization, in particular, follows the additivity principle of convolutions, using complex convolutional structures for training and then equivalently transforming them into traditional convolutions during inference, thus enhancing the expressive power of traditional convolutions. Content-adaptive convolution adapts the convolution to the content of the input image. Its attention mechanism quickly scans the entire image to identify the convolution kernels that require focused attention, then allocates more attention resources to these kernels to capture more detailed information about the target while suppressing other irrelevant information.

[0004] However, both of the above techniques have certain limitations when used alone. These limitations are mainly reflected in the following: the principle of convolution reparameterization is to acquire more feature information at different convolutional scales in the samples during the training phase. This method increases the training time cost of the model, but the performance improvement is limited. Currently, content-adaptive convolution is only limited to assigning dynamic attributes to the convolutional kernel in multiple dimensions of the kernel space (number of kernels, space size, number of input channels, and number of output channels, etc.), without utilizing the correlation between the overall and local aspects of the convolutional kernel. Ultimately, this prevents the model from fully utilizing the local information of feature points. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes an object recognition method based on spatial and channel weighting and dynamic asymmetric convolution.

[0006] This invention improves the ResNet network structure. The module consists of an asymmetric convolutional group (containing a 3x1 convolution and a 1x3 convolution) and a square convolution. These convolutional kernels are aggregated through a dual attention mechanism to generate new convolutional kernels. This method of generating convolutions endows the convolutional kernels with dynamic properties, allowing for different feature extraction methods for different input images. With these dynamic properties, the network can break free from the limitations of static inference paradigms and improve its representational capabilities. The core of the object recognition method based on spatial and channel weighted dynamic asymmetric convolution is to perform convolution operations on the image using asymmetric convolutional kernel groups and square convolutions in parallel. This encodes local salient features in the image from a global perspective, thereby extracting more effective contextual semantic information from the feature map. Addressing the two information dimensions of spatial information and feature channels, this method uses a spatial attention enhancement module to capture spatially salient information in the feature map. The module generates channel attention by integrating relevant features from all channel mappings, thereby selectively emphasizing interdependent channel mappings.

[0007] The method of the present invention specifically includes the following steps:

[0008] Step 1: Input the image and preprocess it to obtain the preprocessed image I;

[0009] Step 2: Obtain new feature maps by improving the ResNet network structure

[0010] Step 2-1: In the improved ResNet network structure, the convolutional layers use global average pooling layers to compress the global spatial information of image I, and then pass through a fully connected layer and the ReLU activation function to obtain the initial attention information G.

[0011] Step 2-2: Pass the attention information G through the channel fully connected layer and the spatial fully connected layer respectively, and calculate the spatial and channel attention weights;

[0012] Steps 2-3: Spatial attention aggregates squared convolutions and asymmetric convolution groups;

[0013] First, zero-padding is used to convert the convolutions in the asymmetric convolution group into equivalent 3x3 convolutions. Then, the three convolution kernels are weighted with their corresponding spatial attention weights. Finally, these weights are summed to obtain the new convolution kernel parameters. If bias is required, set three bias values ​​corresponding to a squared convolution, a 3x1 convolution, and a 1x3 convolution, respectively; these bias values ​​are then aggregated with the spatial attention parameters of their corresponding convolutions to obtain new bias values.

[0014] Steps 2-4: Channel attention is applied channel-by-channel to the initial image I via multiplication to obtain a new feature map.

[0015] Step 3: Feature Extraction

[0016] convolution kernel parameters and bias value Forming a new convolutional layer L, feature map Feature extraction is performed through convolutional layer L to obtain the final feature output.

[0017] Preferably, the attention decoding layer described in step 1 improves the attention generation method based on the traditional SENet module, and consists of three fully connected layers, specifically:

[0018] The original SENet module calculation formula is as follows:

[0019]

[0020] F sq For global average pooling convolution, I is the input image, F ex Given a fully connected layer and a ReLU activation function, This is channel attention; furthermore, the dynamic asymmetric convolution method uses two fully connected layers to generate two types of attention, specifically defined as follows:

[0021] G = F ex (F sq (I)) (2)

[0022]

[0023] π=f(f k (G) / t) (4)

[0024] f c and f kThese are channel and spatial fully connected layers, respectively. f represents the Softmax activation function, π = {π1, π2, π3} is the spatial attention, and t represents the temperature in the annealing mechanism.

[0025] As a preferred approach, the output of spatial attention is normalized using the Softmax function when calculating spatial attention weights to promote the learning of spatial attention; an annealing mechanism is used for spatial attention to suppress the one-hot output of the Softmax function, ensuring that spatial attention is relatively average in the early stages of training, thereby promoting the learning of asymmetric convolution groups and squared convolutions.

[0026] Preferably, the spatial attention aggregates squared convolutions and asymmetric convolution groups, where the convolution sizes in the asymmetric convolution group are (3, 1, C) and (1, 3, C), and the size of the squared convolution is (3, 3, C), where C represents the number of channels. The 3x1 and 1x3 convolution kernels are identically mapped and merged into a 3x3 convolution stack through reconstruction. First, the 3x1 and 1x3 convolutions are equivalently transformed into 3x3 convolutions using zero-padding. Using formula (5), the three convolution kernels are multiplied by the spatial attention and then superimposed to obtain a new 3x3 convolution kernel, as follows:

[0027]

[0028]

[0029] Among them, in formula (5) W represents the addition of nuclear elements. k These represent the parameter matrices of the three convolution kernels, π. k This represents the spatial attention weights corresponding to the convolution kernel. This represents the weights of the convolution kernel in the aggregation.

[0030] Preferably, the aggregation obtains a new bias value, specifically using the following calculation formula:

[0031]

[0032] Among them, b k For the k-th bias value, The bias parameter represents the aggregation.

[0033] As a preferred embodiment, the channel attention is obtained by multiplying and weighting each channel to the initial image I to obtain a new feature map. Its implementation model can make full use of the relevant information between different channels, and the calculation formula is as follows:

[0034]

[0035] Among them, Fscale This is a matrix multiplication operation. For channel attention.

[0036] Preferably, the feature extraction specifically involves: the asymmetric convolution group obtaining local contextual semantic information of the feature map by mimicking the backbone of the squared convolution, and using this local contextual semantic information to enhance the global contextual semantic information of the feature map; the module uses spatial attention and channel attention to enhance the influence factors of significant features in the spatial and channel dimensions of the feature map; the specific formula is as follows:

[0037]

[0038] in For feature maps enhanced by channel attention, The 3x3 convolution is obtained by linearly stacking three convolutions based on spatial attention. The bias parameters are obtained by linearly aggregating the bias parameters corresponding to the three convolutions in the same way; Out represents the output feature, and Conv(·) represents the convolution calculation.

[0039] Preferably, the annealing mechanism is specifically implemented as follows:

[0040] (1) Set the initial temperature t = 31;

[0041] (2) After the model completes one epoch, reduce the temperature t by 3;

[0042] (3) Repeat (2) until the temperature t becomes 1.

[0043] The beneficial effects of this invention are:

[0044] 1. This invention reconstructs the squared convolution using a proposed asymmetric convolution group, strengthening the intrinsic connection between local and global information within the convolution. This allows the network to acquire features from three different convolutional regions, enhancing the model's learning ability. Specifically, the asymmetric convolution group reconstructs the backbone of the squared convolution, increasing the convolution's capture of local salient features. This local salient information helps the model better obtain contextual semantic information about object features, improving the model's accuracy in object recognition.

[0045] 2. This invention, through dual-attention adaptive weighting, enables the model to not only extract more accurate salient features of the target object in terms of spatial information, but also to fully utilize the relevant information between different channels. Therefore, the dual-attention mechanism can effectively solve the problem that convolutional neural networks cannot fully utilize and extract the main features of the target object. Attached Figure Description

[0046] Figure 1The diagram shows the structure of three ResNet network models with different depths.

[0047] Figure 2 This is a diagram of the ResNet10 architecture using dynamic asymmetric convolution.

[0048] Figure 3 This is a diagram of a dynamic asymmetric convolution structure.

[0049] Figure 4 This is a flowchart of the dual attention generation and channel attention mechanism.

[0050] Figure 5 This is a flowchart of the spatial attention mechanism. Detailed Implementation

[0051] To make the technical solution and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0052] Step 1: Dual-attention decoding layer. The dual-attention mechanism flow is as follows, and the structure is attached. Figure 4

[0053] Step 1-1: As attached Figure 4 The global spatial information X∈R of the input image I is acquired through global average pooling. 2x2xC Then, the global spatial information X is passed through a fully connected layer to obtain preliminary attention information G∈R. 1x1x(C / r) , where r is the channel reduction factor. The channel reduction factor r can reduce the number of channels in the attention information G, thereby reducing the computational cost of generating attention.

[0054] Steps 1-2: The attention information G mentioned above is passed through the channel fully connected layer and the spatial fully connected layer respectively to calculate the channel attention. and spatial attention π∈R 1x1x3 The Softmax function normalizes the output of spatial attention to facilitate its learning. During the initial training phase, the module employs an annealing mechanism to suppress the one-hot output of the Softmax function, ensuring that spatial attention is relatively evenly distributed during the early stages of training, thus promoting kernel learning. The annealing mechanism is specifically implemented as follows:

[0055] (1) Set the initial temperature t = 31;

[0056] (2) After the model completes one epoch, reduce the temperature t by 3;

[0057] (3) Repeat (2) until the temperature t becomes 1.

[0058] The attention decoding layer improves upon the traditional SENet module's attention generation method and comprises three fully connected layers. The specific definition of attention generation is as follows:

[0059] G=F ex (F sq (I)) (1)

[0060]

[0061] π=f(f k (G) / t) (3)

[0062] Among them, F sq Global average pooling convolution, where I is the input image and F... ex Given a fully connected layer and a ReLU activation function, f c and f k These are channel and spatial fully connected layers, respectively. f represents the Softmax activation function, π = {π1, π2, π3} is the spatial attention, and t represents the temperature in the annealing mechanism.

[0063] Step 2: Application of Dual Attention

[0064] Step 2-1: Spatial attention aggregates squared convolutions and asymmetric convolution groups. First, zero-padding is used to convert the convolutions in the asymmetric convolution group into equivalent 3x3 convolutions. Then, each of the three convolution kernels is multiplied by its corresponding spatial attention weight. Finally, these are summed to obtain the new convolution kernel parameters.

[0065] Spatial attention reconstructs the convolution parameters as shown in the attached figure. Figure 5 First, the 3x1 and 1x3 convolutional kernels are equivalently converted to 3x3 convolutions using zero-padding. The three kernels are then multiplied by the spatial attention according to formula (4) and stacked to form a 3x3 convolution, ultimately resulting in a new 3x3 convolutional kernel. (See attached...) Figure 5 Asymmetric convolutional groups reconstruct squared convolutions by mimicking the backbone of the squared convolution. This approach strengthens the intrinsic connection between local and global information within the convolution, enhancing the network's ability to acquire local context for feature points and thus improving the model's learning capacity. Specifically, asymmetric convolutional blocks increase the influence of local salient features and selectively focus on important regions within the convolution kernel through spatial attention. This local salient information allows for better acquisition of object feature information, improving the model's accuracy in object recognition. Details are as follows:

[0066]

[0067]

[0068] Among them, in formula (4) W represents the addition of corresponding kernel elements. k These represent the parameter matrices of the three convolution kernels, π. k This represents the spatial attention weights corresponding to the convolution kernel. This represents the weights of the convolution kernel in the aggregation.

[0069] Step 2-2: If the convolutional layer needs to use bias values, then three bias values ​​are set in the dynamic asymmetric convolution, and these bias values ​​correspond one-to-one with the squared convolutions and the convolutions in the asymmetric convolution group in the module. These bias values ​​are aggregated with spatial attention according to formula (6) to obtain new bias values.

[0070]

[0071] Among them, b k For the k-th bias value, The bias parameter represents the aggregation.

[0072] Steps 2-3: Perform matrix multiplication between the channel attention and the initial input image I to obtain a new feature map. According to the appendix Figure 4 As shown, the channel attention is applied to the previous input image I through multiplication according to formula (7) to obtain a new feature map. Channel attention integrates relevant features from all channel maps, selectively emphasizing interdependent channel maps. It leverages the correlation between different channels by enhancing or suppressing information in feature channels, influencing interdependent asymmetric convolutional groups and squared convolutions. The calculation formula is as follows:

[0073]

[0074] Among them, F scale This is a matrix multiplication operation.

[0075] Step 3: Feature Extraction

[0076] According to the above convolution kernel parameters in formula (8) and bias value Forming a new convolutional layer L, feature map Feature extraction is performed through convolutional layer L to obtain the final feature output.

[0077]

[0078] in For feature maps enhanced by channel attention, This is a 3x3 convolution obtained by linearly superimposing asymmetric convolution groups and squared convolutions based on spatial attention. The bias parameters are obtained by linearly aggregating the bias parameters corresponding to the three convolutions in the same way. Out represents the output feature, and Conv(·) represents the convolution calculation.

[0079] According to the appendix Figure 3 The model achieves accurate extraction of local salient information of the target object. Asymmetric convolutional groups obtain local contextual semantic information of the feature map by mimicking the backbone of squared convolution, and use this local contextual semantic information to enhance the global contextual semantic information of the feature map. Spatial attention and channel attention not only help the module extract more accurate spatial salient features of the target object, but also fully utilize the relevant information between different channels.

[0080] According to the appendix Figure 1 We present a detailed usage example of the dynamic asymmetric convolution module. This module can easily replace the standard 3x3 convolutional layer, thereby improving the object recognition performance of the ResNet network. (See attached...) Figure 1 The ResNet10 architecture list is attached, and we can build it as shown in the appendix. Figure 2 The network structure shown.

[0081] Based on Table 1, on the ImageNet dataset, we demonstrate that the dynamic asymmetric convolution module improves the object recognition performance of the ResNet network. Specifically, the DAC module achieves Top-1 accuracies of 66.678%, 72.498%, and 74.784%, respectively, representing improvements of 4.214%, 2.5%, and 1.49% compared to the basic ResNet network, and achieving the best accuracy. Dynamic asymmetric convolution, through the dynamic reconstruction of convolution parameters, increases the computational cost of the model to a limited extent, but significantly improves model performance.

[0082]

[0083] Table 1

Claims

1. A method for object recognition using spatial and channel-weighted dynamic asymmetric convolution, characterized in that, The method specifically includes the following steps: Step 1: Input the image and preprocess it to obtain the preprocessed image I; Step 2: Obtain new feature maps by improving the ResNet network structure ; Step 2-1: In the improved ResNet network structure, the convolutional layers use global average pooling layers to compress the global spatial information of image I, and then pass through a fully connected layer and the ReLU activation function to obtain the initial attention information G. Step 2-2: Pass the attention information G through the channel fully connected layer and the spatial fully connected layer respectively, and calculate the spatial and channel attention weights; Steps 2-3: Spatial attention aggregates squared convolutions and asymmetric convolution groups; First, zero-padding is used to convert the convolutions in the asymmetric convolution group into equivalent 3x3 convolutions. Then, the three convolution kernels are weighted with their corresponding spatial attention weights. Finally, they are summed to obtain the new convolution kernel parameters. Three bias values ​​are set, corresponding to a squared convolution, a 3 x 1 convolution, and a 1 x 3 convolution, respectively. These bias values ​​are then aggregated with the spatial attention parameters of their corresponding convolutions to obtain new bias values. ; Steps 2-4: Channel attention is applied channel-by-channel to the initial image I via multiplication to obtain a new feature map. ; Step 3: Feature Extraction convolution kernel parameters and bias value Forming a new convolutional layer L, feature map Feature extraction is performed through convolutional layer L to obtain the final feature output.

2. The object recognition method based on spatial and channel weighted summaries and dynamic asymmetric convolution according to claim 1, characterized in that: The attention information generation method in step 2 improves upon the traditional SENet module, comprising three fully connected layers: Two fully connected layers are used to generate two types of attention, as defined below: F sq For global average pooling convolution, I is the input image, F ex Given a fully connected layer and a ReLU activation function, For channel attention; and These are the channel and spatial fully connected layers, respectively. f represents the Softmax activation function, π={π1, π2, π3} are the spatial attention weights, and t represents the temperature in the annealing mechanism.

3. The object recognition method based on spatial and channel weighted summaries and dynamic asymmetric convolution according to claim 1, characterized in that: When calculating the spatial attention weights, the output of spatial attention is normalized using the Softmax function to facilitate the learning of spatial attention. An annealing mechanism is used for spatial attention to suppress the one-hot output of the Softmax function, ensuring that spatial attention is relatively average in the early stage of training, so as to promote the learning of asymmetric convolution groups and squared convolutions.

4. The object recognition method based on spatial and channel weighted summaries and dynamic asymmetric convolution according to claim 1, characterized in that: The spatial attention mechanism aggregates squared convolutions and asymmetric convolution groups, where the convolution sizes in the asymmetric convolution group are (3, 1, C) and (1, 3, C), and the size of the squared convolution is (3, 3, C), where C represents the number of channels. The 3 x 1 and 1 x 3 convolution kernels are identically mapped and merged into a 3 x 3 convolution stack through reconstruction. First, the 3 x 1 and 1 x 3 convolutions are equivalently transformed into 3 x 3 convolutions using zero-padding. The three convolution kernels are multiplied by the spatial attention and then stacked to obtain a new 3 x 3 convolution kernel, as detailed below: Where ⨁ represents the addition of nuclear elements, W k These represent the parameter matrices of the three convolution kernels, π. k This represents the spatial attention weights corresponding to the convolution kernel. This represents the weights of the convolution kernel used for aggregation.

5. The object recognition method based on spatial and channel weighted and dynamic asymmetric convolution according to claim 1, characterized in that: The aggregation process yields a new bias value, specifically calculated using the following formula: Among them, b k For the k-th bias value, The bias parameter represents the aggregation.

6. The object recognition method based on spatial and channel weighted summaries and dynamic asymmetric convolution according to claim 1, characterized in that: The channel attention is obtained by multiplying and weighting each channel to the initial image I to obtain a new feature map. Its implementation model can fully utilize the relevant information between different channels, and the calculation formula is as follows: in, This is a matrix multiplication operation. For channel attention.

7. The object recognition method based on spatial and channel weighted summaries and dynamic asymmetric convolution according to claim 1, characterized in that: The feature extraction is specifically as follows: the asymmetric convolution group obtains the local contextual semantic information of the feature map by mimicking the backbone of the squared convolution, and uses the local contextual semantic information to enhance the global contextual semantic information of the feature map; the module uses spatial attention and channel attention to enhance the influence factors of the salient features of the feature map in the spatial and channel dimensions. The specific formula is as follows: in For feature maps enhanced by channel attention, The 3x3 convolution is obtained by linearly stacking three convolutions based on spatial attention. The bias parameters are obtained by linearly aggregating the bias parameters corresponding to the three convolutions in the same way; Out represents the output feature. This indicates convolution calculation.

8. The object recognition method based on spatial and channel weighted and dynamic asymmetric convolution according to claim 3, characterized in that: The annealing mechanism is specifically implemented as follows: (1) Set the initial temperature t=31; (2) After the model completes one epoch, reduce the temperature t by 3; (3) Repeat (2) until the temperature t becomes 1.

Images