A selective convolution method, device and medium based on attention mechanism

Abstract

The present invention provides a selective convolution method based on an attention mechanism, comprising the steps of: S1, performing a first convolution on an input image to generate a feature map; S2, sending the number obtained by global pooling the feature map to a fully connected layer to obtain the importance W of the N channels of the feature map; S3, dividing the feature map into g groups in descending order according to the importance of the feature map, and arranging them in descending order according to the number of feature layers contained in each group to obtain an importance queue; performing convolution on the grouped feature map to obtain a fourth feature map set Y; S4, performing a linear transformation on the fourth feature map set to obtain a third feature map, and using the third feature map for feature recognition. The present invention combines the attention mechanism and designs a selective convolution method based on the attention mechanism based on the principles of "the capable do more work" and "the incompetent can also do something". The selective convolution of the present invention adopts the strategy of grouped convolution, and the amount of calculation and parameters is greatly reduced compared to ordinary convolution.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a selective convolution method, device and medium based on an attention mechanism. Background Art

[0002] The amount of information in an image is huge and scattered, which is not conducive to obtaining the key information in the image. Therefore, it is necessary to process the image to obtain the required key features. Convolution has a strong feature extraction capability, and the features of most images can be extracted through convolution. The features of the image are extracted through the convolution kernel, and then the convolution kernel is continuously fitted by back propagation to enable it to have the ability to extract the required features. At present, the optimization methods for convolution are as follows: grouped convolution, depth-separable convolution, shuffle convolution, etc., but they all have problems to varying degrees in some aspects.

[0003] 1. Ordinary convolution has a large amount of computation, which is equal to the number of channels of the input feature map × the number of channels of the output feature map × the length of the convolution kernel × the width of the convolution kernel × the length of the output feature map × the width of the output feature map, which is six-dimensional. The feature map generated by ordinary convolution may have information redundancy, that is, the features between several channels are very similar, and redundant features will cause additional computational overhead. Ordinary convolution processes the features of all channels in the same way, does not consider the importance of channels, and does not give priority to important channels.

[0004] 2. The channel-level attention mechanism considers the importance of different channels. All features on each channel are multiplied by the corresponding importance of each channel obtained by full connection. However, multiplying more important channels by larger numbers does not necessarily improve the integrity of the network, because the numbers in each feature map are abstract features. In some cases, the smaller the value in the feature map, the lower the loss function.

[0005] 3. Grouped convolution can reduce the number of convolution parameters to one-third of the number of groups of ordinary convolution parameters. Although it can reduce a lot of parameters, it hinders the information exchange between groups and is not conducive to the transmission of information between groups.

[0006] 4. ShuffleNet adopts the idea of ​​channel shuffle, which can exchange feature maps in each group of group convolution to exchange information between groups, alleviating the problem that each group of group convolution cannot transmit information. However, its shuffle method is fixed rather than random, and its shuffle method cannot be learned by the network to determine a better shuffle method.

[0007] 5. The convolution in GhostNet is divided into two steps, namely ordinary convolution and linear transformation. The ordinary convolution in the first step has the common problems of ordinary convolution, namely, large amount of calculation, information redundancy, and indiscriminate treatment of each channel.

[0008] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims of this application and are not admitted to be prior art by inclusion in this section. Summary of the invention

[0009] In view of the above technical problems in the related art, the present invention proposes a selective convolution method based on an attention mechanism, which comprises the following steps:

[0010] S1, obtaining an input image, and performing a first convolution on the input image to generate a feature map;

[0011] S2, the number obtained by performing global pooling on the feature map is sent to the fully connected layer to obtain the importance W of the N channels of the feature map;

[0012] S3, performing convolution on the feature map, wherein for each channel whose importance is greater than a threshold, the output channel is Wi*N, and a first feature map is obtained, wherein Wi represents the importance of the i-th channel; for channels whose importance is less than the threshold, they are grouped so that their output channels are 1, and a second feature map is obtained;

[0013] S4, linearly transform the first feature map and the second feature map to obtain a third feature map, and use the third feature map to perform feature recognition.

[0014] Specifically, step S3 also includes: grouping the channels whose importance is less than a threshold, and when the importance of the group is not 1, adjusting the threshold so that the sum of the weights of the least important first k channels is less than or equal to 1 / n, and the sum of the weights of the least important first k+1 channels is greater than 1 / n.

[0015] In a second aspect, another embodiment of the present invention discloses a selective convolution method based on an attention mechanism, which comprises the following steps:

[0016] S1, obtaining an input image, and performing a first convolution on the input image to generate a feature map;

[0017] S2, the number obtained by performing global pooling on the feature map is sent to the fully connected layer to obtain the importance W of the N channels of the feature map;

[0018] S3, according to the descending order of the importance of the feature graph, the feature graph is divided into g groups, namely X1, ..., Xg, the importance of each group is the sum of the importance of each layer in the group, and the importance queues G1, ...Gg are obtained by arranging them in descending order according to the number of feature layers contained in each group, wherein G1>...>Gg; convolving the grouped feature graphs to obtain a fourth feature graph set Y, wherein the output channel of each feature graph in the fourth feature graph set Y is output according to the importance queue;

[0019] S4, linearly transform the fourth feature map set to obtain a third feature map, and use the third feature map to perform feature recognition.

[0020] Specifically, the g numbers satisfy a geometric progression with a first term of 1 and a common ratio of 2; the gth number among the g numbers may be greater than 2. g -1.

[0021] Specifically, step S3 is as follows: the feature maps are divided into g groups, the first n / g feature maps with the highest importance are divided into the first group according to W, the first n / g feature maps with the highest importance in feature maps other than the first group of feature maps in X are divided into the second group, and so on, until X is divided into g groups, and the importance of each group is the sum of the importance of each layer in the group.

[0022] Specifically, the fourth feature atlas Y is linearly transformed s times to obtain a fifth feature atlas Z; the feature map and the fifth feature atlas are concatenated in channel dimension to obtain a third feature map.

[0023] In a third aspect, another embodiment of the present invention discloses a selective convolution device based on an attention mechanism, which includes the following units:

[0024] A first convolution unit, used to obtain an input image, and perform a first convolution on the input image to generate a feature map;

[0025] An importance acquisition unit, used for performing global pooling on the feature map and sending the obtained number to a fully connected layer to obtain the importance W of N channels of the feature map;

[0026] A second grouping convolution unit is used to divide the feature map into g groups, namely X1, ..., Xg, in descending order according to the importance of the feature map, and the importance of each group is the sum of the importance of each layer in the group, and the importance queues G1, ...Gg are obtained by arranging them in descending order according to the number of feature layers contained in each group, wherein G1>...>Gg; convolve the grouped feature maps to obtain a fourth feature map set Y, wherein the output channel of each feature map in the fourth feature map set Y is output according to the importance queue;

[0027] The feature map generating unit is used to perform a linear transformation on the fourth feature map set to obtain a third feature map, and use the third feature map to perform feature recognition.

[0028] Specifically, the g numbers satisfy a geometric progression with a first term of 1 and a common ratio of 2; the gth number among the g numbers may be greater than 2. g -1.

[0029] Specifically, the second grouping convolution unit is used to divide the feature map into g groups, divide the first n / g feature maps with the highest importance into the first group according to W, divide the first n / g feature maps with the highest importance in the feature maps other than the first group of feature maps in X into the second group, and so on, until X is divided into g groups, and the importance of each group is the sum of the importance of each layer in the group.

[0030] Specifically, the fourth feature atlas Y is linearly transformed s times to obtain a fifth feature atlas Z; the feature map and the fifth feature atlas are concatenated in channel dimension to obtain a third feature map.

[0031] Another embodiment of the present invention discloses a non-volatile memory, which stores instructions. When the instructions are executed by a processor, they are used to implement the above-mentioned selective convolution method based on the attention mechanism.

[0032] The present invention combines the attention mechanism and designs a selective convolution method based on the attention mechanism based on the principles of "the capable should work harder" and "the incapable should also do something". The selective convolution of the present invention adopts the strategy of grouped convolution, and the amount of calculation and parameters are greatly reduced compared with ordinary convolution. The information in the feature map is rich and even redundant, and the redundant information can ensure a comprehensive understanding of the input data. The information between several channels with information redundancy is highly similar, and the similar one can be generated by the other by linear transformation, such as convolution with a convolution kernel of 3×3×1 or 5×5×1. Linear transformation is used in the second step of the selective convolution, and redundant information is used to some extent to ensure a more comprehensive understanding of the data, while also reducing the amount of parameters. The first step of the selective convolution also considers the importance of each channel, but the processing method is different from the attention mechanism, but allows important channels to convolve more features. The more important the channel, the more features it convolutes to reflect its own importance, and also reflects the principle of "the capable should work harder". The more important the channel, the more contribution it makes to this convolution. In view of the problem that grouped convolution does not exchange information between groups, the grouping method of each convolution of the selective convolution is different. It groups the data based on the importance of each channel, and the importance ranking of each feature map is different, which leads to different grouping methods for each selection convolution, greatly promoting information exchange between groups. The selection convolution uses the linear transformation operation in GhostNet, retaining some redundant information to ensure a more comprehensive understanding of the input data, while reducing the number of parameters for the entire convolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 Schematic diagram of a selective convolution method based on an attention mechanism provided by an embodiment of the present invention;

[0035] Figure 2 Schematic diagram of another selective convolution method based on attention mechanism provided by an embodiment of the present invention;

[0036] Figure 3 Schematic diagram of a selective convolution device based on an attention mechanism provided by an embodiment of the present invention;

[0037] Figure 4 is a schematic diagram of another selective convolution device based on the attention mechanism provided by an embodiment of the present invention;

[0038] Figure 5It is a schematic diagram of a selective convolution device based on an attention mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0040] Embodiment 1

[0041] refer to Figure 1 , this embodiment discloses a selective convolution method based on an attention mechanism, which includes the following steps:

[0042] S1, obtaining an input image, and performing a first convolution on the input image to generate a feature map;

[0043] The input image of this embodiment is an image to be recognized, and a first convolution is performed on the input image to generate a feature map, wherein the first

[0044] S2, the number obtained by performing global pooling on the feature map is sent to the fully connected layer to obtain the importance W of the N channels of the feature map;

[0045] The selected convolution of this embodiment is divided into two steps. The first step is the group convolution combined with the attention mechanism, and the second step is the linear transformation operation. The implementation process of the first step of the selected convolution is introduced below.

[0046] Specifically, this embodiment first performs a first convolution on an input image to generate a feature map X. It is assumed that the height, width and number of channels of the feature map X to be convolved are represented by height, width and n respectively.

[0047] The n channels of feature map X are globally pooled and sent to the fully connected layer to get the importance of these n channels (w1, w2, w3, ..., wn), represented by W. The larger the value, the more important it is. We give a threshold α, and channels below the threshold α are considered unimportant channels, and channels above the threshold α are considered important channels.

[0048] The channel-level attention mechanism SE multiplies all numbers on each channel by this importance to reflect the importance of each channel. However, multiplying the more important channels by larger numbers does not necessarily improve the overall performance of the network.

[0049] The numbers in each feature map are abstract features. In some cases, the smaller the value in the feature map, the more likely it is that the loss function will decrease. We perform convolution on important channels, so that more important channels can be convolved with more layers of feature maps to reflect their own importance, which reflects the principle of more work for those who are more capable.

[0050] Multiplying important channels by a larger number may not necessarily reflect their importance, but allowing important channels to convolve more features can reflect their importance.

[0051] Assume that the number of channels of the intermediate feature map generated by the first step of convolution is n, because the sum of (w1,w2,w3,.....,wn) is 1, so n×w i It represents the number of channels of the feature map output after convolution of the i-th channel in X. The larger the wi, the more feature maps are convolved out of the channel.

[0052] S3, performing convolution on the feature map, wherein for each channel whose importance is greater than a threshold, the output channel is Wi*N, and a first feature map is obtained, wherein Wi represents the importance of the i-th channel; for channels whose importance is less than the threshold, they are grouped so that their output channels are 1, and a second feature map is obtained;

[0053] Specifically, step S3 also includes: grouping the channels whose importance is less than a threshold, and when the importance of the group is not 1, adjusting the threshold so that the sum of the weights of the least important first k channels is less than or equal to 1 / n, and the sum of the weights of the least important first k+1 channels is greater than 1 / n.

[0054] But when n is not large enough or wi is very small, n×w i Less than 1, resulting in the number of feature map channels convolved from some unimportant channels below the threshold α being less than 1, that is, these channels will not contribute to the convolution. We follow the principle of "incompetence is also contribution" to allow the least important channels to contribute to the convolution. Therefore, we pack the unimportant channels below the threshold into a group, and add the corresponding weights of each channel to form the weight of the group. However, there is still a situation where the sum of the weights of the packed group multiplied by n is still less than 1, which indicates that the threshold setting is unreasonable.

[0055] We reset the threshold ɑ so that the sum of the weights of the least important first k is less than or equal to 1 / n, and the sum of the weights of the least important first k+1 is greater than 1 / n. This ensures that after all the least important channels are packaged into a group, one layer of feature maps can still be output, which reflects our principle of "incompetence is also action", allowing the least important layers of feature maps to contribute to the convolution.

[0056] S4, linearly transform the first feature map and the second feature map to obtain a third feature map, and use the third feature map to perform feature recognition.

[0057] The second step of the selected convolution uses the linear transformation operation in GhostNet, which retains some redundant information to ensure a more comprehensive understanding of the input data, while reducing the number of parameters of the entire convolution. There may be information redundancy in an ordinary feature map, that is, the feature map features between several channels may be highly similar. Using convolution kernels to generate repeated features will introduce a lot of parameters that are not necessary. Repeated features can be generated by linear transformation of one of the features that are repeated with it. The cost of linear transformation is very low, and the number of parameters required is very small. Linear transformation can be implemented by a single-layer convolution without normalization and activation function. Assume that the m-dimensional feature map generated by the first step of the selected convolution is Y, whose dimensions are width1×height1×m. Y is linearly transformed s times to obtain the final Z. The linear transformation process is shown in the following formula:

[0058]

[0059] yi is the feature map of the i-th layer in Y, f i,j represents the jth linear transformation of yi, s is the multiple of the number of convolution Z channels relative to the number of Y channels, z i,j is the feature map obtained by performing the j-th linear transformation on yi, in particular, f i,s It is an identity mapping used to preserve the original feature map. The feature map of the m layer generated by the first step of the selective convolution is concatenated with the feature map of the m×(s-1) layer generated by the second step of the selective convolution to obtain the final feature map of the selective convolution.

[0060] This embodiment combines the attention mechanism to propose a selective convolution method based on the attention mechanism. (I) In view of the problem of large number of parameters of ordinary convolution, the selective convolution adopts the strategy of grouped convolution, and the amount of calculation and parameters are greatly reduced compared with ordinary convolution. The information in the feature map is rich and even redundant, and the redundant information can ensure a comprehensive understanding of the input data. The information between several channels with information redundancy is highly similar, and the similar one can be generated by the other through linear transformation, such as convolution with a convolution kernel of 3×3×1 or 5×5×1. Linear transformation is used in the second step of the selective convolution, which to some extent utilizes redundant information to ensure a more comprehensive understanding of the data, while also reducing the number of parameters.

[0061] (ii) Improved the traditional channel-level attention mechanism. The traditional attention mechanism between channels generates the importance of each channel through the fully connected layer, and all numbers in each channel are multiplied by this importance to reflect their own importance. However, the larger the number multiplied by all the numbers in each layer of the feature map, the less likely it is to promote the final network performance. The first step of convolution selection also considers the importance of each channel, but the processing method is different from the attention mechanism. Instead, it allows important channels to convolve more features. The more important the channel, the more features it will convolve to reflect its own importance, and it also reflects the principle of "the more capable, the more work". The more important the channel, the more contribution it makes to this convolution.

[0062] (III) The convolution is performed in two steps, where the second step uses the linear transformation operation in GhostNet, retaining some redundant information to ensure a more comprehensive understanding of the input data while reducing the number of parameters of the entire convolution.

[0063] Embodiment 2

[0064] refer to Figure 2 , this embodiment discloses a selective convolution method based on an attention mechanism, which includes the following steps:

[0065] S1, obtaining an input image, and performing a first convolution on the input image to generate a feature map;

[0066] S2, the number obtained by performing global pooling on the feature map is sent to the fully connected layer to obtain the importance W of the N channels of the feature map;

[0067] In the first embodiment, there are still problems. We let each layer of feature maps output a different number of feature maps according to the principle of "the more capable, the more work", but the capacity here is dynamic, that is, it is determined by the fully connected output during the network propagation process rather than in the model definition stage. As a result, a new convolution kernel is generated for each forward propagation of the network, and each convolution kernel is only back-propagated once, resulting in the inability to train.

[0068] We have improved the above convolution method, but still follow the principle of "the capable should work harder" and "the incompetent should also be able to do something".

[0069] S3, according to the descending order of the importance of the feature graph, the feature graph is divided into g groups, namely X1, ..., Xg, the importance of each group is the sum of the importance of each layer in the group, and the importance queues G1, ...Gg are obtained by arranging them in descending order according to the number of feature layers contained in each group, wherein G1>...>Gg; convolving the grouped feature graphs to obtain a fourth feature graph set Y, wherein the output channel of each feature graph in the fourth feature graph set Y is output according to the importance queue;

[0070] In this embodiment, the importance levels W(w1, w2, w3, ..., wm) of the feature graphs are arranged in descending order, and the feature layers corresponding to the importance levels are also arranged in descending order to obtain X'(X'1, ...X'm). In this embodiment, it is assumed that W1>W2>...Wm.

[0071] In this embodiment, the output channel of the Y1th feature map in the fourth feature map set is G1, ..., and the output channel of the Ygth feature map in the fourth feature map set is Gg.

[0072] Specifically, in this embodiment, the m channels of the feature graph X (the feature graph has m channels) are divided into g numbers, which are represented by G. The g numbers satisfy a geometric progression with a first term of 1 and a common ratio of 2. The gth number in the g numbers can be greater than 2. g -1, to ensure that the sum of these g numbers is m. The feature map X is also divided into g groups. According to W, the first n / g most important feature maps are divided into the first group, and the first n / g most important feature maps in X except the first group of feature maps are divided into the second group, and so on, until X is divided into g groups, and the importance of each group is the sum of the importance of each layer in the group. X is grouped and convolved according to the groups. The feature maps in our grouped convolution are not continuous in X, but are selected according to their importance. The number of feature maps generated by each group of our grouped convolution is determined according to the importance of each group. The number of feature maps generated by the most important group is the largest number in G, the number of feature maps generated by the second most important group is the second largest number in G, and so on. Although this method cannot accurately determine the number of feature maps after convolution of each layer of feature maps according to their importance, it can also roughly determine the number of feature maps generated according to the importance of the feature maps, reflecting the principle of "the more capable, the more work".

[0073] S4, linearly transform the fourth feature map set to obtain a third feature map, and use the third feature map to perform feature recognition.

[0074] The second step of the selected convolution uses the linear transformation operation in GhostNet, which retains some redundant information to ensure a more comprehensive understanding of the input data, while reducing the number of parameters of the entire convolution. There may be information redundancy in an ordinary feature map, that is, the feature map features between several channels may be highly similar. Using convolution kernels to generate repeated features will introduce a lot of parameters that are not necessary. Repeated features can be generated by linear transformation of one of the features that are repeated with it. The cost of linear transformation is very low, and the number of parameters required is very small. Linear transformation can be implemented by a single-layer convolution without normalization and activation function. Assume that the m-dimensional feature map generated by the first step of the selected convolution is Y, whose dimensions are width1×height1×m. Y is linearly transformed s times to obtain the final Z. The linear transformation process is shown in the following formula:

[0075]

[0076] yi is the feature map of the i-th layer in Y, f i,j represents the jth linear transformation of yi, s is the multiple of the number of convolution Z channels relative to the number of Y channels, z i,j is the feature map obtained by performing the j-th linear transformation on yi, in particular, f i,s It is an identity mapping used to preserve the original feature map. The feature map of the m layer generated by the first step of the selective convolution is concatenated with the feature map of the m × (s-1) layer generated by the second step of the selective convolution to obtain the final feature map of the selective convolution.

[0077] This embodiment combines the attention mechanism to propose a selective convolution method based on the attention mechanism. (I) In view of the problem of large number of parameters of ordinary convolution, the selective convolution adopts the strategy of grouped convolution, and the amount of calculation and parameters are greatly reduced compared with ordinary convolution. The information in the feature map is rich and even redundant, and the redundant information can ensure a comprehensive understanding of the input data. The information between several channels with information redundancy is highly similar, and the similar one can be generated by the other through linear transformation, such as convolution with a convolution kernel of 3×3×1 or 5×5×1. Linear transformation is used in the second step of the selective convolution, which to some extent utilizes redundant information to ensure a more comprehensive understanding of the data, while also reducing the number of parameters.

[0078] (ii) Improved the traditional channel-level attention mechanism. The traditional attention mechanism between channels generates the importance of each channel through the fully connected layer, and all numbers in each channel are multiplied by this importance to reflect their own importance. However, the larger the number multiplied by all the numbers in each layer of the feature map, the less likely it is to promote the final network performance. The first step of convolution selection also considers the importance of each channel, but the processing method is different from the attention mechanism. Instead, it allows important channels to convolve more features. The more important the channel, the more features it will convolve to reflect its own importance, and it also reflects the principle of "the more capable, the more work". The more important the channel, the more contribution it makes to this convolution.

[0079] (III) To address the problem that grouped convolution does not allow information exchange between groups, the grouping method for each convolution is different. The grouping is based on the importance of each channel, and the importance ranking of each feature map is different, which leads to a different grouping method for each convolution, greatly promoting information exchange between groups.

[0080] (iv) The convolution is performed in two steps, where the second step uses the linear transformation operation in GhostNet, retaining some redundant information to ensure a more comprehensive understanding of the input data while reducing the number of parameters of the entire convolution.

[0081] Embodiment 3

[0082] refer to Figure 3 , this embodiment discloses a selective convolution device based on an attention mechanism, which includes the following units:

[0083] A first convolution unit, used to obtain an input image, and perform a first convolution on the input image to generate a feature map;

[0084] The input image of this embodiment is an image to be recognized, and a first convolution is performed on the input image to generate a feature map, wherein the first convolution is a convolution form commonly used in the art.

[0085] An importance acquisition unit, used for performing global pooling on the feature map and sending the obtained number to a fully connected layer to obtain the importance W of N channels of the feature map;

[0086] The selected convolution of this embodiment is divided into two steps. The first step is the group convolution combined with the attention mechanism, and the second step is the linear transformation operation. The implementation process of the first step of the selected convolution is introduced below.

[0087] Specifically, this embodiment first performs a first convolution on an input image to generate a feature map X. It is assumed that the height, width and number of channels of the feature map X to be convolved are represented by height, width and n respectively.

[0088] The n channels of feature map X are globally pooled and sent to the fully connected layer to get the importance of these n channels (w1, w2, w3, ..., wn), represented by W. The larger the value, the more important it is. We give a threshold α, and channels below the threshold α are considered unimportant channels, and channels above the threshold α are considered important channels.

[0089] The channel-level attention mechanism SE multiplies all numbers on each channel by this importance to reflect the importance of each channel. However, multiplying the more important channels by larger numbers does not necessarily improve the overall performance of the network.

[0090] The numbers in each feature map are abstract features. In some cases, the smaller the value in the feature map, the more likely it is that the loss function will decrease. We perform convolution on important channels, so that more important channels can be convolved with more layers of feature maps to reflect their own importance, which reflects the principle of more work for those who are more capable.

[0091] Multiplying important channels by a larger number may not necessarily reflect their importance, but allowing important channels to convolve more features can reflect their importance.

[0092] Assume that the number of channels of the intermediate feature map generated by the first step of convolution is n, because the sum of (w1,w2,w3,.....,wn) is 1, so n×w i It represents the number of channels of the feature map output after convolution of the i-th channel in X. The larger the wi, the more feature maps are convolved out of the channel.

[0093] A second grouped convolution unit is used to perform convolution on the feature map, wherein for each channel whose importance is greater than a threshold, the output channel is Wi*N, and a first feature map is obtained, wherein Wi represents the number of the i-th channel; for channels whose importance is less than the threshold, they are grouped together so that their output channels are 1, and a second feature map is obtained;

[0094] Specifically, the second group convolution unit also includes: dividing the channels whose importance is less than the threshold into a group, and when the importance of the group is not 1, adjusting the threshold so that the sum of the weights of the least important first k is less than or equal to 1 / n, and the sum of the weights of the least important first k+1 is greater than 1 / n.

[0095] But when n is not large enough or wi is very small, n×w i Less than 1, resulting in the number of feature map channels convolved from some unimportant channels below the threshold α being less than 1, that is, these channels will not contribute to the convolution. We follow the principle of "incompetence is also contribution" to allow the least important channels to contribute to the convolution. Therefore, we pack the unimportant channels below the threshold into a group, and add the corresponding weights of each channel to form the weight of the group. However, there is still a situation where the sum of the weights of the packed group multiplied by n is still less than 1, which indicates that the threshold setting is unreasonable.

[0096] We reset the threshold ɑ so that the sum of the weights of the least important first k is less than or equal to 1 / n, and the sum of the weights of the least important first k+1 is greater than 1 / n. This ensures that after all the least important channels are packaged into a group, one layer of feature maps can still be output. This reflects our principle of "incompetence is also action", so that the least important layers of feature maps can also contribute to the convolution. The flowchart is as follows Figure 1 shown.

[0097] The feature map generating unit is used to perform linear transformation on the first feature map and the second feature map to obtain a third feature map, and use the third feature map to perform feature recognition.

[0098] The second step of the selected convolution uses the linear transformation operation in GhostNet, which retains some redundant information to ensure a more comprehensive understanding of the input data, while reducing the number of parameters of the entire convolution. There may be information redundancy in an ordinary feature map, that is, the feature map features between several channels may be highly similar. Using convolution kernels to generate repeated features will introduce a lot of parameters that are not necessary. Repeated features can be generated by linear transformation of one of the features that are repeated with it. The cost of linear transformation is very low, and the number of parameters required is very small. Linear transformation can be implemented by a single-layer convolution without normalization and activation function. Assume that the m-dimensional feature map generated by the first step of the selected convolution is Y, whose dimensions are width1×height1×m. Y is linearly transformed s times to obtain the final Z. The linear transformation process is shown in the following formula:

[0099]

[0100] yi is the feature map of the i-th layer in Y, f i,j represents the jth linear transformation of yi, s is the multiple of the number of convolution Z channels relative to the number of Y channels, z i,j is the feature map obtained by performing the j-th linear transformation on yi, in particular, f i,s It is an identity mapping used to preserve the original feature map. The feature map of the m layer generated by the first step of the selective convolution is concatenated with the feature map of the m×(s-1) layer generated by the second step of the selective convolution to obtain the final feature map of the selective convolution.

[0101] Embodiment 4

[0102] refer to Figure 4 , this embodiment discloses a selective convolution device based on an attention mechanism, which includes the following units:

[0103] A first convolution unit, used to obtain an input image, and perform a first convolution on the input image to generate a feature map;

[0104] An importance acquisition unit, used for performing global pooling on the feature map and sending the obtained number to a fully connected layer to obtain the importance W of N channels of the feature map;

[0105] A second grouping convolution unit is used to divide the feature map into g groups, namely X1, ..., Xg, in descending order according to the importance of the feature map, and the importance of each group is the sum of the importance of each layer in the group, and the importance queues G1, ...Gg are obtained by arranging them in descending order according to the number of feature layers contained in each group, wherein G1>...>Gg; convolve the grouped feature maps to obtain a fourth feature map set Y, wherein the output channel of each feature map in the fourth feature map set Y is output according to the importance queue;

[0106] In this embodiment, the importance levels W(w1, w2, w3, ..., wm) of the feature graphs are arranged in descending order, and the feature layers corresponding to the importance levels are also arranged in descending order to obtain X'(X'1, ...X'm). In this embodiment, it is assumed that W1>W2>...Wm.

[0107] In this embodiment, the output channel of the Y1th feature map in the fourth feature map set is G1, ..., and the output channel of the Ygth feature map in the fourth feature map set is Gg.

[0108] In the third embodiment, there are still problems. We let each layer of feature maps output a different number of feature maps according to the principle of "the more capable, the more work", but the capacity here is dynamic, that is, it is determined by the fully connected output during the network propagation process rather than in the model definition stage. As a result, a new convolution kernel is generated for each forward propagation of the network, and each convolution kernel is only back-propagated once, resulting in the inability to train.

[0109] We have improved the above convolution method, but still follow the principle of "the capable should work harder" and "the incompetent should also be able to do something".

[0110] Specifically, in this embodiment, the m channels of the feature graph X (the feature graph has m channels) are divided into g numbers, which are represented by G. The g numbers satisfy a geometric progression with a first term of 1 and a common ratio of 2. The gth number in the g numbers can be greater than 2. g -1, to ensure that the sum of these g numbers is m. The feature map X is also divided into g groups. According to W, the first n / g most important feature maps are divided into the first group, and the first n / g most important feature maps in X except the first group of feature maps are divided into the second group, and so on, until X is divided into g groups, and the importance of each group is the sum of the importance of each layer in the group. X is grouped and convolved according to the groups. The feature maps in our grouped convolution are not continuous in X, but are selected according to their importance. The number of feature maps generated by each group of our grouped convolution is determined according to the importance of each group. The number of feature maps generated by the most important group is the largest number in G, the number of feature maps generated by the second most important group is the second largest number in G, and so on. Although this method cannot accurately determine the number of feature maps after convolution of each layer of feature maps according to their importance, it can also roughly determine the number of feature maps generated according to the importance of the feature maps, reflecting the principle of "the more capable, the more work".

[0111] The feature map generating unit is used to perform a linear transformation on the fourth feature map set to obtain a third feature map, and use the third feature map to perform feature recognition.

[0112] The second step of the selected convolution uses the linear transformation operation in GhostNet, which retains some redundant information to ensure a more comprehensive understanding of the input data, while reducing the number of parameters of the entire convolution. There may be information redundancy in an ordinary feature map, that is, the feature map features between several channels may be highly similar. Using convolution kernels to generate repeated features will introduce a lot of parameters that are not necessary. Repeated features can be generated by linear transformation of one of the features that are repeated with it. The cost of linear transformation is very low, and the number of parameters required is very small. Linear transformation can be implemented by a single-layer convolution without normalization and activation function. Assume that the m-dimensional feature map generated by the first step of the selected convolution is Y, whose dimensions are width1×height1×m. Y is linearly transformed s times to obtain the final Z. The linear transformation process is shown in the following formula:

[0113]

[0114] yi is the feature map of the i-th layer in Y, f i,j represents the jth linear transformation of yi, s is the multiple of the number of convolution Z channels relative to the number of Y channels, z i,j is the feature map obtained by performing the j-th linear transformation on yi, in particular, f i,s It is an identity mapping used to preserve the original feature map. The feature map of the m layer generated by the first step of the selective convolution is concatenated with the feature map of the m×(s-1) layer generated by the second step of the selective convolution to obtain the final feature map of the selective convolution.

[0115] Embodiment 5

[0116] refer to Figure 5 , Figure 5 : is a structural diagram of a selective convolution device based on an attention mechanism in this embodiment. The selective convolution device 20 based on the attention mechanism in this embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above method embodiment are implemented. Alternatively, when the processor 21 executes the computer program, the functions of each module / unit in the above device embodiments are implemented.

[0117] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the attention-based selective convolution device 20. For example, the computer program may be divided into the modules in Embodiment 2. For the specific functions of each module, please refer to the working process of the device described in the above embodiment, which will not be repeated here.

[0118] The selective convolution device 20 based on the attention mechanism may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of the selective convolution device 20 based on the attention mechanism, and does not constitute a limitation on the selective convolution device 20 based on the attention mechanism, and may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, the selective convolution device 20 based on the attention mechanism may also include input and output devices, network access devices, buses, and the like.

[0119] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the selective convolution device 20 based on the attention mechanism, and uses various interfaces and lines to connect the various parts of the selective convolution device 20 based on the attention mechanism.

[0120] The memory 22 can be used to store the computer program and / or module, and the processor 21 implements various functions of the selective convolution device 20 based on the attention mechanism by running or executing the computer program and / or module stored in the memory 22, and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0121] Wherein, if the module / unit integrated in the selective convolution device 20 based on the attention mechanism is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, the steps of the above-mentioned method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0122] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A selective convolution method based on an attention mechanism, comprising the following steps: S1, obtaining an input image, and performing a first convolution on the input image to generate a feature map; S2, the number obtained by performing global pooling on the feature map is sent to the fully connected layer to obtain the importance W of the N channels of the feature map; S3, convolution is performed on the feature map, where for each channel whose importance is greater than a threshold, the output channel is , obtaining a first feature map, wherein Wi represents the importance of the i-th channel; dividing the channels whose importance is less than the threshold into a group so that the output channel thereof is 1, obtaining a second feature map; step S3 further includes: Channels with importance less than the threshold are grouped together. When the output channel of the group is not 1, the threshold is adjusted so that the sum of the weights of the least important first k is less than or equal to 1 / N, and the sum of the weights of the least important first k+1 is greater than 1 / N. S4, linearly transform the first feature map and the second feature map to obtain a third feature map, and use the third feature map to perform feature recognition.

2. A selective convolution device based on an attention mechanism, comprising the following units: A first convolution unit, used to obtain an input image, and perform a first convolution on the input image to generate a feature map; An importance acquisition unit, used for performing global pooling on the feature map and sending the obtained number to a fully connected layer to obtain the importance W of N channels of the feature map; The second group convolution unit performs convolution on the feature map, where for each channel whose importance is greater than the threshold, the output channel is , get the first feature map, where Wi represents the importance of the i-th channel; Channels with importance less than the threshold are divided into a group so that their output channels are 1, and a second feature map is obtained; The second group convolution unit further includes: grouping channels whose importance is less than a threshold into a group, wherein when the output channel of the group is not 1, adjusting the threshold so that the sum of the weights of the least important first k channels is less than or equal to 1 / N, and the sum of the weights of the least important first k+1 channels is greater than 1 / N; The feature map generating unit performs linear transformation on the first feature map and the second feature map to obtain a third feature map, and uses the third feature map to perform feature recognition.

Images