A method for extracting discriminative information from feature maps based on convolutional neural networks based on diversity semantic extraction and joint component perception

Abstract

The present invention relates to the field of computer technology, and specifically to a method for extracting discriminative information from a feature map based on a convolutional neural network based on diversity semantic extraction and joint component perception. The network introduces the features of the channel dimension, and combines the features of the spatial dimension with the channel dimension to extract the discriminative information of the input feature map. Specifically, the diversity semantic extraction module proposed in the present invention mines the important semantic information of the feature map from the channel dimension, and in order to make the extracted information non-repetitive, an inter-group difference constraint is designed to extract the diverse discriminative features. At the same time, a joint component perception mechanism is also designed, which uses the feature map output by the diversity semantic extraction module as a guide to mine the fine-grained features in the local feature map from the spatial dimension, thereby improving the system's ability to extract subtle discriminative information from the input feature map in complex and changing environments.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for extracting discriminative information in a feature graph by using a convolutional neural network based on diversity semantic extraction and joint component perception. Background Art

[0002] Usually a feature map has multiple channels, and the semantics expressed by the features in each channel are different, so the semantic information of the same component area in the feature map in the spatial dimension is of different importance in each channel. In order to better mine discriminative fine-grained features, the feature map can be rigidly divided to mine fine-grained features from each component (local feature map). However, the feature map after rigid division will lose the semantic continuity in the overall feature map, which makes the feature extraction in each component more susceptible to interference from background, occlusion and other noise, resulting in the inability of existing convolutional neural networks to effectively learn subtle discriminative information. In this regard, a method for extracting discriminative information in feature maps based on a convolutional neural network based on diversity semantic extraction and joint component perception is proposed to solve the above problems. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a method for designing a convolutional neural network based on diversity semantic extraction and joint component perception to extract discriminative information in feature maps.

[0004] The technical solution adopted by the present invention to solve the technical problem is:

[0005] A method for extracting discriminative information from a feature map based on a convolutional neural network based on diversity semantic extraction and joint component perception is implemented. The ResNet-50 network is used as the backbone network and is divided into three branches after the res_conv4_1 layer: a global branch, a diversity semantic extraction branch, and a joint component perception branch; the diversity semantic extraction branch is embedded in a diversity semantic extraction DSE (Diversity Semantic Extraction) module; the joint component perception branch is embedded in a joint component perception mechanism JSP (Joint Sensing Parts Mechanism), forming a convolutional neural network based on diversity semantic extraction and joint component perception, and the network is used to extract discriminative information from an input feature map;

[0006] The DSE module obtains three tensors by convolution of the input feature map, divides the three tensors into several groups along the channel dimension, and then obtains the channel query matrix, channel key matrix and channel value matrix of each group by reshaping operation;

[0007] Multiply the query matrix of each group by the transpose of the channel key matrix and normalize it with the Softmax function to obtain the channel dependency matrix.

[0008] Compress the channel dependency matrix along the column direction to obtain the channel attention matrix;

[0009] The channel attention matrices of all groups are concatenated into one matrix and the Softmax function is applied to each row of the matrix. Then, the processed matrix is ​​split by column to obtain a diverse channel attention mask after difference constraint.

[0010] Each mask is multiplied element-by-element by the corresponding channel value matrix to obtain an output feature map, and the output feature maps of all groups are concatenated and reshaped, and finally added to the original feature map and batch normalization (BN) and GELU activation function are implemented to obtain the final DSE module output feature map; the DSE module can form a diverse semantic extraction branch to extract diverse semantic information in the input feature map;

[0011] The JSP divides the input feature map to obtain feature representations of several components, compresses the final output feature map of DSE and performs inner product with each component, and establishes a connection between the global feature map with discriminative information and each component;

[0012] The inner product is normalized by applying the Sigmoid function to obtain a spatial attention map, which is multiplied by the corresponding feature representation and then added to each feature representation to generate an output feature map of JSP; JSP uses the feature map finally output by the DSE module as a guide to mine fine-grained features in the local feature map from the spatial dimension.

[0013] Furthermore, the specific process of obtaining the channel query matrix, channel key matrix and channel value matrix of each group is as follows:

[0014] Let tensor X∈R C×H×W As the input feature map of the DSE module, C, H, and W represent the channel dimension, height, and width of the input feature map respectively; three tensors of the same size are obtained by using three unshared 1×1 convolutions, namely the query tensor X Q C , the key tensor X K C Sum value tensor X V C :

[0015] ( X Q C ,X K C , X V C ) = Conv 1×1 ( X );

[0016] Along the channel dimension ( X Q C , X K C , X V C ) is evenly divided, ( X Q C , X K C , X V C ) are divided into G groups, and then the channel query matrix of each group is obtained by reshaping operation Q g ∈R C / G×HW , Channel Key Matrix K g ∈R C / G×HW and the channel value matrix V g ∈R C / G×HW :

[0017] ( Q g , K g , V g ) = Reshape(divide ( X Q C , X K C , X V C ));

[0018] Where g=1,2,...,G.

[0019] Furthermore, the channel dependency matrix is A g ∈R (C / G)×(C / G) :

[0020] ;

[0021] in, Represents matrix multiplication; the Softmax function is along the dependency matrix A g The column direction is implemented; the matrix A g Compress along the column direction to get the channel attention matrix A ’g ∈R (C / G)×1 , the matrix A ’1 , A ’2 , ..., A ’G Splice into a C / G row and G column matrix, and apply the Softmax function to each row of the matrix to get A G ∈R (C / G)×G :

[0022] A G = Softmax( A ’1 , A ’2 , ..., A ’G );

[0023] Will A G Split by column to get diverse channel attention masks A ’’ g , g=1,2,...,G.

[0024] Furthermore, the JSP divides the input feature map to obtain the feature representation of several components in the following specific process:

[0025] Suppose a feature map Y∈R H×W×C , where H and W are the height and width of feature map Y, respectively, and C represents the channel dimension; the input feature map is evenly divided along the H dimension to obtain the feature representation of n components: Y 1 , Y 2 ,..., Y n ,in Y i ∈R M×C , i=1,2,...,n,M=H / n×W.

[0026] Furthermore, the specific process of compressing the final output feature map of DSE and performing inner product with each component is as follows:

[0027] The output feature map F of DSE is compressed into a vector f of size 1×1×C and combined with each Y i Do the inner product:

[0028] h ’i = f * Y i ;

[0029] h ’i ∈R (H / n)×W , * indicates f and Y i The inner product between the feature vectors of each pixel in .

[0030] Furthermore, the global branch output feature map, the output feature map of the DSE module, and the feature map of each component output by the JSP are compressed into a feature vector with a size of 2048 dimensions through average pooling.

[0031] Furthermore, the dimension of the 2048-dimensional feature vector output by the three branches is reduced to 256 through 1×1 convolution, batch normalization (BN) and Gaussian Error Linear Unit (GELU), obtaining a 256-dimensional feature embedding, which greatly reduces the number of parameters.

[0032] Furthermore, the 256-dimensional feature embedding is used for training with triplet loss, and is used for training with cross entropy loss after conversion through a fully connected layer.

[0033] Furthermore, the final output feature maps of the three branches are concatenated to obtain the final feature representation of the input feature map.

[0034] Technical effects of the present invention:

[0035] Compared with the prior art, the present invention proposes a method for extracting discriminative information from feature maps based on a convolutional neural network based on diversity semantic extraction and joint component perception. The network introduces channel dimension features and combines the spatial dimension with the channel dimension features to extract subtle discriminative information from the input feature map. Specifically, the diversity semantic extraction module (DSE) proposed in the present invention mines important semantic information of feature maps from the channel dimension, and in order to make the extracted information non-repetitive, an inter-group difference constraint is designed to extract diverse discriminative features. At the same time, a joint component perception mechanism (JSP) is also designed. This mechanism uses the feature map output by the diversity semantic extraction module as a guide to mine fine-grained features in local feature maps from the spatial dimension, thereby improving the system's ability to extract subtle discriminative information from input feature maps in complex and changing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a structural diagram of a convolutional neural network based on diversity semantic extraction and joint component perception of the present invention;

[0037] Figure 2 This is a structural diagram of the diversity semantic extraction module of the present invention;

[0038] Figure 3 This is a structural diagram of the joint component perception mechanism of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention is clearly and completely described below in conjunction with the accompanying drawings.

[0040] Example

[0041] like Figure 1 As shown, the present embodiment involves a method for extracting discriminative information from a feature map using a convolutional neural network based on diversity semantic extraction and joint component perception. The ResNet-50 network is used as the backbone network, and the part after the res_conv4_1 layer of the ResNet-50 network is divided into three branches: a global branch, a diversity semantic extraction branch, and a joint component perception branch. In order to retain a larger receptive field and more accurately locate important detail information, the present embodiment does not perform downsampling operations in the res_conv5_1 blocks of these three branches;

[0042] The method for extracting discriminative information in a feature map by using a convolutional neural network based on diversity semantic extraction and joint component perception learns global features of an input feature map through a global branch;

[0043] A diversity semantic extraction DSE module is embedded in the diversity semantic extraction branch to form a diversity semantic extraction branch, which extracts diverse semantic information from the input feature map, thereby providing more available clues for feature extraction of the input feature map;

[0044] The joint component perception mechanism JSP is embedded in the joint component perception branch, and the JSP mechanism is used to output feature maps of multiple components. JSP uses the feature map finally output by the DSE module as a guide to mine fine-grained features in the local feature map from the spatial dimension.

[0045] The output feature maps of the three branches are compressed into a 2048-dimensional feature vector by average pooling. To reduce the number of parameters, the 1×1 convolution, batch normalization (BN) and Gaussian Error Linear Unit (GELU) are used to reduce the dimension of the 2048-dimensional feature vector output by the three branches to 256, and a 256-dimensional feature embedding is obtained; the 256-dimensional feature embedding is used for triplet loss training, and after conversion by the fully connected layer, it is used for cross entropy loss training. In the test phase, the feature embeddings output by the three branches are concatenated to obtain the final feature representation of the input feature map.

[0046] 1. Structure of the Diversity Semantic Extraction (DSE) module

[0047] like Figure 2 As shown, let the tensor X∈R C×H×W As the input feature map of the DSE module, C, H, and W represent the channel dimension, height, and width of the input feature map respectively; three tensors of the same size are obtained by using three unshared 1×1 convolutions, namely the query tensor X Q C , the key tensor X K C Sum value tensor X V C :

[0048] ( X Q C , X K C , X V C ) = Conv 1×1 ( X ).

[0049] In order to mine diverse semantic information more efficiently, this embodiment divides the input feature map of the DSE module into multiple groups to explore the features contained in each channel in more detail.

[0050] Specifically, this embodiment is along the channel dimension ( X Q C , X K C , X V C ) is evenly divided, ( X Q C , X K C , X V C ) are divided into G groups, where G represents the number of groups. Then, the channel query matrix of each group is obtained by reshaping operation. Q g ∈R C / G×HW , Channel Key Matrix K g ∈R C / G×HW and the channel value matrix V g ∈R C / G×HW :

[0051] ( Q g , K g , V g ) = Reshape(divide ( X Q C , X K C , X V C ));

[0052] Where g=1,2,...,G.

[0053] Q g and K g represents the matrix composed of the query vector and key vector of all positions of the g-th group. Q g and K gThe channel dependency matrix is ​​obtained by multiplying the transpose of and normalizing with the Softmax function. A g ∈R (C / G)×(C / G) :

[0054] ;

[0055] in, Represents matrix multiplication; the Softmax function is along the dependency matrix A g The matrix is ​​implemented in the column direction. A g The element at (i, j) A g (i, j) Represents the affinity between the i-th channel and the j-th channel in the g-th group of channels. The matrix A g The elements of the i-th row in aggregate the intra-group context information of all channels in the g-th group to the i-th channel. A g Compress along the column direction to get the channel attention matrix A ’g ∈R (C / G)×1 , which models the overall intra-group dependency of the ith channel in the gth group. A ’g The jth element in A ’g j The larger the value, the stronger the dependence of other channels on the jth channel, which in turn indicates that the semantic features expressed by the group are more activated on the jth channel. A ’g The size of the element represents the importance of the channel.

[0056] This embodiment groups channels and calculates the dependencies between channels within a group to discover the channels with the most discriminative information in each group. However, the information of some channels in different groups may be the same, or even focus on exactly the same semantic information, so there will be some redundancy in the dependencies generated by each group. In order to capture the diverse channel dependencies and mine the rich discriminative information in the input feature map, this embodiment designs an inter-group difference constraint and applies it between all groups to ensure that the channel information focused on by each group is not repeated. Specifically, the matrix A ’1 , A ’2 , ..., A ’G Splice into a C / G row and G column matrix, and apply the Softmax function to each row of the matrix to getA G ∈R (C / G)×G :

[0057] A G = Softmax( A ’1 , A ’2 , ..., A ’G ).

[0058] In getting A G After that, it is split into columns to obtain the diverse channel attention mask after difference constraint. A ’’g , g=1,2,...,G. Then each mask is associated with the corresponding channel value matrix V g Perform element-by-element multiplication to reflect the importance of the extracted channels and map them to the feature maps corresponding to each group, and get F g ∈R (C / G)×HW :

[0059] F g = A ’’g ⊙ V g .

[0060] At this time F g By using the dependencies between all channels in the group, the information of the most discriminative channels in the group that are semantically different from other groups is captured. F g After splicing together and reshaping, we get F ’ ∈R C×H×W Finally, it is added to the original feature map X and batch normalization (BN) and GELU activation function are implemented to obtain the final output feature map F∈R C×H×W :

[0061] F = GELU(BN(F ’ +X)).

[0062] 2. Structure of Joint Component Awareness Mechanism JSP

[0063] JSP can better clarify the target semantics in rigid partition parts, reduce the noise interference introduced by rigid partition, and thus mine fine-grained features that are useful for input feature map identification. This mechanism uses the feature map output by DSE as a guide and extracts component-level discriminative features by establishing a connection with the feature map output by DSE. Its structure is as follows: Figure 3 shown.

[0064] Suppose a feature map Y∈R H×W×C , where H and W are the height and width of feature map Y, respectively, and C represents the channel dimension. The input feature map is evenly divided along the H dimension to obtain the feature representation of n components (local feature maps): Y 1 , Y 2 ,..., Y n ,in Y i ∈R M×C , i = 1, 2, ..., n, M = H / n × W. The feature map output by the DSE module is used to generate a spatial attention map for each component, which highlights the characteristics of the spatial region related to the input feature map in each component and suppresses the interference of noise.

[0065] Specifically, the output feature map F of DSE is compressed into a vector f of size 1×1×C and combined with each Y i Do the inner product:

[0066] h ’i = f * Y i ;

[0067] h ’i ∈R (H / n)×W , * indicates f and Y i In this way, the connection between the global feature map with discriminative information and each component is established. h ’i Each pixel in reflects the degree of association between the corresponding fine-grained features in the component and the global features. h ’i The larger the value of a pixel in , the higher the correlation between the component-level feature at that pixel and the global feature map. h ’i Applying the Sigmoid function to normalize the spatial attention map h i :

[0068] h i =Sigmoid( h ’i );

[0069] h i is with Y i matrices with the same height and width. h i and Y i The principal elements are multiplied and added to each component to generate a new feature embedding, denoted as Y ’i :

[0070] Y ’i =( h i ⊙ Y i )+ Y i .

[0071] JSP establishes the semantic connection between the complete features in the global feature map and the fine-grained features in a component. It uses the global features as a guide to aggregate the important features in the component, extract the semantic information in the component and the semantic information of other components related to the component. This not only alleviates the problem of increased difficulty in extracting component features caused by incomplete division, but also effectively reduces the interference of background, occlusion and other noise, and further improves the system's ability to extract discriminative information from the input feature map in complex and changing environments.

[0072] The above-mentioned specific implementation modes are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementation modes. Any appropriate changes or modifications made thereto by any ordinary technician in the technical field in accordance with the claims of the present invention shall fall within the patent protection scope of the present invention.

Claims

1. A method for extracting discriminative information from a feature map based on a convolutional neural network based on diversity semantic extraction and joint component perception, characterized in that: The ResNet-50 network is used as the backbone network, and is divided into three branches after the res_conv4_1 layer: a global branch, a diversity semantic extraction branch, and a joint component perception branch; the diversity semantic extraction branch is embedded in the diversity semantic extraction DSE module; the joint component perception branch is embedded in the joint component perception mechanism JSP, forming a convolutional neural network based on diversity semantic extraction and joint component perception, and the network is used to extract discriminative information in the input feature map; The DSE module obtains three tensors by convolution of the input feature map, divides the three tensors into several groups along the channel dimension, and then obtains the channel query matrix, channel key matrix and channel value matrix of each group by reshaping operation; Multiply the query matrix of each group by the transpose of the channel key matrix and normalize it with the Softmax function to obtain the channel dependency matrix. Compress the channel dependency matrix along the column direction to obtain the channel attention matrix; The channel attention matrices of all groups are concatenated into one matrix and the Softmax function is applied to each row of the matrix. Then, the processed matrix is ​​split by column to obtain a diverse channel attention mask after difference constraint. Each mask is multiplied element-by-element by the corresponding channel value matrix to obtain an output feature map, and the output feature maps of all groups are concatenated together and reshaped, and finally added to the original feature map and batch normalized and GELU activated to obtain the final DSE module output feature map; The JSP divides the input feature map to obtain feature representations of several components, compresses the final output feature map of DSE and performs inner product with each component, and establishes a connection between the global feature map with discriminative information and each component; Applying the Sigmoid function to normalize the inner product to obtain a spatial attention map, multiplying the spatial attention map with the corresponding feature representation and then adding it to each feature representation to generate an output feature map of the JSP; The specific process of obtaining the channel query matrix, channel key matrix and channel value matrix of each group is as follows: Let tensor X∈R C×H×W As the input feature map of the DSE module, C, H, and W represent the channel dimension, height, and width of the input feature map respectively; three tensors of the same size are obtained by using three unshared 1×1 convolutions, namely the query tensor X Q C , the key tensor X K C Sum value tensor X V C : ( X Q C , X K C , X V C ) = Conv 1×1 ( X ); Along the channel dimension ( X Q C , X K C , X V C ) is evenly divided, ( X Q C , X K C , X V C ) are divided into G groups, and then the channel query matrix of each group is obtained by reshaping operation Q g ∈R C / G×HW , Channel Key Matrix K g ∈R C / G×HW and the channel value matrix V g ∈R C / G×HW : ( Q g , K g , V g ) = Reshape(divide ( X Q C , X K C , X V C )); Where g=1,2,...,G; The channel dependency matrix is: A g ∈R (C / G)×(C / G) : ； in, Represents matrix multiplication; the Softmax function is along the dependency matrix A g The column direction is implemented; the matrix A g Compress along the column direction to get the channel attention matrix A ’ g ∈R (C / G)×1 , the matrix A ’1 , A ’2 , ..., A ’G Splice into a C / G row and G column matrix, and apply the Softmax function to each row of the matrix to get A G ∈R (C / G)×G : A G = Softmax( A ’1 , A ’2 , ..., A ’G ); Will A G Split by column to get diverse channel attention masks A ’’ g , g=1,2,...,G; The specific process of the JSP dividing the input feature map to obtain the feature representation of several components is as follows: Suppose a feature map Y∈R H×W×C , where H and W are the height and width of feature map Y, respectively, and C represents the channel dimension; the input feature map is evenly divided along the H dimension to obtain the feature representation of n components: Y 1 , Y 2 ,..., Y n ,in Y i ∈R M ×C , i=1,2,...,n, M=H / n×W; The specific process of compressing the final output feature map of DSE and performing inner product with each component is as follows: The output feature map F of DSE is compressed into a vector f of size 1×1×C and combined with each Y i Do the inner product: h ’i = f * Y i ； h ’i ∈R (H / n)×W , * indicates f and Y i The inner product between the feature vectors of each pixel in ; JSP can clarify the target semantics in the rigid partition components and reduce the noise interference introduced by the rigid partition, thereby mining the fine-grained features that are useful for input feature map identification; the DSE module can form a diverse semantic extraction branch to extract diverse semantic information from the input feature map.

2. The method for extracting discriminative information from a feature graph based on a convolutional neural network based on diversity semantic extraction and joint component perception according to claim 1, characterized in that: The output feature map of the global branch, the output feature map of the DSE module, and the feature map of each component output by the JSP are compressed into a feature vector with a size of 2048 dimensions through average pooling.

3. The method for extracting discriminative information from a feature graph based on a convolutional neural network based on diversity semantic extraction and joint component perception according to claim 2, characterized in that: The dimension of the 2048-dimensional feature vector output by the three branches is reduced to 256 through 1×1 convolution, batch normalization and Gaussian error linear unit to obtain a 256-dimensional feature embedding.

4. The method for extracting discriminative information from a feature graph based on a convolutional neural network based on diversity semantic extraction and joint component perception according to claim 3, characterized in that: The 256-dimensional feature embedding is used for training with triplet loss, and is converted through a fully connected layer for training with cross entropy loss.

5. The method for extracting discriminative information from a feature graph using a convolutional neural network based on diversity semantic extraction and joint component perception according to any one of claims 1 to 4, characterized in that: The final output feature maps of the three branches are concatenated to obtain the final feature representation of the input feature map.

Images