Bird fine-grained image recognition method and device based on Transformer and component feature fusion
By generating and fusing component attention maps of bird images in the Transformer architecture, combining cross entropy and center loss optimization parameters, the problem of bird fine-grained image recognition is solved, high-precision recognition is achieved and the dependence of labeled information is reduced.
Patent Information
- Application Number
- CN202210279684.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-21
- Publication Date
- 2024-12-24
- Estimated Expiration
- 2042-03-21
AI Technical Summary
The bird fine-grained image recognition task is difficult to achieve high-precision recognition due to the subtle differences between objects and large visual differences, and the recognition is difficult for the prior art to achieve high-precision recognition.
The basic feature map is extracted using a feature encoder based on the Transformer architecture, and the component attention map is generated through the attention module, bilinear attention pooling operation is performed, discriminant component features are fused, and the model parameters are optimized by combining cross entropy loss and center loss.
It realizes high recognition accuracy of bird fine-grained images, can pay attention to discriminant components under weak supervision, improve feature expression capabilities, and avoid dependence on other annotated information.
Smart Images
Figure CN114626476B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, in particular to deep learning and fine-grained image recognition technology, and specifically to a method and device for fine-grained bird recognition based on Transformer and component feature fusion. Background Art
[0002] Bird image recognition is a fine-grained image recognition task. Fine-grained image recognition is to distinguish between different subcategories belonging to the same general category. General image recognition is to divide objects into large categories, such as distinguishing between horses and cats. The features between these different types of objects are quite different, and the categories are relatively easy to distinguish. For fine-grained images, the differences between objects usually exist in subtle parts, and the same object presents large visual differences due to scale, perspective, background, etc., so it is more difficult to recognize. Summary of the invention
[0003] The purpose of the present invention is to provide a method and device for bird fine-grained image recognition based on Transformer and component feature fusion, which can obtain higher recognition accuracy of bird fine-grained images.
[0004] To achieve the above object, the present invention provides a bird fine-grained image recognition method based on Transformer and component feature fusion, which comprises:
[0005] Step 1, by inputting the preprocessed image into a feature encoder based on a Transformer architecture network, extracting a basic feature map, and inputting the basic feature map into an attention module to generate a component attention map;
[0006] Step 2, performing a bilinear attention pooling operation on the basic feature map and the component attention map to obtain discriminative component features;
[0007] Step 3, by splicing the discriminative component features in the channel dimension, an enhanced feature representation that incorporates the discriminative component information is obtained;
[0008] In step 4, the category mapping is completed by inputting the enhanced feature representation into the fully connected layer, and the model parameters are optimized through cross entropy loss and center loss.
[0009] Furthermore, the method for extracting the basic feature map in step 1 specifically includes:
[0010] Step 11a, input the preprocessed original image I into the feature extraction network f, and extract the two-dimensional basic feature map F, F∈(H·W)×ψ, where H and W represent the height and width of the basic feature map F respectively, and D represents the embedding dimension size;
[0011] Step 12a, reorganize the basic feature map F to obtain a three-dimensional basic feature map The process is shown in the following formula (1):
[0012]
[0013] Where reshape(·) means reorganizing the basic feature map.
[0014] Furthermore, the method for generating a component attention map in step 1 specifically includes:
[0015] Step 11b, determine the number of channels M of the component attention map to be generated, that is, the number of generated component features;
[0016] Step 12b, the attention module G is composed of a 2D convolution with a convolution kernel of 1×1 and a Sigmoid function, and the feature map Input the attention module G to generate a component attention map A that represents the distribution of different components of the target object, as shown in the following formula (2):
[0017]
[0018] In the formula, A i (i=1,2,...,M) represents the attention map of the i-th component in the target object.
[0019] Furthermore, the step 2 specifically includes:
[0020] Step 21: Attention map A of different components i The dimension is expanded to the same as the basic feature map The same as above, then the expanded component attention map A i With the basic feature map According to the following formula (3), element-by-element multiplication is performed to obtain the discriminative component feature P i :
[0021]
[0022] In the formula, ⊙ represents the element-by-element multiplication operation;
[0023] Step 22: The discriminative component feature P i According to the global average pooling provided by the following formula (4), the aggregation operation is performed to aggregate the discriminative component features P i :
[0024] h i =ψ(P i ) (4)
[0025] In the formula, hi represents the feature after aggregation of the i-th component, and ψ(·) represents the global average pooling (GAP).
[0026] Furthermore, the step 3 specifically includes:
[0027] Step 31: aggregate the discriminative component features h i Splicing is performed in the channel dimension to obtain an enhanced feature representation, namely the global component feature Q, which integrates the discriminative component information and has stronger feature expression ability.
[0028] Q=Concate(h1,h2,…,h M ) (5)
[0029] In the formula, Concate(·) represents feature concatenation;
[0030] Step 32, after the global component feature Q is normalized by the L2 norm, it is passed to the fully connected layer to complete the mapping of the feature vector to the category.
[0031] Furthermore, the step 4 specifically includes:
[0032] Step 41: Input the global component feature Q into the fully connected layer to complete the mapping of bird image categories and obtain the cross entropy loss between the predicted value and the label. It is used to penalize the classification results. The loss of a single sample is shown in formula (6):
[0033]
[0034] In the formula, y represents the category label, y' represents the predicted value, and P represents the probability after Softmax processing;
[0035] In step 42, the center loss of a single sample described by formula (8) is used to weakly supervise the generation process of component attention, so that different component features continue to approach the feature center:
[0036]
[0037] In the formula, q i is the i-th component feature in the global component feature Q, c i is the center of the i-th component feature;
[0038] Step 43, initialize c i , during the model training process, it is updated according to the following formula (9):
[0039] c i ←c i +α(q i -c i) (9)
[0040] Where α∈[0,1] is c i Updated learning rate, overall loss of the model during the training phase It is defined as follows (10):
[0041]
[0042] The present invention also provides a bird fine-grained image recognition device based on Transformer and component feature fusion, which includes:
[0043] A component attention generation unit is used to extract a basic feature map by inputting the preprocessed image into a feature encoder based on a Transformer architecture network, and input the basic feature map into an attention module to generate a component attention map;
[0044] A discriminative component feature generating unit, which is used to perform a bilinear attention pooling operation on the basic feature map and the component attention map to obtain a discriminative component feature;
[0045] A feature fusion unit, which is used to obtain an enhanced feature representation that incorporates discriminative component information by concatenating the discriminative component features in the channel dimension;
[0046] The parameter learning optimization unit is used to complete the category mapping by inputting the enhanced feature representation into the fully connected layer, and optimize the model parameters through the cross entropy loss and the center loss.
[0047] Furthermore, the component attention generation unit includes:
[0048] The basic feature map extraction subunit specifically includes:
[0049] A two-dimensional basic feature map extraction module is used to input the preprocessed original image I into the feature extraction network f to extract a two-dimensional basic feature map F, F∈(H·W)×D, where H and W represent the height and width of the basic feature map F, respectively, and D represents the embedding dimension size;
[0050] The three-dimensional basic feature map module is used to reorganize the basic feature map F to obtain a three-dimensional basic feature map The process is shown in the following formula (1):
[0051]
[0052] In the formula, reshape(·) means reorganizing the feature map;
[0053] The component attention map generation subunit is used to determine the number of channels M of the component attention map to be generated, and the attention module G is composed of a two-dimensional convolution with a convolution kernel of 1×1 and a Sigmoid function. Input the attention module G to generate a component attention map A that represents the distribution of different components of the target object, as shown in the following formula (2):
[0054]
[0055] In the formula, A i (i=1,2,...,M) represents the attention map of the i-th component in the target object.
[0056] Furthermore, the discriminative component feature generating unit specifically includes:
[0057] A single discriminative component feature generation module is used to convert different component attention maps A i The dimension of the component attention map A is expanded to be consistent with the basic feature map F, and then the expanded component attention map A is i Multiply the basic feature map F element by element according to the following formula (3) to obtain the discriminative component feature P i :
[0058]
[0059] In the formula, ⊙ represents the element-by-element multiplication operation;
[0060] The discriminative component feature fusion module is used to combine the discriminative component features P i According to the global average pooling provided by the following formula (4), the aggregation operation is performed to fuse the discriminative component features P i :
[0061] h i =ψ(P i ) (4)
[0062] In the formula, h i represents the feature after aggregation of the i-th component, and ψ(·) represents the global average pooling (GAP).
[0063] Furthermore, the parameter learning optimization unit specifically includes:
[0064] The sample classification loss acquisition module is used to input the global component feature Q into the fully connected layer to complete the mapping of bird image categories and obtain the cross entropy loss of the predicted value and the label. It is used to penalize the classification results. The classification loss of a single sample is shown in formula (6):
[0065]
[0066] In the formula, y represents the category label, y' represents the predicted value, and P represents the probability after Softmax processing;
[0067] The center update module of the component feature is used to weakly supervise the generation process of the component attention using the center loss of a single sample described by formula (8) and initialize c i , during the model training process, it is updated according to the following formula (9), and the overall loss of the model during the training phase is It is defined as follows (10):
[0068]
[0069] c i ←c i +ɑ(q i -c i ) (9)
[0070]
[0071] In the formula, q i is the i-th component feature in the global component feature Q, c i is the center of the i-th component feature, ɑ∈[0,1] is c i The learning rate for the update.
[0072] The present invention adopts the above technical solution, which has the following advantages:
[0073] The present invention generates a component attention map through an attention method, and combines it with a feature extraction network based on the Transformer architecture to realize the fusion of discriminative component features. It can not only pay attention to the discriminative components, but also obtain feature representations with better expressive ability. The model only needs category labels in the training stage, and does not require other annotation information, so as to achieve high recognition accuracy of bird images under weak supervision. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] Figure 1 A schematic diagram of a flow chart of a method provided in an embodiment of the present invention.
[0075] Figure 2 for Figure 1 The corresponding overall model structure diagram.
[0076] Figure 3 for Figure 1 Diagram of the attention module in .
[0077] Figure 4 for Figure 1 Diagram of the component feature extraction and fusion process.
[0078] Figure 5 for Figure 1 A graph showing the effect of the center loss on model performance. DETAILED DESCRIPTION
[0079] In order to make the above-mentioned objects, features and advantages more obvious and understandable in the accompanying drawings, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0080] Terminology explanation: In the field of computer vision, the network based on the Transformer architecture is mainly composed of multi-layer perceptrons, which first divide the image into multiple image blocks and then pass them to other subsequent networks. The self-attention mechanism in the network enables the extracted feature map to contain global information, which is beneficial to downstream tasks.
[0081] like Figure 1 As shown, the bird fine-grained image recognition method based on Transformer and component feature fusion provided by the embodiment of the present invention includes the following steps:
[0082] Step 1: Extract the basic feature map by inputting the preprocessed image into a feature encoder based on the Transformer architecture network, and input the basic feature map into the attention module to generate a component attention map.
[0083] Step 2: Perform a bilinear attention pooling operation on the basic feature map and the component attention map to obtain discriminative component features.
[0084] Step 3: By concatenating the discriminative component features in the channel dimension, an enhanced feature representation that incorporates the discriminative component information is obtained.
[0085] Step 4: By inputting the enhanced feature representation into the fully connected layer, the category mapping is completed, and the model parameters are optimized through cross entropy loss and center loss. The model parameters are applicable to the model consisting of the feature extraction network f, the constructed attention module G, and the fully connected layer.
[0086] Therefore, step 1 is used to obtain the basic feature map of the image and the discriminative component attention map.
[0087] In one embodiment, the method for obtaining the basic feature map of the image in step 1 specifically includes:
[0088] Step 11a, image preprocessing.
[0089] For example, select the public bird datasets CUB-200-2011 and NABirds, and divide the selected bird datasets into training sets and test sets. The following examples illustrate the specific implementation methods of image preprocessing according to the stage, and the image preprocessing methods of the two bird datasets are the same.
[0090] Training phase: First, the images in the training set are resized to 496×496 pixels, and then a 384×384 pixel area is randomly cropped. Then, data augmentation is performed by random horizontal flipping. Finally, the image data is normalized, and the normalized mean and standard deviation are 0.485, 0.456, 0.406], [0.229, 0.224, 0.225], respectively.
[0091] Testing phase: The center of the image is cropped to 384×384 pixels and normalized in the same way as in the training phase.
[0092] Step 12a, input the preprocessed original image I into the feature extraction network f, and extract the two-dimensional basic feature map F, F∈(H·W)×D, where H and W represent the height and width of the basic feature map F respectively, and D represents the embedding dimension size. Figure 2 As shown, the feature extraction network f used in this embodiment is Swin-L based on the Transformer architecture.
[0093] Step 13a, reorganize the basic feature map F to obtain a three-dimensional basic feature map This allows the basic feature map F to be adapted to the attention network constructed later, and the input dimension meets the consistency requirements.
[0094] Then, the method for obtaining the basic feature map of the image in step 1 can be described as the following formula (1):
[0095]
[0096] Where reshape(·) means reorganizing the feature map.
[0097] It should be noted that in the above embodiment, ResNet can also be used to extract image features, and its output is a three-dimensional feature map, which does not need to be reorganized. However, this method extracts local features and has limited feature expression capabilities. The feature extraction network f in step 1 can also use other network structures in the prior art, as long as it can obtain a basic feature map with rich expression, which will not be listed here one by one.
[0098] In one embodiment, the method for obtaining a discriminative component attention map of an image in step 1 specifically includes:
[0099] Step 11b, determine the number of channels M of the component attention map to be generated, that is, the number of generated component features. Different data sets can be selected according to actual conditions.
[0100] Since the number of channels in the component attention map reflects the coverage of the model's attention to the discriminative components of the target object, when the number of paid attention to components is large, the model's discriminative performance for subtle differences in objects will be better. In the case of balancing the number of learnable model parameters and accuracy, the M values on the CUB-200-2011 and NABirds datasets can be set to 64 and 32 respectively. The value of M can be selected according to the experimental results on different datasets.
[0101] Step 12b, build an attention model to generate a component attention map.
[0102] Generally, the image attention module consists of a fully connected layer, a two-dimensional convolution, batch normalization, and an activation function (e.g., ReLU, Sigmoid, Softmax). The attention graphs generated by different attention architectures have different effects on the performance of the model. Through experimental analysis, it is found that the attention generation module G consisting of a two-dimensional convolution with a convolution kernel of 1×1 and a Sigmoid function is more suitable for the backbone network of the architecture of this embodiment. Its specific structure is as follows: Figure 3 As shown, the 1x1 convolution is used to change the number of feature channels to make it equal to the number of components that need to be set. It can be seen that the method of generating attention by combining two-dimensional convolution with activation function is not only applicable to convolutional neural networks, but also effective in networks based on Transformer architecture. The process of generating component attention map A is shown in the following formula (2):
[0103]
[0104] In the formula, A∈H×W×M,A i ∈H×W(i=1,2,...,M) represents the attention map of the i-th component in the target object, such as the head and torso of a bird.
[0105] As another implementation method of obtaining the discriminative component attention map of the image in step 1, without changing step 11b, step 12b can also adopt an attention module G consisting of a two-dimensional convolution with a convolution kernel 1x1, two-dimensional batch normalization and a ReLU function, or an attention module G consisting of a fully connected layer, a one-dimensional layer normalization and a Softmax function, or even an attention module G consisting of a fully connected layer, a one-dimensional layer normalization and a ReLU function. The latter two methods do not require the reorganization of the basic feature map.
[0106] From the above, it can be known that step 2 is used to obtain discriminative component features by bilinear attention pooling (BAP) of the basic feature map and the component attention map. Figure 4 As shown, the method for extracting discriminative component features in step 2 specifically includes:
[0107] Step 21: Attention map A of different components i The dimension is expanded to the same as the basic feature map , that is, i Repeat multiple times in the channel dimension so that the number of channels is equal to are consistent, and then they are multiplied element by element to obtain the discriminative component feature P i ∈H×W×D, at this time, the basic feature map with discriminative component positions will be activated, so that the discriminative component features can be obtained. The specific process is shown in formula (3):
[0108]
[0109] Where ⊙ represents the element-by-element multiplication operation.
[0110] Step 22: For image classification tasks, global average pooling (GAP) is usually used to aggregate features. The discriminative component features obtained in step 31 are aggregated through global average pooling to facilitate the fusion of component features. The feature aggregation process of the i-th component is defined as follows:
[0111] h i =ψ(P i ) (4)
[0112] In the formula, h i ∈D represents the aggregated features of the ith component, and ψ(·) represents the global average pooling.
[0113] In another embodiment, step 21 can be implemented by directly channel-joining the obtained feature map with the attention map. However, this method does not extract the discriminative component and then fuse the features, so the representation ability of the features is limited.
[0114] From the above, we can see that step 3 is used to fuse the discriminative component features, which specifically includes:
[0115] Step 31: aggregate the discriminative component features h i The concatenation is performed in the channel dimension to obtain an enhanced feature representation, namely the global component feature Q∈M·D shown in the following formula (5). This feature integrates the discriminative component information and has a stronger feature expression capability.
[0116] Q=Concate(h1,h2,…,h M ) (5)
[0117] In the formula, Concate(·) represents feature concatenation;
[0118] Step 32: After L2 norm normalization, the global component feature Q is passed to the fully connected layer.
[0119] In the above step 31, feature fusion can also be performed by directly adding the feature maps instead of splicing them in the channel dimension.
[0120] In one embodiment, step 4 specifically includes:
[0121] Step 41, the classification network of the model is composed of a fully connected layer and a Softmax. The global component feature Q is input into the fully connected layer to complete the mapping of the bird image category and obtain the cross entropy loss between the predicted value and the label. It is used to penalize the classification results and measure the gap between categories. The classification loss of a single sample is shown in formula (6):
[0122]
[0123] In the formula, y represents the category label pre-marked in the image, such as 0, 1, 2, ..., y' represents the predicted value obtained by inputting the component feature Q into the fully connected layer, and P represents the classification probability of 0-1 obtained after Softmax processing, which can be shown as formula (7):
[0124]
[0125] In the formula, y' j is the output value corresponding to the jth category, and C is the total number of categories in the data set.
[0126] Step 42, in order to avoid the homogenization of component attention maps during model training, that is, to ensure that attention maps of different layers can represent different target components, a center loss function is used in the model to weakly supervise the generated component attention, and the component attention map is constrained so that the component feature Q continuously approaches the feature center. During the model training process, the center loss function makes the feature expression of the same component of the target as similar as possible, and the difference between different component features is greater. The center loss of a single sample is defined as follows:
[0127]
[0128] In the formula, q i ∈D is the i-th component feature in the global component feature Q, c i ∈D is the feature center of the i-th component. i Initialized to 0, it is updated during model training as follows:
[0129] c i ←ci +α(q i -c i ) (9)
[0130] Where α∈[0,1] is c i The updated learning rate. In the experiment, it was found that when α = 0.05, better results can be achieved. The overall loss of the model during the training phase The definition is as follows:
[0131]
[0132] When the model is in the testing phase, only the cross entropy loss is used as the overall loss.
[0133] The embodiment of the present invention also provides a bird fine-grained image recognition device based on Transformer and component feature fusion, which includes a component attention generation unit, a discriminative component feature generation unit, a feature fusion unit and a parameter learning optimization unit, wherein:
[0134] The component attention generation unit is used to extract a basic feature map by inputting the preprocessed image into a feature encoder based on a Transformer architecture network, and inputting the basic feature map into an attention module to generate a component attention map.
[0135] The discriminative component feature generation unit is used to perform a bilinear attention pooling operation on the basic feature map and the component attention map to obtain discriminative component features.
[0136] The feature fusion unit is used to obtain an enhanced feature representation that incorporates discriminative component information by concatenating the discriminative component features in the channel dimension.
[0137] The parameter learning optimization unit is used to complete the category mapping by inputting the enhanced feature representation into the fully connected layer, and optimize the model parameters through cross entropy loss and center loss.
[0138] In one embodiment, the component attention generation unit includes a basic feature map extraction subunit and a component attention map generation subunit.
[0139] Among them, the two-dimensional basic feature map extraction module and the three-dimensional basic feature map module, the two-dimensional basic feature map extraction module is used to input the preprocessed original image I into the feature extraction network f, and extract the two-dimensional basic feature map F, F∈(H·W)×D, where H and W represent the height and width of the basic feature map F respectively, and D represents the embedding dimension size. The three-dimensional basic feature map module is used to reorganize the basic feature map F to obtain a three-dimensional basic feature map The process is shown in the following formula (1):
[0140]
[0141] Where reshape(·) means reorganizing the feature map.
[0142] The component attention map generation subunit is used to determine the number of channels M of the component attention map to be generated, and an attention module G is composed of a two-dimensional convolution with a convolution kernel of 1×1 and a Sigmoid function. Input the attention module G to generate a component attention map A that represents the distribution of different components of the target object, as shown in the following formula (2):
[0143]
[0144] In the formula, A i (i=1,2,...,M) represents the attention map of the i-th component in the target object.
[0145] In one embodiment, the discriminative component feature generation unit specifically includes a single discriminative component feature generation module and a discriminative component feature fusion module, wherein:
[0146] A single discriminative component feature generation module is used to convert different component attention maps A i The dimension of the component attention map A is expanded to be consistent with the basic feature map F, and then the expanded component attention map A is i Multiply the basic feature map F element by element according to the following formula (3) to obtain the discriminative component feature P i :
[0147]
[0148] Where ⊙ represents the element-by-element multiplication operation.
[0149] The discriminative component feature fusion module is used to transform the discriminative component feature P i According to the global average pooling provided by the following formula (4), the aggregation operation is performed to fuse the discriminative component features P i :
[0150] h i =ψ(P i ) (4)
[0151] In the formula, h i represents the feature after aggregation of the i-th component, and ψ(·) represents the global average pooling (GAP).
[0152] In one embodiment, the parameter learning optimization unit specifically includes a single sample loss acquisition module and a component feature center update module, wherein:
[0153] Sample classification loss acquisition module The acquisition module is used to input the global component feature Q into the fully connected layer, complete the mapping of bird image categories, and obtain the cross entropy loss of the predicted value and the label. It is used to penalize the classification results. The classification loss of a single sample is shown in formula (6):
[0154]
[0155] In the formula, y represents the category label, y' represents the predicted value, and P represents the probability after Softmax processing.
[0156] The center update module of the component feature is used to weakly supervise the generation process of the component attention using the center loss of a single sample described by formula (8) and initialize c i , during the model training process, it is updated according to the following formula (9), and the overall loss of the model during the training phase is It is defined as follows (10):
[0157]
[0158] c i ←c i +α(q i -c i ) (9)
[0159]
[0160] In the formula, q i is the i-th component feature in the global component feature Q, c i is the center of the i-th component feature, α∈[0,1] is c i The learning rate for the update.
[0161] In actual use, the input image is first preprocessed in the same manner as in the above embodiment, and then the trained model parameters are loaded. Finally, the preprocessed image is input into the model to output the category probability.
[0162] In order to verify that the center loss is effective in improving the performance of the model, Grad-CAM is used to visualize the feature map output by the last layer of the feature extraction network. The results are as follows: Figure 5 As shown in the figure, it can be seen that the high energy of the heat map without adding the center loss is scattered on the main body of the bird or in the background area with more content, while the high energy area of the heat map with adding the center loss is more concentrated on the main body of the bird, indicating that this area has a greater impact on the classification result and the classification result is better.
[0163] By adopting the method provided by the present invention, it is possible to focus on discriminative components and achieve high-precision recognition of fine-grained bird images under weak supervision.
[0164] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Those skilled in the art should understand that the technical solutions described in the above embodiments may be modified, or some of the technical features thereof may be replaced by equivalents; these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A bird fine-grained image recognition method based on Transformer and component feature fusion, characterized in that: include: Step 1, by inputting the preprocessed image into a feature encoder based on a Transformer architecture network, extracting a basic feature map, and inputting the basic feature map into an attention module to generate a component attention map; the method for extracting the basic feature map in step 1 specifically includes: Step 11a, input the preprocessed original image I into the feature extraction network f, and extract the two-dimensional basic feature map F, F∈(H·W)×D, where H and W represent the height and width of the basic feature map F respectively, and D represents the embedding dimension size; Step 12a, reorganize the basic feature map F to obtain a three-dimensional basic feature map The process is shown in the following formula (1): In the formula, reshape(·) means to reorganize the basic feature map; Step 2, performing a bilinear attention pooling operation on the basic feature map and the component attention map to obtain discriminative component features; the step 2 specifically includes: Step 21: Attention map A of different components i The dimension is expanded to the same as the basic feature map The same as above, then the expanded component attention map A i With the basic feature map According to the following formula (3), element-by-element multiplication is performed to obtain the discriminative component feature P i : In the formula, ⊙ represents the element-by-element multiplication operation; Step 22: The discriminative component feature P i According to the global average pooling provided by the following formula (4), the aggregation operation is performed to aggregate the discriminative component features P i : h i =ψ(P i ) (4) In the formula, h i represents the feature after aggregation of the i-th component, ψ(·) represents the global average pooling; Step 3, by splicing the discriminative component features in the channel dimension, an enhanced feature representation that incorporates the discriminative component information is obtained; the step 3 specifically includes: Step 31: aggregate the discriminative component features h i Splicing is performed in the channel dimension to obtain an enhanced feature representation, namely the global component feature Q; Q=Concate(h1,h2,...,h M ) (5) In the formula, Concate(·) represents feature concatenation; Step 32, after normalizing the global component feature Q with the L2 norm, it is passed to the fully connected layer to complete the mapping from feature vector to category; In step 4, the category mapping is completed by inputting the enhanced feature representation into the fully connected layer, and the model parameters are optimized through cross entropy loss and center loss.
2. The bird fine-grained image recognition method based on Transformer and component feature fusion as claimed in claim 1, characterized in that: The method for generating a component attention map in step 1 specifically includes: Step 11b, determine the number of channels M of the component attention map to be generated, that is, the number of generated component features; Step 12b, the attention module G is composed of a 2D convolution with a convolution kernel of 1×1 and a Sigmoid function, and the feature map Input the attention module G to generate a component attention map A that represents the distribution of different components of the target object, as shown in the following formula (2): In the formula, A i (i=1,2,...,M) represents the attention map of the i-th component in the target object.
3. The bird fine-grained image recognition method based on Transformer and component feature fusion as described in claim 1 or 2, characterized in that: The step 4 specifically includes: Step 41: Input the global component feature Q into the fully connected layer to complete the mapping of bird image categories and obtain the cross entropy loss between the predicted value and the label. It is used to penalize the classification results. The loss of a single sample is shown in formula (6): In the formula, y represents the category label, y' represents the predicted value, and P represents the probability after Softmax processing; In step 42, the center loss of a single sample described by formula (8) is used to weakly supervise the generation process of component attention, so that different component features continue to approach the feature center: In the formula, q i is the i-th component feature in the global component feature Q, c i is the center of the i-th component feature; Step 43, initialize c i , during the model training process, it is updated according to the following formula (9): c i ←c i +α(q i -c i ) (9) Where α∈[0,1] is c i Updated learning rate, overall loss of the model during the training phase It is defined as follows (10):
4. A bird fine-grained image recognition device based on Transformer and component feature fusion, characterized in that: include: A component attention generation unit is used to extract a basic feature map by inputting the preprocessed image into a feature encoder based on a Transformer architecture network, and input the basic feature map into an attention module to generate a component attention map; The component attention generation unit comprises: The basic feature map extraction subunit specifically includes: A two-dimensional basic feature map extraction module is used to input the preprocessed original image I into the feature extraction network f to extract a two-dimensional basic feature map F, F∈(H·W)×D, where H and W represent the height and width of the basic feature map F, respectively, and D represents the embedding dimension size; The three-dimensional basic feature map module is used to reorganize the basic feature map F to obtain a three-dimensional basic feature map The process is shown in the following formula (1): In the formula, reshape(·) means reorganizing the feature map; A discriminative component feature generation unit is used to perform a bilinear attention pooling operation on the basic feature map and the component attention map to obtain a discriminative component feature; the discriminative component feature generation unit specifically includes: A single discriminative component feature generation module is used to convert different component attention maps A i The dimension of the component attention map A is expanded to be consistent with the basic feature map F, and then the expanded component attention map A is i Multiply the basic feature map F element by element according to the following formula (3) to obtain the discriminative component feature P i : In the formula, ⊙ represents the element-by-element multiplication operation; The discriminative component feature fusion module is used to combine the discriminative component features P i According to the global average pooling provided by the following formula (4), the aggregation operation is performed to fuse the discriminative component features P i : h i =ψ(P i ) (4) In the formula, h i represents the feature after aggregation of the i-th component, ψ(·) represents the global average pooling; The feature fusion unit is used to obtain an enhanced feature representation that integrates the discriminative component information by splicing the discriminative component features in the channel dimension; i Splicing is performed on the channel dimension to obtain an enhanced feature representation, namely the global component feature Q. The global component feature Q is normalized by the L2 norm and then passed to the fully connected layer to complete the mapping of feature vectors to categories. Q=Concate(h1,h2,...,h M ) (5) In the formula, Concate(·) represents feature concatenation; The parameter learning optimization unit is used to complete the category mapping by inputting the enhanced feature representation into the fully connected layer, and optimize the model parameters through the cross entropy loss and the center loss.
5. The bird fine-grained image recognition device based on Transformer and component feature fusion as claimed in claim 4, characterized in that: The component attention generation unit also includes: The component attention map generation subunit is used to determine the number of channels M of the component attention map to be generated, and the attention module G is composed of a two-dimensional convolution with a convolution kernel of 1×1 and a Sigmoid function. Input the attention module G to generate a component attention map A that represents the distribution of different components of the target object, as shown in the following formula (2): In the formula, A i (i=1, 2, ..., M) represents the i-th component attention map in the target object.
6. The bird fine-grained image recognition device based on Transformer and component feature fusion as claimed in claim 4 or 5, characterized in that: The parameter learning optimization unit specifically includes: The sample classification loss acquisition module is used to input the global component feature Q into the fully connected layer to complete the mapping of bird image categories and obtain the cross entropy loss of the predicted value and the label. It is used to penalize the classification results. The classification loss of a single sample is shown in formula (6): In the formula, y represents the category label, y′ represents the predicted value, and P represents the probability after Softmax processing; The center update module of the component feature is used to weakly supervise the generation process of the component attention using the center loss of a single sample described by formula (8) and initialize c i , during the model training process, it is updated according to the following formula (9), and the overall loss of the model during the training phase is It is defined as follows (10): c i ←c i +α(q i -c i ) (9) In the formula, q i is the i-th component feature in the global component feature Q, c i is the center of the i-th component feature, α∈[0,1] is c i The learning rate for the update.