A fine-grained image classification method, device, electronic device and storage medium

Abstract

The present invention relates to a fine-grained image classification method, device, electronic device and storage medium. The fine-grained image classification method described in the present invention comprises: obtaining an image to be classified; inputting the image to be classified into a trained feature extraction network to obtain multi-scale features of the image; wherein the feature extraction network is a ResNet101 neural network with a PCFN module inserted; inputting the multi-scale features of the image into a classification network to obtain a fine-grained classification result of the image to be classified. The fine-grained image classification method described in the present invention learns more discriminative features by associating cross-layer features, combines the characteristics of features at different layers, obtains more expressive multi-scale features, and improves the classification effect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a fine-grained image classification method, device, electronic equipment and storage medium. Background Art

[0002] Fine-grained image classification is one of the more popular research directions in the field of image recognition. Fine-grained image classification is to classify subcategories in a large category of image library or image set. Compared with traditional image classification, fine-grained image classification has the characteristics of small inter-class differences and large intra-class differences. For a long time, it has been considered a challenging task due to the feature learning of distinguishing region localization and recognition region.

[0003] There are already many excellent fine-grained image classification algorithms, but the following problems still exist:

[0004] (1) Using manual annotation information during the training process of convolutional neural networks can improve the classification performance of the model, but labeling information requires a lot of manpower, material resources and time. Therefore, how to make the model have better classification performance when only using image class labels is of great research significance.

[0005] (2) In general, an image usually contains a lot of background information, and the differential information that affects classification often exists in very small component areas. The feature dimension finally extracted by the convolutional neural network is certain. The more differential information is extracted, the more discriminative features there are in the final feature vector, and the higher the classification accuracy. Therefore, how to make the feature vector contain more effective information is one of the problems to be solved to improve the accuracy of fine-grained classification.

[0006] (3) The channel and spatial pixel relationship of different layer features is not fully considered, and the characteristics of different layers are not fully considered. Therefore, how to better use multi-scale features to make the model have better classification performance is of great research significance. Summary of the invention

[0007] Based on this, the purpose of the present invention is to provide a fine-grained image classification method, device, electronic device and storage medium, which can learn more discriminative features by associating cross-layer features, combining the characteristics of features at different layers, and obtaining more expressive multi-scale features to improve the classification effect.

[0008] In a first aspect, the present invention provides a fine-grained image classification method, comprising the following steps:

[0009] Get the image to be classified;

[0010] Inputting the image to be classified into a trained feature extraction network to obtain multi-scale features of the image; wherein the feature extraction network is a ResNet101 neural network into which a PCFN module is inserted;

[0011] The multi-scale features of the image are input into a classification network to obtain a fine-grained classification result of the image to be classified.

[0012] Furthermore, the image is input into a trained feature extraction network to obtain multi-scale features of the image, including:

[0013] The image is input into the first layer of the Resnet101 neural network, and then into layer1, layer2, layer3, and layer4 for convolution processing;

[0014] The output features of layer2 and layer3 are respectively input into the subsequent PCFN modules to obtain semantically weighted output features;

[0015] The semantically weighted output features of layer2 and layer3 are respectively input into the dimension transformation module to obtain output features consistent with the output feature space and channel dimensions of layer4;

[0016] The output features of the dimension transformation module and the output features of layer4 are fused to obtain multi-scale features of the image.

[0017] Furthermore, the output features of layer2 and layer3 are respectively input into the subsequent PCFN modules to obtain semantically weighted output features, including:

[0018] For any one of the output features of layer2 and layer3, the output feature is input into the channel association branch and the spatial association branch of the PCFN module as the response layer of the PCFN module;

[0019] Perform an upsampling operation on the output features of the layer4, and input the channel association branch and the spatial association branch of the PCFN module as the query layer of the PCFN module;

[0020] A weighted operation is performed on the channel association branch and the space association branch to obtain semantic weighted information of the output feature of layer4 to the output feature of any one of the output features of layer2 and layer3 in two dimensions of channel and space;

[0021] The weighted information is output to obtain a semantically weighted output feature corresponding to any one of the output features of layer2 and layer3.

[0022] Furthermore, the output features of the dimension transformation module and the output features of layer4 are subjected to feature fusion, including:

[0023] The output features of the dimension transformation module and the output features of layer4 are fused in an element-wise add manner.

[0024] Furthermore, the training step of the feature extraction network includes:

[0025] Obtaining a training data set for the feature extraction network, and dividing the training data set into a training set and a test set;

[0026] The feature extraction network is trained using the training set, and the SGD optimizer is used for learning. The loss function decreases continuously with the back propagation of the error, and the training accuracy rate increases continuously. When the loss function converges and no longer decreases, a pre-trained feature extraction network is obtained;

[0027] The pre-trained feature extraction network is tested using the test set, and when the accuracy of the test result reaches a preset threshold, the pre-trained feature extraction network is saved to obtain a trained feature extraction network.

[0028] Further, using the training set to train the parameters of the feature extraction network includes:

[0029] Each dataset was trained for 150 epochs, with a gradual warm-up strategy used in the first ten epochs of the training process;

[0030] The batch size is set to 32 and the base learning rate is set to 0.01;

[0031] A fixed step size decay learning rate decay strategy is adopted, and the learning rate is reduced to one tenth of the original every 20 epochs.

[0032] Furthermore, obtaining a training data set for the feature extraction network also includes:

[0033] Performing a preprocessing operation on the training data set to obtain a data-enhanced training data set;

[0034] The preprocessing operations include: translation, scaling, rotation and flipping.

[0035] In a second aspect, the present invention further provides a fine-grained image classification device, comprising:

[0036] An image acquisition module, used for acquiring images to be classified;

[0037] A feature extraction module, used for inputting the image to be classified into a trained feature extraction network to obtain multi-scale features of the image; wherein the feature extraction network is a ResNet101 neural network into which a PCFN module is inserted;

[0038] The classification module is used to input the multi-scale features of the image into a classification network to obtain a fine-grained classification result of the image to be classified.

[0039] In a third aspect, the present invention further provides an electronic device, characterized in that it includes:

[0040] at least one memory and at least one processor;

[0041] The memory is used to store one or more programs;

[0042] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a fine-grained image classification method as described in any one of the first aspects of the present invention.

[0043] In a fourth aspect, the present invention further provides a computer-readable storage medium, characterized in that:

[0044] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a fine-grained image classification method as described in any one of the first aspects of the present invention are implemented.

[0045] The present invention provides a fine-grained image classification method, device, electronic device and storage medium, which form dependencies in space and channels between features at multiple levels, and learn more discriminative features, so as to obtain more discriminative and better expressive features, and improve classification performance. The present invention proposes a plug-and-play module, Polarized Cross-layer Fusion Network (PCFN for short), which models high-resolution dependencies in space and channels across layers, with lower complexity than other methods, but achieves state-of-the-art performance. PCFN can adaptively focus on each part at multiple scales, and by establishing high- and low-level feature dependencies on the CNN backbone, extracting multi-scale features guided by high-level feature semantics, it can extract high-level semantics and low-level detailed information, thereby better classifying and accurately positioning.

[0046] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a process flow of a fine-grained image classification method provided by the present invention;

[0048] Figure 2 A schematic diagram of the structure of the feature extraction network used in the present invention;

[0049] Figure 3 A schematic diagram of the structure of the PCFN module used in the present invention;

[0050] Figure 4 A schematic diagram of the structure of a fine-grained image classification device provided by the present invention. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0052] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0053] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0054] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0055] In addition, in the description of this application, unless otherwise specified, "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0056] In view of the problems in the background technology, the present application embodiment provides a fine-grained image classification method, such as Figure 1 As shown, the method comprises the following steps:

[0057] S01: Obtain an image to be classified.

[0058] In a preferred embodiment, the image is a color picture and is scaled to the input size of the convolutional neural network.

[0059] S02: Inputting the image to be classified into a trained feature extraction network to obtain multi-scale features of the image; wherein the feature extraction network is a ResNet101 neural network into which a PCFN module is inserted.

[0060] In a preferred embodiment, the structure of the feature extraction network is as follows: Figure 2 As shown in the figure, ResNet101 is used as the candidate baseline, the PCFN module is inserted into the backbone network, the features of the last convolutional layer of Resnet are used as the query layer of the PCFN module, and the features of the first few layers of Resnet are used as the response layer of PCFN. Through the PCFN module, the shallow features of Resnet are semantically enhanced at a high level, and the shallow features after semantic enhancement are subjected to spatial dimensionality reduction and channel dimensionality increase operations before being fused with the high-level features.

[0061] The PCFN module described above can model the dependency between high-level features and shallow features in channels and spaces. The module contains two basic components: the query layer and the response layer. Based on the observation that neurons in different layers have different receptive fields, a query layer is associated with multiple response layers.

[0062] S03: Input the multi-scale features of the image into a classification network to obtain a fine-grained classification result of the image to be classified.

[0063] In a specific embodiment, the classification network may be a softmax function layer. In other examples, it may also be a classification network such as a support vector machine or a decision tree.

[0064] The present invention provides a fine-grained image classification method, which can learn more discriminative features by associating cross-layer features, and obtain more expressive multi-scale features by combining the characteristics of features at different layers, thereby improving the classification effect.

[0065] In conjunction with a specific embodiment, the working process and working principle of a fine-grained image classification method provided by the present invention are explained starting from the construction and training of a feature extraction network.

[0066] First, a convolutional neural network is established. The present invention uses ResNet101 as a candidate baseline and uses the PCFN module to perform multi-scale feature fusion to implement a classification model. The preprocessed image is sent to the convolution layer of the Resnet network. The last fully connected layer of Resnet is replaced with a fully connected layer whose output is consistent with the number of categories in the data set. The present invention establishes an association between the features of stage 2, 3 and the features of stage 4 of Resnet, that is, the features of stage 2, 3 are used as the response layer of PCFN and the features of stage 4 are used as the query layer. Figure 3 As shown in the figure, the PCFN module contains a channel association branch and a spatial association branch, through which the information of the query layer weighting the response layer in the two dimensions of the channel space is obtained respectively. By adding the module to the candidate baseline, the shallow features after semantic weighting can be obtained. The shallow features after semantic weighting are subjected to spatial dimensionality reduction operations through maximum pooling and channel-by-channel convolution, and then channel dimensionality is increased. Through spatial dimensionality reduction and channel dimensionality increase, the shallow features after semantic weighting are consistent with the deep features, and the deep and shallow features are fused through element-wise add operations. Based on this module, more discriminative multi-scale features can be obtained.

[0067] For the established feature extraction network model, it is necessary to train it using a data set.

[0068] The process of establishing the data set is as follows:

[0069] The CUB-200-2011, Stanford Cars, and FGVC-Aircraft datasets are used as external datasets. First, each dataset is divided into a training set and a test set. The training set and the test set should be mutually exclusive, that is, there is no intersection between the two subsets. Each image should contain a category label to indicate which category the image belongs to. For example, the CUB-200-2011 dataset has 5994 training sets and 5794 test sets. In addition, all images are scaled to the input size of the convolutional neural network.

[0070] In order to expand the data set and reduce overfitting, in a preferred embodiment, the data set needs to be preprocessed for data enhancement. There are usually four enhancement methods: translation, scaling, rotation, and flipping.

[0071] Specifically, the images in the training set are randomly cropped to 448x448 with a random area ratio of 0.08 to 1.25, and the PIL images are horizontally flipped with a default probability of 0.5. In addition, the scaled images are converted from the input format HxWxC and the pixel value [0,255] to the tensor format CxHxW with the pixel value in the range [0,1]. Among them, C represents the number of channels, and H and W represent the length and width of the image. For example, the images of the CUB-200-2011 dataset are RGB three-channel images, and their C=3. After that, the pixel values ​​are normalized and input into the network. The normalization formula is channel=(channel-mean) / std, and the mean of each channel is (0.485, 0.456, 0.406), and the variance (std) is (0.229, 0.224, 0.225). The preprocessing operation can achieve the purpose of data enhancement and make the network more robust to the features of objects of different forms.

[0072] Based on the feature extraction network and data set established above, the feature extraction network is trained using the training set, and the training results are tested and verified using the test set.

[0073] Preferably, each data set is trained for 150 epochs, and the gradual warm-up strategy is used in the first ten epochs of the training process. The batch size is set to 32, the basic learning rate is set to 0.01, and the fixed step decay learning rate decay strategy is adopted. The learning rate is reduced to one tenth of the original every 20 epochs. The SGD optimizer is used for training. When the loss function converges and no longer continues to decrease, the convolutional neural network model is saved to obtain the pre-trained convolution and the trained convolutional neural network for testing.

[0074] The accuracy of the model is used to evaluate the performance of the test set. The accuracy of the overall classification result is accuracy = TP / Total, where TP represents the number of correctly classified images and Total represents the total number of images.

[0075] Based on the above-trained feature extraction network, the data processing process for classifying an image to be classified includes:

[0076] Step 1: Receive the image and use it as the input data of the neural network and input it into the first layer of the Resnet101 neural network.

[0077] Step 2: Input the output features of layer2 into the subsequent PCFN module, such as Figure 3 The channel association branch and spatial association branch in are used as the response layer of the module.

[0078] At the same time, the output features of layer4 are input into the subsequent PCFN module as the query layer.

[0079] The PCFN module achieves semantic enhancement of layer2 features by associating high- and low-level features to obtain semantically enhanced output features.

[0080] Specifically, PCFN uses the following method to obtain semantically enhanced features for calculation:

[0081] PCFN has two input sources, namely high-level and low-level features. The high-level features are represented as The underlying features are represented as The output feature X of PCFN p for:

[0082] X p =Z ch +Z sp =A ch (X q ,X r )⊙ ch X r +A sp (X q ,X r )⊙ sp X r

[0083] in:

[0084] A ch (X q ,X r )=F SG [W z|θ ((σ1(W v (Xr)))×F SM (σ2(W q (F US (X q ))))))]

[0085] A sp (X q ,X r )=F SG [σ3(F SM (σ1(F GP (W q (F US (X q )))))×σ2(W v (X r )))]

[0086] Where Wq, Wv and Wz are 1×1 convolutional layers, σ1, σ2, σ3 are tensor reshaping operators;

[0087] FSM(·) is the SoftMax operator, and “×” is the matrix dot product operation; F US For X q In order to get r The upsampling operation of the same spatial dimension; FGP(·) is a global pooling operator, “×” is the dot product operation of the matrix. FSG(·) is the Sigmoid function;

[0088] A ch (X q ,X r ) is the channel operation that associates the query layer and the response layer, and obtains the weight of channel semantic enhancement. ch is a channel-level multiplication operator. A ch (X q ,X r )⊙ ch X r Used to obtain shallow features for channel semantic enhancement.

[0089] A sp (X q ,X r ) is the spatial operation that associates the query layer and the response layer, A sp (X q ,X r )⊙ sp X r Used to obtain shallow features for spatial semantic enhancement.

[0090] Step 3: Input the semantically weighted output features of step 2 into the dimension transformation module following the PCFN module to obtain output features that are consistent with the output feature space and channel dimensions of layer 4.

[0091] In step 4, the output features after the dimension change operation in step 3 and the output features of layer 4 are fused by element-wise add.

[0092] PCFN bi (X q ,X r )=X q +W z (F MP (X p )+W d (X p )) (2)

[0093] X p The phase is the feature of the response layer after the channel and spatial correlation query layer. MP (·) and W ZThe symbol is to perform spatial dimensionality reduction and subsequent channel dimensionality increase operation on the response layer features after the associated query layer features. The spatial dimensionality reduction adopts the maximum pooling, which can obtain the key information of the response layer features after the semantic weighting of the query layer. In order to avoid ignoring the local context information, the channel-by-channel convolution W is also used. d Get local context information.

[0094] Step 5: Repeat steps 2 to 4 for the output features of layer3.

[0095] Step 6: Input the output features of the last layer into the classification network.

[0096] Step 7: Take the output features of the classification network as the prediction results and output them.

[0097] The present application also provides a fine-grained image classification device, such as Figure 4 As shown, the fine-grained image classification device 400 includes:

[0098] An image acquisition module 401 is used to acquire an image to be classified;

[0099] A feature extraction module 402 is used to input the image to be classified into a trained feature extraction network to obtain multi-scale features of the image; wherein the feature extraction network is a ResNet101 neural network into which a PCFN module is inserted;

[0100] The classification module 403 is used to input the multi-scale features of the image into a classification network to obtain a fine-grained classification result of the image to be classified.

[0101] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0102] The present application also provides an electronic device, including:

[0103] at least one memory and at least one processor;

[0104] The memory is used to store one or more programs;

[0105] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the fine-grained image classification method as described above.

[0106] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0107] The present application also provides a computer-readable storage medium.

[0108] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the fine-grained image classification method as described above are implemented.

[0109] Computer usable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0110] The present invention provides a fine-grained image classification method, device, electronic device and storage medium, which can learn more discriminative features by associating cross-layer features, and combine the characteristics of features at different layers to obtain multi-scale features with more expressive ability, thereby improving the classification effect. The present invention provides a fine-grained image classification method, device, electronic device and storage medium, which form dependencies in space and channels between features at multiple levels, and learn more discriminative features, thereby obtaining more discriminative and better expressive features, thereby improving classification performance. The present invention proposes a plug-and-play module, Polarized Cross-layer Fusion Network (PCFN for short), which models high-resolution dependencies in space and channels across layers, with lower complexity than other methods, but achieves state-of-the-art performance. PCFN can adaptively focus on each part at multiple scales, and by establishing high- and low-level feature dependencies on the CNN backbone, extracting multi-scale features guided by high-level feature semantics, it can extract high-level semantics and low-level detailed information, thereby better classifying and accurately locating.

[0111] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A fine-grained image classification method, characterized in that: The following steps are involved: Get the image to be classified; Inputting the image to be classified into a trained feature extraction network to obtain multi-scale features of the image; wherein the feature extraction network is a ResNet101 neural network into which a PCFN module is inserted; Inputting the multi-scale features of the image into a classification network to obtain a fine-grained classification result of the image to be classified; The image is input into a trained feature extraction network to obtain multi-scale features of the image, including: The image is input into the first layer of the Resnet101 neural network, and then into layer1, layer2, layer3, and layer4 for convolution processing; The output features of layer2 and layer3 are respectively input into the subsequent PCFN modules to obtain semantically weighted output features; The semantically weighted output features of layer2 and layer3 are respectively input into the dimension transformation module to obtain output features consistent with the spatial dimension and channel dimension of the output features of layer4; The output features of the dimension transformation module and the output features of layer4 are subjected to feature fusion to obtain multi-scale features of the image; The output features of layer2 and layer3 are respectively input into the subsequent PCFN modules to obtain the semantically weighted output features, including: For any one of the output features of layer2 and layer3, the output feature is input into the channel association branch and the spatial association branch of the PCFN module as the response layer of the PCFN module; Perform an upsampling operation on the output features of the layer4, and input the channel association branch and the spatial association branch of the PCFN module as the query layer of the PCFN module; A weighted operation is performed on the channel association branch and the space association branch to obtain semantic weighted information of the output feature of layer4 to the output feature of any one of the output features of layer2 and layer3 in two dimensions of channel and space; Output the weighted information to obtain a semantically weighted output feature corresponding to any one of the output features of layer2 and layer3; The output features of the dimension transformation module and the output features of layer4 are subjected to feature fusion, including: The output features of the dimension transformation module and the output features of layer4 are fused in an element-wise add manner; PCFN uses the following method to obtain semantically enhanced features for calculation: PCFN has two input sources, namely high-level and low-level features; high-level features are represented as The underlying features are represented as The output feature X of PCFN p for: X p =Z ch +Z sp =A ch (X q ,X r )⊙ ch X r +A sp (X q ,X r )⊙ sp X r in: A ch (X q ,X r )=F SG [W z|θ ((σ1(W v (X r )))×F SM (σ2(W q (F US (X q )))))] Where Wq, Wv and Wz are 1×1 convolutional layers, σ1, σ2, σ3 are tensor reshaping operators; FSM(·) is the SoftMax operator, and "×" is the matrix dot product operation; F US For X q In order to get r The upsampling operation of the same spatial dimension; FGP(·) is a global pooling operator, "×" is the dot product operation of the matrix; FSG(·) is the Sigmoid function; A ch (X q ,X r ) is the channel operation that associates the query layer and the response layer, and obtains the weight of channel semantic enhancement; ⊙ ch is a channel-level multiplication operator; A ch (X q ,X r )⊙ ch X r Used to obtain shallow features for channel semantic enhancement; A sp (X q ,X r ) is the spatial operation that associates the query layer and the response layer, A sp (X q ,X r )⊙ sp X r Used to obtain shallow features for spatial semantic enhancement; Step 3: Input the semantically weighted output features of step 2 into the dimension transformation module following the PCFN module to obtain output features consistent with the output feature space and channel dimensions of layer 4; Step 4: The output features after the dimension change operation in step 3 and the output features of layer 4 are fused by element-wise add. PCFN bi (X q ,X r )=X q +W z (F MP (X p )+W d (X p )) (2) X p Phase is the feature of the response layer after the channel and spatial correlation query layer; F MP (·) and W Z The symbol is to perform spatial dimensionality reduction and subsequent channel dimensionality increase operation on the response layer features after the associated query layer features; the spatial dimensionality reduction adopts the maximum pooling, which can obtain the key information of the response layer features after the semantic weighting of the query layer. In order to avoid ignoring the local context information, the channel-by-channel convolution W is also used. d Get local context information.

2. A fine-grained image classification method according to claim 1, characterized in that: The training steps of the feature extraction network include: Obtaining a training data set for the feature extraction network, and dividing the training data set into a training set and a test set; The feature extraction network is trained using the training set, and the SGD optimizer is used for learning. The loss function decreases continuously with the back propagation of the error, and the training accuracy rate increases continuously. When the loss function converges and no longer decreases, a pre-trained feature extraction network is obtained; The pre-trained feature extraction network is tested using the test set, and when the accuracy of the test result reaches a preset threshold, the pre-trained feature extraction network is saved to obtain a trained feature extraction network.

3. A fine-grained image classification method according to claim 2, characterized in that: The parameters of the feature extraction network trained using the training set include: Each dataset was trained for 150 epochs, with a gradual warm-up strategy used in the first ten epochs of the training process; The batch size is set to 32 and the base learning rate is set to 0.01; A fixed step size decay learning rate decay strategy is adopted, and the learning rate is reduced to one tenth of the original every 20 epochs.

4. The fine-grained image classification method according to claim 2, characterized in that: Acquiring a training data set for the feature extraction network also includes: Performing a preprocessing operation on the training data set to obtain a data-enhanced training data set; The preprocessing operations include: translation, scaling, rotation and flipping.

5. A fine-grained image classification device for implementing a fine-grained image classification method as claimed in any one of claims 1 to 4, characterized in that: include: An image acquisition module, used for acquiring images to be classified; A feature extraction module, used for inputting the image to be classified into a trained feature extraction network to obtain multi-scale features of the image; wherein the feature extraction network is a ResNet101 neural network into which a PCFN module is inserted; The classification module is used to input the multi-scale features of the image into a classification network to obtain a fine-grained classification result of the image to be classified.

6. An electronic device, characterized in that: include: at least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a fine-grained image classification method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a fine-grained image classification method as described in any one of claims 1 to 4 are implemented.

Images