Zero sample classification method for animal image based on multi-scale feature fusion

Abstract

The invention discloses a zero sample classification method for animal images based on multi-scale feature fusion, and the method comprises the steps: carrying out the mining of local features in a zero sample classification task through employing different scale features of a third layer and a fourth layer of ResNet101; inputting the features of the third layer and the fourth layer into a multi-scale feature enhancement module to enhance feature discrimination; a cross-layer fusion module is set, interaction is generated by learning features of opposite layers, and the generalization ability of the model is enhanced; and fusing the features of the same layer, and inputting the fused features into an attribute attention module. According to the method, feature information of different layers is fully utilized, prejudice of the model to visible classes in a zero sample classification task is reduced, and the generalized zero sample and zero sample classification accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the field of zero-sample learning, and in particular relates to a zero-sample classification method for animal images based on multi-scale feature fusion. Background Art

[0002] Zero-shot learning (ZSL) is a key research direction in the field of machine learning. Traditional supervised learning usually requires exposure to all possible category sample data during the training phase, but in reality, it is not feasible to obtain sample data of all categories. This leads to an important challenge: when the model encounters unseen categories in practical applications, traditional methods fail because they cannot learn information about new categories from existing data. In practical scenarios, sample data of new categories may be extremely limited or even non-existent. For example, an animal recognition model may only be exposed to image data of cats and dogs during the training phase, but in reality it may encounter animals that the model has never seen, such as zebras and elephants. In this case, traditional supervised learning models cannot correctly classify or recognize these new categories.

[0003] To solve this problem, the emergence of zero-shot learning has become inevitable. Its core idea is to use the information of known categories to assist the model in inferring and classifying unseen categories. This method is regarded as a manifestation of generalization ability. Even when there is no directly observed category, the model can make reasonable inferences. To achieve this goal, researchers have tried a variety of methods and techniques. A key idea of ​​zero-shot learning is the attribute-based method. This method believes that each category can be described by a set of attributes. For example, animals can be characterized by attributes such as hair color and body size. By using the attribute information of known categories, the model can infer the attributes of new categories, thereby realizing the recognition of unknown categories. Another important idea is generative models, such as generative adversarial networks (GANs). Generative models can generate new samples similar to training data. Even if there is no directly observed category, samples of the category can be generated through the generative model to help the model learn the characteristics of the new category.

[0004] Zero-shot learning can solve the problems of limitations and data scarcity in traditional supervised learning. And zero-shot learning has its own advantages. When faced with unknown concepts, humans are often able to infer and generalize based on existing knowledge. Zero-shot learning attempts to imitate the human cognitive process, especially the ability to infer when faced with new concepts.

[0005] In zero-shot classification tasks, it is very important to capture and mine local information. Local information is usually associated with specific categories or attributes. In zero-shot classification, even if samples of a certain category have not been seen, local information can still provide important clues about the characteristics of that category. Local information such as texture, shape, edges, etc. can help provide a more specific and detailed image description. Through this information, the model can better understand the image content and associate it with known categories. And local information can provide more discriminative features, which helps to distinguish the differences between different categories. In zero-shot classification, these features can help the model accurately identify and classify unknown categories and improve the generalization ability of the model. Summary of the invention

[0006] The purpose of the present invention is to provide a zero-sample classification method for animal images based on multi-scale feature fusion.

[0007] The technical solution to achieve the purpose of the present invention is: a zero-sample classification method for animal images based on multi-scale feature fusion, comprising the following steps:

[0008] Step S1: Input an animal image x into the ResNet101 network and obtain the feature output f of the third and fourth layers of the network 3 and f 4 ;

[0009] Step S2: Extract the feature f 3 and f 4 Input them into the multi-scale feature enhancement module respectively to obtain the output f of the multi-scale feature enhancement module ` 3 and f ` 4 ;

[0010] Step S3: Considering the information interaction between the features of the two layers, a cross-layer fusion module is established to transform f` 3 and f` 4 Input together to get the output f`` 3 and f`` 4 ;

[0011] Step S4: In order not to lose the feature information of each original dimension, f` 3 and f`` 3 Fusion obtains f 3_final ,f` 4 and f`` 4 Fusion obtains f 4_final ;

[0012] Step S5: 3_final and f 4_finalInput the attribute attention module respectively, introduce the semantic word vector obtained by the Glove model into the module, and obtain the loss L reg3 and L reg4 ;

[0013] Step S6: 3_final and f 4_final Input the embedding layer respectively, calculate the cosine similarity, and obtain the loss L cls3 and L cls4 , and the classification results and

[0014] Step S7: and Fusion Acquisition As the final output result, the animal category.

[0015] Furthermore, the step S1 includes: for the image data set Where x is the input image and y is the corresponding true sample label. is the attribute information contained in the data set, X is the data set, Y s is the category label of the visible class. These images are used as input in the training phase to train and update the network; in the test phase, there will be y∈Y u The image is taken as input, Y u is the category label of the unseen class, is known, but {y|y∈Y u} is unknown;

[0016] Input an image into the ResNet101 network and select the features of the third and fourth layers as output, i.e., f 3 and f 4 , The ResNet101 network reads the parameters pre-trained on other image datasets, and the parameters of the entire network are updated during training.

[0017] Furthermore, the step S2 includes: extracting the feature f 3 and f 4 The inputs are respectively input into the multi-scale feature enhancement module, which contains five branches to process the input features and finally concatenate the five branches to obtain the output f after feature enhancement. ` 3 and f ` 4 ;

[0018] For the input feature f 3 or 4, input five different branches; the first branch is a convolution with a convolution kernel size of 1 and obtains the output, and the other three branches are three dilated convolutions with a convolution kernel size of 3. The dilation rates of the dilated convolutions in different branches are different, which are [3, 6, 9] respectively. Different dilation rates will also have different convolution padding to ensure the consistency of the input feature and output feature sizes:

[0019]

[0020]

[0021] Among them, H in , W in are the length and width of the input features, H out , W out are the length and width of the output features, padding is the padding, dilation is the dilation rate, kernel and stride are the convolution kernel size and stride of the convolution respectively;

[0022] The last branch includes adaptive mean pooling, convolution with kernel size 1, and nearest neighbor interpolation upsampling operation; finally, the outputs of the five branches are concatenated to the same dimension as the input features to obtain the output f ` 3 or ` 4 .

[0023] Furthermore, the step S3 includes:

[0024] The input f ` 3 and f ` 4 Through two different 1×1 convolution kernels, the dimensions are transformed to each other’s dimensions, that is, f ` 3 Transform the number of channels to 2048 and obtain f ` 4 Change the number of channels to 1024 and get The same number of channels f ` 3 and Perform cross multiplication to obtain a fusion matrix, which represents the similarity of features between different layers, and then and Multiply them with the fusion matrix respectively to get the final output f ` 3 ` and f ` 4 ` .

[0025] Furthermore, step S4 includes: fusing the original features of each layer and combining the features of the other layer information, and obtaining the feature f through an adaptive parameter α. 3_final and f 4_final

[0026] f final =α*f`+(1-α)*f``.

[0027] Furthermore, the step S5 includes: inputting the attribute description into the Glove model to obtain an output V with a dimension of K×300, where K is the number of attributes in the data set; obtaining V` from the semantic word vector V through a set dimension conversion layer T, where V` represents the semantic information of the attribute; and combining V` with the input visual feature f 3_final or 4_final After cross-producting, we obtain the attribute information that the network pays attention to from the visual features, namely the attention map M. M k is the attention map for the kth attribute, k∈K, W×H is the size of the attention map, with width W and height H; then the attention map M is max-pooled to obtain the network’s attention attribute for the input image x.

[0028]

[0029] Where i and j represent different pixel positions on the attention map; the attention attribute The true value of the attribute with the image Calculate the mean square error:

[0030]

[0031] So we get the loss L reg3 and L reg4 .

[0032] Furthermore, the step S6 comprises: 3_final and f 4_final Input different embedding layers separately C×K is the dimension of the embedding layer, C is the number of channels of the input feature, and K is the number of attributes in the data set. The cosine similarity measurement is then used to obtain the classification result. and And calculate L through cross entropy loss cls3 and L cls4 :

[0033]

[0034] Where σ is the scaling factor.

[0035] Furthermore, the step S7 includes: and Combine in different proportions and traverse

[0036] {0.1:0.9,0.2:0.8,…,0.9:0.1} select the best combination Output as the final classification result.

[0037] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0038] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0039] Compared with the prior art, the significant advantages of the present invention are as follows: the present invention designs a universal zero-shot classification network, which uses feature information of different layers to mine and obtain local information, thereby improving the accuracy of the network. At the same time, by using feature information of different scales at different layers, the features between different layers of the network can be complementary, which is conducive to improving the generalization ability of the model, and can improve the embedded zero-shot classification network and enhance the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is an architecture diagram of a zero-shot classification method based on multi-scale feature fusion.

[0041] Figure 2 Schematic diagram of the multi-scale feature enhancement module.

[0042] Figure 3 Schematic diagram of the cross-layer fusion module.

[0043] Figure 4 This is a schematic diagram of the attribute attention module. DETAILED DESCRIPTION

[0044] Multi-scale feature extraction can extract features from different resolutions or layers of an image. For local features, this means being able to capture detailed information at different scales, from subtle textures at smaller scales to overall shapes at larger scales. It can enhance the robustness of local features in an image. Features at different scales can complement and verify each other, making the model more stable and reliable. Features extracted from different scales help provide more diverse and varied local feature descriptions. These features at different scales can enrich the model's understanding of local features and help distinguish and identify different categories.

[0045] The purpose of this invention is to utilize multi-scale information in zero-shot classification tasks, make full use of feature information of different scales, so as to learn better local information and enhance the discriminability and robustness of the model.

[0046] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0047] like Figure 1 As shown, a zero-shot classification method for animal images based on multi-scale feature fusion specifically includes the following steps:

[0048] S1: For image dataset Where x is the input image and y is the corresponding true sample label. It is the attribute information contained in the data set. These images are used as input in the training phase to train and update the network. In the test phase, there will be y∈Y u As input, is known, but {y|y∈Y u} is unknown.

[0049] Input an image into the ResNet101 network and select the features of the third and fourth layers as output, i.e., f 3 and f 4 , C represents the number of channels, f 3 In the middle is 1024, f 4 is 2048, W×H represents the image size, f 3 is 28×28, f 4 The ResNet101 network reads the parameters pre-trained on other large image datasets, and the parameters of the entire network are updated during training.

[0050] S2: Extract the features f 3 and f 4 The inputs are respectively input into the multi-scale feature enhancement module, which contains five branches to process the input features and finally concatenate the five branches to obtain the output f after feature enhancement. ` 3 and f ` 4 .

[0051] like Figure 2 Shown is a schematic diagram of the multi-scale feature enhancement module.

[0052] Wherein, step S2 is specifically as follows:

[0053] For the input feature f 3 or 4, input five different branches. The first branch is a convolution with a kernel size of 1 and obtains the output. The other three branches are three dilated convolutions with a kernel size of 3. The dilation rates of the dilated convolutions in different branches are different, which are [3, 6, 9] respectively. Different dilation rates will also have different convolution padding to ensure the consistency of the input feature and output feature sizes:

[0054]

[0055]

[0056] Among them, H in , W in are the length and width of the input features, H out , W out are the length and width of the output features, padding is the padding, dilation is the dilation rate, kernel and stride are the convolution kernel size and step size, respectively.

[0057] The last branch includes adaptive mean pooling, convolution with kernel size 1, and nearest neighbor interpolation upsampling. Finally, the outputs of the five branches are concatenated to the same dimension as the input features to obtain the output f ` 3 or ` 4 .

[0058] S3: f ` 3 and f ` 4 Input the cross-layer fusion module together, and change f through different convolutions in the module ` 3 and f ` 4 The dimension of the matrix is ​​obtained, and the fusion matrix is ​​obtained. Then the matrix is ​​multiplied with the feature after the transformation dimension to obtain the feature f that combines the information of the other layer. ` 3 ` and f ` 4 ` .

[0059] like Figure 3 Shown is a schematic diagram of the cross-layer fusion module.

[0060] Wherein, step S3 is specifically as follows:

[0061] The input f ` 3 and f ` 4 Through two different 1×1 convolution kernels, the dimensions are transformed to each other’s dimensions, that is, f` 3 Transform the number of channels to 2048 and obtain f ` 4 Change the number of channels to 1024 and get The same number of channels f ` 3 and Perform cross multiplication to obtain a fusion matrix, which represents the similarity of features between different layers, and then and Multiply them with the fusion matrix respectively to get the final output f`` 3 and f`` 4 , where f` 3 and f`` 3 ,f` 4 and f`` 4 The dimensions are consistent.

[0062] S4: Features are fused through an adaptive parameter α:

[0063] f final =α*f`+(1-α)*f``

[0064] Where f` and f`` are the outputs of steps S2 and S3, and are operated between the features of the same layer to obtain f 3_final and f 4_final .

[0065] S5: Put f 3_final or 4_final The semantic word vectors in the module are obtained by inputting the attributes in the dataset, such as the sharp beak and white belly in the CUB dataset, into the Glove model and outputting them. 3_final or 4_final After combining with the semantic word vector, the mean square error loss is calculated with the attribute true value to obtain L reg3 and L reg4 .

[0066] like Figure 4 This is a schematic diagram of the attribute attention module.

[0067] Wherein, step S5 is specifically as follows:

[0068] Input the attribute description into the Glove model, and the output V is K×300, where K is the number of attributes in the dataset. The semantic word vector V is obtained by passing it through the set dimension conversion layer T. ` ,V ` Represents the semantic information of the attribute. ` With the input visual feature f 3_final or4_final After cross-producting, we obtain the attribute information that the network pays attention to from the visual features, namely the attention map M. Then perform maximum pooling on the attention map M to obtain the network's attention attribute on the input image x

[0069]

[0070] Where i and j represent different pixel positions on the attention map. The true value of the attribute with the image Calculate the mean square error:

[0071]

[0072] So we get the loss L reg3 and L reg4 .

[0073] S6: Put f 3_final and f 4_final Input different embedding layers separately Transfer features from feature space to attribute space, and then perform cosine similarity measurement to obtain classification results and And calculate L through cross entropy loss cls3 and L cls4 :

[0074]

[0075] Where σ is the scaling factor.

[0076] The step S7 comprises: and Combine in different proportions, traverse {0.1:0.9,0.2:0.8,…,0.9:0.1} to select the best combination The final classification result is output as the animal category.

[0077] The following table shows the performance comparison with the advanced methods in recent years. Experiments were conducted on two animal datasets, CUB and AWA2, and compared with this year's embedded methods. The results of zero-shot recognition (ZSL) and generalized zero-shot recognition (GZSL) were compared. The highest accuracy (T1) was selected under zero-shot recognition, and the average accuracy (H) of the seen and unseen classes was selected under generalized zero-shot recognition.

[0078]

Claims

1. A zero-shot classification method for animal images based on multi-scale feature fusion, characterized in that: The method comprises the following steps: Step S1: Input an animal image x into the ResNet101 network and obtain the feature outputs f3 and f4 of the third and fourth layers of the network; Step S2: input the extracted features f3 and f4 into the multi-scale feature enhancement module respectively to obtain outputs f'3 and f'4 of the multi-scale feature enhancement module; Step S3: Considering the information interaction between the features of the two layers, a cross-layer fusion module is set up to input f`3 and f`4 together to obtain outputs f``3 and f``4; Step S4: In order not to lose the feature information of their original dimensions, f`3 and f``3 are fused to obtain f 3_final , f`4 and f``4 are fused to obtain f 4_final ; Step S5: 3_final and f 4_final Input the attribute attention module respectively, introduce the semantic word vector obtained by the Glove model into the module, and obtain the loss L reg3 and L reg4 ; Step S6: 3_final and f 4_final Input the embedding layer respectively, calculate the cosine similarity, and obtain the loss L cls3 and L cls4 , and the classification results and Step S7: and Fusion Acquisition As the final output result, the animal category.

2. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 1, characterized in that: The step S1 comprises: for the image data set Where x is the input image and y is the corresponding true sample label. is the attribute information contained in the data set, X is the data set, Y s is the category label of the visible class. These images are used as input in the training phase to train and update the network; in the test phase, there will be y∈Y u The image is taken as input, Y u is the category label of the unseen class, is known, but {y|y∈Y u } is unknown; Input an image into the ResNet101 network and select the features of the third and fourth layers as output, i.e. f3 and f4. The ResNet101 network reads the parameters pre-trained on other image datasets, and the parameters of the entire network are updated during training.

3. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 2, characterized in that: The step S2 comprises: the extracted features f3 and f4 are respectively input into a multi-scale feature enhancement module, the module comprises five branches, the input features are processed, and finally the five branches are spliced ​​to obtain feature enhanced outputs f'3 and f'4; For the input feature f3 or f4, five different branches are input; the first branch is a convolution with a kernel size of 1 and obtains the output, and the other three branches are three dilated convolutions with a kernel size of 3. The dilation rates of the dilated convolutions in different branches are different, which are [3, 6, 9] respectively. Different dilation rates will also have different convolution padding to ensure the consistency of the input feature and output feature sizes: Among them, H in , W in are the length and width of the input features, H out , W out are the length and width of the output features, padding is the padding, dilation is the dilation rate, kernel and stride are the convolution kernel size and stride of the convolution respectively; The last branch includes adaptive mean pooling, convolution with a kernel size of 1, and nearest neighbor interpolation upsampling operations; finally, the outputs of the five branches are concatenated to the same dimension as the input features to obtain the output f`3 or f`4.

4. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 3, characterized in that: The step S3 comprises: The input f`3 and f`4 are transformed into each other's dimensions through two different 1×1 convolution kernels, that is, the number of channels of f`3 is transformed to 2048, and we get Transform f4 to 1024 channels and obtain f3 and Perform cross multiplication to obtain a fusion matrix, which represents the similarity of features between different layers, and then and Multiply them with the fusion matrix respectively to obtain the final output f``3 and f``4.

5. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 4, characterized in that: The step S4 includes: fusing the original features of each layer and combining the features of the other layer information, and obtaining the feature f through an adaptive parameter α 3_final and f 4_final f final = α*f`+(1-α)*f``.

6. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 5, characterized in that: The step S5 includes: inputting the attribute description into the Glove model to obtain an output V with a dimension of K×300, where K is the number of attributes in the data set; obtaining V' through a set dimension conversion layer T for the semantic word vector V, where V' represents the semantic information of the attribute; and combining V' with the input visual feature f 3_final or 4_final After cross-producting, we obtain the attribute information that the network pays attention to from the visual features, namely the attention map M. M k is the attention map for the kth attribute, k∈K, W×H is the size of the attention map, with width W and height H; then the attention map M is max-pooled to obtain the network’s attention attribute for the input image x. Where i and j represent different pixel positions on the attention map; the attention attribute The true value of the attribute with the image Calculate the mean square error: So we get the loss L reg3 and L reg4 .

7. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 6, characterized in that: The step S6 comprises: 3_final and f 4_final Input different embedding layers separately C×K is the dimension of the embedding layer, C is the number of channels of the input feature, and K is the number of attributes in the data set. The cosine similarity measurement is then used to obtain the classification result. and And calculate L through cross entropy loss cls3 and L cls4 : Where σ is the scaling factor.

8. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 7, characterized in that: The step S7 comprises: and Combine in different proportions, traverse {0.1:0.9, 0.2:0.8, ..., 0.9:0.1} to select the best combination Output as the final classification result.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images