Zero-shot classification method for animal images based on multi-scale feature fusion

Abstract

The application discloses a zero-shot classification method for animal images based on multi-scale feature fusion, and the method comprises the following steps: mining local features in a zero-shot classification task by using different scale features of the third layer and the fourth layer of ResNet101; inputting the third layer features and the fourth layer features into a multi-scale feature enhancement module to enhance the discriminativeness of the features; setting a cross-layer fusion module to learn the features of the other layer to generate interaction and enhance the generalization ability of the model; and fusing the features of the same layer and inputting the features into an attribute attention module. The application fully utilizes the feature information of different layers, reduces the bias of the model to visible classes in the zero-shot classification task, and improves the accuracy of generalized zero-shot classification and zero-shot classification.

Description

Technical Field

[0001] This invention belongs to the field of zero-shot learning, specifically relating to a zero-shot classification method for animal images based on multi-scale feature fusion. Background Technology

[0002] Zero-shot learning (ZSL) is a key research direction in machine learning. Traditional supervised learning typically requires exposure to sample data of all possible categories during the training phase, but in reality, obtaining sample data for all categories is impractical. This leads to a significant challenge: when the model encounters an unseen category in a real-world application, traditional methods fail because they cannot learn information about the new category from the existing data. In real-world scenarios, sample data for new categories may be extremely limited or even nonexistent. For example, an animal recognition model might only be exposed to image data of cats and dogs during training, but in reality, it might encounter animals such as zebras and elephants that the model has never seen before. In such cases, traditional supervised learning models cannot correctly classify or recognize these new categories.

[0003] To address this problem, zero-shot learning became inevitable. Its core idea is to use information from known categories to assist the model in inferring and classifying unseen categories. This approach is considered a manifestation of generalization ability; even without directly observed categories, the model can make reasonable inferences. To achieve this goal, researchers have tried various methods and techniques. A key idea in zero-shot learning is attribute-based methods. This approach assumes that each category can be described by a set of attributes; for example, animals can be characterized by attributes such as fur color and body size. By utilizing attribute information from known categories, the model can infer the attributes of new categories, thus achieving the identification of unknown categories. Another important approach is generative models, such as Generative Adversarial Networks (GANs). Generative models can generate new samples similar to the training data. Even without directly observed categories, the generative model can generate samples of that category, helping the model learn the features of new categories.

[0004] Zero-shot learning can address the limitations and data scarcity issues inherent in traditional supervised learning. Furthermore, zero-shot learning possesses its own inherent potential. When faced with unknown concepts, humans are often able to infer and generalize based on existing knowledge. Zero-shot learning attempts to mimic human cognitive processes, particularly the inference ability when encountering new concepts.

[0005] In zero-shot classification tasks, capturing and mining local information is crucial. Local information is typically associated with a specific category or attribute. In zero-shot classification, even without having seen samples of a particular category, local information can still provide important clues about the characteristics of that category. Local information such as texture, shape, and edges helps provide a more specific and detailed description of the image. Through this information, the model can better understand the image content and associate it with known categories. Furthermore, local information can provide more discriminative features, helping to distinguish differences between different categories. In zero-shot classification, these features can help the model accurately identify and classify unknown categories and improve the model's generalization ability. Summary of the Invention

[0006] The purpose of this invention is to provide a zero-shot classification method for animal images based on multi-scale feature fusion.

[0007] The technical solution to achieve the objective of this invention is: a zero-shot classification method for animal images based on multi-scale feature fusion, comprising the following steps:

[0008] Step S1: Input an animal image x into the ResNet101 network and obtain the feature outputs f3 and f4 of the third and fourth layers of the network;

[0009] Step S2: Input the extracted features f3 and f4 into the multi-scale feature enhancement module respectively, and obtain the output f of the multi-scale feature enhancement module. ` 3 and f ` 4;

[0010] Step S3: Considering the information interaction between the features of the two layers, a cross-layer fusion module is set up to input f`3 and f`4 together to obtain the outputs f``3 and f``4;

[0011] Step S4: To avoid losing the feature information of their original dimensions, f`3 and f``3 are fused to obtain f. 3_final f`4 and f``4 are fused to obtain f 4_final ;

[0012] Step S5: f 3_final and f 4_final The input is attribute attention module, which incorporates semantic word vectors obtained from the GloVe model, and the loss L is obtained. reg3 and L reg4 ;

[0013] Step S6: f 3_final and f 4_final The inputs are then fed into the embedding layer, and cosine similarity is calculated to obtain the loss L. cls3 and L cls4 and classification results and

[0014] Step S7: [The sentence is incomplete and requires more context to be translated accurately.] and Fusion The final output is the animal category.

[0015] Furthermore, step S1 includes: for the image dataset Where x is the input image and y is its corresponding ground truth sample label. It refers to the attribute information contained in the dataset, where X is the dataset and Y is the attribute information contained in the dataset. s These images, representing the category labels for the visible class, are used as input during the training phase to train and update the network; during the testing phase, y∈Y... u The image is used as input, Y u Category labels for invisible classes. It is known, but {y|y∈Y} u It is unknown;

[0016] An image is input into a ResNet101 network, and features from the third and fourth layers are selected as the output, namely f3 and f4. The ResNet101 network reads parameters pre-trained on other image datasets, and the parameters of the entire network are updated as it is trained.

[0017] Furthermore, step S2 includes: inputting the extracted features f3 and f4 into a multi-scale feature enhancement module, which contains five branches to process the input features, and finally concatenating the five branches to obtain the feature-enhanced output f. ` 3 and f ` 4;

[0018] For input features f3 or f4, five different branches are input; the first branch is a convolution with a kernel size of 1 to obtain the output, and the other three branches are three dilated convolutions with a kernel size of 3. The dilation rates of the dilated convolutions in different branches are different, namely [3, 6, 9]. Different dilation rates will also have different convolution padding to ensure that the input feature and output feature sizes are consistent.

[0019]

[0020]

[0021] Where H in W in H represents the length and width of the input feature, respectively. out W out, respectively, represent the length and width of the output feature, padding is the padding, dilation is the dilation rate, and kernel and stride are the kernel size and stride of the convolution, respectively;

[0022] The last branch includes adaptive mean pooling, convolution with a kernel size of 1, and nearest neighbor interpolation upsampling; finally, the outputs of the five branches are concatenated to the same dimension as the input features to obtain the output f. ` 3 or f ` 4.

[0023] Furthermore, step S3 includes:

[0024] Input f ` 3 and f ` 4. By using two different 1×1 convolution kernels, the dimension is transformed to the other dimension, that is, for f ` 3. Transform its channel count to 2048 to obtain For f ` 4. Transform its channel count to 1024 to obtain f with the same number of channels ` 3 and Perform a cross product to obtain a fusion matrix, representing the similarity of features between different layers, and then... and Multiply each of these by the fusion matrix to obtain the final output f. ` 3 ` and f ` 4 ` .

[0025] Furthermore, step S4 includes: fusing the original features from each layer and the features combined with information from other layers, and obtaining feature f through an adaptive parameter α. 3_final and f 4_final

[0026] f final =α*f`+(1-α)*f``.

[0027] Further, step S5 includes: inputting the attribute description into the GloVe model to obtain the output V, which has a dimension of K×300, where K is the number of attributes in the dataset; passing the semantic word vector V through a set dimension transformation layer T to obtain V', where V' represents the semantic information of the attribute; and comparing V' with the input visual features f. 3_final or f 4_final After performing cross products, the attribute information that the network focuses on from visual features is obtained, i.e., the attention map M. M kFor the attention map of the k-th attribute, k∈K, W×H is the size of the attention map, with width W and height H; then, max pooling is performed on the attention map M to obtain the network's attention attributes for the input image x.

[0028]

[0029] Where i and j represent different pixel positions on the attention map; the attributes to be focused on The true values ​​of the attributes that an image possesses Calculate the mean square error:

[0030]

[0031] Thus, the loss L is obtained reg3 and L reg4 .

[0032] Furthermore, step S6 includes: f 3_final and f 4_final Input different embedding layers respectively C×K represents the embedding layer dimension, where C is the number of channels in the input features and K is the number of attributes in the dataset. The classification result is then obtained by applying a cosine similarity measure. and L is calculated using cross-entropy loss. cls3 and L cls4 :

[0033]

[0034] Where σ is the scaling factor.

[0035] Furthermore, step S7 includes: classifying the results and Combine in different proportions and traverse

[0036] Select the optimal combination from {0.1:0.9,0.2:0.8,…,0.9:0.1}. This will be output as the final classification result.

[0037] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0038] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the above-described method.

[0039] Compared with existing technologies, the significant advantages of this invention are as follows: This invention designs a general zero-shot classification network, which improves the accuracy of the network by utilizing feature information from different layers to mine and obtain local information. Simultaneously, utilizing feature information at different scales from different layers allows for feature complementarity between different layers, which is beneficial for improving the model's generalization ability. This can enhance embedded zero-shot classification networks and strengthen the model's robustness. Attached Figure Description

[0040] Figure 1 This is an architecture diagram of a zero-shot classification method based on multi-scale feature fusion.

[0041] Figure 2 This is a schematic diagram of the multi-scale feature enhancement module.

[0042] Figure 3 This is a schematic diagram of a cross-layer fusion module.

[0043] Figure 4 This is a schematic diagram of the attribute attention module. Detailed Implementation

[0044] Multi-scale feature extraction enables the extraction of features from different resolutions or levels of an image. For local features, this means capturing detailed information at different scales, from subtle textures at smaller scales to overall shape at larger scales. This enhances robustness to local features in an image. Features at different scales can complement and validate each other, making the model more stable and reliable. Features extracted from different scales help provide more diverse and varied descriptions of local features. These features at different scales enrich the model's understanding of local features, helping to distinguish and identify different categories.

[0045] The purpose of this invention is to utilize multi-scale information in zero-shot classification tasks, making full use of feature information at different scales to learn better local information and enhance the model's discriminative power and robustness.

[0046] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] like Figure 1 As shown, a zero-shot classification method for animal images based on multi-scale feature fusion specifically includes the following steps:

[0048] S1: For image datasets Where x is the input image and y is its corresponding ground truth sample label. This refers to the attribute information contained in the dataset. These images are used as input during the training phase to train and update the network. During the testing phase, there will be y∈Y. u The image is used as input. It is known, but {y|y∈Y} u It is unknown.

[0049] An image is input into a ResNet101 network, and features from the third and fourth layers are selected as the output, namely f3 and f4. C represents the number of channels, which is 1024 in f3 and 2048 in f4. W×H represents the image size, which is 28×28 in f3 and 14×14 in f4. The ResNet101 network reads parameters pre-trained on other large image datasets, and the parameters of the entire network are updated during training.

[0050] S2: The extracted features f3 and f4 are input into the multi-scale feature enhancement module, which contains five branches to process the input features. Finally, the five branches are concatenated to obtain the feature-enhanced output f. ` 3 and f ` 4.

[0051] like Figure 2 The diagram shown is a schematic of the multi-scale feature enhancement module.

[0052] Specifically, step S2 is as follows:

[0053] For input features f3 or f4, five different branches are input. The first branch is a convolution with a kernel size of 1 to obtain the output. The other three branches are three dilated convolutions with a kernel size of 3. The dilation rates of the dilated convolutions in different branches are different, namely [3, 6, 9]. Different dilation rates will also have different convolution padding to ensure that the input and output feature sizes are consistent.

[0054]

[0055]

[0056] Where H in W in H represents the length and width of the input feature, respectively. out W out , , represent the length and width of the output feature, respectively; padding represents the padding; dilation represents the dilation rate; and kernel and stride represent the kernel size and stride of the convolution, respectively.

[0057] The final branch includes adaptive mean pooling, a convolution with a kernel size of 1, and nearest neighbor interpolation upsampling. Finally, the outputs of the five branches are concatenated to the same dimension as the input features to obtain the output f. ` 3 or f ` 4.

[0058] S3: f ` 3 and f` 4. Input the data into the cross-layer fusion module together, and change f using different convolutions within the module. ` 3 and f ` The dimension is reduced to 4, and a fusion matrix is ​​obtained. Then, the matrix is ​​multiplied by the features after the dimension transformation to obtain the feature f that incorporates information from the other layer. ` 3 ` and f ` 4 ` .

[0059] like Figure 3 The diagram shown is a schematic of the cross-layer fusion module.

[0060] Specifically, step S3 is as follows:

[0061] Input f ` 3 and f ` 4. By using two different 1×1 convolution kernels, the dimension is transformed to the other dimension, that is, for f ` 3. Transform its channel count to 2048 to obtain For f ` 4. Transform its channel count to 1024 to obtain f with the same number of channels ` 3 and Perform a cross product to obtain a fusion matrix, representing the similarity of features between different layers, and then... and Multiply by the fusion matrix respectively to obtain the final outputs f``3 and f``4, where the dimensions of f`3 and f``3, and f`4 and f``4 are the same.

[0062] S4: Features are fused using an adaptive parameter α.

[0063] f final =α*f`+(1-α)*f``

[0064] Where f` and f`` are the outputs of steps S2 and S3, respectively, and are obtained by performing operations between features at the same layer. 3_final and f 4_final .

[0065] S5: f 3_final or f 4_final The inputs are attribute-focused modules. The semantic word vectors within these modules are obtained by inputting attributes from the dataset, such as the sharp beak and white belly information from the CUB dataset, into the GloVe model and then outputting the feature f. 3_final or f 4_final After being combined with semantic word vectors, mean squared error loss is calculated by comparing it with the attribute truth values ​​to obtain L. reg3 and L reg4 .

[0066] like Figure 4 This is a schematic diagram of the attribute attention module.

[0067] Specifically, step S5 is as follows:

[0068] Inputting attribute descriptions into the GloVe model yields an output V with dimensions K×300, where K is the number of attributes in the dataset. The semantic word vectors V are then transformed through a defined dimension transformation layer T to obtain Vo. ` V ` Represents the semantic information of attributes. V ` Visual features f of the input 3_final or f 4_final After performing cross products, the attribute information that the network focuses on from visual features is obtained, i.e., the attention map M. By performing max pooling on the attention map M, we can obtain the network's attention attributes for the input image x.

[0069]

[0070] Where i and j represent different pixel positions on the attention map. The attributes to be focused on... The true values ​​of the attributes that an image possesses Calculate the mean square error:

[0071]

[0072] Thus, the loss L is obtained reg3 and L reg4 .

[0073] S6: f 3_final and f 4_final Input different embedding layers respectively The features are transferred from the feature space to the attribute space, and then the cosine similarity is used to obtain the classification result. and L is calculated using cross-entropy loss. cls3 and L cls4 :

[0074]

[0075] Where σ is the scaling factor.

[0076] Step S7 includes: classifying the results and By combining different proportions, we iterate through {0.1:0.9, 0.2:0.8, ..., 0.9:0.1} to select the optimal combination. The final classification result is output as the animal category.

[0077] Attached is a performance comparison table of recent advanced methods. Experiments were conducted on the CUB and AWA2 animal datasets, and compared with this year's embedded methods. The results of zero-shot recognition (ZSL) and generalized zero-shot recognition (GZSL) were compared. The highest accuracy (T1) was selected under zero-shot recognition, and the average accuracy (H) for visible and unseen classes was selected under generalized zero-shot recognition.

[0078]

Claims

1. A zero-shot classification method for animal images based on multi-scale feature fusion, characterized in that, The method includes the following steps: Step S1: Input an animal image x into the ResNet101 network and obtain the feature outputs f3 and f4 of the third and fourth layers of the network; Step S2: Input the extracted features f3 and f4 into the multi-scale feature enhancement module respectively to obtain the outputs f`3 and f`4 of the multi-scale feature enhancement module; Step S3: Considering the information interaction between the features of the two layers, a cross-layer fusion module is set up to input f`3 and f`4 together to obtain the outputs f``3 and f``4; Step S4: To avoid losing the feature information of their original dimensions, f`3 and f``3 are fused to obtain f. 3_final f`4 and f``4 are fused to obtain f 4_final Specifically, it includes: By fusing the original features from each layer and the features incorporating information from other layers, feature f is obtained through an adaptive parameter α. 3_final and f 4_final f final =α*f`+(1-α)*f``; Step S5: f 3_final and f 4_final The input is attribute attention module, which incorporates semantic word vectors obtained from the GloVe model, and the loss L is obtained. reg3 and L reg4 Specifically, it includes: The attribute descriptions from the dataset are input into the GloVe model, yielding an output V with dimensions K×300, where K is the number of attributes in the dataset. The semantic word vector V is then transformed through a defined dimension transformation layer T to obtain V', where V' represents the semantic information of the attributes. V' is then compared with the input visual features f. 3_final or f 4_final After performing cross products, the attribute information that the network focuses on from visual features is obtained, i.e., the attention map M. M k For the attention map of the k-th attribute, k∈K, W×H is the size of the attention map, with width W and height H; then, max pooling is performed on the attention map M to obtain the network's attention attributes for the input image x. Where i and i represent different pixel positions on the attention map; the attributes to be focused on The true values ​​of the attributes that an image possesses Calculate the mean square error: Thus, the loss L is obtained reg3 and L reg4 ; Step S6: f 3_final and f 4_final The inputs are then fed into the embedding layer, and cosine similarity is calculated to obtain the loss L. cls3 and L cls4 and classification results and Step S7: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] and Fusion The final output is the animal category.

2. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 1, characterized in that, Step S1 includes: for the image dataset Where x is the input image and y is its corresponding ground truth sample label. It refers to the attribute information contained in the dataset, where X is the dataset and Y is the attribute information contained in the dataset. s These images, representing the category labels for the visible class, are used as input during the training phase to train and update the network; during the testing phase, y∈Y... u The image is used as input, Y u Category labels for invisible classes. It is known, but {y|y∈Y} u It is unknown; An image is input into a ResNet101 network, and features from the third and fourth layers are selected as the output, namely f3 and f4. The ResNet101 network reads parameters pre-trained on other image datasets, and the parameters of the entire network are updated as it is trained.

3. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 2, characterized in that, Step S2 includes: the extracted features f3 and f4 are respectively input into the multi-scale feature enhancement module, which contains five branches to process the input features, and finally the five branches are concatenated to obtain the feature-enhanced outputs f`3 and f`4; For input features f3 or f4, five different branches are input; the first branch is a convolution with a kernel size of 1 to obtain the output, and the other three branches are three dilated convolutions with a kernel size of 3. The dilation rates of the dilated convolutions in different branches are different, namely [3, 6, 9]. Different dilation rates will also have different convolution padding to ensure that the input feature and output feature sizes are consistent. Where H in W in H represents the length and width of the input feature, respectively. out W out , respectively, represent the length and width of the output feature, padding is the padding, dilation is the dilation rate, and kernel and stride are the kernel size and stride of the convolution, respectively; The last branch includes adaptive mean pooling, convolution with a kernel size of 1, and nearest neighbor interpolation upsampling; finally, the outputs of the five branches are concatenated to the same dimension as the input features to obtain the output f`3 or f`4.

4. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 3, characterized in that, Step S3 includes: The inputs f`3 and f`4 are passed through two different 1×1 convolution kernels to transform their dimensions to each other's dimensions. Specifically, f`3 is transformed to have 2048 channels to obtain... The f`4 transform has 1024 channels, yielding... f`3 and, with the same number of channels Perform a cross product to obtain a fusion matrix, representing the similarity of features between different layers, and then... and Multiply by the fusion matrix respectively to obtain the final outputs f``3 and f``4.

5. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 1, characterized in that, Step S6 includes: f 3_final and f 4_final Input different embedding layers respectively C×K represents the embedding layer dimension, where C is the number of channels in the input features and K is the number of attributes in the dataset. The classification result is then obtained by applying a cosine similarity measure. and L is calculated using cross-entropy loss. cls3 and L cls4 : Where σ is the scaling factor.

6. The zero-shot classification method for animal images based on multi-scale feature fusion according to claim 5, characterized in that, Step S7 includes: classifying the results and By combining different proportions, we iterate through {0.1:0.9, 0.2:0.8, ..., 0.9:0.1} to select the optimal combination. This will be output as the final classification result.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-6.

Images