Zero-shot image classification method, system, device and computer readable storage medium

Abstract

The application discloses a zero-shot image classification method, system, device and computer readable storage medium, and relates to the field of artificial intelligence.The method comprises the following steps: performing feature extraction on a to-be-classified image to obtain visual features; modeling based on the visual features, preset attribute prototypes and a first learnable matrix to obtain spatial activation value weights; obtaining first semantic features based on the spatial activation value weights and the visual features; obtaining attribute features based on the visual features and the attribute prototypes; obtaining channel activation value weights based on the attribute features, a second learnable matrix and a third learnable matrix; obtaining second semantic features based on the channel activation value weights and the first semantic features; obtaining attribute prediction values based on the second semantic features, a fourth learnable matrix and the attribute prototypes; obtaining class scores based on the attribute prediction values, hyperparameters and attribute semantic vectors, and realizing zero-shot image classification according to the class scores.The application realizes more accurate attribute prediction, so that the purpose of more accurate zero-shot image classification is achieved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a zero-shot image classification method, system, device, and computer-readable storage medium. Background Technology

[0002] With the development of image processing and computer vision technologies, deep neural network models have been widely used due to their ability to provide end-to-end solutions from feature extraction to classification. However, traditional computer vision methods typically rely on large amounts of labeled data for training, making them difficult to implement in many practical applications. Therefore, the concept of Zero-Shot Learning (ZSL) has been introduced into image classification to address the problem of training models in the absence of suitable training samples.

[0003] Existing zero-shot learning methods based on embedding networks achieve the association between visual features and corresponding semantic features by studying how to construct embedding functions. However, these methods only align the global visual representation with the corresponding category semantics, losing many fine-grained features that are beneficial for capturing subtle differences between known and unknown categories. To address this issue, a method based on spatial attention mechanisms has been proposed. However, this method only uses attribute localization and spatial attention mechanisms for semantic feature prediction. When different attributes share similar attention regions in an image, highly similar semantic feature predictions occur, leading to inaccurate attribute predictions. Therefore, how to achieve more accurate attribute prediction to complete accurate zero-shot image classification tasks is a pressing problem that needs to be solved. Summary of the Invention

[0004] This application provides a zero-shot image classification method, system, device, and computer-readable storage medium, which can solve the technical problem of inaccurate attribute prediction in the prior art.

[0005] In a first aspect, embodiments of this application provide a zero-shot image classification method, the zero-shot image classification method comprising:

[0006] Feature extraction is performed on the images to be classified to obtain visual features;

[0007] Modeling is performed based on the visual features, the preset attribute prototypes, and the preset first learnable matrix to obtain spatial activation value weights. The attribute prototypes are used to represent attributes shared between different categories.

[0008] The first semantic feature is determined based on the spatial activation value weight and the visual feature;

[0009] The attribute features are determined based on the visual features and the attribute prototype;

[0010] Based on the aforementioned attribute features, a preset second learnable matrix, and a preset third learnable matrix, the channel activation value weights are determined.

[0011] The second semantic feature is determined based on the channel activation value weight and the first semantic feature;

[0012] The attribute prediction value is determined based on the second semantic feature, the preset fourth learnable matrix, and the attribute prototype.

[0013] The category score is determined based on the attribute prediction value, the preset hyperparameters, and the preset attribute semantic vector, and zero-shot image classification is achieved based on the category score.

[0014] In conjunction with the first aspect, in one implementation, the step of modeling based on the visual features, a preset attribute prototype, and a preset first learnable matrix to obtain spatial activation value weights includes:

[0015] The spatial activation value weights are obtained by substituting the visual features, the preset attribute prototypes, and the preset first learnable matrix into the first calculation formula, which is:

[0016]

[0017] In the formula, f i j For the visual features of the j-th region in the i-th image, r a Let W1 be the preset attribute prototype, W1 be the preset first learnable matrix, and ω(r) be the default attribute prototype. a ,f i j ) represents the spatial activation value weight.

[0018] In conjunction with the first aspect, in one implementation, determining the first semantic feature based on the spatial activation value weights and the visual features includes:

[0019] Substituting the spatial activation value weights and the visual features into the second calculation formula yields the first semantic feature, where the second calculation formula is:

[0020]

[0021] In the formula, ω(r) a ,f i j ) represents the spatial activation value weight, f i j For the visual features of the j-th region in the i-th image, Let be the first semantic feature of the i-th image.

[0022] In conjunction with the first aspect, in one implementation, determining the channel activation value weights based on the attribute features, a preset second learnable matrix, and a preset third learnable matrix includes:

[0023] Substituting the attribute features, the preset second learnable matrix, and the preset third learnable matrix into the third calculation formula yields the channel activation value weights. The third calculation formula is as follows:

[0024]

[0025] In the formula, Let W1 represent the attribute features of the i-th image, W2 be a preset second learnable matrix, and W3 be a preset third learnable matrix. Let δ be the channel activation weights of the i-th image, δ be the ReLU activation function, and σ be the Sigmoid activation function.

[0026] In conjunction with the first aspect, in one implementation, determining the second semantic feature based on the channel activation value weight and the first semantic feature includes:

[0027] Substituting the channel activation value weight and the first semantic feature into the fourth calculation formula yields the second semantic feature. The fourth calculation formula is as follows:

[0028]

[0029] In the formula, Let the channel activation values ​​be weights for the i-th image. The first semantic feature of the i-th image is... Let be the second semantic feature of the i-th image.

[0030] In conjunction with the first aspect, in one implementation, determining the attribute prediction value based on the second semantic feature, a preset fourth learnable matrix, and the attribute prototype includes:

[0031] Substituting the second semantic feature, the preset fourth learnable matrix, and the attribute prototype into the fifth calculation formula yields the attribute prediction value. The fifth calculation formula is as follows:

[0032]

[0033] In the formula, W4 is the preset fourth learnable matrix, and r a For attribute prototypes, The second semantic feature of the i-th image, Let be the attribute prediction value for the i-th image.

[0034] In conjunction with the first aspect, in one implementation, determining the category score based on the attribute prediction value, preset hyperparameters, and preset attribute semantic vectors includes:

[0035] The predicted attribute value, the preset hyperparameters, and the preset attribute semantic vector are substituted into the sixth calculation formula to obtain the category score. The sixth calculation formula is as follows:

[0036]

[0037] In the formula, Let be the predicted attribute value for the i-th image, γ be a preset hyperparameter, and h be... c This is a preset attribute semantic vector. Let be the category score of the i-th image.

[0038] Secondly, embodiments of this application provide a zero-shot image classification system, which includes a multi-scale feature fusion module, a spatial attention mechanism module, a channel attention mechanism module, and a hyperspherical classifier;

[0039] The multi-scale feature fusion module is used to extract features from the images to be classified, thereby obtaining visual features;

[0040] The spatial attention mechanism module is used to model spatial activation value weights based on the visual features, preset attribute prototypes and preset first learnable matrices, and to determine the first semantic features based on the spatial activation value weights and the visual features. The attribute prototypes are used to represent attributes shared between different categories.

[0041] The channel attention mechanism module is used to determine attribute features based on the visual features and the attribute prototype, determine channel activation value weights based on the attribute features, a preset second learnable matrix and a preset third learnable matrix, and determine second semantic features based on the channel activation value weights and the first semantic feature.

[0042] The hypersphere classifier is used to determine the attribute prediction value based on the second semantic feature, the preset fourth learnable matrix and the attribute prototype, determine the category score based on the attribute prediction value, the preset hyperparameters and the preset attribute semantic vector, and achieve zero-shot image classification based on the category score.

[0043] Thirdly, embodiments of this application provide a zero-shot image classification device, the zero-shot image classification device including a processor, a memory, and a zero-shot image classification program stored in the memory and executable by the processor, wherein when the zero-shot image classification program is executed by the processor, it implements the steps of the zero-shot image classification method as described in any of the preceding claims.

[0044] Fourthly, embodiments of this application provide a computer-readable storage medium storing a zero-shot image classification program, wherein when the zero-shot image classification program is executed by a processor, it implements the steps of the zero-shot image classification method as described in any of the preceding claims.

[0045] The beneficial effects of the technical solutions provided in this application include:

[0046] Visual features are obtained by feature extraction from the image to be classified, preserving visual information at different granularities. Based on these visual features, a preset attribute prototype, and a preset first learnable matrix, spatial activation value weights are obtained through modeling. First semantic features are determined based on these spatial activation value weights and the visual features, ensuring that even fine-grained visual information can be associated with corresponding semantic attributes. Attribute features are determined based on these visual features and the attribute prototype. Channel activation value weights are determined based on these attribute features, a preset second learnable matrix, and a preset third learnable matrix, enabling different attributes to correspond to different channel attention weights. Second semantic features with higher discriminative power are determined based on these channel activation value weights and the first semantic features. More accurate attribute prediction values ​​are determined based on these second semantic features, a preset fourth learnable matrix, and the attribute prototype. A category score is determined based on the attribute prediction value, preset hyperparameters, and a preset attribute semantic vector, and more accurate zero-shot image classification is achieved based on the category score. This application solves the technical problem of inaccurate attribute prediction in existing technologies and achieves a more accurate zero-shot image classification task. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating an embodiment of the zero-shot image classification method of this application;

[0048] Figure 2 This is a schematic diagram of the overall model structure of the zero-shot image classification method of this application;

[0049] Figure 3 This is a flowchart illustrating an embodiment of the zero-shot image classification method of this application;

[0050] Figure 4 This is a schematic diagram of the architecture of an embodiment of the zero-shot image classification system of this application;

[0051] Figure 5 This is a schematic diagram of the hardware structure of the zero-sample image classification device involved in the embodiments of this application. Detailed Implementation

[0052] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0054] In a first aspect, embodiments of this application provide a zero-shot image classification method.

[0055] In one embodiment, reference is made to Figure 1 , Figure 1 This is a schematic flowchart illustrating an embodiment of the zero-shot image classification method of this application. Figure 1 As shown, zero-shot image classification methods include:

[0056] Step S10: Extract features from the image to be classified to obtain visual features.

[0057] As an example, in this embodiment of the application, ResNet101 is used as the image feature extraction network. ResNet101 can extract four layers of visual features: the conv2 (first residual block) layer preferably contains 3 residual blocks, and each residual block preferably contains two convolutional layers, for extracting the first layer of high-level features of the image; the conv3 (second residual block) layer preferably contains 4 residual blocks, and each residual block preferably contains two convolutional layers, for extracting the second layer of high-level features of the image; the conv4 (third residual block) layer preferably contains 23 residual blocks, and each residual block preferably contains two convolutional layers, for extracting the third layer of high-level features of the image; and the conv5 (fourth residual block) layer preferably contains 3 residual blocks, and each residual block preferably contains two convolutional layers, for extracting the fourth layer of high-level features of the image.

[0058] Specifically, the image to be classified is input into the trained image feature extraction network. The image feature extraction network obtains visual features of different granularities in the i-th image to be classified. The four layers of visual features extracted by the four main modules (i.e., the four residual block layers) of ResNet101 are mapped to obtain the mapped visual features. The four mapped visual features are then summed to obtain the fused visual features. In order to balance the visual information of different granularities, the four layers of visual features can be mapped to the same dimension as the visual features of the conv4 layer extracted by ResNet101. The specific dimension alignment process is as follows: feature dimension alignment is achieved by using upsampling and downsampling. Upsampling uses a 3×3 deconvolution operation, and downsampling and same-dimensional mapping both use a 3×3 convolution operation. At the same time, residual connections are added to the conv4 layer to stabilize the training process and improve the performance of the image feature extraction network.

[0059] It should be noted that the embodiments of this application can fuse visual features of different scales extracted by the image feature network based on the multi-scale visual feature fusion module, thereby preserving visual information of different granularities.

[0060] Step S20: Model based on the visual features, the preset attribute prototype and the preset first learnable matrix to obtain spatial activation value weights. The attribute prototype is used to represent attributes shared between different categories.

[0061] In this embodiment, as an example, an attribute is a high-level semantic information that describes the characteristics of an object. For example, an attribute can be color, shape, and texture. A prototype is a typical representation of a target, and its application to an attribute constitutes the concept of an attribute prototype. For example, the representation of the same attribute in different images is diverse, but the prototype of the attribute can generally summarize this semantic information. In a neural network, an attribute prototype can be represented by a vector. An attribute prototype represents an abstract semantic feature of each attribute in the neural network model. The dimension of the attribute prototype is determined by the number of attributes shared by the visible and invisible categories labeled in the dataset. In this embodiment, the attribute prototype is preferably a learnable vector obtained after training a zero-shot image classification model, which is shared by all images. The preset first learnable matrix is ​​a learnable parameter matrix used to align the dimensions of each visual region feature and the attribute prototype feature. Its specific value can be determined according to actual needs and is not limited here.

[0062] Specifically, a spatial attention mechanism module is designed to model the relationship between visual and semantic information. At the same time, a cross-attention mechanism is used to interactively process the fused visual and semantic features. That is, based on a preset first learnable matrix, the dimensions of the visual features and the dimensions of the attribute prototypes are aligned, and then the similarity between the fused visual features and the attribute prototypes is calculated, thereby determining the spatial activation value weight of each semantic attribute on the image.

[0063] Step S30: Determine the first semantic feature based on the spatial activation value weight and the visual feature.

[0064] As an example, in this embodiment, different attributes have different spatial activation value weights in the image region. Therefore, corresponding semantic features can be extracted from visual features based on these spatial activation value weights. Specifically, after obtaining the spatial activation value weights, these weights are multiplied by the fused visual features to obtain the first semantic feature.

[0065] Step S40: Determine the attribute features based on the visual features and the attribute prototype.

[0066] As an example, in this embodiment of the application, the visual features and the attribute prototype are substituted into the following formula to calculate the attribute features, the calculation formula being:

[0067]

[0068] In the formula, f i j Let r be the visual feature of the j-th region in the i-th image. The total number of regions in the i-th image is J, and r a Here is the property prototype, and 'a' is the property. Let represent the attribute features of the i-th image, and norm represents the Gaussian normalization operation.

[0069] Step S50: Determine the channel activation value weights based on the attribute features, the preset second learnable matrix, and the preset third learnable matrix.

[0070] As an example, in the embodiments of this application, the preset second learnable matrix is ​​a learnable parameter matrix used to reduce the dimensionality of attribute features; the preset third learnable matrix is ​​a learnable parameter matrix used to increase the dimensionality of attribute features to the original dimension; it should be noted that the specific values ​​of the second and third learnable matrices can be determined according to actual needs, and are not limited here.

[0071] Specifically, the channel activation weights are obtained by processing attribute features using MLP (Multilayer Perceptron, a multilayer perceptron in deep learning). The specific operation can be broken down into the following steps: The channel activation weights are obtained by multiplying the attribute features, the preset second learnable matrix, and the preset third learnable matrix. That is, when different attributes share similar image regions, the attribute features are obtained by summing the normalized semantic attribute features and visual features. Then, the channel activation weights are obtained by multiplying the attribute features, the preset second learnable matrix, and the preset third learnable matrix, so that different attributes can obtain different channel activation weights.

[0072] Step S60: Determine the second semantic feature based on the channel activation value weight and the first semantic feature.

[0073] As an example, in this embodiment of the application, a channel attention mechanism module is designed to decouple the first semantic feature to obtain the second semantic feature. Specifically, after obtaining the channel activation value weight, the channel activation value weight is multiplied by the first semantic feature (i.e., the semantic feature extracted by the spatial attention mechanism) to obtain the second semantic feature (i.e., the decoupled semantic feature).

[0074] Step S70: Determine the attribute prediction value based on the second semantic feature, the preset fourth learnable matrix, and the attribute prototype.

[0075] As an example, in this embodiment, the preset fourth learnable matrix is ​​a learnable parameter matrix used to maintain the second semantic feature having the same dimension as the attribute prototype feature. Its specific value can be determined according to actual needs and is not limited here. The attribute prediction value is obtained by calculating the similarity between the attribute prototype and the second semantic feature. Specifically, the attribute prediction value is obtained by performing a dot product of the second semantic feature, the preset fourth learnable matrix, and the attribute prototype.

[0076] Step S80: Determine the category score based on the attribute prediction value, the preset hyperparameters and the preset attribute semantic vector, and perform zero-shot image classification based on the category score.

[0077] As an example, in this embodiment, the preset hyperparameter is used to scale the normalized attribute prediction values. The preset attribute semantic vector refers to the attribute semantic vector of each category labeled in the dataset, and the distribution of attribute semantic vectors of different categories is different. The specific values ​​of the hyperparameter and attribute semantic vector can be determined according to actual needs and are not limited here. Specifically, after obtaining the attribute prediction value, the similarity with the preset attribute semantic vector is calculated to obtain the category score, and the category score with the highest similarity is selected as the predicted category.

[0078] It should be noted that, referring to Figure 2 and Figure 3 As shown in this embodiment, the image feature extraction network extracts visual features from the image to be classified. A multi-scale visual feature fusion module fuses visual features at different scales, preserving visual information of different granularities. A spatial attention mechanism module models the relationship between visual and semantic information and uses learnable attribute prototypes as semantic representations to achieve the fusion and interaction of visual and semantic features, thereby extracting corresponding semantic features from the visual features. A channel attention mechanism module then obtains different channel activation weights based on different semantic attributes, decouples the extracted first semantic features, and separates similar semantic features to obtain a second semantic feature with higher discriminative power. A hyperspherical classifier predicts semantic attributes and completes knowledge transfer and image classification tasks in zero-shot learning based on shared semantic attributes.

[0079] In this embodiment, visual features are obtained by feature extraction from the image to be classified, preserving visual information at different granularities. Based on the visual features, a preset attribute prototype, and a preset first learnable matrix, spatial activation value weights are obtained through modeling. First semantic features are determined based on the spatial activation value weights and the visual features, ensuring that even fine-grained visual information can be associated with corresponding semantic attributes. Attribute features are determined based on the visual features and the attribute prototype. Channel activation value weights are determined based on the attribute features, a preset second learnable matrix, and a preset third learnable matrix, enabling different attributes to correspond to different channel attention weights. Second semantic features with higher discriminative power are determined based on the channel activation value weights and the first semantic features. More accurate attribute prediction values ​​are determined based on the second semantic features, a preset fourth learnable matrix, and the attribute prototype. A category score is determined based on the attribute prediction value, preset hyperparameters, and a preset attribute semantic vector, and more accurate zero-shot image classification is achieved based on the category score. This application solves the technical problem of inaccurate attribute prediction in the prior art and achieves a more accurate zero-shot image classification task.

[0080] Furthermore, in some embodiments, the step of modeling based on the visual features, a preset attribute prototype, and a preset first learnable matrix to obtain spatial activation value weights includes:

[0081] The spatial activation value weights are obtained by substituting the visual features, the preset attribute prototypes, and the preset first learnable matrix into the first calculation formula, which is:

[0082]

[0083] In the formula, fi j For the visual features of the j-th region in the i-th image, r a Let W1 be the preset attribute prototype, W1 be the preset first learnable matrix, and ω(r) be the default attribute prototype. a ,f i j ) represents the spatial activation value weight.

[0084] As an example, in the embodiments of this application, the semantic attribute r is... a Defined as a learnable attribute prototype, representing an abstract semantic feature of each attribute in the neural network model. The dimension of the attribute prototype is determined by the number of attributes shared by the visible and invisible categories labeled in the dataset; f i j Representing the fused visual features, containing visual information at different granularities, the subscript i represents the i-th image, and the superscript j represents the visual feature region in the image. Each image can be divided into J regions based on pixels. W1 is a learnable parameter matrix used to align the dimensions of each visual region feature and attribute prototype feature. ω(r a ,f i j ) represents the computed property prototype r a With image visual features f i j The spatial activation value weights are obtained from the similarity of the positions, and different attributes have different spatial activation value weights in different image regions. In this embodiment, the spatial attention weights are processed by using the SoftMax activation value function in deep learning to ensure that the sum of the spatial activation value weights in the image region is 1.

[0085] Furthermore, in some embodiments, determining the first semantic feature based on the spatial activation value weights and the visual features includes:

[0086] Substituting the spatial activation value weights and the visual features into the second calculation formula yields the first semantic feature, where the second calculation formula is:

[0087]

[0088] In the formula, ω(r) a ,f i j ) represents the spatial activation value weight, f i j For the visual features of the j-th region in the i-th image, Let be the first semantic feature of the i-th image.

[0089] As an example, in the embodiments of this application, the spatial activation value weight ω(r) is...a ,f i j ) and visual features f i j Substituting into the following formula yields the first semantic feature.

[0090]

[0091] Furthermore, in some embodiments, determining the channel activation value weights based on the attribute features, a preset second learnable matrix, and a preset third learnable matrix includes:

[0092] Substituting the attribute features, the preset second learnable matrix, and the preset third learnable matrix into the third calculation formula yields the channel activation value weights. The third calculation formula is as follows:

[0093]

[0094] In the formula, Let W1 represent the attribute features of the i-th image, W2 be a preset second learnable matrix, and W3 be a preset third learnable matrix. Let δ be the channel activation weights of the i-th image, δ be the ReLU activation function, and σ be the Sigmoid activation function.

[0095] As an example, in this embodiment of the application, after obtaining the initial value of the channel activation value weight, the weight is processed using a multilayer perceptron, and the specific processing is as follows:

[0096]

[0097] Here, MLP stands for Multilayer Perceptron in deep learning, specifically referring to attribute features. Substituting the preset second learnable matrix W2 and the preset third learnable matrix W3 into the following formula yields the channel activation weights. The calculation formula is as follows:

[0098]

[0099] Furthermore, in some embodiments, determining the second semantic feature based on the channel activation value weight and the first semantic feature includes:

[0100] Substituting the channel activation value weight and the first semantic feature into the fourth calculation formula yields the second semantic feature. The fourth calculation formula is as follows:

[0101]

[0102] In the formula, Let the channel activation values ​​be weights for the i-th image. The first semantic feature of the i-th image is... Let be the second semantic feature of the i-th image.

[0103] As an example, in the embodiments of this application, the channel activation value weight is... and first semantic features Substituting into the following calculation formula yields the second semantic feature. The calculation formula is as follows:

[0104]

[0105] Furthermore, in some embodiments, determining the attribute prediction value based on the second semantic feature, the preset fourth learnable matrix, and the attribute prototype includes:

[0106] Substituting the second semantic feature, the preset fourth learnable matrix, and the attribute prototype into the fifth calculation formula yields the attribute prediction value. The fifth calculation formula is as follows:

[0107]

[0108] In the formula, W4 is the preset fourth learnable matrix, and r a For attribute prototypes, The second semantic feature of the i-th image, Let be the attribute prediction value for the i-th image.

[0109] As an example, in the embodiments of this application, the second semantic feature is... The pre-defined fourth learnable matrix W4 and attribute prototype r a Substitute the values ​​into the following formula to obtain the attribute prediction values. The calculation formula is as follows:

[0110]

[0111] Furthermore, in some embodiments, determining the category score based on the attribute prediction value, preset hyperparameters, and preset attribute semantic vectors includes:

[0112] The predicted attribute value, the preset hyperparameters, and the preset attribute semantic vector are substituted into the sixth calculation formula to obtain the category score. The sixth calculation formula is as follows:

[0113]

[0114] In the formula, Let be the predicted attribute value for the i-th image, γ be a preset hyperparameter, and h be... c This is a preset attribute semantic vector. Let be the category score of the i-th image.

[0115] As an example, in the embodiments of this application, the attribute prediction value is... Preset hyperparameter γ and preset attribute semantic vector h c Substitute into the following formula to obtain the category score. The calculation formula is as follows:

[0116]

[0117] It should be noted that, during the training process, the loss function used by the zero-shot image classification model based on visual and semantic relationship modeling proposed in this application involves the attribute-based classification loss function L. AC and calibration loss function L SC The overall objective loss function can be expressed as L = L AC +L SC This loss function can achieve excellent zero-shot image classification results without the need for weight parameter adjustment.

[0118] Specifically, the attribute-based classification loss function L AC The method primarily relies on the semantic attribute information labeled in the dataset to determine the attribute prediction results using cross-entropy, thereby achieving zero-shot image classification. The specific operation can be defined as follows:

[0119]

[0120] Using only attribute-based classification loss functions can easily lead to overfitting on visible categories; therefore, a calibration loss function L is designed. SC This is used to shift the prediction result towards the unknown category during training. The specific operation can be defined as follows:

[0121]

[0122] By training the zero-shot image classification model using the two loss functions mentioned above, the resulting model can achieve excellent zero-shot image classification performance.

[0123] It is understood that the zero-shot image classification method provided in this application embodiment can better model the relationship between visual information and semantic information, enhance the prediction ability of visual information of different granularities, obtain semantic features with higher discriminativeness, and have accurate prediction effect on attributes. It has excellent performance in both traditional zero-shot learning and generalized zero-shot learning.

[0124] Secondly, embodiments of this application also provide a zero-shot image classification system.

[0125] In one embodiment, reference is made to Figure 4 , Figure 4 This is a schematic diagram of the functional modules of an embodiment of the zero-shot image classification system of this application. Figure 4 As shown, a zero-shot image classification system includes:

[0126] Multi-scale feature fusion module, spatial attention mechanism module, channel attention mechanism module, and hyperspherical classifier;

[0127] The multi-scale feature fusion module is used to extract features from the images to be classified, thereby obtaining visual features;

[0128] The spatial attention mechanism module is used to model spatial activation value weights based on the visual features, preset attribute prototypes and preset first learnable matrices, and to determine the first semantic features based on the spatial activation value weights and the visual features. The attribute prototypes are used to represent attributes shared between different categories.

[0129] The channel attention mechanism module is used to determine attribute features based on the visual features and the attribute prototype, determine channel activation value weights based on the attribute features, a preset second learnable matrix and a preset third learnable matrix, and determine second semantic features based on the channel activation value weights and the first semantic feature.

[0130] The hypersphere classifier is used to determine the attribute prediction value based on the second semantic feature, the preset fourth learnable matrix and the attribute prototype, determine the category score based on the attribute prediction value, the preset hyperparameters and the preset attribute semantic vector, and achieve zero-shot image classification based on the category score.

[0131] Furthermore, in some embodiments, the spatial attention mechanism module is specifically used for:

[0132] The spatial activation value weights are obtained by substituting the visual features, the preset attribute prototypes, and the preset first learnable matrix into the first calculation formula, which is:

[0133]

[0134] In the formula, f i j For the visual features of the j-th region in the i-th image, r a Let W1 be the preset attribute prototype, W1 be the preset first learnable matrix, and ω(r) be the default attribute prototype. a ,f i j ) represents the spatial activation value weight.

[0135] Furthermore, in some embodiments, the spatial attention mechanism module is specifically used for:

[0136] Substituting the spatial activation value weights and the visual features into the second calculation formula yields the first semantic feature, where the second calculation formula is:

[0137]

[0138] In the formula, ω(r) a ,f i j ) represents the spatial activation value weight, f i j For the visual features of the j-th region in the i-th image, Let be the first semantic feature of the i-th image.

[0139] Furthermore, in some embodiments, the channel attention mechanism module is specifically used to: substitute the attribute features, the preset second learnable matrix, and the preset third learnable matrix into a third calculation formula to obtain the channel activation value weights, wherein the third calculation formula is:

[0140]

[0141] In the formula, Let W1 represent the attribute features of the i-th image, W2 be a preset second learnable matrix, and W3 be a preset third learnable matrix. Let δ be the channel activation weights of the i-th image, δ be the ReLU activation function, and σ be the Sigmoid activation function.

[0142] Furthermore, in some embodiments, the channel attention mechanism module is specifically used to: substitute the channel activation value weights and the first semantic feature into a fourth calculation formula to obtain a second semantic feature, wherein the fourth calculation formula is:

[0143]

[0144] In the formula, Let the channel activation values ​​be weights for the i-th image. The first semantic feature of the i-th image is... Let be the second semantic feature of the i-th image.

[0145] Furthermore, in some embodiments, the hyperspherical classifier is specifically used for:

[0146] Substituting the second semantic feature, the preset fourth learnable matrix, and the attribute prototype into the fifth calculation formula yields the attribute prediction value. The fifth calculation formula is as follows:

[0147]

[0148] In the formula, W4 is the preset fourth learnable matrix, and r a For attribute prototypes, The second semantic feature of the i-th image, Let be the attribute prediction value for the i-th image.

[0149] Furthermore, in some embodiments, the hyperspherical classifier is specifically used for:

[0150] The predicted attribute value, the preset hyperparameters, and the preset attribute semantic vector are substituted into the sixth calculation formula to obtain the category score. The sixth calculation formula is as follows:

[0151]

[0152] In the formula, Let be the predicted attribute value for the i-th image, γ be a preset hyperparameter, and h be... c This is a preset attribute semantic vector. Let be the category score of the i-th image.

[0153] In this embodiment, visual features are obtained by feature extraction from the image to be classified, preserving visual information at different granularities. Based on the visual features, a preset attribute prototype, and a preset first learnable matrix, spatial activation value weights are obtained through modeling. First semantic features are determined based on the spatial activation value weights and the visual features, ensuring that even fine-grained visual information can be associated with corresponding semantic attributes. Attribute features are determined based on the visual features and the attribute prototype. Channel activation value weights are determined based on the attribute features, a preset second learnable matrix, and a preset third learnable matrix, enabling different attributes to correspond to different channel attention weights. Second semantic features with higher discriminative power are determined based on the channel activation value weights and the first semantic features. More accurate attribute prediction values ​​are determined based on the second semantic features, a preset fourth learnable matrix, and the attribute prototype. A category score is determined based on the attribute prediction value, preset hyperparameters, and a preset attribute semantic vector, and more accurate zero-shot image classification is achieved based on the category score. This application solves the technical problem of inaccurate attribute prediction in the prior art and achieves a more accurate zero-shot image classification task.

[0154] The functions of each module in the zero-shot image classification system correspond to the steps in the zero-shot image classification method embodiment, and their functions and implementation processes will not be described in detail here.

[0155] Thirdly, embodiments of this application provide a zero-shot image classification device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0156] Reference Figure 5 , Figure 5 This is a schematic diagram of the hardware structure of the zero-shot image classification device involved in the embodiments of this application. In the embodiments of this application, the zero-shot image classification device may include a processor, a memory, a communication interface, and a communication bus.

[0157] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0158] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of the zero-shot image classification device, as well as interfaces used for interconnecting the zero-shot image classification device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0159] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0160] The processor can be a general-purpose processor, which can call a zero-shot image classification program stored in memory and execute the zero-shot image classification method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU) or a graphics processing unit (GPU). The method executed when the zero-shot image classification program is called can be referred to in various embodiments of the zero-shot image classification method of this application, and will not be repeated here.

[0161] Those skilled in the art will understand that Figure 5The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0162] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0163] The present application provides a computer-readable storage medium storing a zero-shot image classification program, wherein when the zero-shot image classification program is executed by a processor, it implements the steps of the zero-shot image classification method as described above.

[0164] The method implemented when the zero-shot image classification procedure is executed can be referred to in various embodiments of the zero-shot image classification method of this application, and will not be repeated here.

[0165] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0166] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0167] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0168] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0169] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0171] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A zero-shot image classification method, characterized in that, The zero-shot image classification method includes: Feature extraction is performed on the images to be classified to obtain visual features; Modeling is performed based on the visual features, the preset attribute prototypes, and the preset first learnable matrix to obtain spatial activation value weights. The attribute prototypes are used to represent attributes shared between different categories. The first semantic feature is determined based on the spatial activation value weight and the visual feature; The attribute features are determined based on the visual features and the attribute prototype; Based on the aforementioned attribute features, a preset second learnable matrix, and a preset third learnable matrix, the channel activation value weights are determined. The second semantic feature is determined based on the channel activation value weight and the first semantic feature; The attribute prediction value is determined based on the second semantic feature, the preset fourth learnable matrix, and the attribute prototype. The category score is determined based on the attribute prediction value, the preset hyperparameters, and the preset attribute semantic vector, and zero-shot image classification is achieved based on the category score.

2. The zero-shot image classification method as described in claim 1, characterized in that, The spatial activation value weights obtained by modeling based on the visual features, preset attribute prototypes, and a preset first learnable matrix include: The spatial activation value weights are obtained by substituting the visual features, the preset attribute prototypes, and the preset first learnable matrix into the first calculation formula, which is: In the formula, f i j For the visual features of the j-th region in the i-th image, r a Let W1 be the preset attribute prototype, W1 be the preset first learnable matrix, and ω(r) be the default attribute prototype. a f i j ) represents the spatial activation value weight.

3. The zero-shot image classification method as described in claim 1, characterized in that, The determination of the first semantic feature based on the spatial activation value weights and the visual features includes: Substituting the spatial activation value weights and the visual features into the second calculation formula yields the first semantic feature, where the second calculation formula is: In the formula, ω(r) a f i j ) represents the spatial activation value weight, f i j For the visual features of the j-th region in the i-th image, Let be the first semantic feature of the i-th image.

4. The zero-shot image classification method as described in claim 1, characterized in that, The determination of channel activation value weights based on the attribute features, a preset second learnable matrix, and a preset third learnable matrix includes: Substituting the attribute features, the preset second learnable matrix, and the preset third learnable matrix into the third calculation formula yields the channel activation value weights. The third calculation formula is as follows: In the formula, Let W1 represent the attribute features of the i-th image, W2 be a preset second learnable matrix, and W3 be a preset third learnable matrix. Let δ be the channel activation weights of the i-th image, δ be the ReLU activation function, and σ be the Sigmoid activation function.

5. The zero-shot image classification method as described in claim 1, characterized in that, The determination of the second semantic feature based on the channel activation value weight and the first semantic feature includes: Substituting the channel activation value weight and the first semantic feature into the fourth calculation formula yields the second semantic feature. The fourth calculation formula is as follows: In the formula, Let the channel activation values ​​be weights for the i-th image. The first semantic feature of the i-th image is... Let be the second semantic feature of the i-th image.

6. The zero-shot image classification method as described in claim 1, characterized in that, The step of determining the attribute prediction value based on the second semantic feature, the preset fourth learnable matrix, and the attribute prototype includes: Substituting the second semantic feature, the preset fourth learnable matrix, and the attribute prototype into the fifth calculation formula yields the attribute prediction value. The fifth calculation formula is as follows: In the formula, W4 is the preset fourth learnable matrix, and r a For attribute prototypes, The second semantic feature of the i-th image, Let be the attribute prediction value for the i-th image.

7. The zero-shot image classification method as described in claim 1, characterized in that, The process of determining the category score based on the predicted attribute value, preset hyperparameters, and preset attribute semantic vectors includes: The predicted attribute value, the preset hyperparameters, and the preset attribute semantic vector are substituted into the sixth calculation formula to obtain the category score. The sixth calculation formula is as follows: In the formula, Let be the predicted attribute value for the i-th image, γ be a preset hyperparameter, and h be... c This is a preset attribute semantic vector. Let be the category score of the i-th image.

8. A zero-shot image classification system, characterized in that, The zero-shot image classification system includes a multi-scale feature fusion module, a spatial attention mechanism module, a channel attention mechanism module, and a hyperspherical classifier; The multi-scale feature fusion module is used to extract features from the images to be classified, thereby obtaining visual features; The spatial attention mechanism module is used to model spatial activation value weights based on the visual features, preset attribute prototypes and preset first learnable matrices, and to determine the first semantic features based on the spatial activation value weights and the visual features. The attribute prototypes are used to represent attributes shared between different categories. The channel attention mechanism module is used to determine attribute features based on the visual features and the attribute prototype, determine channel activation value weights based on the attribute features, a preset second learnable matrix and a preset third learnable matrix, and determine second semantic features based on the channel activation value weights and the first semantic feature. The hypersphere classifier is used to determine the attribute prediction value based on the second semantic feature, the preset fourth learnable matrix and the attribute prototype, determine the category score based on the attribute prediction value, the preset hyperparameters and the preset attribute semantic vector, and achieve zero-shot image classification based on the category score.

9. A zero-shot image classification device, characterized in that, The zero-shot image classification device includes a processor, a memory, and a zero-shot image classification program stored in the memory and executable by the processor, wherein when the zero-shot image classification program is executed by the processor, it implements the steps of the zero-shot image classification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a zero-shot image classification program, wherein when the zero-shot image classification program is executed by a processor, it implements the steps of the zero-shot image classification method as described in any one of claims 1 to 7.

Images