Zero-shot building image classification method based on double attention mechanism

Abstract

The application relates to a zero sample image classification method based on a double attention mechanism, and belongs to the technical field of image processing, solves the problems of few labeled data and inaccurate positioning of local discriminative features in building style classification, and the solution is as follows: firstly, two models of channel attention and spatial attention are introduced to enhance the representation of specific regions of an image; then, in order to reduce information loss in the spatial mapping process, a generator is used to reconstruct visual features; finally, a common space embedding zero sample classification model is designed, visual features and semantic features are aligned in a subspace, and a nearest neighbor matching is used to realize a classification task. Experiments are carried out on a zero sample dataset and a building style dataset, and the results show that the method has higher classification accuracy than current zero sample learning methods, and can effectively solve the classification problem when building style samples are insufficient.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a zero-sample building image classification method based on a double-attention mechanism. BACKGROUND

[0002] Buildings form different styles of buildings from the aspects of geographical location, safety factors, building materials, etc. Accurate identification of the style of a building is of great significance to building history research, building heritage protection and urban construction. In building style classification, due to the similarity of different building styles and the difference of the same building style, it is more difficult to label. Especially for building heritage images, because they need to be protected, building sites cannot be opened to the public, such as some palaces in the Forbidden City, image data is difficult to obtain and extremely valuable, and the label samples in the data set are far from enough to cover all categories. Therefore, under the condition of lacking enough training data, how to classify the unknown building image style by using known building style instances has become a difficulty.

[0003] Zero-sample classification technology aims to classify samples that have not appeared in the training stage. According to the semantic correlation between seen classes and unseen classes, the knowledge of seen classes is transferred for the identification of unseen classes, which can effectively solve the classification problem when sample labels are lacking. The key of zero-sample learning technology is to learn an embedding space. According to the different embedding spaces, it is mainly divided into semantic space embedding method, visual space embedding method and common subspace embedding method:

[0004] 1. The semantic space embedding method learns an embedding function in the training stage, maps the image features into the semantic space, measures the matching degree with the semantic description vector, and the class label with the highest matching degree is the label of the test class input image;

[0005] 2. The visual space embedding method maps the semantic vector into the visual feature space to retain more description information and prevent feature space from shrinking, which can alleviate the hub problem of semantic space embedding to a certain extent. However, directly learning the mapping function between visual space and semantic space makes the model have weak generalization ability, resulting in poor performance of zero-sample classification;

[0006] 3. The public space embedding method fully utilizes the complementarity and consistency of visual and semantic modal information, mapping visual and semantic features to a public subspace. This method has good decision-making and generalization abilities and can effectively alleviate the domain shift problem. However, in architectural image classification tasks, while the overall appearance features of each architectural style are similar, there are differences in detailed elements. For example, Gothic architecture consists of spires, rose windows, flying buttresses, and pointed arches from top to bottom, while Baroque architecture consists of round arches and domes. From a spatial composition perspective, the importance of each architectural element to the classification task varies. If a traditional convolutional neural network is used, with the features of the last convolutional layer as the feature representation, it may lack specificity for the architectural structure, ignoring the importance of each channel and spatial location in the image, making it difficult to extract robust element features.

[0007] Common attention mechanisms include channel attention, spatial attention, and temporal attention. Channel attention learns the weights of channels and interacts with them, while spatial attention learns important regions in space by embedding location information. A hybrid attention network combining channel and spatial attention learns the weights of various dimensions of image features, and through feature weighting, it can capture detailed features of different objects at different locations in an image. Summary of the Invention

[0008] The main objective of this invention is to overcome the shortcomings of existing technologies and solve the technical problems of missing building image labels and inaccurate local discriminative region localization. This invention provides a zero-shot building image classification method based on a dual-attention mechanism. For situations where there is little or no labeled data for building images, zero-shot learning is used to classify architectural styles. Figure 1 This approach aims to identify categories that are difficult to label, alleviate the low recognition rate caused by the uneven distribution of samples across different styles, and further improve the accuracy of architectural image classification. Visual attention can notice task-relevant areas, remove redundant information, and extract more discriminative visual features.

[0009] The technical solution adopted in this invention is as follows:

[0010] A zero-shot building image classification method based on dual attention mechanism includes the following steps:

[0011] S1, discriminative visual feature extraction: the discriminative visual feature extraction is composed of a backbone network, a channel attention network and a spatial attention network. A ResNeXt-101 network (backbone network) is used to extract global features, a channel attention network (CAN) is used to learn different channel weight positioning building main body, and a spatial attention network (SAN) is used to embed position information into a channel attention map to capture detailed features in the target;

[0012] S2, attribute encoding: semantic features are composed of semantic attributes of images, represent the relationship between categories, and are key information for zero-shot learning. All attribute features of categories are encoded by one-hot, 0 represents no attribute, and 1 represents an attribute. A mapping function from a visual space to a subspace is learned, and a semantic attribute vector is also mapped to a subspace.

[0013] S3, space mapping and classification: visual features are composed of feature maps obtained by double attention, which retain discriminative information in images. The visual features are mapped to a common subspace by a fully connected layer, a mapping function from the visual features to the common space is learned, and the encoded attribute vectors of all categories are also mapped to the same subspace by a fully connected layer. A semantic mapping function is learned, and a nearest neighbor algorithm is used to find a vector matching the visual features in the mapped semantic vector to predict the category label of the sample. The classification task is realized by nearest neighbor matching.

[0014] Further, the step S1 includes the following steps:

[0015] S1-1: the image size of the data set is initialized to 256mm x 256mm, input into the backbone network ResNeXt-101 for preprocessing, and the global features of the image are extracted. The size of the last convolution feature map of ResNeXt-101 is 2048 x 8 x 8.

[0016] S1-2: input the global feature map extracted in step S1-1 into the channel attention network, calculate the weight of each channel, and weight the global feature map:

[0017] W channel =SoftMax(f c_attention (F tmp )),W channel ∈R;

[0018]

[0019] In the formula, W channelSoftMax(·) makes the sum of the weights of each channel to be 1, f c_attention (·) is a channel attention network, F tmp represents the compressed global feature map, F ca is a channel attention map;

[0020] S1-3: the channel attention map F ca In the input spatial attention network, important regions in the image space are found for processing, in order to avoid the destruction of the spatial structure, a spatial attention model composed of multiple convolution layers is used to extract the spatial weight W of the pixel spatial , the spatial weight is applied to the feature map to calculate the local feature attention map F att :

[0021] W spatial =f s_attention (F ca ), W spatial ∈R 8×8 ;

[0022] F att =F ca ×W spatial ,F att ∈R 8×8×16 ;

[0023] In the formula, W spatial is the spatial weight of the pixel, f s_attention (·) is a spatial attention network, F ca is a channel attention map, and F att is a local feature attention map.

[0024] Further, the step S3 includes the following steps:

[0025] S3-1: the extracted local feature attention map F att is mapped into a common subspace using a fully connected layer, and a visual mapping function φ(x i )=W1×f att is learned, wherein W1 is a visual mapping matrix;

[0026] S3-2: in order to reduce the loss of some discriminative information in the knowledge transfer process, a generator is used to reconstruct the mapped features;

[0027] S3-3: the attribute vectors of all categories after encoding are mapped into the same subspace through a fully connected layer, and a semantic mapping function is learned, wherein W2 is a semantic mapping matrix;

[0028] S3-4: the class label of the unseen class sample x u is predicted in the common subspace, that is In the formula, D represents the Euclidean distance metric function, x u For images of unseen classes, φ(x) u ) is the visual mapping vector of the unseen class image. Let be the mapping vector of the semantic attributes of the j-th category in the common subspace. argmin D(·) denotes the minimum value among the vectors.

[0029] The definitions and terms involved in this invention are as follows:

[0030] In zero-shot style classification tasks, the visible class sample is defined as... s represents the visible class, Represents the i-th image of the visible class. This represents the class label of the i-th image. Unseen class samples are defined as... 'u' represents an unseen class. Represents the j-th image of the unseen class. Let X represent the class label of the j-th image, and X and Y represent the image set and class label set, respectively. Visible classes and unseen classes are disjoint, i.e., ... User-defined attribute set A = {a1, a2, a3, ..., a...} n}, n = i + j, a i is an attribute vector, where n is the total number of categories.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] This invention adaptively learns the weights of each channel using a channel attention network, selecting the buildings themselves in the image while ignoring background noise. Spatial attention is then used to generate a mask for each location in the feature map and output a weighted sum to extract detailed features relevant to the classification task. Furthermore, considering that information loss is inevitable due to dimensional differences during spatial mapping, resulting in the loss of discriminative information, a generator is used to reconstruct the mapped features, preserving more original information and mitigating the information loss problem during spatial mapping, thereby improving the accuracy of building image classification. Attached Figure Description

[0033] Figure 1 A schematic diagram of zero-sample building images;

[0034] Figure 2 This is a flowchart of the process framework of the present invention;

[0035] Figure 3 A comparison chart of zero-shot classification results on the CUB dataset;

[0036] Figure 4A comparison chart for zero-shot classification results on the Architecture Style Dataset;

[0037] Figure 5 A comparison chart for attention map visualization. DETAILED DESCRIPTION

[0038] The application will be further described in detail below in conjunction with the accompanying drawings and examples.

[0039] This embodiment is based on the attribute characteristics of American house style categories and professional knowledge in the field of architectural style proposed by Yi, and the semantic attributes of the categories are added in the Architecture Style Dataset, with an attribute dimension of 31, a total of 5000 building images, including 25 style categories, of which the training set has 20 categories and a total of 4042 images, and the test set has 5 categories and a total of 958 images. When training the network, the SGD optimizer (momentum is 0.9, and decay rate is 0.0001) is used to optimize the model of this embodiment, the batch size is set to 50, the model is trained for 200 iterations, and the learning rate is set to 0.0001.

[0040] As shown in the zero-shot building image classification method based on the double attention mechanism, the method comprises the following steps: Figure 2

[0041] S1, discriminative visual feature extraction: a ResNeXt-101 network is used to extract global features, a channel attention network is used to learn different channel weight positioning of the building main body, and a spatial attention network is used to embed position information into a channel attention map to capture detailed features in the target; the step S1 comprises the following steps:

[0042] S1-1: the image size of the data set is initialized to 256mm*256mm, input into the backbone network ResNeXt-101 for preprocessing, and the global features of the image are extracted, and the size of the last convolution feature map of ResNeXt-101 is 2048*8*8;

[0043] S1-2: input the global feature map extracted in step S1-1 into the channel attention network, use 1*1 convolution to compress the channel F tmp =conv 1×1 (f ResNeXt ), without changing the spatial information of the feature map, redundant channels are deleted. As shown in Figure 1 , the channel attention network uses global average pooling (GAP) to calculate the feature value V of each channel feature map:

[0044] V=GAP(F tmp ), V∈R 1×1×16 ​(1)

[0045] where GAP(·) is global average pooling, F tmp is the compressed global feature representation.

[0046] The attention weight W channel of each channel is calculated again.

[0047] W channel =SoftMax(f c_attention (F tmp )), W channel ∈R; (2)

[0048] where SoftMax(·) makes the sum of the weights of each channel equal to 1, f c_attention (·) is the channel attention network, and F tmp is the compressed global feature representation.

[0049] The channel weight is applied to the global feature map, and the global feature map is weighted to obtain the channel attention map F ca .

[0050] F ca =F tmp ×W channel ,F ca ∈R 8×8×16 ; (3)

[0051] where F tmp is the compressed global feature representation, and W channel is the channel attention weight.

[0052] S1-3: The channel attention map F ca is input into the spatial attention network to find important regions in the image space for processing. To avoid the destruction of the spatial structure, a spatial attention model composed of multiple layers of convolution is used to extract the spatial weight W spatial of the pixels.

[0053] W spatial =f s_attention (F ca ), W spatial ∈R 8×8 ; (4)

[0054] where f s_attention (·) is the spatial attention network, and F ca is the channel attention map.

[0055] The spatial weight is applied to the feature map to calculate the local feature attention map F att .

[0056] F att= F ca x W spatial att ∈ R 8×8×16 ; (5)

[0057] where F att is the local feature attention map, F ca is the channel attention map, and W spatial is the spatial weight of pixels;

[0058] S2, attribute encoding: semantic features are composed of semantic attributes of images, representing the relationship between categories, which is the key information of zero-shot learning, and all category attribute features are encoded by one-hot, 0 representing no attribute and 1 representing having the attribute, for example, the attributes of Gothic architecture [rose-windows, narrow-window, glazing,..., Symmetrical, curve, gable] are encoded as [1, 0, 1,..., 1, 1, 0], a mapping function from visual space to subspace is learned, and the semantic attribute vector is also mapped to the subspace;

[0059] S3, spatial mapping and classification: visual features are composed of feature maps obtained by double attention, which retain discriminative information in images, and the visual features are mapped to a common subspace through a fully connected layer, a mapping function from the visual features to the common space is learned, and the encoded attribute vectors of all categories are also mapped to the same subspace through a fully connected layer, a semantic mapping function is learned, and a nearest neighbor algorithm is used in the mapped semantic vector to find a vector matching the visual features, the class label of the sample is predicted, and the classification task is realized through nearest neighbor matching; the step S3 includes the following steps:

[0060] S3-1: a fully connected layer is used to map the extracted local feature attention map F att to a common subspace, and a visual mapping function is learned, where W1 is a visual mapping matrix;

[0061] In the training task of zero-shot learning, a visual embedding function is used to map visual features to a common space, and a class prototype feature C k of a visible class is learned, which is the average vector of visual features:

[0062]

[0063] where m is the total number of samples of each category, x i is the i-th image, and φ(x i ) represents the vector of the i-th image mapped to the common space;

[0064] ​The data set has the characteristics of large intra-class difference and small inter-class difference, so the center loss function is used to reduce the intra-class distance (public

[0065] Formula 7) to pull the samples of the same class closer;

[0066]

[0067] In the formula, φ(x i ) represents the vector of the i-th image mapped into the common space. C k is the class prototype feature. m represents that there are m images in the class;

[0068] S3-2: In order to reduce the loss of some discriminative information in the process of knowledge transfer, a generator is used to reconstruct the mapped features; the reconstruction loss (formula 8) is calculated, is the visual feature vector reconstructed by the generator;

[0069]

[0070] By minimizing the reconstruction loss, the generator is trained to make the reconstructed features closer to the actual data, so as to alleviate the problem of information loss in the feature mapping process;

[0071] S3-3: Map all encoded attribute vectors of the categories to the same subspace through a fully connected layer, and learn the semantic mapping function In the formula, W2 is the semantic mapping matrix;

[0072] In order to make the embedded visual features close to the corresponding semantic attribute embedding vectors, a regression loss is used to minimize the error between the embedding vectors:

[0073]

[0074] In the formula, represents the mean square error of the visual feature mapping vector and the semantic feature mapping vector;

[0075] In the classification task, the cross-entropy loss is often used to calculate the loss value between the prediction and the true label:

[0076]

[0077] In the formula, N represents the total number of categories, represents the probability between the predicted label and the true label; S3-4: Predict the class label of the unseen class sample x u in the common subspace, that is In the formula, D represents the Euclidean distance metric function, x u is the unseen class image, and φ(x u ) is the visual mapping vector of the unseen class image. is the mapping vector of the j-th category semantic attribute in the common subspace. argminD(·) calculates the minimum value between vectors;

[0078] The present specific embodiment performs experimental evaluation on the zero-shot general dataset CUB-200-2011 (CUB) and the architecture style dataset Architecture Style Dataset, and performs comparative experiments with some previous conventional methods, including DAP, ALE, AREN, APN, LDF and TransZero.

[0079] As shown in Figure 3 , on the general dataset CUB, the average precision of the DAM-ZSL model is 75%, which is improved by 1.3% compared with the TransZero model, indicating that DAM-ZSL can learn a visual feature representation highly related to attribute information, and the learned visual region can better represent the main object of the image.

[0080] In Figure 4 , zero-shot learning is used on the Architecture Style Dataset, and the classification precision of the DAM-ZSL model is 39.1%, which is improved by 0.7% and 0.9% compared with the TransZero model and the AREN model, indicating that applying channel and spatial attention to the zero-shot classification model can focus on the detailed regions of the image. However, compared with the APN attribute prototype network, the classification result of the model in the present specific embodiment is slightly worse, because APN learns attribute prototypes, and positioning the attribute prototypes in the visual region can more effectively reduce the number of matching samples, which has a greater impact on small data sets.

[0081] To further evaluate the influence of the double attention mechanism on the zero-shot classification result, an ablation experiment is performed, and the results are shown in Table 1.

[0082] Table 1 Influence of double attention mechanism on ACA precision (%)

[0083]

[0084] When the dual attention mechanism is not used, the classification accuracy is obviously less than that of the complete model (DAM-ZSL). In the CUB dataset, the accuracy decreases by 1.2%, and in the Architecture Style Dataset, the accuracy decreases by 2.6%. When the channel attention is used, due to the uneven images collected in the dataset, the target subject of the images in the CUB dataset is clearer, while the images in the Architecture Style Dataset contain building subjects and non-building elements such as trees, people and cars, so the classification result has little effect on the CUB dataset, but it is improved by 1.8% on the Architecture Style small dataset, which shows that the channel attention model can effectively remove the influence of background and other non-building elements and extract the building itself. When the spatial attention is used without using the channel attention, the extracted visual features are the regions with strong spatial structure in the image. For the Architecture Style Dataset, the spatial composition of the image is obvious, so the classification accuracy is improved by 2.1%. When the channel attention and the spatial attention are combined to extract the features of the image, the visual regions related to the attributes in the image can be extracted, so that the classification result is more accurate.

[0085] In order to intuitively represent the effectiveness of the dual attention mechanism in extracting local discriminative features, Grad-CAM is used to visualize the attention feature map extracted by the DAM-ZSL model, as shown in Figure 5

[0086] The model provided by the application can extract local detail features related to building style classification, such as rose windows of Gothic architecture, which shows that the introduction of the channel-spatial dual attention network into the zero-shot classification task makes the learned visual features more discriminative.

[0087] The above is only a specific embodiment of the application, but the protection scope of the application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.​

Claims

1. A zero-shot building image classification method based on a dual attention mechanism, characterized in that, Includes the following steps: S1. Discriminative visual feature extraction: Global features are extracted using the ResNeXt-101 network, and the main body of the building is located by learning different channel weights through the channel attention network. The spatial attention network embeds the location information into the channel attention map to capture the detailed features in the target. S2. Attribute Encoding: Semantic features are composed of the semantic attributes of the image, representing the relationship between categories. One-hot encoding is used to encode the attribute features of all categories, where 0 represents the absence of the attribute and 1 represents the presence of the attribute. The mapping function from the visual space to the subspace is learned, and the semantic attribute vector is mapped to the subspace at the same time. S3. Spatial Mapping and Classification: Visual features are composed of feature maps obtained by dual attention, preserving discriminative information in the image. The visual features are mapped to a common subspace through a fully connected layer, and the mapping function from the visual features to the common space is learned. At the same time, the attribute vectors encoded by all categories are also mapped to the same subspace through a fully connected layer, and the semantic mapping function is learned. In the mapped semantic vectors, the nearest neighbor algorithm is used to find the vector that matches the visual features, predict the category label of the sample, and achieve the classification task through nearest neighbor matching.

2. The zero-shot building image classification method based on dual attention mechanism according to claim 1, characterized in that: Step S1 includes the following steps: S1-1: Initialize the image size of the dataset to 256mm×256mm, input it into the backbone network ResNeXt-101 for preprocessing, extract global features of the image, and the size of the last convolutional feature map of ResNeXt-101 is 2048×8×8. S1-2: Input the global feature map extracted in step S1-1 into the channel attention network, calculate the weights of each channel, and weight them with the global feature map: W channel =SoftMax(f c_attention (F tmp )),W channel ∈R; F ca =F tmp ×W channel ,F ca ∈R 8×8×16 ; In the formula, f c_attention (·) represents a channel attention network, and SoftMax(·) makes the sum of the weights of each channel equal to 1; F ca For channel attention diagram, F tmp W represents the compressed global feature map. channel Pay attention to the channel weights; S1-3: Pay attention to the channel diagram F ca In the input spatial attention network, important regions in the image space are identified and processed. A spatial attention model composed of multiple convolutional layers is used to extract the spatial weights W of pixels. spatial Spatial weights are applied to the feature map to calculate the local feature attention map F. att : W spatial =f s_attention (F ca ),W spatial ∈R 8×8 ; F att =F ca ×W spatial ,F att ∈R 8×8×16 ; In the formula, W spatial f represents the spatial weights of a pixel. s_attention (·) represents a spatial attention network; F att For attention maps of local features, F ca This is a diagram for channel observation.

3. The zero-shot building image classification method based on dual attention mechanism according to claim 1, characterized in that: Step S3 includes the following steps: S3-1: Use a fully connected layer to focus on the extracted local features in the attention map F. att Mapping to a common subspace, learning the visual mapping function φ(x) i )=W1×f att In the formula, W1 is the visual mapping matrix; S3-2: Proposes using a generator to reconstruct the mapped features; S3-3: Map all category-encoded attribute vectors to the same subspace through a fully connected layer, and learn the semantic mapping function. In the formula, W2 is the semantic mapping matrix; S3-4: Predict unseen class samples x in the common subspace u Category labels, i.e. In the formula, D represents the Euclidean distance metric function, x u For images of unseen classes, φ(x) u ) represents the visual mapping vector of the unseen class image. Let be the mapping vector of the semantic attributes of the j-th category in the common subspace, and argminD(·) denotes the minimum value among the vectors.

Images