An image classification method, device, equipment and readable storage medium

Abstract

The application relates to an image classification method and device, equipment and a readable storage medium, and relates to the technical field of computer vision. The application comprises patch segmentation on an original feature map to obtain a plurality of patches; local attention calculation and global attention calculation are respectively performed on each patch based on an overlapping attention mechanism to obtain a grid-shaped feature map; and label embedding in the grid-shaped feature map is identified and predicted based on a self-supervised learning mechanism to obtain an image classification result corresponding to the original feature map. The application enhances information exchange between two adjacent windows in the feature map through the overlapping attention mechanism to improve the aggregation of global information. Meanwhile, the self-supervised learning is used to strengthen the learning of spatial information by the model, thereby improving the classification precision caused by the lack of inductive bias. Therefore, the combination of the overlapping attention mechanism and the self-supervised learning can effectively optimize the classification precision of the network, thereby improving the classification effect.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image classification method, apparatus, device, and readable storage medium. Background Technology

[0002] Image classification is an image processing method that distinguishes different categories of objects based on the different features they reflect in image information. It uses computers to perform quantitative analysis on images, classifying each pixel or region in an image into one of several categories to replace human visual interpretation. It is an important fundamental problem in computer vision, and is the basis for visual tasks such as object detection, image segmentation, and object tracking.

[0003] With the rapid development of deep learning, image classification based on deep learning can be divided into two main categories: one is image classification based on convolutional neural networks (CNNs), which is the standard network model for computer vision networks, consisting of several convolutional layers and pooling layers, and performs exceptionally well in image processing; the other is image classification methods based on Transformer models. Due to the great success of Transformer in the field of natural language processing, various vision Transformers for solving visual tasks have been proposed. These vision Transformers have not only achieved excellent results in image classification, semantic segmentation, instance segmentation, object detection, and other directions, but have also gradually become an alternative to convolutional neural networks in the field of computer vision.

[0004] However, traditional deep learning-based image classification algorithms often suffer from poor classification accuracy due to a lack of inductive bias when the sample size is insufficient. Summary of the Invention

[0005] This application provides an image classification method, apparatus, device, and readable storage medium to solve the problems of low classification accuracy and poor performance in related technologies.

[0006] Firstly, an image classification method is provided, including the following steps:

[0007] The original feature map is segmented into patches to obtain multiple patches;

[0008] Based on an overlapping attention mechanism, local attention and global attention are calculated for each patch to obtain a grid-like feature map;

[0009] The grid-like feature map is identified and predicted based on a self-supervised learning mechanism to obtain the image classification result corresponding to the original feature map.

[0010] In some embodiments, the overlap-based attention mechanism separately performs local attention calculation and global attention calculation on each patch to obtain a grid-shaped feature map, including:

[0011] performing linear embedding and local attention calculation on the patch to obtain a local feature map;

[0012] performing global attention calculation on the local feature map based on a global attention module, wherein there is an overlapping part between the key vector and the value vector in the global attention module;

[0013] fusing the local feature map and the global feature map to obtain a fused feature map;

[0014] performing multiple times of downsampling processing, local attention calculation and global attention calculation on the fused feature map to obtain a grid-shaped feature map.

[0015] In some embodiments, the performing multiple times of downsampling processing, local attention calculation and global attention calculation on the fused feature map to obtain a grid-shaped feature map includes:

[0016] performing downsampling processing on the fused feature map based on a patch merging module to obtain a down-sampled fused feature map;

[0017] performing local attention calculation on the down-sampled fused feature map based on a local Transformer module to obtain a local fused feature map;

[0018] performing global attention calculation on the local fused feature map based on a global attention module to obtain a global fused feature map;

[0019] fusing the local fused feature map and the global fused feature map to obtain a new fused feature map;

[0020] after repeating the above steps a preset number of times based on the new fused feature map, obtaining a final fused feature map;

[0021] performing downsampling processing on the final fused feature map based on a patch merging module to obtain a final down-sampled fused feature map;

[0022] performing local attention calculation on the final down-sampled fused feature map based on a local Transformer module to obtain a grid-shaped feature map.

[0023] In some embodiments, the performing global attention calculation on the local feature map based on a global attention module to obtain a global feature map includes:

[0024] perform global attention calculation on the local feature map based on relative position encoding in a global attention module to obtain a global feature map;

[0025] The calculation formula of the relative position encoding is:

[0026]

[0027] In the formula, Q, K and V respectively represent a query vector matrix, a key vector matrix and a value vector matrix, d represents a hidden layer dimension, and B represents an offset.

[0028] In some embodiments, when identification prediction is performed based on a self-supervised learning mechanism, the token embedding in the grid-shaped feature map is identified and predicted by the following target loss function:

[0029] Ltotalloss=Lc+μLr

[0030] In the formula, Ltotalloss represents a target loss function, Lc represents a cross-entropy loss function, Lr represents a relative positioning loss function, and μ represents a weight.

[0031] In some embodiments, the method further comprises:

[0032] randomly sampling token embeddings in a grid-shaped training feature map obtained through a training data set to obtain a plurality of groups of embedding pairs;

[0033] performing relative position prediction on each group of embedding pairs by a positioning multi-layer perception to obtain a predicted relative position of each group of embedding pairs;

[0034] determining a relative positioning loss function L r based on the predicted relative position of each group of embedding pairs and an actual relative position of each group of embedding pairs.

[0035] In some embodiments, the calculation formula of the relative positioning loss function L r is:

[0036]

[0037] In the formula, m represents an original image corresponding to the grid-shaped training feature map, D represents a training data set, e ij represents one token embedding in an embedding pair, e kl represents another token embedding in the embedding pair, (d u ,d v ) represents a predicted relative position between the token embeddings in the embedding pair, and (t u ,t v ) represents an actual relative position between the token embeddings in the embedding pair.

[0038] Secondly, an image classification device is provided, comprising:

[0039] The patch partitioning unit is used to segment the original feature map into multiple patches.

[0040] The feature processing unit is used to perform local attention calculation and global attention calculation for each patch based on an overlapping attention mechanism to obtain a grid-like feature map;

[0041] An image classification unit is used to perform recognition and prediction on the grid-like feature map based on a self-supervised learning mechanism to obtain an image classification result corresponding to the original feature map.

[0042] Thirdly, an image classification device is provided, comprising: a memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the aforementioned image classification method.

[0043] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned image classification method.

[0044] The beneficial effects of the technical solution provided in this application include: effectively improving classification accuracy and classification results.

[0045] This application provides an image classification method, apparatus, device, and readable storage medium, comprising: patching an original feature map to obtain multiple patches; performing local attention calculation and global attention calculation on each patch based on an overlapping attention mechanism to obtain a grid-like feature map; and performing recognition and prediction on the grid-like feature map based on a self-supervised learning mechanism to obtain an image classification result corresponding to the original feature map. This application enhances the information exchange between two adjacent windows in the feature map through an overlapping attention mechanism to improve the aggregation of global information, while simultaneously strengthening the model's learning of spatial information through self-supervised learning, thereby improving the classification accuracy decline caused by the lack of inductive bias. Therefore, this application effectively optimizes the classification accuracy of the network by combining an overlapping attention mechanism and self-supervised learning, thereby improving the classification effect. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1A flowchart of an image classification method provided by an embodiment of the present application is shown in the figure;

[0048] Figure 2 A schematic diagram of an image classification model architecture provided by an embodiment of the present application is shown in the figure;

[0049] Figure 3 A patch based on query vector size cropping provided by an embodiment of the present application is shown in the figure;

[0050] Figure 4 A patch based on key vector and value vector size cropping provided by an embodiment of the present application is shown in the figure;

[0051] Figure 5 An auxiliary task of self-supervised learning provided by an embodiment of the present application is shown in the figure;

[0052] Figure 6 A structural schematic diagram of an image classification device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0054] The embodiments of the present application provide an image classification method, device, equipment and readable storage medium, which can solve the problems of low classification accuracy and poor effect in related technologies.

[0055] Referring to Figure 1 and Figure 2 , an image classification method provided by an embodiment of the present application includes the following steps:

[0056] Step S10: performing patch segmentation on the original feature map to obtain a plurality of patches;

[0057] Exemplarily, it should be understood that, before performing image classification, a neural network model of a Transformer architecture based on self-supervised learning and overlapping attention mechanism is constructed, and the neural network model is trained through a preset training data set to generate a final image classification model; and then the image classification model is used for image recognition and classification prediction. Since the construction of the image classification model and the principle of actual classification application are similar, for the sake of simplicity of description, the principle of image classification will be explained and described below by taking the actual classification application of the image classification model as an example.

[0058] It can be understood that when a certain image needs to be classified and recognized, the original feature map corresponding to the image can be input into the patch partition module, so that the patch partition module can perform patch segmentation on the original feature map to obtain a plurality of non-overlapping patches. Wherein, a patch can be regarded as a label, and if the patch size is set to 4x4, the feature dimension of each patch is 4x4x3=48, and "3" represents the length dimension H, the width dimension W and the number of channels C.

[0059] Step S20: performing local attention calculation and global attention calculation on each patch based on the overlapping attention mechanism to obtain a grid-shaped feature map;

[0060] For example, in the embodiment, the local attention calculation and the global attention calculation of each patch are sequentially performed in multiple stages by the overlapping attention mechanism to finally obtain the grid-shaped feature map.

[0061] Further, the overlapping attention mechanism performs local attention calculation and global attention calculation on each patch to obtain a grid-shaped feature map, including:

[0062] performing linear embedding and local attention calculation on the patch to obtain a local feature map;

[0063] performing global attention calculation on the local feature map based on a global attention module to obtain a global feature map, wherein there is an overlapping part between the key vector and the value vector in the global attention module;

[0064] fusing the local feature map and the global feature map to obtain a fused feature map;

[0065] performing multiple downsampling processing, local attention calculation and global attention calculation on the fused feature map to obtain a grid-shaped feature map.

[0066] Exemplarily, in the present embodiment, after obtaining the patch, the patch will be subjected to a first-stage linear embedding and local attention calculation to obtain a local feature map. The first stage includes a linear embedding layer and a local Transformer module. The linear embedding layer is used to embed the patch to project the feature dimension of the patch to an arbitrary dimension, i.e., to change the feature dimension of the patch to an arbitrary dimension. Specifically, the linear embedding layer can be implemented by a two-dimensional convolution layer, while the stride and the kernel size (i.e., the convolution kernel size) are set to patch_size, and the output channel is set to determine the size of the embedding vector, i.e., the original feature map is cropped to a window size of patch_size x patch_size, then embedded, and finally the H dimension and the W dimension are unfolded and moved to the first dimension.

[0067] It should be understood that, assuming that the output after embedding is (N, 96, 224 / 4, 224 / 4), the corresponding meanings are as follows: N is the number of windows, 96 is the number of channels, and the two 224 / 4 represent H and W, respectively. Then, the H and W are unfolded and changed to (N, 96, 56x56). At this time, the H dimension and the W dimension are unfolded to the first dimension, i.e., the channel dimension is moved to the end and changed to (N, 56x56, 96).

[0068] After the linear embedding layer embeds the patch, a local Transformer module is used to perform local attention calculation on the embedding result to obtain a local feature map of the first stage. The dimension of the local feature map is Then, the local feature map enters a first global attention calculation module. The global attention calculation module reduces the dimension of the hidden layer by a 1x1 convolution layer to reduce the calculation complexity. The dimension of the hidden layer is reduced to reduction represents a changeable hyperparameter, and P represents the number of layers of the hidden layer, which can be equal to 96 or 192 or 384, and can also be determined according to actual needs, which is not limited here.

[0069] It can be understood that, in the global attention calculation module, the key vector key, the value vector value, and the query vector query are set to different sizes in the present embodiment, so as to generate patches of different sizes for key and value embedding, and the size of the patch for query embedding is larger than that of the patch for key and value embedding, and there is an overlapping part between the key and the value. For example, as shown in Figure 3 and Figure 4 The size of the query is set to 2x2, and the size of the key and the value is set to 3x3. Then, the patches are embedded based on the above-mentioned sizes.

[0070] It should be noted that if the query size is set to 4 and the size of the key and the value is 6, the total number of queries is And the total number of keys and values is Padding represents padding, and stride represents stride.

[0071] The multi-head self-attention calculation is performed on the embeddings of the query, the key and the value through the above overlapping attention mechanism, that is, even if the local feature map sequentially passes through a multi-layer perceptron containing one hidden layer, a LayerNorm layer which is the same as the local Transformer module, and a residual connection layer, and then a 1x1 convolution operation is performed on the obtained feature to obtain a global feature map. Specifically, the local feature map is first normalized by LayerNorm, and the newly obtained feature map is windowed; secondly, self-attention is calculated for each window, and then the windows are merged, followed by dropout and residual connection; then a global feature map is formed through a layer of LayerNorm, a full connection layer, and dropout and residual connection. It can be understood that since the calculation of self-attention in this embodiment is limited within the window, the computational complexity is effectively reduced.

[0072] Then, the global feature map and the local feature map are broadcasted and added to form a fusion feature map, and finally the fusion feature map is subjected to multiple downsampling processing, local attention calculation and global attention calculation to obtain a grid-shaped feature map.

[0073] Further, the multiple downsampling processing, local attention calculation and global attention calculation on the fusion feature map to obtain the grid-shaped feature map comprise:

[0074] Downsampling processing is performed on the fusion feature map based on the patch merging module to obtain a downsampled fusion feature map;

[0075] Local attention calculation is performed on the downsampled fusion feature map based on the local Transformer module to obtain a local fusion feature map;

[0076] Global attention calculation is performed on the local fusion feature map based on the global attention module to obtain a global fusion feature map;

[0077] The local fusion feature map and the global fusion feature map are fused to obtain a new fusion feature map;

[0078] After repeating the above steps based on the new fusion feature map for a preset number of times, a final fusion feature map is obtained;

[0079] The patch merging module is used for downsampling processing on the final fusion feature map to obtain a final downsampled fusion feature map.

[0080] The local Transformer module is used for local attention calculation on the final downsampled fusion feature map to obtain a grid-shaped feature map.

[0081] For example, in this embodiment, after the fusion feature map (assuming the fusion feature map is X1) is obtained through the first-stage processing and the calculation of the global attention module, the patch merging module and the local Transformer module are used for three-stage downsampling processing and local attention calculation on the fusion feature map X1.

[0082] Specifically, the patch merging module in the second stage is used for downsampling processing on the fusion feature map X1, that is, the elements are selected with an interval of 2 in the row direction and the column direction of the fusion feature map X1 (that is, each downsampling operation is twice), and then the elements are spliced into a tensor, and then the channel dimension is adjusted to twice the original through a fully connected layer, thereby obtaining a downsampled fusion feature map Y1; the local Transformer module in the second stage is used for local attention calculation on the downsampled fusion feature map Y1 to obtain a local fusion feature map Z1, so as to complete the processing in the second stage; then the global attention module is used for global attention calculation on the local fusion feature map Z1 to obtain a global fusion feature map P1, and the local fusion feature map Z1 and the global fusion feature map P1 are fused to obtain a fusion feature map X2 with a dimension of H / 8×W / 8×2C.

[0083] The patch merging module in the third stage is used for downsampling processing on the new fusion feature map X2 to obtain a downsampled fusion feature map Y2; the local Transformer module in the third stage is used for local attention calculation on the downsampled fusion feature map Y2 to obtain a local fusion feature map Z2, so as to complete the processing in the third stage; then the global attention module is used for global attention calculation on the local fusion feature map Z2 to obtain a global fusion feature map P2, and the local fusion feature map Z2 and the global fusion feature map P2 are fused to obtain a final fusion feature map X3 with a dimension of H / 16×W / 16×4C.

[0084] The patch merging module in the fourth stage is used for downsampling processing on the final fusion feature map X3 to obtain a final downsampled fusion feature map Y3; the local Transformer module in the fourth stage is used for local attention calculation on the final downsampled fusion feature map Y3 to obtain a final local fusion feature map Z3, so as to complete the processing in the fourth stage, that is, a grid-shaped feature map with a dimension of H / 32×W / 32×8C is finally formed (for example, Figure 5a feature map of size p x p comprising p x p token embeddings.

[0085] It can be seen that the subsequent three stages all comprise a patch merging module to reduce the resolution, reduce the number of tokens, adjust the channel dimension, thereby form a hierarchical design, and save the calculation amount. For example, the number of tokens input into the first stage is H / 4 x W / 4, and the number of output channels is C. After each stage, the number of tokens is reduced by 1 / 2, and the channel becomes 2 times the original. That is, the number of tokens in the second, third, and fourth stages is reduced to H / 8 x W / 8, H / 16 x W / 16, and H / 32 x W / 32, respectively, and the number of output channels is 2C, 4C, and 8C, respectively.

[0086] Further, the global attention module is used to perform global attention calculation on the local feature map to obtain a global feature map, including:

[0087] The global attention module is used to perform global attention calculation on the local feature map based on relative position encoding to obtain a global feature map.

[0088] The calculation formula of the relative position encoding is:

[0089]

[0090] In the formula, Q, K, and V represent query vector matrix, key vector matrix, and value vector matrix respectively, d represents hidden layer dimension, and B represents offset.

[0091] In the embodiment, the global attention module adds relative position encoding when performing global attention calculation on query, key, and value. The calculation formula of the relative position encoding is:

[0092]

[0093] In the formula, Q, K, and V represent query vector matrix, key vector matrix, and value vector matrix respectively, and K, R represents a space set, M 2 is the number of queries in a window, N 2 is the number of keys, d represents hidden layer dimension, and B represents offset.

[0094] It should be noted that for relative position encoding, a learnable variable with a shape of (2xwindow_size-1x2xwindow_size-1, numHeads) can be set; then the calculated relative encoding position index is selected to obtain a shape of (window_sizexwindow_size, window_sizexwindow_size, numHeads) encoding, and is added to the Attention tensor, that is, through the calculation of a numerical value, the coordinates of different positions of the matrix when the query vector and the key vector perform Attention calculation can be distinguished, thereby improving the accuracy.

[0095] Step S30: performing identification prediction on the grid-shaped feature map based on the self-supervised learning mechanism to obtain an image classification result corresponding to the original feature map.

[0096] Exemplarily, in the embodiment, when the prediction layer in the image classification model performs identification prediction on the grid-shaped feature map, the identification prediction process is constrained by the auxiliary task corresponding to the self-supervised learning mechanism in the image classification model to obtain a more accurate image classification result. It should be understood that self-supervised learning aims to mine the representation characteristics of unlabeled data as supervision information by designing auxiliary tasks, thereby improving the feature extraction capability of the model. In the natural language processing task, one of the main uses of the auxiliary task is to predict the randomly deleted words in the training set sentences to improve the extraction capability of the model for the order of features, and in the visual field, because the patch sequence of the image is longer than the sentence sequence, the feature information contained is more rich, and the replacement mode is more complex, therefore, unlike masking words in natural language processing: image reorganization will divide the image into different patches, for example, a nine-square grid, and then let the network predict the relative position information of different patches, and this process can improve the extraction capability of the model for local features and global spatial information, thereby improving the classification accuracy of the model.

[0097] The specific identification process of the prediction layer is: taking the entire grid-shaped feature map as input through a standard small multi-layer perception head, and setting an additional class token vector (i.e. class token), collecting the context information in the entire grid through the class token vector, and then outputting a posterior distribution of a target class set, the process is constrained by a loss function, and finally the above posterior distribution and the true label of the image are calculated to obtain the image classification result.

[0098] Further, when identification prediction is performed based on the self-supervised learning mechanism, the token embedding in the grid-shaped feature map is identified and predicted by the following target loss function:

[0099] Ltotal loss =L c +μL r

[0100] In the formula, L total loss Let L represent the target loss function. c L represents the cross-entropy loss function. r Let μ represent the relative positioning loss function, and μ represent the weights.

[0101] Specifically, the process of determining the target loss function is as follows:

[0102] Random sampling is performed on the label embeddings in the grid-like training feature map obtained from the training dataset to obtain multiple sets of embedding pairs;

[0103] The relative position of each embedding pair is predicted by using a localization multilayer perceptron to predict the relative position of each embedding pair.

[0104] The relative positioning loss function L is determined based on the predicted relative position and the actual relative position of each embedding pair. r Wherein, the relative positioning loss function L r The calculation formula is:

[0105]

[0106] In the formula, m represents the original image corresponding to the grid-like training feature map, D represents the training dataset, and e ij Denotes a tag embedding in an embedding pair, e kl Indicates another tokenized embedding in the embedding pair, (d u ,d v ) represents the predicted relative position between the embedded pairs labeled embeddings, (t u ,t v ) indicates the actual relative position between the embedding pairs marked by the embeddings.

[0107] As an example, in this embodiment, a localization multilayer perceptron is used to enhance the model's learning of spatial information. Specifically, during the training of the model using the training dataset, an auxiliary task based on self-supervised dashed lines is used to randomly sample the grid of token embeddings output by the model, so that the network can predict the relative distance between the sampled pairs, thereby determining the target loss function.

[0108] For example, see Figure 5 As shown, for an image m, it is divided into a p×p grid using a visual self-attention model. A set of labeled embeddings is randomly sampled from each grid to form embedding pairs, which can be represented as (e... ij ,e kl) ; then the actual relative position of the embedding pair is calculated as:

[0109]

[0110] where t u represents the actual relative horizontal distance between the two token embeddings in the embedding pair, t v represents the actual relative vertical distance between the two token embeddings in the embedding pair, (t u ,t v ) represents the actual relative position between the token embeddings in the embedding pair.

[0111] The selected embedding pair (e ij ,e kl ) is then concatenated and fed into the Localization MLP (i.e., localization multi-layer perceptron). The Localization MLP has 3 layers, with two hidden layers and GELU as the activation function, and the output layer has two neurons to predict the relative distance between the two token embedding coordinates in the embedding pair. The output of the Localization MLP is calculated as:

[0112] d u ,d v ) T = f(e ij ,e kl ) T

[0113] where (d u ,d v ) represents the predicted relative position between the token embeddings in the embedding pair.

[0114] Given a small batch of samples D (i.e., training dataset D) with a batch size of n images, for each image m, λ pairs of embedding pairs are randomly sampled, and the average loss of all embedding pairs is calculated. The relative positioning loss function L r is defined as:

[0115]

[0116] where E represents a spatial set containing multiple groups of embedding pairs, and G represents a spatial set containing multiple embedding pairs corresponding to image m.

[0117] Finally, the relative positioning loss function L r is combined with the cross-entropy loss function L c used by the original model to form a new target loss function, i.e., the target loss function can be defined as the sum of the cross-entropy loss function L c and the relative positioning loss function L r multiplied by a weight μ:

[0118] L total loss =L c +μL r

[0119] It can be seen that the embodiment adopts a combination of self-supervised learning and overlapping attention mechanism for image classification: a module for calculating global attention is added at each stage of the hierarchical local Transformer, which utilizes overlapping attention mechanism and only contains simple multiplication and addition operations, thus not increasing the computational complexity; and the overlapping key vectors and value vectors can enhance the information exchange between two adjacent windows, which is conducive to the aggregation of global information in the local Transformer; in addition, a self-supervised learning auxiliary task is added, and random sampling is performed in the label embedding output by the model to establish the dependency between the label embeddings, and the random sampling strengthens the connection between elements in the spatial position, thereby making up for the lack of convolution-specific inductive bias in the Transformer-based architecture to some extent, thereby optimizing the classification accuracy of the network.

[0120] In summary, the embodiment fully utilizes the overlapping attention mechanism to capture the global information of the Transformer model, and combines the auxiliary task of self-supervised learning to strengthen the capture of spatial information by the model, so as to improve the network performance, reduce the dependence of the model on the amount of training data, and thus effectively improve the accuracy and effect of image classification.

[0121] In addition, the embodiment is also run on a computer with 2xIntel(R)Xeon(R)E5-2640 v4 x86_64, 2.4GHz, 20 cores and 4xNvidia TeslaV100 16GB GPU, and uses public datasets ImageNet, CIFAR-10, CIFAR-100 and Oxford Flowers102 to compare with traditional image classification methods, and the results show that the accuracy of the image classification of the embodiment is improved and the parameter amount is reduced, so the image classification method provided by the embodiment can be applied to the field of autonomous driving.

[0122] The embodiment of the present application also provides an image classification device, which comprises:

[0123] a patch partition unit configured to perform patch segmentation on the original feature map to obtain a plurality of patches;

[0124] a feature processing unit configured to perform local attention calculation and global attention calculation on each patch based on the overlapping attention mechanism to obtain a grid-shaped feature map;

[0125] an image classification unit configured to perform recognition prediction on the grid-shaped feature map based on a self-supervised learning mechanism to obtain an image classification result corresponding to the original feature map.

[0126] Further, the feature processing unit is specifically configured to:

[0127] perform linear embedding and local attention calculation on the patch to obtain a local feature map;

[0128] perform global attention calculation on the local feature map based on a global attention module to obtain a global feature map, wherein there is an overlapping part between the key vector and the value vector in the global attention module;

[0129] fuse the local feature map and the global feature map to obtain a fused feature map;

[0130] perform multiple times of down-sampling processing, local attention calculation and global attention calculation on the fused feature map to obtain a grid-shaped feature map.

[0131] Further, the feature processing unit is specifically configured to:

[0132] perform down-sampling processing on the fused feature map based on a patch merging module to obtain a down-sampled fused feature map;

[0133] perform local attention calculation on the down-sampled fused feature map based on a local Transformer module to obtain a local fused feature map;

[0134] perform global attention calculation on the local fused feature map based on a global attention module to obtain a global fused feature map;

[0135] fuse the local fused feature map and the global fused feature map to obtain a new fused feature map;

[0136] after repeating the above steps based on the new fused feature map for a preset number of times, obtain a final fused feature map;

[0137] perform down-sampling processing on the final fused feature map based on a patch merging module to obtain a final down-sampled fused feature map;

[0138] perform local attention calculation on the final down-sampled fused feature map based on a local Transformer module to obtain a grid-shaped feature map.

[0139] Further, the feature processing unit is specifically configured to:

[0140] perform global attention calculation on the local feature map based on relative position encoding in the global attention module to obtain a global feature map;

[0141] The calculation formula of the relative position coding is:

[0142]

[0143] In the formula, Q, K and V respectively represent a query vector matrix, a key vector matrix and a value vector matrix, d represents a hidden layer dimension, and B represents an offset.

[0144] Further, when identifying and predicting based on the self-supervised learning mechanism, the mark embedding in the grid-shaped feature map is identified and predicted through the following target loss function:

[0145] L total loss = L c + μL r

[0146] In the formula, L total loss represents a target loss function, L c represents a cross-entropy loss function, L r represents a relative positioning loss function, and μ represents a weight.

[0147] Further, the device further comprises a training unit configured to:

[0148] randomly sample the mark embedding in the grid-shaped training feature map obtained through the training data set to obtain a plurality of embedding pairs;

[0149] predict the relative position of each embedding pair through the positioning multi-layer perception to obtain the predicted relative position of each embedding pair;

[0150] determine the relative positioning loss function L r based on the predicted relative position of each embedding pair and the actual relative position of each embedding pair.

[0151] Further, the calculation formula of the relative positioning loss function L r is:

[0152]

[0153] In the formula, m represents an original image corresponding to the grid-shaped training feature map, D represents a training data set, e ij represents one mark embedding in the embedding pair, e kl represents another mark embedding in the embedding pair, (d u ,d v ) represents the predicted relative position between the mark embeddings in the embedding pair, (t u ,t v ) represents the actual relative position between the mark embeddings in the embedding pair.

[0154] It should be noted that the skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each unit can refer to the corresponding process in the foregoing image classification method embodiments, which will not be described here.

[0155] The apparatus provided by the foregoing embodiments can be implemented as a computer program in the form of a program product, which can run on an image classification device as shown in the specification. Figure 6

[0156] The embodiments of the present application also provide an image classification device, which comprises a memory, a processor and a network interface connected through a system bus, and the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement all or part of the steps of the foregoing image classification method.

[0157] The network interface is configured to perform network communication, such as sending the assigned task. Those skilled in the art can understand that, Figure 6 The structure shown in the specification is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0158] The processor can be a CPU, and can also be other general-purpose processors, DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc. The processor is the control center of the computer device, and connects all parts of the computer device through various interfaces and lines.

[0159] ​The memory can be used to store computer programs and / or modules, and the processor realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a video playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as video data, image data, etc.), and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, an SMC (Smart Media Card), an SD (Secure Digital) card, a Flash Card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0160] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize all steps or part of steps of the image classification method.

[0161] The embodiment of the present application realizes all or part of the above processes, and can also be realized by a computer program to instruct related hardware, and the computer program can be stored in a computer readable storage medium, and the computer program is executed by a processor to realize the steps of the above methods. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, ROM (Read-Only memory), RAM (Random Access memory), electric carrier wave signal, telecommunication signal and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electric carrier wave signal and telecommunication signal.

[0162] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, a server, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage and so forth) embodying computer-readable program code.

[0163] The present application is described in reference to the flow diagrams and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing unit, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagram and / or block diagram in the flow diagrams and / or block diagrams illustrate the architecture, functionality, and operation of possible implementations of apparatuses (systems), methods and computer program products according to the present application. In this regard, each block in the flow diagrams and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable Figure 1 The flow diagram and / or block diagram in the flow diagrams and / or block diagrams illustrate the architecture, functionality, and operation of possible implementations of apparatuses (systems), methods and computer program products according to the present application. In this regard, each block in the flow diagrams and / or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable

[0164] It should be noted that, in this document, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0165] The above descriptions are only specific embodiments of the present application to make those skilled in the art understand or implement the present application. Various modifications of these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image classification method, characterized in that, Includes the following steps: The original feature map is segmented into patches to obtain multiple patches; Based on an overlapping attention mechanism, local attention and global attention are calculated for each patch to obtain a grid-like feature map; The grid-like feature map is identified and predicted based on a self-supervised learning mechanism to obtain the image classification result corresponding to the original feature map; The overlapping attention mechanism performs local and global attention calculations on each patch to obtain a grid-like feature map, including: Linear embedding and local attention computation are performed on the patch to obtain local feature maps; Global attention is calculated on the local feature map based on the global attention module to obtain the global feature map. There is an overlap between the key vector and the value vector in the global attention module. The local feature map and the global feature map are fused to obtain a fused feature map; The fused feature map is subjected to multiple downsampling processes, local attention calculations, and global attention calculations to obtain a grid-like feature map; The step of performing multiple downsampling processes, local attention calculations, and global attention calculations on the fused feature map to obtain a grid-like feature map includes: The fused feature map is downsampled based on the patch merging module to obtain the downsampled fused feature map. Local attention calculation is performed on the downsampled fused feature map based on the local Transformer module to obtain the local fused feature map; Global attention is calculated on the local fusion feature map based on the global attention module to obtain the global fusion feature map; The local fusion feature map and the global fusion feature map are fused to obtain a new fusion feature map; After repeating the above steps a preset number of times based on the new fused feature map, the final fused feature map is obtained; The final fused feature map is downsampled based on the patch merging module to obtain the final downsampled fused feature map. The local attention calculation is performed on the final downsampled fused feature map based on the local Transformer module to obtain a grid-like feature map.

2. The image classification method as described in claim 1, characterized in that, The step of performing global attention calculation on the local feature map based on the global attention module to obtain the global feature map includes: Global attention is calculated on the local feature map based on the relative position encoding in the global attention module to obtain the global feature map; The formula for calculating the relative position code is: In the formula, Q, K, and V represent the query vector matrix, key vector matrix, and value vector matrix, respectively, d represents the hidden layer dimension, and B represents the offset.

3. The image classification method as described in claim 1, characterized in that, When performing recognition and prediction based on a self-supervised learning mechanism, the following objective loss function is used to perform recognition and prediction on the label embeddings in the grid-like feature map: In the formula, Represents the target loss function. Represents the cross-entropy loss function. This represents the relative positioning loss function. Indicates the weight.

4. The image classification method as described in claim 3, characterized in that, The method further includes: Random sampling is performed on the label embeddings in the grid-like training feature map obtained from the training dataset to obtain multiple sets of embedding pairs; The relative position of each embedding pair is predicted by using a localization multilayer perceptron to predict the relative position of each embedding pair. The relative positioning loss function is determined based on the predicted relative position and the actual relative position of each embedding pair. .

5. The image classification method as described in claim 4, characterized in that, The relative positioning loss function The calculation formula is: In the formula, m represents the original image corresponding to the grid-like training feature map, and D represents the training dataset. This represents a tag embedding in an embedding pair. This indicates another tag embedding in the embedding pair. This indicates the predicted relative position between the labeled embeddings in the embedding pair. This indicates the actual relative position between the embedding pairs marked by the embeddings.

6. An image classification device, characterized in that, include: The patch partitioning unit is used to segment the original feature map into multiple patches. The feature processing unit is used to perform local attention calculation and global attention calculation for each patch based on an overlapping attention mechanism to obtain a grid-like feature map; An image classification unit is used to identify and predict the grid-like feature map based on a self-supervised learning mechanism, so as to obtain the image classification result corresponding to the original feature map; The overlapping attention mechanism performs local and global attention calculations on each patch to obtain a grid-like feature map, including: Linear embedding and local attention computation are performed on the patch to obtain local feature maps; Global attention is calculated on the local feature map based on the global attention module to obtain the global feature map. There is an overlap between the key vector and the value vector in the global attention module. The local feature map and the global feature map are fused to obtain a fused feature map; The fused feature map is subjected to multiple downsampling processes, local attention calculations, and global attention calculations to obtain a grid-like feature map; The step of performing multiple downsampling processes, local attention calculations, and global attention calculations on the fused feature map to obtain a grid-like feature map includes: The fused feature map is downsampled based on the patch merging module to obtain the downsampled fused feature map. Local attention calculation is performed on the downsampled fused feature map based on the local Transformer module to obtain the local fused feature map; Global attention is calculated on the local fusion feature map based on the global attention module to obtain the global fusion feature map; The local fusion feature map and the global fusion feature map are fused to obtain a new fusion feature map; After repeating the above steps a preset number of times based on the new fused feature map, the final fused feature map is obtained; The final fused feature map is downsampled based on the patch merging module to obtain the final downsampled fused feature map. The local attention calculation is performed on the final downsampled fused feature map based on the local Transformer module to obtain a grid-like feature map.

7. An image classification device, characterized in that, include: A memory and a processor, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the image classification method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the image classification method according to any one of claims 1 to 5.

Images