Image recognition method and system based on self-supervised multi-scale attention learning

Abstract

The invention provides an image recognition method and system based on self-supervised multi-scale attention learning, and the method comprises the steps: inputting an input image into ResNet for processing, and obtaining the output features of each stage; inputting the output feature of a predetermined target stage in each stage into the corresponding convolution block for processing to obtain a first feature of the predetermined target stage; inputting the first feature of the predetermined target stage and the output feature of the final stage in each stage into a corresponding attention module, processing to obtain a corresponding multi-scale attention map, and inputting the multi-scale attention map into a self-supervised multi-scale attention learning module; obtaining global feature representation of the corresponding stage according to the multi-scale attention maps corresponding to the predetermined target stage and the final stage and the output features, and inputting the global features corresponding to the predetermined target stage and the final stage into corresponding classifiers to obtain probability distribution of prediction categories; and obtaining a final prediction category of the picture according to the probability distribution of the prediction categories of the predetermined target stage and the final stage.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition, and more specifically, to an image recognition method and system based on self-supervised multi-scale attention learning. Background Art

[0002] In recent years, traffic sign recognition has achieved good results in terms of accuracy and robustness. The deep learning framework integrates three spatial transformer network (STN) layers into the convolutional neural network (CNN) to achieve geometric transformation of the target object in the feature layer, effectively reducing the interference of background and geometric noise. A highly compact deep convolutional neural network architecture called MicronNet uses design principles and optimization strategies at the micro and macro architecture levels to optimize information density. A micro convolutional neural network model MicronNet-BF based on the MicronNet design concept combines batch normalization and decomposition techniques to take advantage of MicronNet. However, the above methods still have difficulty recognizing traffic signs with only slight differences. In addition, due to the variability in the real world, such as poor viewing angles, motion blur, color fading, and lighting conditions, the recognition of traffic signs with only slight differences will be further difficult.

[0003] Fine-grained visual classification (FGVC) is different from traditional image classification. The challenges it faces are large intra-class differences and small inter-class differences. Specifically, objects in the same category may show significantly different postures and perspectives, while the visual differences between different subclasses are often very subtle. Therefore, in FGVC, an effective strategy is to learn effective features in discriminative areas. This enables fine-grained visual classification methods to capture tiny features in images, and thus to more accurately distinguish traffic signs with only subtle differences, thereby improving the accuracy and robustness of traffic sign recognition.

[0004] In the early exploration stage of FGVC, some research methods rely on finding discriminative regions under the guidance of manual annotation. However, manual annotation often requires professional knowledge and a lot of time investment, and it is easy to make mistakes during the annotation process. These factors limit the feasibility of manual annotation methods in practical application scenarios. Therefore, in recent years, researchers have focused more on weakly supervised learning methods that only use image labels for supervision. Among them, the introduction of attention mechanism has achieved good results. A novel recurrent attention convolutional neural network (RA-CNN) achieves mutual reinforcement of region localization and feature learning by cyclically learning discriminative region attention and region-based feature representation at multiple scales. Multi-attention convolutional neural network (MA-CNN) clusters spatially correlated fine-grained patterns into compact and discriminative parts by designing a channel grouping and weighted sub-network. Weakly supervised attention learning generates multiple attention maps to represent the discriminative object parts, and extracts sequence local features through bilinear attention pooling. At the same time, based on the attention map, an attention-guided data augmentation method is proposed to improve the efficiency of data augmentation. However, these methods lack strong supervisory signals or additional effective information to assist the training process of the attention module, which can easily lead to distracted and misleading attention. To encourage the network to learn more useful attention for fine-grained visual classification, a GradCAM-guided channel-spatial attention module supervises and constrains the attention weights by generating a coarse localization map. A self-enhanced attention mechanism encourages the network to use features from the commonly attended region for prediction by using the generated attention map as the prediction target and fitting the attention map with a category-independent projection. A counterfactual attention learning (CAL) method quantifies the quality of attention by comparing the impact of the learned attention and the uncorrected attention on the final prediction, and maximizes this difference to encourage the network to learn more effective visual attention. However, such strategies mainly focus on the high-level semantic features in the deepest layer of the network, which mainly focus on the overall structure of the target and inevitably ignore the detailed information from the tiny discriminative regions.

[0005] Therefore, how to learn multi-scale attention that can better capture subtle differences to improve fine-grained recognition performance is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of the embodiments of the present disclosure is to provide an image recognition method and system based on self-supervised multi-scale attention learning to improve fine-grained image recognition performance.

[0007] In a first aspect, the present invention provides an image recognition method based on self-supervised multi-scale attention learning, comprising: inputting an input image into ResNet for processing to obtain output features at each stage;

[0008] Inputting the output features of the predetermined target stage in each stage into the corresponding convolution block for processing to obtain the first feature of the predetermined target stage;

[0009] Inputting the first feature of the predetermined target stage and the output feature of the final stage in each stage into the corresponding attention module, processing to obtain the corresponding multi-scale attention map, and inputting the map into the self-supervised multi-scale attention learning module to assist the training of the self-supervised multi-scale attention learning module;

[0010] According to the multi-scale attention map and output features corresponding to the predetermined target stage and the final stage, the global feature representation of the corresponding stage is obtained, and the global features corresponding to the predetermined target stage and the final stage are input into the corresponding classifier to obtain the probability distribution of the predicted category; and according to the probability distribution of the predicted category of the predetermined target stage and the final stage, the final predicted category of the image is obtained.

[0011] Furthermore, the predetermined target stage is the first two stages of the final stage.

[0012] Furthermore, the step of obtaining the global feature representation of the corresponding stage according to the multi-scale attention map and output features corresponding to the predetermined target stage and the final stage includes:

[0013] The multi-scale attention maps corresponding to the predetermined target stage and the final stage are subjected to bilinear attention pooling processing with the output features of the corresponding stages to obtain a global feature representation of the corresponding stages.

[0014] Furthermore, the step of obtaining the global feature representation of the corresponding stage according to the multi-scale attention map and output features corresponding to the predetermined target stage and the final stage also includes:

[0015] The global feature representations corresponding to the predetermined target stage and the final stage are concatenated in the channel dimension to obtain a new global feature representation;

[0016] Inputting the new global feature representation into a corresponding classifier to obtain a first probability distribution of the predicted category;

[0017] The final predicted category of the picture is obtained according to the probability distribution of the predicted categories in the predetermined target stage and the final stage, and the first probability distribution.

[0018] Further, obtaining the final predicted category of the picture according to the probability distribution of the predicted categories in the predetermined target stage and the final stage and the first probability distribution includes:

[0019] The probability distributions of the predicted categories at the predetermined target stage and the final stage, and an average value of the first probability distribution are used as the final predicted category of the picture.

[0020] In a second aspect, the present invention provides an image recognition system based on self-supervised multi-scale attention learning, comprising: an image processing module, for inputting an input image into ResNet for processing to obtain output features at each stage;

[0021] A convolution processing module, used for inputting the output features of the predetermined target stage in each stage into the corresponding convolution block for processing, so as to obtain the first features of the predetermined target stage;

[0022] An attention processing module, used for inputting the first feature of the predetermined target stage and the output feature of the final stage in each stage into the corresponding attention module, processing to obtain the corresponding multi-scale attention map, and inputting the map into the self-supervised multi-scale learning attention module to assist the training of the self-supervised multi-scale attention learning module;

[0023] The category prediction module is used to obtain the global feature representation of the corresponding stage according to the multi-scale attention map and output features corresponding to the predetermined target stage and the final stage, and input the global features corresponding to the predetermined target stage and the final stage into the corresponding classifier to obtain the probability distribution of the predicted category; the final predicted category of the image is obtained according to the probability distribution of the predicted category of the predetermined target stage and the final stage.

[0024] Furthermore, the predetermined target stage is the first two stages of the final stage.

[0025] Furthermore, the category prediction module is specifically used to: perform bilinear attention pooling processing on the multi-scale attention maps corresponding to the predetermined target stage and the final stage and the output features of the corresponding stage to obtain a global feature representation of the corresponding stage.

[0026] Furthermore, the category prediction module is also used to: concatenate the global feature representations corresponding to the predetermined target stage and the final stage on the channel dimension to obtain a new global feature representation; input the new global feature representation into the corresponding classifier to obtain a first probability distribution of the predicted category; and obtain the final predicted category of the image based on the probability distribution of the predicted category of the predetermined target stage and the final stage, and the first probability distribution.

[0027] Furthermore, the category prediction module is also used to: use the probability distribution of the predicted categories in the predetermined target stage and the final stage, and the average value of the first probability distribution as the final predicted category of the picture.

[0028] The image recognition method and system based on self-supervised multi-scale attention learning of the present invention utilizes the characteristics of different deep features of the network to learn from each other and obtain a multi-scale attention map that can better distinguish subtle differences. Multi-scale attention maps are generated on feature layers of different depths of the network, and these attention maps carry the key information and characteristics of the features of their respective deep layers. Then, the generation of the middle-layer attention map is supervised by the deep-layer attention map to filter background noise. At the same time, in this supervision process, the middle-layer convolution filter can be prompted to mainly learn the feature representation on the target object, so that the network can locate the key information of the target object at an early stage, thereby enhancing the ability of the deep-layer attention map to extract local features and improving the performance of fine-grained image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 is a flowchart of an image recognition method based on self-supervised multi-scale attention learning according to an embodiment of the present disclosure.

[0031] Figure 2 This is a network architecture diagram of an image recognition method based on self-supervised multi-scale attention learning according to an embodiment of the present disclosure.

[0032] Figure 3 It is a schematic diagram of the principle of self-supervised multi-scale attention learning in the image recognition method based on self-supervised multi-scale attention learning according to an embodiment of the present disclosure.

[0033] Figure 4 is a structural diagram of an image recognition system based on self-supervised multi-scale attention learning according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0035] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0036] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described in the present embodiment may be embodied in a wide variety of forms, and any specific structure and / or function described in the present embodiment is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described in the present embodiment may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described in the present embodiment may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described in the present embodiment may be used to implement this device and / or practice this method.

[0037] Figure 1 is a flow chart of an image recognition method based on self-supervised multi-scale attention learning according to an embodiment of the present disclosure. Figure 1 As shown, the image recognition method based on self-supervised multi-scale attention learning includes:

[0038] Step 101: Input the input image into ResNet for processing to obtain output features of each stage;

[0039] Step 102: Input the output features of the predetermined target stage in each stage into the corresponding convolution block for processing to obtain the first features of the predetermined target stage;

[0040] Step 103: Input the first feature of the predetermined target stage and the output feature of the final stage in each stage into the corresponding attention module, process to obtain the corresponding multi-scale attention map, and input it into the self-supervised multi-scale attention learning module to assist the training of the attention module;

[0041] Step 104: obtaining a global feature representation of the corresponding stage according to the multi-scale attention map and output features corresponding to the predetermined target stage and the final stage, and inputting the global features corresponding to the predetermined target stage and the final stage into the corresponding classifier to obtain the probability distribution of the predicted category; and obtaining the final predicted category of the image according to the probability distribution of the predicted category of the predetermined target stage and the final stage.

[0042] This embodiment can use the characteristics of different deep features of the network to learn from each other and obtain a multi-scale attention map that can better distinguish subtle differences. Multi-scale attention maps are generated on feature layers of different depths of the network. These attention maps carry the key information and characteristics of the features of their respective deep layers. Then, the generation of the middle-layer attention map is supervised by the deep attention map to filter background noise. At the same time, in this supervision process, the middle-layer convolution filter can be prompted to mainly learn the feature representation on the target object, so that the network can locate the key information of the target object at an early stage, thereby enhancing the ability of the deep attention map to extract local features and improving the performance of fine-grained image recognition.

[0043] There are also multiple preferred embodiments of the above-mentioned image recognition method based on self-supervised multi-scale attention learning.

[0044] In order to retain the detail information from the tiny discriminative area, the middle-level features are introduced to generate the middle-level attention. At the same time, by utilizing the respective characteristics of the deep features and the middle-level features, that is, the deep features contain global semantic information that is highly relevant to the task and can effectively separate the target object from the background noise, and the middle-level features have a higher spatial resolution and can capture the detail information in the image, so as to promote mutual learning between the deep and middle-level attention maps, so as to overcome the limitations of the middle-level attention map itself and enhance the feature extraction capability of the deep attention map for discriminative local areas. The network architecture diagram in the image recognition method based on self-supervised multi-scale attention learning according to the embodiment of the present disclosure is as follows: Figure 2 As shown in Figure 1, ResNet is used as the backbone network. ResNet is divided into S stages, each of which is a collection of layers operating on feature maps with the same spatial dimensions. This embodiment promotes the network to learn multi-scale attention maps that can better distinguish subtle differences through mutual learning between multi-scale attention maps. To this end, Figure 2 Based on the baseline model Baseline shown in the middle dotted box, the middle-level features are introduced to generate a multi-scale attention map together with the deep-level features. At the same time, the classification results are output in both the middle and deep layers, and a multi-scale feature network MSFN is constructed, as shown in Figure 2 To ensure that the introduced mid-level features have sufficient discriminative information, two stages before the final stage S, namely stage S-2 and stage S-1, are selected as the source of mid-level features. The output features of these stages can be defined as Among them, H l , W l and C l They represent the height, width and number of channels of the feature map of the first stage, respectively, l∈{S-2, S-1, S}. The specific process of the entire network is as follows.

[0045] First, the input image is processed by ResNet to obtain the output features F at each stage l Then, the middle-level feature F u Lost

[0046] Enter its corresponding convolution block B u After being processed by formula (1), the feature F is obtained u '', u∈{S-2,S-1}. Specifically, the feature F at any stage l is input to the corresponding convolution block B l It is processed in and can be expressed as:

[0047] F″ l =B l (F l ) (1)

[0048]

[0049] Where l∈{1,...,S}, Relu represents the activation function, and bn represents batch normalization. Represents a convolution operation using a 1x1 convolution kernel, C n is the number of input channels, is the number of output channels. l '' and F l ' indicates F l After convolution block B l The output feature map.

[0050] Next, F u ''With the output feature F of the last stage S Input to the attention module, after being processed by formula (4), a multi-scale attention map is obtained Among them, H l , W l and M represent the attention map A l The height, width and number of. This example uses a multi-head attention module W, which can pass the feature map X∈R H×W×C Generate attention map A∈E H×W×M , which can be expressed as:

[0051]

[0052] The multi-head attention module W is implemented by a convolutional layer followed by ReLU activation, A K ∈R H×W represents the attention map covering a certain part, such as the wing of a bird or the wheel of a car, and M is the number of attention maps.

[0053] Multi-scale Attention MapA lIt will be input into the self-supervised multi-scale attention learning module SMAL to assist the training process of the attention module, such as Figure 2 Then, the output features and attention map of the corresponding layer are subjected to bilinear attention pooling BAP, and processed by formulas (5) and (6) to obtain the global feature representation V of the corresponding layer. l Specifically, using the attention map A K Softly weight the feature map X and aggregate it through the global average pooling operation to obtain the corresponding part feature representation V K ∈R c , which can be expressed as:

[0054]

[0055] Finally, all the part feature representations are summarized into a global feature representation V∈R MC , which can be expressed as:

[0056] V=Cat(V 1 ,V 2 ,...,V M ) (6)

[0057] Where Cat represents concatenation in the channel dimension. The generation process of the attention map is performed through the attention regularization loss L A Weak supervision can be expressed as:

[0058]

[0059] C K ←C K +β(V K -C K ) (8)

[0060] Among them, C K is the feature center of the kth object part. K It can be initialized from zero and updated via a moving average, with β controlling the update rate.

[0061] Finally, the global feature representation is sent to the corresponding classifier l According to formula (9), the probability distribution P of the predicted category is obtained. l Specifically, the classifier contains a global maximum pooling layer GMP, a certain number of fully connected layers FC, and an exponential linear unit Elu. The feature map X is input to the Classifier l The corresponding prediction score P will be obtained l , which can be expressed as:

[0062] P l=Classifier l (F″ l )=Elu(FC(GMP(X))) (9)

[0063] At the same time, in order to make full use of multi-scale features, the global feature representations of the last three stages are combined to obtain a new global feature representation V S+1 , which can be expressed as:

[0064] V S+1 =Cat(V S-2 ,V S-1 ,V S ) (10)

[0065] Among them, Cat represents concatenation in the channel dimension. This feature representation is then also sent to the corresponding classifier Classifier S+1 In the above example, we obtain the probability distribution P of the predicted category. S+1 Finally, in order to obtain stable generalization performance, the final classification result is the average of the prediction outputs of multiple stages:

[0066]

[0067] It is a very effective strategy to use middle-level features to extract detailed information of small discriminant areas. However, it is not enough to rely solely on middle-level features because they themselves have certain limitations. These features have weak semantic expression capabilities and are insufficient in their ability to distinguish target objects from background noise, so they are easily disturbed by background noise. Relatively speaking, deep features carry knowledge that is highly relevant to the task and can effectively separate target objects from background noise. The attention map generated based on middle-level features and deep features will also reflect these characteristics. Therefore, this embodiment proposes a self-supervised multi-scale attention learning method, which uses a deep attention map to supervise the generation of a middle-level attention map to help the middle-level attention map filter background noise. Algorithm 1 is as follows.

[0068]

[0069]

[0070] First, for attention maps of different scales Perform average pooling operation on the channel to obtain the aggregated attention map Among them, H l , W l and M represent the attention map A l The height, width and number of, l∈{S-2,S-1,S}, can be expressed as:

[0071]

[0072] This embodiment calculates the average of all attention maps in this way to preserve the overall structure of the target, because it comprehensively considers all the discriminative parts, thereby more comprehensively expressing the information in the attention map. Next, in order to achieve the supervision of the deep attention map on the middle-level attention map, their sizes need to be adjusted. Non-overlapping pooling operations are performed on the spatial dimensions to make them the same size as the deep attention map Consistent, thereby reducing the parameters in subsequent operations, can be expressed as:

[0073]

[0074] where n u ×n u is the pooling window size, u∈{S-2,S-1}.

[0075] Then, and Reshape into a vector V l ∈R HW×1 , and transpose it to get vector V l T ∈R 1×HW Next, a subtraction operation Sub is defined, which behaves similarly to matrix multiplication. Specifically, by applying V l and V l T Sub will generate a new matrix S l ∈R HW×HW , which can be expressed as:

[0076] S l =Sub(V l ,V l T ) (14)

[0077] Matrix S l Each element of represents the difference between the pixel values ​​of two elements in the corresponding aggregated attention map. For example, Indicates V l The i-th element in V l T The difference between the jth elements in . Means V l The pixel value of the i-th element in is less than By comparing V l The i-th element in V l TThe difference between all other elements in the array can be used to determine the sorting order of the element among all elements.

[0078] Since the deep attention map is highly sensitive to the target object and less sensitive to background noise, the middle-level attention map can be made to approximate this feature. However, directly approximating the pixel difference between the two is not advisable because the absolute size of the pixel values ​​in the attention maps of different scales lacks comparability. Nevertheless, the pixel value sorting order of elements at the same position between them can be the same. Therefore, the pixel value sorting order of each element in the deep attention map is taken as the target, and the elements in the middle-level attention map are made to approximate this sorting order. The specific operation can be expressed as:

[0079]

[0080] Where * represents multiplication, u∈{S-2,S-1}, max(·) represents selecting the maximum value from a set of values, and sign(·) is defined as a sign function, which returns the corresponding sign according to the positive or negative value of the input value: when the input value is greater than zero, the return value is 1; when the input value is less than zero, the return value is -1; if the input value is exactly zero, the return value is set to 0. At the same time, only the upper triangular part of the matrix is ​​processed, because the elements of the upper and lower triangles have the same absolute value, but opposite signs. L SMAL Encourage S u With S S have the same sort order. For example, when and When the signs are the same, max(·) returns 0, indicating that this element does not need to be adjusted. and When the signs are opposite, that is, the sorting order is different, max(·) returns The absolute value of , thereby increasing the loss value, prompting the network to optimize these inconsistent elements. In this way, the mid-level attention map will have a high response on the target object and a low response on the background noise during the generation process, thereby effectively filtering out the background noise.

[0081] The optimization of the deep attention map is reflected in the generation process of the middle-level attention map. Specifically, the middle-level attention map is generated based on the middle-level features. As defined in formula (4), A u =W(X u ), where X u Assumed to be a mid-level feature. Under the supervision of the deep attention map, the mid-level attention map gradually focuses on the target object and effectively filters the background noise. In this process, X uIt will also gradually be learned as a good feature representation that can focus on the target object, which will help the network lock key information at an early stage and further enhance the deep attention map's ability to extract features from discriminative local areas.

[0082] For the output and attention module of each stage, the loss is calculated by combining the cross-entropy loss and formula (7):

[0083]

[0084] Cross entropy loss is a commonly used loss function in classification problems, especially in deep learning. It measures the difference between the predicted distribution and the true distribution. The smaller the value, the closer the model's prediction is to the actual situation. In multi-classification problems, the formula for cross entropy loss is:

[0085]

[0086] Where N is the number of categories, y i is the true label of the sample belonging to the i-th category, P i is the probability that the network predicts that the sample belongs to the i-th class.

[0087] Then, the self-supervised multi-scale attention learning method is used to assist the training of the attention module. The final loss function can be written as:

[0088]

[0089] where α u It is a hyperparameter used to control the supervision strength of the deep attention map on the middle attention map.

[0090] This example conducts generalization experiments on multiple recognized FGVC benchmark datasets to evaluate the effectiveness of the proposed self-supervised multi-scale attention learning method. The traditional spatial attention is used as the baseline, and the self-supervised multi-scale attention learning method is compared with the baseline method and other state-of-the-art methods. The experimental setup, implementation details, and results are described below.

[0091] Table 1 Detailed information of datasets in fine-grained visual classification tasks

[0092]

[0093] This embodiment conducts generalization experiments on three extensive and competitive datasets, namely CUB-200-2011, Stanford Cars, and FGVC-Aircraft, as shown in Table 1. The CUB-200-2011 dataset covers 200 species of birds, including 5,994 images for training and 5,794 images for testing. The Stanford Cars dataset contains 16,185 images of 196 different types of cars, of which 8,144 images are classified as training sets and 8,041 images are used for testing. As for the FGVC-Aircraft dataset, it contains 6,667 training images and 3,333 testing images of 100 different types of aircraft.

[0094] This example uses ResNet50 as the backbone network. For the attention model, the number of attention points is set to 32, and a weakly supervised data enhancement method is used. The network training uses the stochastic gradient descent (SGD) algorithm for 200 cycles, with a momentum of 0.9 and a weight decay of 5×10 -4 The batch size of the FGVC-Aircraft dataset is 4, and the batch size of the other two datasets is 6, and the image size is 448×448. The initial learning rate of the backbone network is set to 1×10 -4 The initial learning rate of the newly added convolutional layer and the fully connected layer is 1×10 -3 , and gradually decreases during training according to the cosine annealing strategy. On the CUB-200-2011 dataset, the loss L SMAL It is designed to be added after the second cycle of training. On the FGVC-Aircraft and Stanford Cars datasets, the loss L SMAL It is designed to be added after the 4th cycle of training. The reason for this is that the loss L SMAL It can only work after the attention map is initially formed.

[0095] In addition, this embodiment conducts an ablation experiment and uses ResNet50 as the backbone network.

[0096] (1) This embodiment compares the classification accuracy of the baseline network, the multi-scale feature network MSFN, and the multi-scale feature network after introducing self-supervised multi-scale attention learning SMAL to explore the impact of self-supervised multi-scale attention learning on network performance. The specific results are shown in Table 2. Compared with the baseline network, the multi-scale feature network achieved 0.84%, 1.02%, and 0.51% classification accuracy improvements on the CUB-200-2011, FGVC-Aircraft, and Stanford Cars datasets. Furthermore, by incorporating self-supervised multi-scale attention learning, the mutual learning between multi-scale attention maps is promoted, thereby increasing the classification accuracy by 1.52%, 1.32%, and 0.82%. This result effectively confirms that self-supervised multi-scale attention learning can improve multi-scale attention maps, thereby further improving the performance of the network.

[0097] Table 2 Effect of self-supervised multi-scale attention learning on network performance

[0098]

[0099] (2) This example evaluates the hyperparameter α in detail on the CUB-200-2011 dataset. u Impact on the network, α u Represents the deep attention map A S For the middle-level attention map A u The supervision strength is u∈{S-2,S-1}. The specific results are shown in Table 3, where P l represents the classification accuracy of each stage, l∈{S-2,S-1,S,S+1,final}. The results show that when A S-2 and A S-1 When supervision is performed simultaneously, as the supervision intensity increases, the classification accuracy of each stage is improved, P S-2 The maximum improvement is 0.96%, P S-1 The maximum energy can be increased by 1.04%, P S The maximum improvement is 0.31%, P S+1 The maximum improvement is 0.58%, P final The maximum improvement is 0.68%. S-2 =0,α S-2 =0.01, that is, only for A S-1 When conducting supervision, P S-2 Not much change, P S-1 To P final All of them are improved; when α S-2 =0.01,α S-2 =0, that is, only for A S-2 When conducting supervision, P S-2 There is a significant improvement, while PS-1 To P final Based on the above data, it can be concluded that using the deep attention map to supervise the middle-level attention map can help the middle-level attention map filter out background noise and focus on the target object, thereby improving the classification accuracy of the middle-level stage. For the deep stage S, although no additional processing is performed, it can still be seen that the deep classification accuracy is also improved after supervising the middle-level attention map. This verifies that the supervision of the middle-level attention map can also enable the convolution filter in the middle-level stage to learn good features, which is beneficial for the network to lock key information in the early stage and further enhance the deep attention map's ability to extract features for discriminative local areas.

[0100] Table 3 Different α u Impact on network performance

[0101]

[0102] (3) Effectiveness of combining multi-stage prediction outputs. Since the information of each stage is complementary, the final classification result of this embodiment is the average of the prediction outputs of multiple stages. In order to verify the effectiveness of this method, this embodiment shows the classification accuracy of each stage and the classification accuracy of their different combinations in Table 4. S-1～S+1 Indicates P S-1 , P S and P S+1 The average value of , and the rest are expressed in a similar way. It can be seen that for all data sets, when the prediction outputs of each stage are combined, the classification accuracy is higher than the classification accuracy of a single classifier. For the CUB-200-2011 data set, P S-2～S+1 The value of is the largest, indicating that each stage can provide useful information, thus maximizing the comprehensive classification accuracy. In contrast, on the FGVC-Aircraft and Stanford Cars datasets, P S-1～S+1 The value of is the largest, and P S-2～S+1 The value of P has decreased. The reason is that the receptive field of the attention map in the S-2 stage is relatively small, and it can only capture relatively small details. For objects with rigid structural characteristics and large sizes such as airplanes and cars, classification needs to rely on the observation of larger parts, and the attention map in the S-2 stage is not suitable for this, resulting in P S-2～S+1 Therefore, for these datasets, only combining the S-1 to S+1 stages can achieve the best performance.

[0103] Table 4 Performance of prediction output combinations at different stages

[0104]

[0105] On the CUB-200-2011, Stanford Cars and FGVC-Aircraft datasets, this embodiment compares the proposed method with the existing methods. For detailed comparison data, please refer to Table 5. The classification accuracy data of the existing methods are derived from the results reported in their respective papers. The results show that the method of this embodiment performs better than the existing well-performing methods on the CUB-200-2011 and FGVC-Aircraft datasets. For example, the classification accuracy of CMAL-Net is 0.2% higher than that of AP-CNN on the FGVC-Aircraft dataset, and the classification accuracy is 2.1% higher than that of AP-CNN on the CUB-200-2011 dataset. On the Stanford Cars dataset, the classification accuracy comparable to that of the existing methods is also obtained. These data strongly confirm the effectiveness of the method proposed in this embodiment.

[0106] Table 5 Comparison with existing methods

[0107]

[0108] Traffic sign recognition has achieved good results in terms of accuracy and robustness. However, for those traffic signs with only minor differences, the recognition process still faces challenges. In addition, the collection of traffic sign data sets is often affected by factors such as the visual environment and shooting angle. The blurring of traffic signs, lighting problems, and sign deformation caused by these factors will further hinder the recognition of traffic signs with only minor differences. This embodiment applies the proposed method to the field of traffic sign recognition, aiming to effectively distinguish traffic signs with only minor differences even when affected by environmental factors.

[0109] This embodiment uses the TT100K dataset to verify the effectiveness of the proposed method. The reason for choosing the TT100K dataset instead of other datasets is that it contains more types of traffic signs, and there are a large number of traffic signs with only slight differences. In addition, the collection process of this dataset is affected by factors such as the visual environment and shooting angle, which increases the complexity of the data. The TT100K dataset was originally used for traffic sign detection tasks, in which traffic signs occupy a small area in the entire image, so all images were cropped and resized to 48×48. In addition, in order to alleviate the possible impact of data imbalance, data enhancement strategies were implemented for categories with a small number of samples in the training set, and the categories in the training set and the test set were aligned. Finally, the TT100K dataset contains 149 categories, with 22,285 images in the training set and 8,255 images in the test set.

[0110] This example uses ResNet34 as the backbone network. For the attention model, the number of attention points is set to 32, and a weakly supervised data enhancement method is used. The network training uses the stochastic gradient descent (SGD) algorithm, with 100 cycles, a momentum of 0.9, and a weight decay of 5×10 -4 The batch size is 32, and the initial learning rate of the backbone network is set to 2×10 -4 The initial learning rate of the newly added convolutional layer and the fully connected layer is 2×10 -3 , and gradually decreases during the training process according to the cosine annealing strategy. SMAL Designed to be added after the second epoch of training.

[0111] This example compares the proposed method with the existing fine-grained recognition method. To ensure the fairness of the comparison, this example makes appropriate adjustments to the existing fine-grained recognition method, uniformly uses ResNet34 as the backbone network, and ensures that the network can process 48×48 input images.

[0112] In terms of overall classification accuracy, the specific results are shown in Table 7. The fine-grained recognition methods are superior to the traditional traffic sign recognition methods, indicating that the fine-grained methods can effectively recognize traffic signs with slight differences in complex environments, which are often difficult to distinguish by traditional methods. The method of this embodiment is superior to the existing fine-grained recognition method, further demonstrating the superiority of the method of this embodiment.

[0113] Table 6 Comparison of TT100K overall classification accuracy

[0114]

[0115] This embodiment selects some representative categories from all the categories for detailed comparison. In i10 to w59 in Table 8, they represent traffic signs of different directions respectively. Traditional traffic sign recognition methods are sometimes unable to accurately distinguish similar direction signs, such as i10 and i12, p19 and p23, w58 and w59, which often indicate opposite directions. A common result is that the classification accuracy of one direction is higher, while the classification accuracy of the relative direction is lower. Existing fine-grained recognition methods have improved this problem to a certain extent, but some signs still cannot be accurately recognized. The method proposed in this embodiment can more accurately determine the directions represented by these signs.

[0116] Table 7 Comparison of classification accuracy of traffic signs in different directions on the TT100K dataset

[0117]

[0118] In Table 9, from pl20 to pm35, there are some subtle differences between these traffic signs. For example, the weight limit sign pm20 has an additional letter "t" in the lower right corner compared to the speed limit sign pl20; and the height limit sign ph5 has an additional letter "m" compared to the speed limit sign pl5. Traditional traffic sign recognition methods, which mainly rely on overall appearance features, often ignore these small but important details, resulting in low classification accuracy. Existing fine-grained recognition methods are also unable to identify them well. The method proposed in this embodiment can more accurately distinguish these similar signs through deep mining of subtle features, and its classification accuracy is higher than that of traditional traffic sign recognition methods and existing fine-grained recognition methods.

[0119] Table 8 Comparison of classification accuracy of subtle traffic signs in TT100K dataset

[0120]

[0121]

[0122] In pl60 to il110 of Table 10, the differences between these traffic signs are mainly reflected in the different background colors. For example, the maximum speed limit sign pl60 and the minimum speed limit sign il60, although they have the same internal numbers, have different borders and background information. Traditional traffic sign recognition methods mainly rely on the overall shape and characteristics of the sign for recognition. When encountering lighting changes, image blur or noise interference, the color and border details of the sign are often ignored, resulting in low classification accuracy. Existing fine-grained visual classification methods are sometimes still not accurate enough to distinguish similar signs. The method of this embodiment can focus on detailed information and improve the ability to resist such changes, thereby achieving higher classification accuracy when classifying similar signs.

[0123] Table 9 Comparison of classification accuracy of traffic signs with different background colors in the TT100K dataset

[0124]

[0125] In ph2.2 to i4l in Table 11, ph represents traffic signs of height limit. The difficulty in their recognition lies in whether the letter "m" and the decimal point can be found accurately. The difference between i4 and i4l is that i4l has an additional directional arrow. According to the results, the traditional traffic sign recognition method has obvious deficiencies in the classification of traffic signs of height limit, and the classification accuracy is generally low. It is also impossible to distinguish i4 from i4l. Existing fine-grained visual classification methods are sometimes unable to identify them well. The method proposed in this embodiment can capture these details and has a higher classification accuracy.

[0126] In addition, in the recognition process of certain traffic signs, the classification accuracy of the traditional traffic sign recognition method is more significantly different from that of the method of this embodiment. This is because in a complex environment, it becomes more difficult to capture subtle differences. For example, signs such as PL20, PH4, and I4 are relatively blurry, PH5, PH4.5, PL60, and PL110 are affected by insufficient light, and PH2.5 and PL80 are not only blurry, but also tilted. In contrast, the method of this embodiment can focus more on the details of the target object and is less affected by the external environment, so it can still maintain a high classification accuracy under these complex conditions.

[0127] Table 10 Comparison of classification accuracy of height-limited traffic signs in TT100K dataset

[0128]

[0129] Figure 4 is a structural diagram of an image recognition system based on self-supervised multi-scale attention learning according to an embodiment of the present disclosure. Figure 4 The embodiment shown can be used to explain this embodiment. Figure 4 Shown: An image recognition system based on self-supervised multi-scale attention learning, comprising:

[0130] Image processing module 401, used to input the input image into ResNet for processing to obtain output features of each stage;

[0131] A convolution processing module 402, configured to input the output features of the predetermined target stage in each stage into the corresponding convolution block for processing, so as to obtain the first features of the predetermined target stage;

[0132] An attention processing module 403 is used to input the first feature of the predetermined target stage and the output feature of the final stage in each stage into the corresponding attention module, process to obtain a corresponding multi-scale attention map, and input it into a self-supervised multi-scale attention learning module to assist the training of the attention module;

[0133] The category prediction module 404 is used to obtain the global feature representation of the corresponding stage according to the multi-scale attention map and output features corresponding to the predetermined target stage and the final stage, and input the global features corresponding to the predetermined target stage and the final stage into the corresponding classifier to obtain the probability distribution of the predicted category; the final predicted category of the image is obtained according to the probability distribution of the predicted category of the predetermined target stage and the final stage.

[0134] Preferably, the predetermined target stages are the first two stages of the final stage.

[0135] Preferably, the category prediction module 404 is specifically used to: perform bilinear attention pooling processing on the multi-scale attention maps corresponding to the predetermined target stage and the final stage and the output features of the corresponding stage to obtain a global feature representation of the corresponding stage.

[0136] Preferably, the category prediction module 404 is also used to: concatenate the global feature representations corresponding to the predetermined target stage and the final stage on the channel dimension to obtain a new global feature representation; input the new global feature representation into the corresponding classifier to obtain a first probability distribution of the predicted category; and obtain the final predicted category of the image based on the probability distribution of the predicted category of the predetermined target stage and the final stage, and the first probability distribution.

[0137] Preferably, the category prediction module 404 is further configured to: use the probability distribution of the predicted categories in the predetermined target stage and the final stage, and an average value of the first probability distribution as the final predicted category of the picture.

[0138] This embodiment uses the characteristics of different deep features of the network to learn from each other and obtain a multi-scale attention map that can better distinguish subtle differences. Multi-scale attention maps are generated on feature layers of different depths of the network. These attention maps carry the key information and characteristics of the features of their respective deep layers. Then, the generation of the middle-layer attention map is supervised by the deep attention map to filter background noise. At the same time, in this supervision process, the middle-layer convolution filter can be prompted to mainly learn the feature representation on the target object, so that the network can locate the key information of the target object at an early stage, thereby enhancing the ability of the deep attention map to extract local features and improving the performance of fine-grained image recognition.

[0139] It should be noted that not all steps and modules in the above-mentioned processes and system structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0140] In the above embodiments, the hardware unit can be realized by mechanical means or electrical means. For example, a hardware unit can include permanent dedicated circuits or logic (such as special processors, FPGA or ASIC) to complete the corresponding operation. The hardware unit can also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set by software to complete the corresponding operation. Concrete implementation (mechanical means or dedicated permanent circuits or temporarily set circuits) can be determined based on cost and time considerations.

[0141] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. An image recognition method based on self-supervised multi-scale attention learning, characterized in that: include: Input the input image into ResNet for processing to obtain the output features of each stage; Inputting the output features of the predetermined target stage in each stage into the corresponding convolution block for processing to obtain the first feature of the predetermined target stage; Inputting the first feature of the predetermined target stage and the output feature of the final stage in each stage into the corresponding attention module, processing to obtain the corresponding multi-scale attention map, and inputting the map into the self-supervised multi-scale attention learning module to assist the training of the self-supervised multi-scale attention learning module; According to the multi-scale attention map and output features corresponding to the predetermined target stage and the final stage, the global feature representation of the corresponding stage is obtained, and the global features corresponding to the predetermined target stage and the final stage are input into the corresponding classifier to obtain the probability distribution of the predicted category; and according to the probability distribution of the predicted category of the predetermined target stage and the final stage, the final predicted category of the image is obtained.

2. The image recognition method based on self-supervised multi-scale attention learning according to claim 1, characterized in that: The predetermined target stages are the first two stages of the final stage.

3. The image recognition method based on self-supervised multi-scale attention learning according to claim 2, characterized in that: The step of obtaining the global feature representation of the corresponding stage according to the multi-scale attention map and output features corresponding to the predetermined target stage and the final stage includes: The multi-scale attention maps corresponding to the predetermined target stage and the final stage are subjected to bilinear attention pooling processing with the output features of the corresponding stages to obtain a global feature representation of the corresponding stages.

4. The image recognition method based on self-supervised multi-scale attention learning according to claim 3, characterized in that: The step of obtaining the global feature representation of the corresponding stage according to the multi-scale attention map and output features corresponding to the predetermined target stage and the final stage also includes: The global feature representations corresponding to the predetermined target stage and the final stage are concatenated in the channel dimension to obtain a new global feature representation; Inputting the new global feature representation into a corresponding classifier to obtain a first probability distribution of the predicted category; The final predicted category of the picture is obtained according to the probability distribution of the predicted categories in the predetermined target stage and the final stage, and the first probability distribution.

5. The image recognition method based on self-supervised multi-scale attention learning according to claim 4, characterized in that: The final predicted category of the picture obtained according to the probability distribution of the predicted category at the predetermined target stage and the final stage and the first probability distribution includes: The probability distributions of the predicted categories at the predetermined target stage and the final stage, and an average value of the first probability distribution are used as the final predicted category of the picture.

6. An image recognition system based on self-supervised multi-scale attention learning, characterized in that: include: The image processing module is used to input the input image into ResNet for processing and obtain the output features of each stage; A convolution processing module, used for inputting the output features of the predetermined target stage in each stage into the corresponding convolution block for processing, so as to obtain the first features of the predetermined target stage; An attention processing module, used for inputting the first feature of the predetermined target stage and the output feature of the final stage in each stage into the corresponding attention module, processing to obtain the corresponding multi-scale attention map, and inputting the map into the self-supervised multi-scale attention learning module to assist the training of the self-supervised multi-scale attention learning module; The category prediction module is used to obtain the global feature representation of the corresponding stage according to the multi-scale attention map and output features corresponding to the predetermined target stage and the final stage, and input the global features corresponding to the predetermined target stage and the final stage into the corresponding classifier to obtain the probability distribution of the predicted category; the final predicted category of the image is obtained according to the probability distribution of the predicted category of the predetermined target stage and the final stage.

7. The image recognition system based on self-supervised multi-scale attention learning according to claim 6, characterized in that: The predetermined target stages are the first two stages of the final stage.

8. The image recognition system based on self-supervised multi-scale attention learning according to claim 7, characterized in that: The category prediction module is specifically used to perform bilinear attention pooling processing on the multi-scale attention maps corresponding to the predetermined target stage and the final stage and the output features of the corresponding stage to obtain a global feature representation of the corresponding stage.

9. The image recognition system based on self-supervised multi-scale attention learning according to claim 8, characterized in that: The category prediction module is also used to: concatenate the global feature representations corresponding to the predetermined target stage and the final stage on the channel dimension to obtain a new global feature representation; input the new global feature representation into the corresponding classifier to obtain a first probability distribution of the predicted category; and obtain the final predicted category of the image based on the probability distribution of the predicted category of the predetermined target stage and the final stage, and the first probability distribution.

10. The image recognition system based on self-supervised multi-scale attention learning according to claim 9, characterized in that: The category prediction module is further used to: use the probability distribution of the predicted categories in the predetermined target stage and the final stage, and the average value of the first probability distribution as the final predicted category of the picture.

Images